sahara-8.0.0/0000775000175100017510000000000013245515027013006 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/0000775000175100017510000000000013245515026015476 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/notes/0000775000175100017510000000000013245515026016626 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/notes/substring-matching-1d5981b8e5b1d919.yaml0000666000175100017510000000041013245514472025472 0ustar zuulzuul00000000000000--- fixes: - Add regular expression matching on search values for certain string fields of sahara objects. This applies to list operations through the REST API and therefore applies to the dashboard and sahara client as well. Closes bug 1503345. sahara-8.0.0/releasenotes/notes/cdh_5_9_0_image_generation_validation-19d10e6468e30b4f.yaml0000666000175100017510000000024013245514472031071 0ustar zuulzuul00000000000000--- features: - Enables the creation and validation of CDH 5.9.0 images using the new image generation process where libguestfs replaces the use of DIB. sahara-8.0.0/releasenotes/notes/add-impala-2.2-c1649599649aff5c.yaml0000666000175100017510000000006013245514472024167 0ustar zuulzuul00000000000000--- features: - Add impala 2.2 to MapR plugin sahara-8.0.0/releasenotes/notes/add-mapr-sentry-6012c08b55d679de.yaml0000666000175100017510000000005413245514472024666 0ustar zuulzuul00000000000000--- features: - Add Sentry to MapR plugin sahara-8.0.0/releasenotes/notes/nova-network-removal-debe306fd7c61268.yaml0000666000175100017510000000055713245514472026125 0ustar zuulzuul00000000000000--- issues: - Ironic integration might be broken if floating IPs are used, due to the use of pre-created ports by the Sahara engine. The status of Ironic support was untested for this release. deprecations: - Support for nova-network is removed, reflective of its removal from nova itself and from python-novaclient. use_neutron=False is unsupported. sahara-8.0.0/releasenotes/notes/mapr-services-new-versions-b32c2e8fe07d1600.yaml0000666000175100017510000000025313245514472027140 0ustar zuulzuul00000000000000--- features: - The following service versions were added to MapR 5.2.0 plugin - HBase 1.1 - Drill 1.6 - Mahout 0.11 0.12 - Spark 1.6.1 - Impala 2.5 sahara-8.0.0/releasenotes/notes/cdh_5_7_image_generation_validation-308e7529a9018663.yaml0000666000175100017510000000023713245514472030447 0ustar zuulzuul00000000000000--- features: - Enables the creation and validation of CDH 5.7.0 images using the new image generation process where libguestfs replaces the use of DIB. sahara-8.0.0/releasenotes/notes/fix-install-provision-events-c1bd2e05bf2be6bd.yaml0000666000175100017510000000011513245514472027777 0ustar zuulzuul00000000000000--- fixes: - Fix uncompleted event logs for Oozie and Drill in MapR plugin.sahara-8.0.0/releasenotes/notes/optional-project-id-apiv1-2e89756f6f16bd5e.yaml0000666000175100017510000000012113245514472026653 0ustar zuulzuul00000000000000--- other: - The presence of project ID in Sahara APIv1 paths is now optional. sahara-8.0.0/releasenotes/notes/transport_url-5bbbf0bb54d81727.yaml0000666000175100017510000000030213245514472024721 0ustar zuulzuul00000000000000--- features: - Separate transport url can be used for notifications purposes now, to enable this feature 'transport_url' should be provided in 'oslo_messaging_notifications' section. sahara-8.0.0/releasenotes/notes/add-mapr-520-3ed6cd0ae9688e17.yaml0000666000175100017510000000007113245514472024017 0ustar zuulzuul00000000000000--- features: - MaR 5.2.0 is supported in MapR plugin. sahara-8.0.0/releasenotes/notes/add_kafka_in_cdh-774c7c051480c892.yaml0000666000175100017510000000007213245514472024714 0ustar zuulzuul00000000000000--- features: - Kafka was added in CDH 5.5 and CDH 5.7 sahara-8.0.0/releasenotes/notes/hdp-removed-from-defaults-31d1e1f15973b682.yaml0000666000175100017510000000025613245514472026560 0ustar zuulzuul00000000000000--- upgrade: - HDP plugin removed from default configuration list. End users who are using HDP should ensure that their configuration files continue to list "hdp". sahara-8.0.0/releasenotes/notes/cdh-5-5-35e582e149a05632.yaml0000666000175100017510000000007013245514472022554 0ustar zuulzuul00000000000000--- features: - CDH 5.5.0 is supported in CDH plugin. sahara-8.0.0/releasenotes/notes/refactor-floating-ips-logic-9d37d9297f3621b3.yaml0000666000175100017510000000026713245514472027111 0ustar zuulzuul00000000000000--- features: - Refactoring the logic on how floating ips are used by Sahara. This change will allow the coexistence of cluster using floating ips with cluster that do not. sahara-8.0.0/releasenotes/notes/remove-cdh_5.0_5.3_5.4-b5f140e9b0233c07.yaml0000666000175100017510000000012113245514472025221 0ustar zuulzuul00000000000000--- features: - Versions 5.0.0 5.3.0 and 5.4.0 of Cloudera plugin are removed. sahara-8.0.0/releasenotes/notes/kerberos-76dd297462b7337c.yaml0000666000175100017510000000026013245514472023423 0ustar zuulzuul00000000000000--- features: - Kerberos support implemented for Cloudera and Ambari plugins. New oozie client implemented to support authentication for oozie in kerberized cluster. sahara-8.0.0/releasenotes/notes/deprecate-storm-version-092.yaml-b9ff2b9ebbb983fc.yaml0000666000175100017510000000007113245514472030301 0ustar zuulzuul00000000000000--- deprecations: - Storm version 0.9.2 is deprecated. sahara-8.0.0/releasenotes/notes/add-wsgi-server-support-c8fbc3d76d4e42f6.yaml0000666000175100017510000000020313245514472026616 0ustar zuulzuul00000000000000--- features: - Added support of running sahara-api as wsgi application. Use 'sahara-wsgi-api' command for use this feature. sahara-8.0.0/releasenotes/notes/remove-use-neutron-2499b661dce041d4.yaml0000666000175100017510000000025513245514472025436 0ustar zuulzuul00000000000000--- upgrade: - | Nova network has been fully removed from the OpenStack codebase, remove all instances of switches on use_neutron and the configuration value. sahara-8.0.0/releasenotes/notes/cdh-labels-5695d95bce226051.yaml0000666000175100017510000000036113245514472023600 0ustar zuulzuul00000000000000--- features: - Versions 5.5.0 and 5.7.0 of Cloudera plugin are declared as stable. deprecations: - Versions 5, 5.3.0, 5.4.0 of Cloudera plugin are deprecated. It is no longer maintainted and supposed to be removed in P release. sahara-8.0.0/releasenotes/notes/hadoop-swift-jar-for-ambari-4439913b01d42468.yaml0000666000175100017510000000012713245514472026630 0ustar zuulzuul00000000000000--- fixes: - This patch adds ability to work with swift by using Keystone API v3 sahara-8.0.0/releasenotes/notes/remove-hdp-137d0ad3d2389b7a.yaml0000666000175100017510000000013613245514472023776 0ustar zuulzuul00000000000000--- deprecations: - Support of HDP 2.0.6 plugin was removed. Use Ambari plugin instead. sahara-8.0.0/releasenotes/notes/neutron-default-a6baf93d857d86b3.yaml0000666000175100017510000000022413245514472025142 0ustar zuulzuul00000000000000--- upgrade: - Neutron is used by default now (use_neutron=True). Nova-network is not functionaly for most use cases starting from Ocata. sahara-8.0.0/releasenotes/notes/remove-hardcoded-password-from-hive-eb923b518974e853.yaml0000666000175100017510000000014413245514472030554 0ustar zuulzuul00000000000000--- fixes: - Fixed issues with hardcoded password during starting hive process, bug 1498035. sahara-8.0.0/releasenotes/notes/apiv2-preview-release-b1ee8cc9b2fb01da.yaml0000666000175100017510000000063013245514472026340 0ustar zuulzuul00000000000000--- features: - | Sahara's APIv2 is now exposed by default (although its state is still experimental). It has feature parity with Sahara's APIv1.1, but APIv2 brings better REST semantics, tweaks to some response payloads, and some other improvements. APIv2 will remain labeled experimental until it is stabilized following the addition of new features to it in the coming cycle(s). sahara-8.0.0/releasenotes/notes/sahara-cfg-location-change-7b61454311b16ce8.yaml0000666000175100017510000000021613245514472026614 0ustar zuulzuul00000000000000--- upgrade: - | Sample configuration files previously installed in share/sahara will now be installed into etc/sahara instead. sahara-8.0.0/releasenotes/notes/add-storm-version-1_1_0-3e10b34824706a62.yaml0000666000175100017510000000007413245514472025656 0ustar zuulzuul00000000000000--- features: - Storm 1.1.0 is supported in Storm plugin. sahara-8.0.0/releasenotes/notes/cdh_5_11_0_image_generation_validation-6334ef6d04950935.yaml0000666000175100017510000000024013245514472031012 0ustar zuulzuul00000000000000--- features: - Enables the creation and validation of CDH 5.11.0 images using the new image generation process where libguestfs replaces the use of DIB. sahara-8.0.0/releasenotes/notes/spark-2.2-d7c3a84bd52f735a.yaml0000666000175100017510000000006613245514472023437 0ustar zuulzuul00000000000000--- features: - Adding Spark version 2.2 to Sahara. sahara-8.0.0/releasenotes/notes/add-mapr-kafka-3a808bbc1aa21055.yaml0000666000175100017510000000005313245514472024446 0ustar zuulzuul00000000000000--- features: - Add Kafka to MapR plugin sahara-8.0.0/releasenotes/notes/mapr5.2.0-image-gen-c850e74977b00abe.yaml0000666000175100017510000000015413245514472025107 0ustar zuulzuul00000000000000--- features: - Adding ability to create and validate MapR 5.2.0 images using the new image gen tool. sahara-8.0.0/releasenotes/notes/auto_configs_for_hdp-011d460d37dcdf02.yaml0000666000175100017510000000020613245514472026075 0ustar zuulzuul00000000000000--- features: - Add ability to automatically generate better configurations for Ambari cluster by using 'ALWAYS_APPLY' strategy sahara-8.0.0/releasenotes/notes/ambari_2_4_image_generation_validation-47eabb9fa90384c8.yaml0000666000175100017510000000024013245514472031507 0ustar zuulzuul00000000000000--- features: - Enables the creation and validation of Ambari 2.4 images using the new image generation process where libguestfs replaces the use of DIB. sahara-8.0.0/releasenotes/notes/mapr-labels-5cc318616db59403.yaml0000666000175100017510000000036613245514472024000 0ustar zuulzuul00000000000000--- features: - MapR 5.1.0.mrv2 is now Enabled. deprecations: - MapR 5.0.0.mrv2 is now Deprecated. It is not recommended for usage. It is better to use MapR 5.1.0.mrv2 instead. This version of plugin will be removed in Ocata release. sahara-8.0.0/releasenotes/notes/mapr-health-check-2eba3d742a2b853f.yaml0000666000175100017510000000007513245514472025265 0ustar zuulzuul00000000000000--- features: - Custom health check is added to MapR pluginsahara-8.0.0/releasenotes/notes/vanilla-2.8.2-support-84c89aad31105584.yaml0000666000175100017510000000011113245514472025374 0ustar zuulzuul00000000000000--- features: - | Support deploy hadoop 2.8.2 with vanilla plugin. sahara-8.0.0/releasenotes/notes/add_mapr_repo_configs-04af1a67350bfd24.yaml0000666000175100017510000000015413245514472026225 0ustar zuulzuul00000000000000--- features: - MapR repositories now can be configured in general section of cluster template configs sahara-8.0.0/releasenotes/notes/policy_in_code-5847902775ff9861.yaml0000666000175100017510000000061213245514472024445 0ustar zuulzuul00000000000000--- features: - This feature allows the policy enforcement to be done in code thus facilitating better maintenance of the policy file. In code the default policies are set and the operator only needs to change the policy file if they wish to override the rule or role for a specific policy or operation. Also, a complete policy file can be generated using genconfig tool. sahara-8.0.0/releasenotes/notes/remove-hard-coded-oozie-password-b97475c8772aa1bd.yaml0000666000175100017510000000015713245514472030216 0ustar zuulzuul00000000000000--- fixes: - Fixed issues with hardcoded password during creation MySQL database for Oozie, bug 1541122. sahara-8.0.0/releasenotes/notes/support-s3-job-binary-6d91267ae11d09d3.yaml0000666000175100017510000000013313245514472025742 0ustar zuulzuul00000000000000--- features: - An EDP job binary may reference a file stored in a S3-like object store. sahara-8.0.0/releasenotes/notes/ambari-server-start-856403bc280dfba3.yaml0000666000175100017510000000007613245514472025623 0ustar zuulzuul00000000000000--- fixes: - Starting Ambari clusters on Centos 7 is fixed. sahara-8.0.0/releasenotes/notes/honor-endpoint-type-neutron-4583128c383d9745.yaml0000666000175100017510000000015613245514472027066 0ustar zuulzuul00000000000000--- fixes: - Fixed issue with handling endpoint_type during creation neutron client, closed bug 1564805 sahara-8.0.0/releasenotes/notes/hdp25-b35ef99c240fc127.yaml0000666000175100017510000000010713245514472022660 0ustar zuulzuul00000000000000--- features: - Implemented support of HDP 2.5 in the Ambari plugin. sahara-8.0.0/releasenotes/notes/ambari-downscaling-b9ba759ce9c7325e.yaml0000666000175100017510000000007613245514472025576 0ustar zuulzuul00000000000000--- fixes: - Fixed incorrect down scaling of ambari cluster sahara-8.0.0/releasenotes/notes/keystoneclient-to-keystonauth-migration-c75988975ad1a506.yaml0000666000175100017510000000016013245514472031634 0ustar zuulzuul00000000000000--- upgrade: - migration from keystoneclient to keystoneauth is done for using auth features of keystone. sahara-8.0.0/releasenotes/notes/rack_awareness_for_cdh-e0cd5d4ab46aa1b5.yaml0000666000175100017510000000010713245514472026607 0ustar zuulzuul00000000000000--- features: - Added rack awareness feature for CDH 5.5 and CDH 5.7 sahara-8.0.0/releasenotes/notes/trustee-conf-section-5994dcd48a9744d7.yaml0000666000175100017510000000053013245514472025761 0ustar zuulzuul00000000000000--- deprecations: - | The use of [keystone_authtoken] credentials for trust creation is now deprecated. Please use the new [trustee] config section. The options ``username``, ``password``, ``project_name``, ``user_domain_name``, ``project_domain_name``, and ``auth_url`` (with version) are obligatory within that section. sahara-8.0.0/releasenotes/notes/cdh_5_9_support-b603a2648b2e7b32.yaml0000666000175100017510000000007013245514472024647 0ustar zuulzuul00000000000000--- features: - CDH 5.9.0 is supported in CDH plugin. sahara-8.0.0/releasenotes/notes/deprecate-mapr-51-090423438e3dda20.yaml0000666000175100017510000000022513245514472024674 0ustar zuulzuul00000000000000--- deprecations: - MapR 5.1.0.mrv2 is now deprecated and will be removed in Ocata release. It is recommended to use MapR 5.2.0.mrv2 instead. sahara-8.0.0/releasenotes/notes/remove-mapr-500-3df3041be99a864c.yaml0000666000175100017510000000010113245514472024465 0ustar zuulzuul00000000000000--- deprecations: - Removed support for the MapR 5.0.0 plugin. sahara-8.0.0/releasenotes/notes/ironic-support-79e7ecad05f54029.yaml0000666000175100017510000000020613245514472024740 0ustar zuulzuul00000000000000--- other: - We are assuring that ironic support was tested after latest updates to nova and sahara and it is fully functional. sahara-8.0.0/releasenotes/notes/mapr-services-new-versions-dc7652e33f26bbdc.yaml0000666000175100017510000000022613245514472027311 0ustar zuulzuul00000000000000--- features: - The following service versions were added to MapR 5.2.0 plugin - Pig 0.16 - Spark 2.0.1 - Hue 3.10 - Drill 1.8, 1.9 sahara-8.0.0/releasenotes/notes/event_log_for_hdp-a114511c477ef16d.yaml0000666000175100017510000000006113245514472025326 0ustar zuulzuul00000000000000--- features: - Added event log for HDP plugin sahara-8.0.0/releasenotes/notes/remove_enable_notifications_opt-4c0d46e8e79eb06f.yaml0000666000175100017510000000030713245514472030445 0ustar zuulzuul00000000000000--- deprecations: - The 'enable' option of the 'oslo_messaging_notifications' section has been removed. To enable notifications now please specify the 'driver' option in the same section. sahara-8.0.0/releasenotes/notes/api-insecure-cbd4fd5da71b29a3.yaml0000666000175100017510000000011413245514472024527 0ustar zuulzuul00000000000000--- fixes: - Fixed api_insecure handling in sessions. Closed bug 1539498. sahara-8.0.0/releasenotes/notes/designate-integration-784c5f7f29546015.yaml0000666000175100017510000000014113245514472026013 0ustar zuulzuul00000000000000--- features: - Added integration of Designate for hostname resolution through dns servers sahara-8.0.0/releasenotes/notes/ambari-hive-92b911e0a759ee88.yaml0000666000175100017510000000007313245514472024056 0ustar zuulzuul00000000000000--- fixes: - Fixed launching Hive jobs in Ambari plugin. sahara-8.0.0/releasenotes/notes/deprecate-cdh_5_5-0da56b562170566f.yaml0000666000175100017510000000010213245514472024724 0ustar zuulzuul00000000000000--- features: - Version 5.5.0 of Cloudera plugin is deprecated. sahara-8.0.0/releasenotes/notes/mapr-remove-spark-standalone-293ca864de9a7848.yaml0000666000175100017510000000010413245514472027370 0ustar zuulzuul00000000000000--- features: - Remove support for Spark standalone in MapR pluginsahara-8.0.0/releasenotes/notes/force-delete-apiv2-e372392bbc8639f8.yaml0000666000175100017510000000025213245514472025247 0ustar zuulzuul00000000000000--- features: - The ability to force delete clusters is exposed in Sahara APIv2. The Heat service must support Stack Abandon for force delete to function properly. sahara-8.0.0/releasenotes/notes/engine-opt-258ff1ae9b04d628.yaml0000666000175100017510000000012513245514472024005 0ustar zuulzuul00000000000000--- deprecations: - Option 'infrastructure engine' is removed from sahara configs. sahara-8.0.0/releasenotes/notes/options-to-oslo_messaging_notifications-cee206fc4f74c217.yaml0000666000175100017510000000011613245514472032066 0ustar zuulzuul00000000000000--- upgrade: - Move notifications options into oslo_messaging_notifications sahara-8.0.0/releasenotes/notes/cdh_5_11_support-10d4abb91bc4475f.yaml0000666000175100017510000000007113245514472025061 0ustar zuulzuul00000000000000--- features: - CDH 5.11.0 is supported in CDH plugin. sahara-8.0.0/releasenotes/notes/convert-to-cluster-template-43d502496d18625e.yaml0000666000175100017510000000014513245514472027111 0ustar zuulzuul00000000000000--- deprecations: - Convert to cluster template feature is no longer supported by all plugins. sahara-8.0.0/releasenotes/notes/deprecate-hdp-a9ff0ecf6006da49.yaml0000666000175100017510000000023613245514472024605 0ustar zuulzuul00000000000000--- deprecations: - The HDP 2.0.6 plugin is deprecated in Mitaka release and will be removed in Newton release. Please, use the Ambari 2.3 instead. sahara-8.0.0/releasenotes/notes/rack_awareness_for_hdp-6e3d44468cc141a5.yaml0000666000175100017510000000007613245514472026347 0ustar zuulzuul00000000000000--- features: - Added rack awareness feature for HDP plugin sahara-8.0.0/releasenotes/notes/key_manager_integration-e32d141809c8cc46.yaml0000666000175100017510000000024613245514472026544 0ustar zuulzuul00000000000000--- features: - OpenStack Key Manager service can now be used by sahara to enable storage of sensitive information in an external service such as barbican. sahara-8.0.0/releasenotes/notes/remove-spark-100-44f3d5efc3806410.yaml0000666000175100017510000000010213245514472024555 0ustar zuulzuul00000000000000--- deprecations: - Removed support for the Spark 1.0.0 plugin. sahara-8.0.0/releasenotes/notes/config-groups-ambari-837de6d33eb0fa87.yaml0000666000175100017510000000015713245514472026043 0ustar zuulzuul00000000000000--- fixes: - After decommissioning hosts all assoicated configs groups will be removed in ambari plugin. sahara-8.0.0/releasenotes/notes/deprecate-spark-version-131-98eccc79b13b6b8f.yaml0000666000175100017510000000007113245514472027147 0ustar zuulzuul00000000000000--- deprecations: - Spark version 1.3.1 is deprecated. sahara-8.0.0/releasenotes/notes/.placeholder0000666000175100017510000000000013245514472021105 0ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/notes/cdh_5_7_support-9522cb9b4dce2378.yaml0000666000175100017510000000006613245514472024751 0ustar zuulzuul00000000000000--- features: - CDH 5.7 is supported in CDH plugin. sahara-8.0.0/releasenotes/notes/ca-cert-fix-5c434a82f9347039.yaml0000666000175100017510000000015513245514472023627 0ustar zuulzuul00000000000000--- fixes: - CA certificate handling in keystone, nova, neutron and cinder clients are fixed (#330635) sahara-8.0.0/releasenotes/notes/zookeeper-configuration-steps-48c3d9706c86f227.yaml0000666000175100017510000000036213245514472027620 0ustar zuulzuul00000000000000--- prelude: > Documentation about distributed periodics are extended with steps about installation additional libs required for correct work of coordination backend. Please refer Advanced Configuration Guide for details. sahara-8.0.0/releasenotes/notes/add-scheduler-edp-job-9eda17dd174e53fa.yaml0000666000175100017510000000010013245514472026111 0ustar zuulzuul00000000000000--- features: - Add ability of scheduling EDP jobs for sahara sahara-8.0.0/releasenotes/notes/novaclient_images_to_glanceclient-0266a2bd92b4be05.yaml0000666000175100017510000000010513245514472030616 0ustar zuulzuul00000000000000--- upgrade: - Migration from novaclient.v2.images to glanceclient sahara-8.0.0/releasenotes/notes/deprecate-plugin-vanilla260-46e4b8fe96e8fe68.yaml0000666000175100017510000000007713245514472027162 0ustar zuulzuul00000000000000--- deprecations: - Removed support of Vanilla 2.6.0 plugin. sahara-8.0.0/releasenotes/notes/remove_custom_auth_domainname-984fd2d931e306cc.yaml0000666000175100017510000000035513245514472030046 0ustar zuulzuul00000000000000--- deprecations: - The custom admin_user_domain_name and admin_project_domain_name configuration options have been removed; they are provided by keystone_authtoken as user_domain_name and project_domain_name respectively. sahara-8.0.0/releasenotes/source/0000775000175100017510000000000013245515026016776 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/source/newton.rst0000666000175100017510000000023213245514472021045 0ustar zuulzuul00000000000000=================================== Newton Series Release Notes =================================== .. release-notes:: :branch: origin/stable/newton sahara-8.0.0/releasenotes/source/_static/0000775000175100017510000000000013245515026020424 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/source/_static/.placeholder0000666000175100017510000000000013245514472022703 0ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/source/liberty.rst0000666000175100017510000000022213245514472021204 0ustar zuulzuul00000000000000============================== Liberty Series Release Notes ============================== .. release-notes:: :branch: origin/stable/liberty sahara-8.0.0/releasenotes/source/pike.rst0000666000175100017510000000021713245514472020466 0ustar zuulzuul00000000000000=================================== Pike Series Release Notes =================================== .. release-notes:: :branch: stable/pike sahara-8.0.0/releasenotes/source/conf.py0000666000175100017510000001562413245514472020313 0ustar zuulzuul00000000000000# -*- coding: utf-8 -*- # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # Sahara Release Notes documentation build configuration file extensions = [ 'reno.sphinxext', 'openstackdocstheme' ] # openstackdocstheme options repository_name = 'openstack/sahara' bug_project = 'sahara' bug_tag = 'releasenotes' html_last_updated_fmt = '%Y-%m-%d %H:%M' # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The master toctree document. master_doc = 'index' # General information about the project. project = u'Sahara Release Notes' copyright = u'2015, Sahara Developers' # Release do not need a version number in the title, they # cover multiple versions. # The full version, including alpha/beta/rc tags. release = '' # The short X.Y version. version = '' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = [] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # -- Options for HTML output ---------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'openstackdocs' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. # html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. # html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". # html_title = None # A shorter title for the navigation bar. Default is the same as html_title. # html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. # html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. # html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. # html_extra_path = [] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. # html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. # html_use_smartypants = True # Custom sidebar templates, maps document names to template names. # html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. # html_additional_pages = {} # If false, no module index is generated. # html_domain_indices = True # If false, no index is generated. # html_use_index = True # If true, the index is split into individual pages for each letter. # html_split_index = False # If true, links to the reST sources are added to the pages. # html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. # html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. # html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. # html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). # html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'SaharaReleaseNotesdoc' # -- Options for LaTeX output --------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). # 'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). # 'pointsize': '10pt', # Additional stuff for the LaTeX preamble. # 'preamble': '', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ ('index', 'SaharaReleaseNotes.tex', u'Sahara Release Notes Documentation', u'Sahara Developers', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. # latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. # latex_use_parts = False # If true, show page references after internal links. # latex_show_pagerefs = False # If true, show URL addresses after external links. # latex_show_urls = False # Documents to append as an appendix to all manuals. # latex_appendices = [] # If false, no module index is generated. # latex_domain_indices = True # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'saharareleasenotes', u'Sahara Release Notes Documentation', [u'Sahara Developers'], 1) ] # If true, show URL addresses after external links. # man_show_urls = False # -- Options for Texinfo output ------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'SaharaReleaseNotes', u'Sahara Release Notes Documentation', u'Sahara Developers', 'SaharaReleaseNotes', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. # texinfo_appendices = [] # If false, no module index is generated. # texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. # texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. # texinfo_no_detailmenu = False # -- Options for Internationalization output ------------------------------ locale_dirs = ['locale/'] sahara-8.0.0/releasenotes/source/locale/0000775000175100017510000000000013245515026020235 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/source/locale/fr/0000775000175100017510000000000013245515026020644 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/source/locale/fr/LC_MESSAGES/0000775000175100017510000000000013245515026022431 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/source/locale/fr/LC_MESSAGES/releasenotes.po0000666000175100017510000000247513245514472025500 0ustar zuulzuul00000000000000# Gérald LONLAS , 2016. #zanata msgid "" msgstr "" "Project-Id-Version: Sahara Release Notes 7.0.0\n" "Report-Msgid-Bugs-To: \n" "POT-Creation-Date: 2017-08-11 19:13+0000\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" "PO-Revision-Date: 2016-10-22 06:34+0000\n" "Last-Translator: Gérald LONLAS \n" "Language-Team: French\n" "Language: fr\n" "X-Generator: Zanata 3.9.6\n" "Plural-Forms: nplurals=2; plural=(n > 1)\n" msgid "3.0.1" msgstr "3.0.1" msgid "4.0.0" msgstr "4.0.0" msgid "4.0.1" msgstr "4.0.1" msgid "4.1.0" msgstr "4.1.0" msgid "5.0.0" msgstr "5.0.0" msgid "Bug Fixes" msgstr "Corrections de bugs" msgid "Current Series Release Notes" msgstr "Note de la release actuelle" msgid "Deprecation Notes" msgstr "Notes dépréciées " msgid "Known Issues" msgstr "Problèmes connus" msgid "Liberty Series Release Notes" msgstr "Note de release pour Liberty" msgid "Mitaka Series Release Notes" msgstr "Note de release pour Mitaka" msgid "New Features" msgstr "Nouvelles fonctionnalités" msgid "Newton Series Release Notes" msgstr "Note de release pour Newton" msgid "Other Notes" msgstr "Autres notes" msgid "Sahara Release Notes" msgstr "Note de release de Sahara" msgid "Upgrade Notes" msgstr "Notes de mises à jours" sahara-8.0.0/releasenotes/source/unreleased.rst0000666000175100017510000000016013245514472021662 0ustar zuulzuul00000000000000============================== Current Series Release Notes ============================== .. release-notes:: sahara-8.0.0/releasenotes/source/index.rst0000666000175100017510000000024113245514476020646 0ustar zuulzuul00000000000000====================== Sahara Release Notes ====================== .. toctree:: :maxdepth: 1 unreleased pike ocata newton mitaka liberty sahara-8.0.0/releasenotes/source/mitaka.rst0000666000175100017510000000023213245514472021001 0ustar zuulzuul00000000000000=================================== Mitaka Series Release Notes =================================== .. release-notes:: :branch: origin/stable/mitaka sahara-8.0.0/releasenotes/source/ocata.rst0000666000175100017510000000023013245514472020620 0ustar zuulzuul00000000000000=================================== Ocata Series Release Notes =================================== .. release-notes:: :branch: origin/stable/ocata sahara-8.0.0/releasenotes/source/_templates/0000775000175100017510000000000013245515026021133 5ustar zuulzuul00000000000000sahara-8.0.0/releasenotes/source/_templates/.placeholder0000666000175100017510000000000013245514472023412 0ustar zuulzuul00000000000000sahara-8.0.0/bandit.yaml0000666000175100017510000001174713245514472015152 0ustar zuulzuul00000000000000# optional: after how many files to update progress #show_progress_every: 100 # optional: plugins directory name #plugins_dir: 'plugins' # optional: plugins discovery name pattern plugin_name_pattern: '*.py' # optional: terminal escape sequences to display colors #output_colors: # DEFAULT: '\033[0m' # HEADER: '\033[95m' # LOW: '\033[94m' # WARN: '\033[93m' # ERROR: '\033[91m' # optional: log format string #log_format: "[%(module)s]\t%(levelname)s\t%(message)s" # globs of files which should be analyzed include: - '*.py' - '*.pyw' # a list of strings, which if found in the path will cause files to be excluded # for example /tests/ - to remove all all files in tests directory exclude_dirs: profiles: sahara_default: include: - hardcoded_password_string - hardcoded_password_funcarg # - hardcoded_password_default - blacklist_calls - blacklist_imports - subprocess_popen_with_shell_equals_true - subprocess_without_shell_equals_true - any_other_function_with_shell_equals_true - start_process_with_a_shell - start_process_with_no_shell - hardcoded_sql_expressions - jinja2_autoescape_false - use_of_mako_templates blacklist_calls: bad_name_sets: - pickle: qualnames: [pickle.loads, pickle.load, pickle.Unpickler, cPickle.loads, cPickle.load, cPickle.Unpickler] message: "Pickle library appears to be in use, possible security issue." - marshal: qualnames: [marshal.load, marshal.loads] message: "Deserialization with the marshal module is possibly dangerous." - md5: qualnames: [hashlib.md5] message: "Use of insecure MD5 hash function." - mktemp_q: qualnames: [tempfile.mktemp] message: "Use of insecure and deprecated function (mktemp)." - eval: qualnames: [eval] message: "Use of possibly insecure function - consider using safer ast.literal_eval." - mark_safe: qualnames: [mark_safe] message: "Use of mark_safe() may expose cross-site scripting vulnerabilities and should be reviewed." - httpsconnection: qualnames: [httplib.HTTPSConnection] message: "Use of HTTPSConnection does not provide security, see https://wiki.openstack.org/wiki/OSSN/OSSN-0033" - yaml_load: qualnames: [yaml.load] message: "Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load()." - urllib_urlopen: qualnames: [urllib.urlopen, urllib.urlretrieve, urllib.URLopener, urllib.FancyURLopener, urllib2.urlopen, urllib2.Request] message: "Audit url open for permitted schemes. Allowing use of file:/ or custom schemes is often unexpected." shell_injection: # Start a process using the subprocess module, or one of its wrappers. subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call, subprocess.check_output, utils.execute, utils.execute_with_timeout] # Start a process with a function vulnerable to shell injection. shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4, popen2.popen2, popen2.popen3, popen2.popen4, popen2.Popen3, popen2.Popen4, commands.getoutput, commands.getstatusoutput] # Start a process with a function that is not vulnerable to shell injection. no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv,os.execve, os.execvp, os.execvpe, os.spawnl, os.spawnle, os.spawnlp, os.spawnlpe, os.spawnv, os.spawnve, os.spawnvp, os.spawnvpe, os.startfile] blacklist_imports: bad_import_sets: - telnet: imports: [telnetlib] level: ERROR message: "Telnet is considered insecure. Use SSH or some other encrypted protocol." - info_libs: imports: [pickle, cPickle, subprocess, Crypto] level: LOW message: "Consider possible security implications associated with {module} module." hardcoded_tmp_directory: tmp_dirs: [/tmp, /var/tmp, /dev/shm] hardcoded_password: word_list: "wordlist/default-passwords" ssl_with_bad_version: bad_protocol_versions: - 'PROTOCOL_SSLv2' - 'SSLv2_METHOD' - 'SSLv23_METHOD' - 'PROTOCOL_SSLv3' # strict option - 'PROTOCOL_TLSv1' # strict option - 'SSLv3_METHOD' # strict option - 'TLSv1_METHOD' # strict option password_config_option_not_marked_secret: function_names: - oslo.config.cfg.StrOpt - oslo_config.cfg.StrOpt execute_with_run_as_root_equals_true: function_names: - ceilometer.utils.execute - cinder.utils.execute - neutron.agent.linux.utils.execute - nova.utils.execute - nova.utils.trycmd try_except_pass: check_typed_exception: True sahara-8.0.0/doc/0000775000175100017510000000000013245515026013552 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/0000775000175100017510000000000013245515026015052 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/configuration/0000775000175100017510000000000013245515026017721 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/configuration/sampleconfig.rst0000666000175100017510000000027113245514472023130 0ustar zuulzuul00000000000000Sample sahara.conf file ======================= This is an automatically generated sample of the sahara.conf file. .. literalinclude:: ../sample.config :language: ini :linenos: sahara-8.0.0/doc/source/configuration/descriptionconfig.rst0000666000175100017510000000034713245514472024176 0ustar zuulzuul00000000000000Configuration options ===================== This section provides a list of the configuration options that can be set in the sahara configuration file. .. show-options:: :config-file: tools/config/config-generator.sahara.conf sahara-8.0.0/doc/source/configuration/index.rst0000666000175100017510000000021513245514472021566 0ustar zuulzuul00000000000000======================= Configuration Reference ======================= .. toctree:: :maxdepth: 1 descriptionconfig sampleconfig sahara-8.0.0/doc/source/user/0000775000175100017510000000000013245515026016030 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/user/cdh-plugin.rst0000666000175100017510000001475113245514476020636 0ustar zuulzuul00000000000000Cloudera Plugin =============== The Cloudera plugin is a Sahara plugin which allows the user to deploy and operate a cluster with Cloudera Manager. The Cloudera plugin is enabled in Sahara by default. You can manually modify the Sahara configuration file (default /etc/sahara/sahara.conf) to explicitly enable or disable it in "plugins" line. You need to build images using :doc:`cdh-imagebuilder` to produce images used to provision cluster or you could download prepared images from http://sahara-files.mirantis.com/images/upstream/ They already have Cloudera Express installed (version 5.0.0, 5.3.0, 5.4.0, 5.5.0, 5.7.x and 5.9.x). The cloudera plugin requires an image to be tagged in Sahara Image Registry with two tags: 'cdh' and '' (e.g. '5', '5.3.0', '5.4.0', '5.5.0', '5.7.0', '5.9.0' or '5.9.1', here '5' stands for '5.0.0'). The default username specified for these images is different for each distribution: for 5.0.0, 5.3.0 and 5.4.0 version: +--------------+------------+ | OS | username | +==============+============+ | Ubuntu 12.04 | ubuntu | +--------------+------------+ | CentOS 6.6 | cloud-user | +--------------+------------+ for 5.5.0 and higher versions: +--------------+------------+ | OS | username | +==============+============+ | Ubuntu 14.04 | ubuntu | +--------------+------------+ | CentOS 6.6 | cloud-user | +--------------+------------+ | CentOS 7 | centos | +--------------+------------+ Services Supported ------------------ Currently below services are supported in both versions of Cloudera plugin: HDFS, Oozie, YARN, Spark, Zookeeper, Hive, Hue, HBase. 5.3.0 version of Cloudera Plugin also supported following services: Impala, Flume, Solr, Sqoop, and Key-value Store Indexer. In version 5.4.0 KMS service support was added based on version 5.3.0. Kafka 2.0.2 was added for CDH 5.5 and higher. .. note:: Sentry service is enabled in Cloudera plugin. However, as we do not enable Kerberos authentication in the cluster for CDH version < 5.5 (which is required for Sentry functionality) then using Sentry service will not really take any effect, and other services depending on Sentry will not do any authentication too. High Availability Support ------------------------- Currently HDFS NameNode High Availability is supported beginning with Cloudera 5.4.0 version. You can refer to :doc:`features` for the detail info. YARN ResourceManager High Availability is supported beginning with Cloudera 5.4.0 version. This feature adds redundancy in the form of an Active/Standby ResourceManager pair to avoid the failure of single RM. Upon failover, the Standby RM become Active so that the applications can resume from their last check-pointed state. Cluster Validation ------------------ When the user performs an operation on the cluster using a Cloudera plugin, the cluster topology requested by the user is verified for consistency. The following limitations are required in the cluster topology for all cloudera plugin versions: + Cluster must contain exactly one manager. + Cluster must contain exactly one namenode. + Cluster must contain exactly one secondarynamenode. + Cluster must contain at least ``dfs_replication`` datanodes. + Cluster can contain at most one resourcemanager and this process is also required by nodemanager. + Cluster can contain at most one jobhistory and this process is also required for resourcemanager. + Cluster can contain at most one oozie and this process is also required for EDP. + Cluster can't contain oozie without datanode. + Cluster can't contain oozie without nodemanager. + Cluster can't contain oozie without jobhistory. + Cluster can't contain hive on the cluster without the following services: metastore, hive server, webcat and resourcemanager. + Cluster can contain at most one hue server. + Cluster can't contain hue server without hive service and oozie. + Cluster can contain at most one spark history server. + Cluster can't contain spark history server without resourcemanager. + Cluster can't contain hbase master service without at least one zookeeper and at least one hbase regionserver. + Cluster can't contain hbase regionserver without at least one hbase maser. In case of 5.3.0, 5.4.0, 5.5.0, 5.7.x or 5.9.x version of Cloudera Plugin there are few extra limitations in the cluster topology: + Cluster can't contain flume without at least one datanode. + Cluster can contain at most one sentry server service. + Cluster can't contain sentry server service without at least one zookeeper and at least one datanode. + Cluster can't contain solr server without at least one zookeeper and at least one datanode. + Cluster can contain at most one sqoop server. + Cluster can't contain sqoop server without at least one datanode, nodemanager and jobhistory. + Cluster can't contain hbase indexer without at least one datanode, zookeeper, solr server and hbase master. + Cluster can contain at most one impala catalog server. + Cluster can contain at most one impala statestore. + Cluster can't contain impala catalogserver without impala statestore, at least one impalad service, at least one datanode, and metastore. + If using Impala, the daemons must be installed on every datanode. In case of version 5.5.0, 5.7.x or 5.9.x of Cloudera Plugin additional services in the cluster topology are available: + Cluster can have the kafka service and several kafka brokers. Enabling Kerberos security for cluster -------------------------------------- If you want to protect your clusters using MIT Kerberos security you have to complete a few steps below. * If you would like to create a cluster protected by Kerberos security you just need to enable Kerberos by checkbox in the ``General Parameters`` section of the cluster configuration. If you prefer to use the OpenStack CLI for cluster creation, you have to put the data below in the ``cluster_configs`` section: .. sourcecode:: console "cluster_configs": { "Enable Kerberos Security": true, } Sahara in this case will correctly prepare KDC server and will create principals along with keytabs to enable authentication for Hadoop services. * Ensure that you have the latest hadoop-openstack jar file distributed on your cluster nodes. You can download one at ``http://tarballs.openstack.org/sahara/dist/`` * Sahara will create principals along with keytabs for system users like ``hdfs`` and ``spark`` so that you will not have to perform additional auth operations to execute your jobs on top of the cluster. sahara-8.0.0/doc/source/user/cdh-imagebuilder.rst0000666000175100017510000000726313245514472021765 0ustar zuulzuul00000000000000.. _cdh-diskimage-builder-label: Building Images for Cloudera Plugin =================================== In this document you will find instructions on how to build Ubuntu and CentOS images with Cloudera Express (now only versions {5.0.0, 5.3.0 5.4.0, 5.5.0, 5.7.x, 5.9.x} are supported). To simplify the task of building such images we use `Disk Image Builder `_. `Disk Image Builder` builds disk images using elements. An element is a particular set of code that alters how the image is built, or runs within the chroot to prepare the image. Elements for building Cloudera images are stored in `Sahara extra repository `_ .. note:: Sahara requires images with cloud-init package installed: * `For CentOS 6 `_ * `For CentOS 7 `_ * `For Ubuntu 14 `_ To create cloudera images follow these steps: 1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements" locally. 2. Use tox to build images. You can run "tox -e venv -- sahara-image-create" command in sahara-image-elements directory to build images. By default this script will attempt to create cloud images for all versions of supported plugins and all operating systems (subset of Ubuntu, Fedora, and CentOS depending on plugin). To only create Cloudera images, you should use the "-p cloudera" parameter in the command line. If you want to create the image only for a specific operating system, you should use the "-i ubuntu|centos|centos7" parameter to assign the operating system (the cloudera plugin only supports Ubuntu and Centos). If you want to create the image only for a specific Cloudera version, you should use the "-v 5.0|5.3|5.4|5.5|5.7|5.9" parameter to assign the version. Note that Centos 7 can only be used with CDH 5.5 and higher. Below is an example to create Cloudera images for both Ubuntu and CentOS with Cloudera Express 5.5.0 version. .. sourcecode:: console tox -e venv -- sahara-image-create -p cloudera -v 5.5 If you want to create only an Ubuntu image, you may use following example for that: .. sourcecode:: console tox -e venv -- sahara-image-create -p cloudera -i ubuntu -v 5.5 For CDH 5.7 and higher we support minor versions. If you want to build a minor version just export DIB_CDH_MINOR_VERSION before sahara-image-create launch, e.g.: .. sourcecode:: console export DIB_CDH_MINOR_VERSION=5.7.1 NOTE: If you don't want to use default values, you should explicitly set the values of your required parameters. The script will create required cloud images using image elements that install all the necessary packages and configure them. You will find the created images in the parent directory. .. note:: Disk Image Builder will generate QCOW2 images, used with the default OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different hypervisor, the generated image should be converted to an appropriate format. The VMware Nova backend requires the VMDK image format. You may use qemu-img utility to convert a QCOW2 image to VMDK. .. sourcecode:: console qemu-img convert -O vmdk .qcow2 .vmdk For finer control of diskimage-create.sh see the `official documentation `_ sahara-8.0.0/doc/source/user/sahara-on-ironic.rst0000666000175100017510000001125613245514472021727 0ustar zuulzuul00000000000000How to run a Sahara cluster on bare metal servers ================================================= Hadoop clusters are designed to store and analyze extremely large amounts of unstructured data in distributed computing environments. Sahara enables you to boot Hadoop clusters in both virtual and bare metal environments. When Booting Hadoop clusters with Sahara on bare metal servers, you benefit from the bare metal performance with self-service resource provisioning. 1. Create a new OpenStack environment using Devstack as described in the `Devstack Guide `_ 2. Install Ironic as described in the `Ironic Installation Guide `_ 3. Install Sahara as described in the `Sahara Installation Guide <../install/installation-guide.html>`_ 4. Build the Sahara image and prepare it for uploading to Glance: - Build an image for Sahara plugin with the ``-b`` flag. Use sahara image elements when building the image. See ``_ - Convert the qcow2 image format to the raw format. For example: .. sourcecode:: console $ qemu-img convert -O raw image-converted.qcow image-converted-from-qcow2.raw .. - Mount the raw image to the system. - ``chroot`` to the mounted directory and remove the installed grub. - Build grub2 from sources and install to ``/usr/sbin``. - In ``/etc/sysconfig/selinux``, disable selinux ``SELINUX=disabled`` - In the configuration file, set ``onboot=yes`` and ``BOOTPROTO=dhcp`` for every interface. - Add the configuration files for all interfaces in the ``/etc/sysconfig/network-scripts`` directory. 5. Upload the Sahara disk image to Glance, and register it in the Sahara Image Registry. Referencing its separate kernel and initramfs images. 6. Configure the bare metal network for the Sahara cluster nodes: - Add bare metal servers to your environment manually referencing their IPMI addresses (Ironic does not detect servers), for Ironic to manage the servers power and network. For example: .. code-block:: bash $ ironic node-create -d pxe_ipmitool \ -i ipmi_address=$IP_ADDRESS \ -i ipmi_username=$USERNAME \ -i ipmi_password=$PASSWORD \ -i pxe_deploy_kernel=$deploy.kernel.id \ -i pxe_deploy_ramdisk=$deploy.ramfs.id $ ironic port-create -n $NODE_ID -a "$MAC_eth1" .. - Add the hardware information: .. code-block:: bash $ ironic node-update $NODE_ID add properties/cpus=$CPU \ properties/memory_mb=$RAM properties/local_gb=$ROOT_GB \ properties/cpu_arch='x86_64' .. 7. Add a special flavor for the bare metal instances with an arch meta parameter to match the virtual architecture of the server's CPU with the metal one. For example: .. code-block:: bash $ nova flavor-create baremetal auto $RAM $DISK_GB $CPU $ nova flavor-key baremetal set cpu_arch=x86_64 .. Note: +++++ The vCPU ad vRAM parameters (x86_64 in the example) will not be applied because the operating system has access to the real CPU cores and RAM. Only the root disk parameter is applied, and Ironic will resize the root disk partition. Ironic supports only a flat network topology for the bare metal provisioning, you must use Neutron to configure it. 8. Launch your Sahara cluster on Ironic from the cluster template: * Log in to Horizon. * Go to Data Processing > Node Group Templates. * Find the templates that belong to the plugin you would like to use * Update those templates to use 'bare metal' flavor instead of the default one * Go to Data Processing > Cluster Templates. * Click Launch Cluster. * On the Launch Cluster dialog: * Specify the bare metal network for cluster nodes The cluster provisioning time is slower compared to the cluster provisioning of the same size that runs on VMs. Ironic does real hardware reports which is time consuming, and the whole root disk is filled from ``/dev/zero`` for security reasons. Known limitations: ------------------ * Security groups are not applied. * When booting a nova instance with a bare metal flavor, the user can not provide a pre-created neutron port to ``nova boot`` command. `LP1544195 `_ * Nodes are not isolated by projects. * VM to Bare Metal network routing is not allowed. * The user has to specify the count of ironic nodes before Devstack deploys an OpenStack. * The user cannot use the same image for several ironic node types. For example, if there are 3 ironic node types, the user has to create 3 images and 3 flavors. * Multiple interfaces on a single node are not supported. Devstack configures only one interface. sahara-8.0.0/doc/source/user/ambari-plugin.rst0000666000175100017510000001024513245514476021325 0ustar zuulzuul00000000000000 Ambari Plugin ============= The Ambari sahara plugin provides a way to provision clusters with Hortonworks Data Platform on OpenStack using templates in a single click and in an easily repeatable fashion. The sahara controller serves as the glue between Hadoop and OpenStack. The Ambari plugin mediates between the sahara controller and Apache Ambari in order to deploy and configure Hadoop on OpenStack. Core to the HDP Plugin is Apache Ambari which is used as the orchestrator for deploying HDP on OpenStack. The Ambari plugin uses Ambari Blueprints for cluster provisioning. Apache Ambari Blueprints ------------------------ Apache Ambari Blueprints is a portable document definition, which provides a complete definition for an Apache Hadoop cluster, including cluster topology, components, services and their configurations. Ambari Blueprints can be consumed by the Ambari plugin to instantiate a Hadoop cluster on OpenStack. The benefits of this approach is that it allows for Hadoop clusters to be configured and deployed using an Ambari native format that can be used with as well as outside of OpenStack allowing for clusters to be re-instantiated in a variety of environments. Images ------ The sahara Ambari plugin is using minimal (operating system only) images. For more information about Ambari images, refer to ``_. You could download well tested and up-to-date prepared images from http://sahara-files.mirantis.com/images/upstream/ HDP plugin requires an image to be tagged in sahara Image Registry with two tags: 'ambari' and '' (e.g. '2.5'). Also in the Image Registry you will need to specify username for an image. The username specified should be 'cloud-user' in case of CentOS 6.x image, 'centos' for CentOS 7 images and 'ubuntu' for Ubuntu images. High Availability for HDFS and YARN ----------------------------------- High Availability (Using the Quorum Journal Manager) can be deployed automatically with the Ambari plugin. You can deploy High Available cluster through UI by selecting ``NameNode HA`` and/or ``ResourceManager HA`` options in general configs of cluster template. The NameNode High Availability is deployed using 2 NameNodes, one active and one standby. The NameNodes use a set of JournalNodes and Zookepeer Servers to ensure the necessary synchronization. In case of ResourceManager HA 2 ResourceManagers should be enabled in addition. A typical Highly available Ambari cluster uses 2 separate NameNodes, 2 separate ResourceManagers and at least 3 JournalNodes and at least 3 Zookeeper Servers. HDP Version Support ------------------- The HDP plugin currently supports deployment of HDP 2.3, 2.4 and 2.5. Cluster Validation ------------------ Prior to Hadoop cluster creation, the HDP plugin will perform the following validation checks to ensure a successful Hadoop deployment: * Ensure the existence of Ambari Server process in the cluster; * Ensure the existence of a NameNode, Zookeeper, ResourceManagers processes HistoryServer and App TimeLine Server in the cluster Enabling Kerberos security for cluster -------------------------------------- If you want to protect your clusters using MIT Kerberos security you have to complete a few steps below. * If you would like to create a cluster protected by Kerberos security you just need to enable Kerberos by checkbox in the ``General Parameters`` section of the cluster configuration. If you prefer to use the OpenStack CLI for cluster creation, you have to put the data below in the ``cluster_configs`` section: .. sourcecode:: console "cluster_configs": { "Enable Kerberos Security": true, } Sahara in this case will correctly prepare KDC server and will create principals along with keytabs to enable authentication for Hadoop services. * Ensure that you have the latest hadoop-openstack jar file distributed on your cluster nodes. You can download one at ``http://tarballs.openstack.org/sahara/dist/`` * Sahara will create principals along with keytabs for system users like ``oozie``, ``hdfs`` and ``spark`` so that you will not have to perform additional auth operations to execute your jobs on top of the cluster. sahara-8.0.0/doc/source/user/storm-plugin.rst0000666000175100017510000000432213245514472021231 0ustar zuulzuul00000000000000Storm Plugin ============ The Storm plugin for sahara provides a way to provision Apache Storm clusters on OpenStack in a single click and in an easily repeatable fashion. Currently Storm is installed in standalone mode, with no YARN support. Images ------ For cluster provisioning, prepared images should be used. The Storm plugin has been developed and tested with the images generated by sahara-image-elements: * https://git.openstack.org/cgit/openstack/sahara-image-elements The Storm plugin requires an image to be tagged in the sahara image registry with two tags: 'storm' and '' (e.g. '1.1.0'). Also you should specify the username of the default cloud-user used in the image. Note that the Storm cluster is deployed using the scripts available in the Storm distribution, which allow the user to start all services (nimbus, supervisors and zookeepers), stop all services and so on. As such Storm is not deployed as a standard Ubuntu service and if the virtual machines are rebooted, Storm will not be restarted. Storm configuration ------------------- Storm needs few parameters to work and has sensible defaults. If needed they can be changed when creating the sahara cluster template. No node group options are available. Once the cluster is ready, connect with ssh to the master using the `ubuntu` user and the appropriate ssh key. Storm is installed in `/usr/local/storm` and should be completely configured and ready to start executing jobs. At the bottom of the cluster information page from the OpenStack dashboard, a link to the Storm web interface is provided. Cluster Validation ------------------ When a user creates a Storm cluster using the Storm plugin, the cluster topology requested by user is verified for consistency. Currently there are the following limitations in cluster topology for the Storm plugin: + Cluster must contain exactly one Storm nimbus + Cluster must contain at least one Storm supervisor + Cluster must contain at least one Zookeeper node The tested configuration has nimbus, supervisor, and Zookeeper processes each running on their own nodes. Another possible configuration is one node with nimbus alone, and additional nodes each with supervisor and Zookeeper processes together. sahara-8.0.0/doc/source/user/quickstart.rst0000666000175100017510000006461413245514472020775 0ustar zuulzuul00000000000000================ Quickstart guide ================ Launching a cluster via Sahara CLI commands =========================================== This guide will help you setup a vanilla Hadoop cluster using a combination of OpenStack command line tools and the sahara :doc:`REST API <../reference/restapi>`. 1. Install sahara ----------------- * If you want to hack the code follow :doc:`../contributor/development-environment`. OR * If you just want to install and use sahara follow :doc:`../install/installation-guide`. 2. Identity service configuration --------------------------------- To use the OpenStack command line tools you should specify environment variables with the configuration details for your OpenStack installation. The following example assumes that the Identity service is at ``127.0.0.1:5000``, with a user ``admin`` in the ``admin`` project whose password is ``nova``: .. sourcecode:: console $ export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/ $ export OS_PROJECT_NAME=admin $ export OS_USERNAME=admin $ export OS_PASSWORD=nova 3. Upload an image to the Image service --------------------------------------- You will need to upload a virtual machine image to the OpenStack Image service. You can download pre-built images with vanilla Apache Hadoop installed, or build the images yourself. This guide uses the latest available Ubuntu upstream image, referred to as ``sahara-vanilla-latest-ubuntu.qcow2`` and the latest version of vanilla plugin as an example. Sample images are available here: `Sample Images `_ * Download a pre-built image **Note:** For the steps below, substitute ```` with the appropriate OpenStack release and ```` with the image of your choice. .. sourcecode:: console $ ssh user@hostname $ wget http://sahara-files.mirantis.com/images/upstream//.qcow2 Upload the image downloaded above into the OpenStack Image service: .. sourcecode:: console $ openstack image create sahara-vanilla-latest-ubuntu --disk-format qcow2 \ --container-format bare --file sahara-vanilla-latest-ubuntu.qcow2 +------------------+--------------------------------------+ | Field | Value | +------------------+--------------------------------------+ | checksum | 3da49911332fc46db0c5fb7c197e3a77 | | container_format | bare | | created_at | 2016-02-29T10:15:04.000000 | | deleted | False | | deleted_at | None | | disk_format | qcow2 | | id | 71b9eeac-c904-4170-866a-1f833ea614f3 | | is_public | False | | min_disk | 0 | | min_ram | 0 | | name | sahara-vanilla-latest-ubuntu | | owner | 057d23cddb864759bfa61d730d444b1f | | properties | | | protected | False | | size | 1181876224 | | status | active | | updated_at | 2016-02-29T10:15:41.000000 | | virtual_size | None | +------------------+--------------------------------------+ OR * Build the image using: `diskimage-builder script `_ Remember the image name or save the image ID. This will be used during the image registration with sahara. You can get the image ID using the ``openstack`` command line tool as follows: .. sourcecode:: console $ openstack image list --property name=sahara-vanilla-latest-ubuntu +--------------------------------------+------------------------------+ | ID | Name | +--------------------------------------+------------------------------+ | 71b9eeac-c904-4170-866a-1f833ea614f3 | sahara-vanilla-latest-ubuntu | +--------------------------------------+------------------------------+ 4. Register the image with the sahara image registry ---------------------------------------------------- Now you will begin to interact with sahara by registering the virtual machine image in the sahara image registry. Register the image with the username ``ubuntu``. .. note:: The username will vary depending on the source image used, as follows: Ubuntu: ``ubuntu`` CentOS 7: ``centos`` CentOS 6: ``cloud-user`` Fedora: ``fedora`` Note that the Sahara team recommends using CentOS 7 instead of CentOS 6 as a base OS wherever possible; it is better supported throughout OpenStack image maintenance infrastructure and its more modern filesystem is much more appropriate for large-scale data processing. For more please see :doc:`../user/vanilla-plugin` .. sourcecode:: console $ openstack dataprocessing image register sahara-vanilla-latest-ubuntu \ --username ubuntu Tag the image to inform sahara about the plugin and the version with which it shall be used. .. note:: For the steps below and the rest of this guide, substitute ```` with the appropriate version of your plugin. .. sourcecode:: console $ openstack dataprocessing image tags add sahara-vanilla-latest-ubuntu \ --tags vanilla +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | Description | None | | Id | 71b9eeac-c904-4170-866a-1f833ea614f3 | | Name | sahara-vanilla-latest-ubuntu | | Status | ACTIVE | | Tags | , vanilla | | Username | ubuntu | +-------------+--------------------------------------+ 5. Create node group templates ------------------------------ Node groups are the building blocks of clusters in sahara. Before you can begin provisioning clusters you must define a few node group templates to describe node group configurations. You can get information about available plugins with the following command: .. sourcecode:: console $ openstack dataprocessing plugin list Also you can get information about available services for a particular plugin with the ``plugin show`` command. For example: .. sourcecode:: console $ openstack dataprocessing plugin show vanilla --plugin-version +---------------------+-----------------------------------------------------------------------------------------------------------------------+ | Field | Value | +---------------------+-----------------------------------------------------------------------------------------------------------------------+ | Description | The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any | | | management consoles. It can also deploy the Oozie component. | | Name | vanilla | | Required image tags | , vanilla | | Title | Vanilla Apache Hadoop | | | | | Service: | Available processes: | | | | | HDFS | datanode, namenode, secondarynamenode | | Hadoop | | | Hive | hiveserver | | JobFlow | oozie | | Spark | spark history server | | MapReduce | historyserver | | YARN | nodemanager, resourcemanager | +---------------------+-----------------------------------------------------------------------------------------------------------------------+ .. note:: These commands assume that floating IP addresses are being used. For more details on floating IP please see :ref:`floating_ip_management`. Create a master node group template with the command: .. sourcecode:: console $ openstack dataprocessing node group template create \ --name vanilla-default-master --plugin vanilla \ --plugin-version --processes namenode resourcemanager \ --flavor 2 --auto-security-group --floating-ip-pool +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | Auto security group | True | | Availability zone | None | | Flavor id | 2 | | Floating ip pool | dbd8d1aa-6e8e-4a35-a77b-966c901464d5 | | Id | 0f066e14-9a73-4379-bbb4-9d9347633e31 | | Is default | False | | Is protected | False | | Is proxy gateway | False | | Is public | False | | Name | vanilla-default-master | | Node processes | namenode, resourcemanager | | Plugin name | vanilla | | Security groups | None | | Use autoconfig | False | | Version | | | Volumes per node | 0 | +---------------------+--------------------------------------+ Create a worker node group template with the command: .. sourcecode:: console $ openstack dataprocessing node group template create \ --name vanilla-default-worker --plugin vanilla \ --plugin-version --processes datanode nodemanager \ --flavor 2 --auto-security-group --floating-ip-pool +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | Auto security group | True | | Availability zone | None | | Flavor id | 2 | | Floating ip pool | dbd8d1aa-6e8e-4a35-a77b-966c901464d5 | | Id | 6546bf44-0590-4539-bfcb-99f8e2c11efc | | Is default | False | | Is protected | False | | Is proxy gateway | False | | Is public | False | | Name | vanilla-default-worker | | Node processes | datanode, nodemanager | | Plugin name | vanilla | | Security groups | None | | Use autoconfig | False | | Version | | | Volumes per node | 0 | +---------------------+--------------------------------------+ Alternatively you can create node group templates from JSON files: If your environment does not use floating IPs, omit defining floating IP in the template below. Sample templates can be found here: `Sample Templates `_ Create a file named ``my_master_template_create.json`` with the following content: .. sourcecode:: json { "plugin_name": "vanilla", "hadoop_version": "", "node_processes": [ "namenode", "resourcemanager" ], "name": "vanilla-default-master", "floating_ip_pool": "", "flavor_id": "2", "auto_security_group": true } Create a file named ``my_worker_template_create.json`` with the following content: .. sourcecode:: json { "plugin_name": "vanilla", "hadoop_version": "", "node_processes": [ "nodemanager", "datanode" ], "name": "vanilla-default-worker", "floating_ip_pool": "", "flavor_id": "2", "auto_security_group": true } Use the ``openstack`` client to upload the node group templates: .. sourcecode:: console $ openstack dataprocessing node group template create \ --json my_master_template_create.json $ openstack dataprocessing node group template create \ --json my_worker_template_create.json List the available node group templates to ensure that they have been added properly: .. sourcecode:: console $ openstack dataprocessing node group template list --name vanilla-default +------------------------+--------------------------------------+-------------+--------------------+ | Name | Id | Plugin name | Version | +------------------------+--------------------------------------+-------------+--------------------+ | vanilla-default-master | 0f066e14-9a73-4379-bbb4-9d9347633e31 | vanilla | | | vanilla-default-worker | 6546bf44-0590-4539-bfcb-99f8e2c11efc | vanilla | | +------------------------+--------------------------------------+-------------+--------------------+ Remember the name or save the ID for the master and worker node group templates, as they will be used during cluster template creation. For example: * vanilla-default-master: ``0f066e14-9a73-4379-bbb4-9d9347633e31`` * vanilla-default-worker: ``6546bf44-0590-4539-bfcb-99f8e2c11efc`` 6. Create a cluster template ---------------------------- The last step before provisioning the cluster is to create a template that describes the node groups of the cluster. Create a cluster template with the command: .. sourcecode:: console $ openstack dataprocessing cluster template create \ --name vanilla-default-cluster \ --node-groups vanilla-default-master:1 vanilla-default-worker:3 +----------------+----------------------------------------------------+ | Field | Value | +----------------+----------------------------------------------------+ | Anti affinity | | | Description | None | | Id | 9d871ebd-88a9-40af-ae3e-d8c8f292401c | | Is default | False | | Is protected | False | | Is public | False | | Name | vanilla-default-cluster | | Node groups | vanilla-default-master:1, vanilla-default-worker:3 | | Plugin name | vanilla | | Use autoconfig | False | | Version | | +----------------+----------------------------------------------------+ Alternatively you can create cluster template from JSON file: Create a file named ``my_cluster_template_create.json`` with the following content: .. sourcecode:: json { "plugin_name": "vanilla", "hadoop_version": "", "node_groups": [ { "name": "worker", "count": 3, "node_group_template_id": "6546bf44-0590-4539-bfcb-99f8e2c11efc" }, { "name": "master", "count": 1, "node_group_template_id": "0f066e14-9a73-4379-bbb4-9d9347633e31" } ], "name": "vanilla-default-cluster", "cluster_configs": {} } Upload the cluster template using the ``openstack`` command line tool: .. sourcecode:: console $ openstack dataprocessing cluster template create --json my_cluster_template_create.json Remember the cluster template name or save the cluster template ID for use in the cluster provisioning command. The cluster ID can be found in the output of the creation command or by listing the cluster templates as follows: .. sourcecode:: console $ openstack dataprocessing cluster template list --name vanilla-default +-------------------------+--------------------------------------+-------------+--------------------+ | Name | Id | Plugin name | Version | +-------------------------+--------------------------------------+-------------+--------------------+ | vanilla-default-cluster | 9d871ebd-88a9-40af-ae3e-d8c8f292401c | vanilla | | +-------------------------+--------------------------------------+-------------+--------------------+ 7. Create cluster ----------------- Now you are ready to provision the cluster. This step requires a few pieces of information that can be found by querying various OpenStack services. Create a cluster with the command: .. sourcecode:: console $ openstack dataprocessing cluster create --name my-cluster-1 \ --cluster-template vanilla-default-cluster --user-keypair my_stack \ --neutron-network private --image sahara-vanilla-latest-ubuntu +----------------------------+----------------------------------------------------+ | Field | Value | +----------------------------+----------------------------------------------------+ | Anti affinity | | | Cluster template id | 9d871ebd-88a9-40af-ae3e-d8c8f292401c | | Description | | | Id | 1f0dc6f7-6600-495f-8f3a-8ac08cdb3afc | | Image | 71b9eeac-c904-4170-866a-1f833ea614f3 | | Is protected | False | | Is public | False | | Is transient | False | | Name | my-cluster-1 | | Neutron management network | fabe9dae-6fbd-47ca-9eb1-1543de325efc | | Node groups | vanilla-default-master:1, vanilla-default-worker:3 | | Plugin name | vanilla | | Status | Validating | | Use autoconfig | False | | User keypair id | my_stack | | Version | | +----------------------------+----------------------------------------------------+ Alternatively you can create a cluster template from a JSON file: Create a file named ``my_cluster_create.json`` with the following content: .. sourcecode:: json { "name": "my-cluster-1", "plugin_name": "vanilla", "hadoop_version": "", "cluster_template_id" : "9d871ebd-88a9-40af-ae3e-d8c8f292401c", "user_keypair_id": "my_stack", "default_image_id": "71b9eeac-c904-4170-866a-1f833ea614f3", "neutron_management_network": "fabe9dae-6fbd-47ca-9eb1-1543de325efc" } The parameter ``user_keypair_id`` with the value ``my_stack`` is generated by creating a keypair. You can create your own keypair in the OpenStack Dashboard, or through the ``openstack`` command line client as follows: .. sourcecode:: console $ openstack keypair create my_stack --public-key $PATH_TO_PUBLIC_KEY If sahara is configured to use neutron for networking, you will also need to include the ``--neutron-network`` argument in the ``cluster create`` command or the ``neutron_management_network`` parameter in ``my_cluster_create.json``. If your environment does not use neutron, you should omit these arguments. You can determine the neutron network id with the following command: .. sourcecode:: console $ openstack network list Create and start the cluster: .. sourcecode:: console $ openstack dataprocessing cluster create --json my_cluster_create.json Verify the cluster status by using the ``openstack`` command line tool as follows: .. sourcecode:: console $ openstack dataprocessing cluster show my-cluster-1 -c Status +--------+--------+ | Field | Value | +--------+--------+ | Status | Active | +--------+--------+ The cluster creation operation may take several minutes to complete. During this time the "status" returned from the previous command may show states other than ``Active``. A cluster also can be created with the ``wait`` flag. In that case the cluster creation command will not be finished until the cluster is moved to the ``Active`` state. 8. Run a MapReduce job to check Hadoop installation --------------------------------------------------- Check that your Hadoop installation is working properly by running an example job on the cluster manually. * Login to the NameNode (usually the master node) via ssh with the ssh-key used above: .. sourcecode:: console $ ssh -i my_stack.pem ubuntu@ * Switch to the hadoop user: .. sourcecode:: console $ sudo su hadoop * Go to the shared hadoop directory and run the simplest MapReduce example: .. sourcecode:: console $ cd /opt/hadoop-/share/hadoop/mapreduce $ /opt/hadoop-/bin/hadoop jar hadoop-mapreduce-examples-.jar pi 10 100 Congratulations! Your Hadoop cluster is ready to use, running on your OpenStack cloud. Elastic Data Processing (EDP) ============================= Job Binaries are the entities you define/upload the source code (mains and libraries) for your job. First you need to download your binary file or script to swift container and register your file in Sahara with the command: .. code:: bash (openstack) dataprocessing job binary create --url "swift://integration.sahara/hive.sql" \ --username username --password password --description "My first job binary" hive-binary Data Sources ------------ Data Sources are entities where the input and output from your jobs are housed. You can create data sources which are related to Swift, Manila or HDFS. You need to set the type of data source (swift, hdfs, manila, maprfs), name and url. The next two commands will create input and output data sources in swift. .. code:: bash $ openstack dataprocessing data source create --type swift --username admin --password admin \ --url "swift://integration.sahara/input.txt" input $ openstack dataprocessing data source create --type swift --username admin --password admin \ --url "swift://integration.sahara/output.txt" output If you want to create data sources in hdfs, use valid hdfs urls: .. code:: bash $ openstack dataprocessing data source create --type hdfs --url "hdfs://tmp/input.txt" input $ openstack dataprocessing data source create --type hdfs --url "hdfs://tmp/output.txt" output Job Templates (Jobs in API) --------------------------- In this step you need to create a job template. You have to set the type of the job template using the `type` parameter. Choose the main library using the job binary which was created in the previous step and set a name for the job template. Example of the command: .. code:: bash $ openstack dataprocessing job template create --type Hive \ --name hive-job-template --main hive-binary Jobs (Job Executions in API) ---------------------------- This is the last step in our guide. In this step you need to launch your job. You need to pass the following arguments: * The name or ID of input/output data sources for the job * The name or ID of the job template * The name or ID of the cluster on which to run the job For instance: .. code:: bash $ openstack dataprocessing job execute --input input --output output \ --job-template hive-job-template --cluster my-first-cluster You can check status of your job with the command: .. code:: bash $ openstack dataprocessing job show Once the job is marked as successful you can check the output data source. It will contain the output data of this job. Congratulations! sahara-8.0.0/doc/source/user/overview.rst0000666000175100017510000000631113245514472020437 0ustar zuulzuul00000000000000Getting Started ================ Clusters -------- A cluster deployed by sahara consists of node groups. Node groups vary by their role, parameters and number of machines. The picture below illustrates an example of a Hadoop cluster consisting of 3 node groups each having a different role (set of processes). .. image:: ../images/hadoop-cluster-example.jpg Node group parameters include Hadoop parameters like ``io.sort.mb`` or ``mapred.child.java.opts``, and several infrastructure parameters like the flavor for instances or storage location (ephemeral drive or cinder volume). A cluster is characterized by its node groups and its parameters. Like a node group, a cluster has data processing framework and infrastructure parameters. An example of a cluster-wide Hadoop parameter is ``dfs.replication``. For infrastructure, an example could be image which will be used to launch cluster instances. Templates --------- In order to simplify cluster provisioning sahara employs the concept of templates. There are two kinds of templates: node group templates and cluster templates. The former is used to create node groups, the latter - clusters. Essentially templates have the very same parameters as corresponding entities. Their aim is to remove the burden of specifying all of the required parameters each time a user wants to launch a cluster. In the REST interface, templates have extended functionality. First you can specify node-scoped parameters, they will work as defaults for node groups. Also with the REST interface, during cluster creation a user can override template parameters for both cluster and node groups. Templates are portable - they can be exported to JSON files and imported either on the same deployment or on another one. To import an exported template, replace the placeholder values with appropriate ones. This can be accomplished easily through the CLI or UI, or manually editing the template file. Provisioning Plugins -------------------- A provisioning plugin is a component responsible for provisioning a data processing cluster. Generally each plugin is capable of provisioning a specific data processing framework or Hadoop distribution. Also the plugin can install management and/or monitoring tools for a cluster. Since framework configuration parameters vary depending on the distribution and the version, templates are always plugin and version specific. A template cannot be used if the plugin, or framework, versions are different than the ones they were created for. You may find the list of available plugins on that page: :doc:`plugins` Image Registry -------------- OpenStack starts VMs based on a pre-built image with an installed OS. The image requirements for sahara depend on the plugin and data processing framework version. Some plugins require just a basic cloud image and will install the framework on the instance from scratch. Some plugins might require images with pre-installed frameworks or Hadoop distributions. The Sahara Image Registry is a feature which helps filter out images during cluster creation. See :doc:`registering-image` for details on how to work with Image Registry. Features -------- Sahara has several interesting features. The full list could be found here: :doc:`features` sahara-8.0.0/doc/source/user/spark-plugin.rst0000666000175100017510000000477213245514472021216 0ustar zuulzuul00000000000000Spark Plugin ============ The Spark plugin for sahara provides a way to provision Apache Spark clusters on OpenStack in a single click and in an easily repeatable fashion. Currently Spark is installed in standalone mode, with no YARN or Mesos support. Images ------ For cluster provisioning, prepared images should be used. The Spark plugin has been developed and tested with the images generated by sahara-image-elements: * https://git.openstack.org/cgit/openstack/sahara-image-elements The latest Ubuntu images generated by sahara-image-elements have Cloudera CDH 5.4.0 HDFS and Apache Spark installed. A prepared image for Spark can be found at the following location: * http://sahara-files.mirantis.com/images/upstream/ The Spark plugin requires an image to be tagged in the sahara image registry with two tags: 'spark' and '' (e.g. '1.6.0'). Also you should specify the username of the default cloud-user used in the image. For the images available at the URLs listed above and for all the ones generated with the DIB it is `ubuntu`. Note that the Spark cluster is deployed using the scripts available in the Spark distribution, which allow the user to start all services (master and slaves), stop all services and so on. As such (and as opposed to CDH HDFS daemons), Spark is not deployed as a standard Ubuntu service and if the virtual machines are rebooted, Spark will not be restarted. Spark configuration ------------------- Spark needs few parameters to work and has sensible defaults. If needed they can be changed when creating the sahara cluster template. No node group options are available. Once the cluster is ready, connect with ssh to the master using the `ubuntu` user and the appropriate ssh key. Spark is installed in `/opt/spark` and should be completely configured and ready to start executing jobs. At the bottom of the cluster information page from the OpenStack dashboard, a link to the Spark web interface is provided. Cluster Validation ------------------ When a user creates an Hadoop cluster using the Spark plugin, the cluster topology requested by user is verified for consistency. Currently there are the following limitations in cluster topology for the Spark plugin: + Cluster must contain exactly one HDFS namenode + Cluster must contain exactly one Spark master + Cluster must contain at least one Spark slave + Cluster must contain at least one HDFS datanode The tested configuration co-locates the NameNode with the master and a DataNode with each slave to maximize data locality. sahara-8.0.0/doc/source/user/vanilla-plugin.rst0000666000175100017510000000373413245514472021521 0ustar zuulzuul00000000000000Vanilla Plugin ============== The vanilla plugin is a reference implementation which allows users to operate a cluster with Apache Hadoop. Since the Newton release Spark is integrated into the Vanilla plugin so you can launch Spark jobs on a Vanilla cluster. For cluster provisioning prepared images should be used. They already have Apache Hadoop 2.7.1 installed. You may build images by yourself using :doc:`vanilla-imagebuilder` or you could download prepared images from http://sahara-files.mirantis.com/images/upstream Vanilla plugin requires an image to be tagged in Sahara Image Registry with two tags: 'vanilla' and '' (e.g. '2.7.1'). The default username specified for these images is different for each distribution: +--------------+------------+ | OS | username | +==============+============+ | Ubuntu 14 | ubuntu | +--------------+------------+ | Fedora 20 | fedora | +--------------+------------+ | CentOS 6 | cloud-user | +--------------+------------+ | CentOS 7 | centos | +--------------+------------+ Cluster Validation ------------------ When user creates or scales a Hadoop cluster using a Vanilla plugin, the cluster topology requested by user is verified for consistency. Currently there are the following limitations in cluster topology for Vanilla plugin: For Vanilla Hadoop version 2.x.x: + Cluster must contain exactly one namenode + Cluster can contain at most one resourcemanager + Cluster can contain at most one secondary namenode + Cluster can contain at most one historyserver + Cluster can contain at most one oozie and this process is also required for EDP + Cluster can't contain oozie without resourcemanager and without historyserver + Cluster can't have nodemanager nodes if it doesn't have resourcemanager + Cluster can have at most one hiveserver node. + Cluster can have at most one spark history server and this process is also required for Spark EDP (Spark is available since the Newton release). sahara-8.0.0/doc/source/user/dashboard-user-guide.rst0000666000175100017510000004627713245514472022606 0ustar zuulzuul00000000000000Sahara (Data Processing) UI User Guide ====================================== This guide assumes that you already have the sahara service and Horizon dashboard up and running. Don't forget to make sure that sahara is registered in Keystone. If you require assistance with that, please see the `installation guide <../install/installation-guide-html>`_. The sections below give a panel by panel overview of setting up clusters and running jobs. For a description of using the guided cluster and job tools, look at `Launching a cluster via the Cluster Creation Guide`_ and `Running a job via the Job Execution Guide`_. Launching a cluster via the sahara UI ------------------------------------- Registering an Image -------------------- 1) Navigate to the "Project" dashboard, then to the "Data Processing" tab, then click on the "Clusters" panel and finally the "Image Registry" tab. 2) From that page, click on the "Register Image" button at the top right 3) Choose the image that you'd like to register with sahara 4) Enter the username of the cloud-init user on the image 5) Choose plugin and version to make the image available only for the intended clusters 6) Click the "Done" button to finish the registration Create Node Group Templates --------------------------- 1) Navigate to the "Project" dashboard, then to the "Data Processing" tab, then click on the "Clusters" panel and then the "Node Group Templates" tab. 2) From that page, click on the "Create Template" button at the top right 3) Choose your desired Plugin name and Version from the dropdowns and click "Next" 4) Give your Node Group Template a name (description is optional) 5) Choose a flavor for this template (based on your CPU/memory/disk needs) 6) Choose the storage location for your instance, this can be either "Ephemeral Drive" or "Cinder Volume". If you choose "Cinder Volume", you will need to add additional configuration 7) Switch to the Node processes tab and choose which processes should be run for all instances that are spawned from this Node Group Template 8) Click on the "Create" button to finish creating your Node Group Template Create a Cluster Template ------------------------- 1) Navigate to the "Project" dashboard, then to the "Data Processing" tab, then click on the "Clusters" panel and finally the "Cluster Templates" tab. 2) From that page, click on the "Create Template" button at the top right 3) Choose your desired Plugin name and Version from the dropdowns and click "Next" 4) Under the "Details" tab, you must give your template a name 5) Under the "Node Groups" tab, you should add one or more nodes that can be based on one or more templates - To do this, start by choosing a Node Group Template from the dropdown and click the "+" button - You can adjust the number of nodes to be spawned for this node group via the text box or the "-" and "+" buttons - Repeat these steps if you need nodes from additional node group templates 6) Optionally, you can adjust your configuration further by using the "General Parameters", "HDFS Parameters" and "MapReduce Parameters" tabs 7) If you have Designate DNS service you can choose the domain name in "DNS" tab for internal and external hostname resolution 8) Click on the "Create" button to finish creating your Cluster Template Launching a Cluster ------------------- 1) Navigate to the "Project" dashboard, then to the "Data Processing" tab, then click on the "Clusters" panel and lastly, click on the "Clusters" tab. 2) Click on the "Launch Cluster" button at the top right 3) Choose your desired Plugin name and Version from the dropdowns and click "Next" 4) Give your cluster a name (required) 5) Choose which cluster template should be used for your cluster 6) Choose the image that should be used for your cluster (if you do not see any options here, see `Registering an Image`_ above) 7) Optionally choose a keypair that can be used to authenticate to your cluster instances 8) Click on the "Create" button to start your cluster - Your cluster's status will display on the Clusters table - It will likely take several minutes to reach the "Active" state Scaling a Cluster ----------------- 1) From the Data Processing/Clusters page (Clusters tab), click on the "Scale Cluster" button of the row that contains the cluster that you want to scale 2) You can adjust the numbers of instances for existing Node Group Templates 3) You can also add a new Node Group Template and choose a number of instances to launch - This can be done by selecting your desired Node Group Template from the dropdown and clicking the "+" button - Your new Node Group will appear below and you can adjust the number of instances via the text box or the "+" and "-" buttons 4) To confirm the scaling settings and trigger the spawning/deletion of instances, click on "Scale" Elastic Data Processing (EDP) ----------------------------- Data Sources ------------ Data Sources are where the input and output from your jobs are housed. 1) From the Data Processing/Jobs page (Data Sources tab), click on the "Create Data Source" button at the top right 2) Give your Data Source a name 3) Enter the URL of the Data Source - For a swift object, enter / (ie: *mycontainer/inputfile*). sahara will prepend *swift://* for you - For an HDFS object, enter an absolute path, a relative path or a full URL: + */my/absolute/path* indicates an absolute path in the cluster HDFS + *my/path* indicates the path */user/hadoop/my/path* in the cluster HDFS assuming the defined HDFS user is *hadoop* + *hdfs://host:port/path* can be used to indicate any HDFS location 4) Enter the username and password for the Data Source (also see `Additional Notes`_) 5) Enter an optional description 6) Click on "Create" 7) Repeat for additional Data Sources Job Binaries ------------ Job Binaries are where you define/upload the source code (mains and libraries) for your job. 1) From the Data Processing/Jobs (Job Binaries tab), click on the "Create Job Binary" button at the top right 2) Give your Job Binary a name (this can be different than the actual filename) 3) Choose the type of storage for your Job Binary - For "swift", enter the URL of your binary (/) as well as the username and password (also see `Additional Notes`_) - For "manila", choose the share and enter the path for the binary in this share. This assumes that you have already stored that file in the appropriate path on the share. The share will be automatically mounted to any cluster nodes which require access to the file, if it is not mounted already. - For "Internal database", you can choose from "Create a script" or "Upload a new file" (**only API v1.1**) 4) Enter an optional description 5) Click on "Create" 6) Repeat for additional Job Binaries Job Templates (Known as "Jobs" in the API) ------------------------------------------ Job templates are where you define the type of job you'd like to run as well as which "Job Binaries" are required. 1) From the Data Processing/Jobs page (Job Templates tab), click on the "Create Job Template" button at the top right 2) Give your Job Template a name 3) Choose the type of job you'd like to run 4) Choose the main binary from the dropdown - This is required for Hive, Pig, and Spark jobs - Other job types do not use a main binary 5) Enter an optional description for your Job Template 6) Click on the "Libs" tab and choose any libraries needed by your job template - MapReduce and Java jobs require at least one library - Other job types may optionally use libraries 7) Click on "Create" Jobs (Known as "Job Executions" in the API) ------------------------------------------- Jobs are what you get by "Launching" a job template. You can monitor the status of your job to see when it has completed its run 1) From the Data Processing/Jobs page (Job Templates tab), find the row that contains the job template you want to launch and click either "Launch on New Cluster" or "Launch on Existing Cluster" the right side of that row 2) Choose the cluster (already running--see `Launching a Cluster`_ above) on which you would like the job to run 3) Choose the Input and Output Data Sources (Data Sources defined above) 4) If additional configuration is required, click on the "Configure" tab - Additional configuration properties can be defined by clicking on the "Add" button - An example configuration entry might be mapred.mapper.class for the Name and org.apache.oozie.example.SampleMapper for the Value 5) Click on "Launch". To monitor the status of your job, you can navigate to the Data Processing/Jobs panel and click on the Jobs tab. 6) You can relaunch a Job from the Jobs page by using the "Relaunch on New Cluster" or "Relaunch on Existing Cluster" links - Relaunch on New Cluster will take you through the forms to start a new cluster before letting you specify input/output Data Sources and job configuration - Relaunch on Existing Cluster will prompt you for input/output Data Sources as well as allow you to change job configuration before launching the job Example Jobs ------------ There are sample jobs located in the sahara repository. In this section, we will give a walkthrough on how to run those jobs via the Horizon UI. These steps assume that you already have a cluster up and running (in the "Active" state). You may want to clone into https://git.openstack.org/cgit/openstack/sahara-tests/ so that you will have all of the source code and inputs stored locally. 1) Sample Pig job - https://git.openstack.org/cgit/openstack/sahara-tests/tree/sahara_tests/scenario/defaults/edp-examples/edp-pig/cleanup-string/example.pig - Load the input data file from https://git.openstack.org/cgit/openstack/sahara-tests/tree/sahara_tests/scenario/defaults/edp-examples/edp-pig/cleanup-string/data/input into swift - Click on Project/Object Store/Containers and create a container with any name ("samplecontainer" for our purposes here) - Click on Upload Object and give the object a name ("piginput" in this case) - Navigate to Data Processing/Jobs/Data Sources, Click on Create Data Source - Name your Data Source ("pig-input-ds" in this sample) - Type = Swift, URL samplecontainer/piginput, fill-in the Source username/password fields with your username/password and click "Create" - Create another Data Source to use as output for the job - Name = pig-output-ds, Type = Swift, URL = samplecontainer/pigoutput, Source username/password, "Create" - Store your Job Binaries in Swift (you can choose another type of storage if you want) - Navigate to Project/Object Store/Containers, choose "samplecontainer" - Click on Upload Object and find example.pig at /sahara-tests/scenario/defaults/edp-examples/ edp-pig/cleanup-string/, name it "example.pig" (or other name). The Swift path will be swift://samplecontainer/example.pig - Click on Upload Object and find edp-pig-udf-stringcleaner.jar at /sahara-tests/scenario/defaults/edp-examples/ edp-pig/cleanup-string/, name it "edp-pig-udf-stringcleaner.jar" (or other name). The Swift path will be swift://samplecontainer/edp-pig-udf-stringcleaner.jar - Navigate to Data Processing/Jobs/Job Binaries, Click on Create Job Binary - Name = example.pig, Storage type = Swift, URL = samplecontainer/example.pig, Username = , Password = - Create another Job Binary: Name = edp-pig-udf-stringcleaner.jar, Storage type = Swift, URL = samplecontainer/edp-pig-udf-stringcleaner.jar, Username = , Password = - Create a Job Template - Navigate to Data Processing/Jobs/Job Templates, Click on Create Job Template - Name = pigsample, Job Type = Pig, Choose "example.pig" as the main binary - Click on the "Libs" tab and choose "edp-pig-udf-stringcleaner.jar", then hit the "Choose" button beneath the dropdown, then click on "Create" - Launch your job - To launch your job from the Job Templates page, click on the down arrow at the far right of the screen and choose "Launch on Existing Cluster" - For the input, choose "pig-input-ds", for output choose "pig-output-ds". Also choose whichever cluster you'd like to run the job on - For this job, no additional configuration is necessary, so you can just click on "Launch" - You will be taken to the "Jobs" page where you can see your job progress through "PENDING, RUNNING, SUCCEEDED" phases - When your job finishes with "SUCCEEDED", you can navigate back to Object Store/Containers and browse to the samplecontainer to see your output. It should be in the "pigoutput" folder 2) Sample Spark job - https://git.openstack.org/cgit/openstack/sahara-tests/tree/sahara_tests/scenario/defaults/edp-examples/edp-spark You can clone into https://git.openstack.org/cgit/openstack/sahara-tests/ for quicker access to the files for this sample job. - Store the Job Binary in Swift (you can choose another type of storage if you want) - Click on Project/Object Store/Containers and create a container with any name ("samplecontainer" for our purposes here) - Click on Upload Object and find spark-wordcount.jar at /sahara-tests/scenario/defaults/edp-examples/ edp-spark/, name it "spark-wordcount.jar" (or other name). The Swift path will be swift://samplecontainer/spark-wordcount.jar - Navigate to Data Processing/Jobs/Job Binaries, Click on Create Job Binary - Name = sparkexample.jar, Storage type = Swift, URL = samplecontainer/spark-wordcount.jar, Username = , Password = - Create a Job Template - Name = sparkexamplejob, Job Type = Spark, Main binary = Choose sparkexample.jar, Click "Create" - Launch your job - To launch your job from the Job Templates page, click on the down arrow at the far right of the screen and choose "Launch on Existing Cluster" - Choose whichever cluster you'd like to run the job on - Click on the "Configure" tab - Set the main class to be: sahara.edp.spark.SparkWordCount - Under Arguments, click Add and fill url for the input file, once more click Add and fill url for the output file. - Click on Launch - You will be taken to the "Jobs" page where you can see your job progress through "PENDING, RUNNING, SUCCEEDED" phases - When your job finishes with "SUCCEEDED", you can see your results in your output file. - The stdout and stderr files of the command used for executing your job are located at /tmp/spark-edp// on Spark master node in case of Spark clusters, or on Spark JobHistory node in other cases like Vanilla, CDH and so on. Additional Notes ---------------- 1) Throughout the sahara UI, you will find that if you try to delete an object that you will not be able to delete it if another object depends on it. An example of this would be trying to delete a Job Template that has an existing Job. In order to be able to delete that job, you would first need to delete any Job Templates that relate to that job. 2) In the examples above, we mention adding your username/password for the swift Data Sources. It should be noted that it is possible to configure sahara such that the username/password credentials are *not* required. For more information on that, please refer to: :doc:`Sahara Advanced Configuration Guide <../admin/advanced-configuration-guide>` Launching a cluster via the Cluster Creation Guide -------------------------------------------------- 1) Under the Data Processing group, choose "Clusters" and then click on the "Clusters" tab. The "Cluster Creation Guide" button is above that table. Click on it. 2) Click on the "Choose Plugin" button then select the cluster type from the Plugin Name dropdown and choose your target version. When done, click on "Select" to proceed. 3) Click on "Create a Master Node Group Template". Give your template a name, choose a flavor and choose which processes should run on nodes launched for this node group. The processes chosen here should be things that are more server-like in nature (namenode, oozieserver, spark master, etc). Optionally, you can set other options here such as availability zone, storage, security and process specific parameters. Click on "Create" to proceed. 4) Click on "Create a Worker Node Group Template". Give your template a name, choose a flavor and choose which processes should run on nodes launched for this node group. Processes chosen here should be more worker-like in nature (datanode, spark slave, task tracker, etc). Optionally, you can set other options here such as availability zone, storage, security and process specific parameters. Click on "Create" to proceed. 5) Click on "Create a Cluster Template". Give your template a name. Next, click on the "Node Groups" tab and enter the count for each of the node groups (these are pre-populated from steps 3 and 4). It would be common to have 1 for the "master" node group type and some larger number of "worker" instances depending on you desired cluster size. Optionally, you can also set additional parameters for cluster-wide settings via the other tabs on this page. Click on "Create" to proceed. 6) Click on "Launch a Cluster". Give your cluster a name and choose the image that you want to use for all instances in your cluster. The cluster template that you created in step 5 is already pre-populated. If you want ssh access to the instances of your cluster, select a keypair from the dropdown. Click on "Launch" to proceed. You will be taken to the Clusters panel where you can see your cluster progress toward the Active state. Running a job via the Job Execution Guide ----------------------------------------- 1) Under the Data Processing group, choose "Jobs" and then click on the "Jobs" tab. The "Job Execution Guide" button is above that table. Click on it. 2) Click on "Select type" and choose the type of job that you want to run. 3) If your job requires input/output data sources, you will have the option to create them via the "Create a Data Source" button (Note: This button will not be shown for job types that do not require data sources). Give your data source a name and choose the type. If you have chosen swift, you may also enter the username and password. Enter the URL for your data source. For more details on what the URL should look like, see `Data Sources`_. 4) Click on "Create a job template". Give your job template a name. Depending on the type of job that you've chosen, you may need to select your main binary and/or additional libraries (available from the "Libs" tab). If you have not yet uploaded the files to run your program, you can add them via the "+" icon next to the "Choose a main binary" select box. 5) Click on "Launch job". Choose the active cluster where you want to run you job. Optionally, you can click on the "Configure" tab and provide any required configuration, arguments or parameters for your job. Click on "Launch" to execute your job. You will be taken to the Jobs tab where you can monitor the state of your job as it progresses. sahara-8.0.0/doc/source/user/vanilla-imagebuilder.rst0000666000175100017510000000755013245514472022654 0ustar zuulzuul00000000000000.. _diskimage-builder-label: Building Images for Vanilla Plugin ================================== In this document you will find instruction on how to build Ubuntu, Fedora, and CentOS images with Apache Hadoop version 2.x.x. As of now the vanilla plugin works with images with pre-installed versions of Apache Hadoop. To simplify the task of building such images we use `Disk Image Builder `_. `Disk Image Builder` builds disk images using elements. An element is a particular set of code that alters how the image is built, or runs within the chroot to prepare the image. Elements for building vanilla images are stored in the `Sahara image elements repository `_ .. note:: Sahara requires images with cloud-init package installed: * `For Fedora `_ * `For Ubuntu 14 `_ * `For CentOS 6 `_ * `For CentOS 7 `_ To create vanilla images follow these steps: 1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements" locally. 2. Use tox to build images. You can run the command below in sahara-image-elements directory to build images. By default this script will attempt to create cloud images for all versions of supported plugins and all operating systems (subset of Ubuntu, Fedora, and CentOS depending on plugin). .. sourcecode:: console tox -e venv -- sahara-image-create -u If you want to build Vanilla 2.7.1 image with centos 7 just execute: .. sourcecode:: console tox -e venv -- sahara-image-create -p vanilla -v 2.7.1 -i centos7 Tox will create a virtualenv and install required python packages in it, clone the repositories "https://git.openstack.org/cgit/openstack/diskimage-builder" and "https://git.openstack.org/cgit/openstack/sahara-image-elements" and export necessary parameters. * ``DIB_HADOOP_VERSION`` - version of Hadoop to install * ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin) * ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built Oozie libs here: ``http://sahara-files.mirantis.com/oozie-4.2.0-hadoop-2.7.1.tar.gz``) * ``SPARK_DOWNLOAD_URL`` - download link for Spark * ``HIVE_VERSION`` - version of Hive to install (currently supports only 0.11.0) * ``ubuntu_image_name`` * ``fedora_image_name`` * ``DIB_IMAGE_SIZE`` - parameter that specifies a volume of hard disk of instance. You need to specify it only for Fedora because Fedora doesn't use all available volume * ``DIB_COMMIT_ID`` - latest commit id of diskimage-builder project * ``SAHARA_ELEMENTS_COMMIT_ID`` - latest commit id of sahara-image-elements project NOTE: If you don't want to use default values, you should set your values of parameters. Then it will create required cloud images using image elements that install all the necessary packages and configure them. You will find created images in the parent directory. .. note:: Disk Image Builder will generate QCOW2 images, used with the default OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different hypervisor, the generated image should be converted to an appropriate format. VMware Nova backend requires VMDK image format. You may use qemu-img utility to convert a QCOW2 image to VMDK. .. sourcecode:: console qemu-img convert -O vmdk .qcow2 .vmdk For finer control of diskimage-create.sh see the `official documentation `_ sahara-8.0.0/doc/source/user/hadoop-swift.rst0000666000175100017510000001241213245514476021200 0ustar zuulzuul00000000000000.. _swift-integration-label: Swift Integration ================= Hadoop and Swift integration are the essential continuation of the Hadoop/OpenStack marriage. The key component to making this marriage work is the Hadoop Swift filesystem implementation. Although this implementation has been merged into the upstream Hadoop project, Sahara maintains a version with the most current features enabled. * The original Hadoop patch can be found at https://issues.apache.org/jira/browse/HADOOP-8545 * The most current Sahara maintained version of this patch can be found in the `Sahara Extra repository `_ * The latest compiled version of the jar for this component can be downloaded from http://tarballs.openstack.org/sahara/dist/hadoop-openstack/master/ Now the latest version of this jar (which uses Keystone API v3) is used in the plugins' images automatically during build of these images. But for Ambari plugin we need to explicitly put this jar into /opt directory of the base image **before** cluster launching. Hadoop patching --------------- You may build the jar file yourself by choosing the latest patch from the Sahara Extra repository and using Maven to build with the pom.xml file provided. Or you may get the latest jar pre-built at http://tarballs.openstack.org/sahara/dist/hadoop-openstack/master/ You will need to put this file into the hadoop libraries (e.g. /usr/lib/share/hadoop/lib, it depends on the plugin which you use) on each ResourceManager and NodeManager node (for Hadoop 2.x) in the cluster. Hadoop configurations --------------------- In general, when Sahara runs a job on a cluster it will handle configuring the Hadoop installation. In cases where a user might require more in-depth configuration all the data is set in the ``core-site.xml`` file on the cluster instances using this template: .. sourcecode:: xml ${name} + ${config} ${value} ${not mandatory description} There are two types of configs here: 1. General. The ``${name}`` in this case equals to ``fs.swift``. Here is the list of ``${config}``: * ``.impl`` - Swift FileSystem implementation. The ${value} is ``org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem`` * ``.connect.timeout`` - timeout for all connections by default: 15000 * ``.socket.timeout`` - how long the connection waits for responses from servers. by default: 60000 * ``.connect.retry.count`` - connection retry count for all connections. by default: 3 * ``.connect.throttle.delay`` - delay in millis between bulk (delete, rename, copy operations). by default: 0 * ``.blocksize`` - blocksize for filesystem. By default: 32Mb * ``.partsize`` - the partition size for uploads. By default: 4608*1024Kb * ``.requestsize`` - request size for reads in KB. By default: 64Kb 2. Provider-specific. The patch for Hadoop supports different cloud providers. The ``${name}`` in this case equals to ``fs.swift.service.${provider}``. Here is the list of ``${config}``: * ``.auth.url`` - authorization URL * ``.auth.endpoint.prefix`` - prefix for the service url, e.g. ``/AUTH_`` * ``.tenant`` - project name * ``.username`` * ``.password`` * ``.domain.name`` - Domains can be used to specify users who are not in the project specified. * ``.domain.id`` - You can also specify domain using id. * ``.trust.id`` - Trusts are optionally used to scope the authentication tokens of the supplied user. * ``.http.port`` * ``.https.port`` * ``.region`` - Swift region is used when cloud has more than one Swift installation. If region param is not set first region from Keystone endpoint list will be chosen. If region param not found exception will be thrown. * ``.location-aware`` - turn On location awareness. Is false by default * ``.apikey`` * ``.public`` Example ------- For this example it is assumed that you have setup a Hadoop instance with a valid configuration and the Swift filesystem component. Furthermore there is assumed to be a Swift container named ``integration`` holding an object named ``temp``, as well as a Keystone user named ``admin`` with a password of ``swordfish``. The following example illustrates how to copy an object to a new location in the same container. We will use Hadoop's ``distcp`` command (http://hadoop.apache.org/docs/stable/hadoop-distcp/DistCp.html) to accomplish the copy. Note that the service provider for our Swift access is ``sahara``, and that we will not need to specify the project of our Swift container as it will be provided in the Hadoop configuration. Swift paths are expressed in Hadoop according to the following template: ``swift://${container}.${provider}/${object}``. For our example source this will appear as ``swift://integration.sahara/temp``. Let's run the job: .. sourcecode:: console $ hadoop distcp -D fs.swift.service.sahara.username=admin \ -D fs.swift.service.sahara.password=swordfish \ swift://integration.sahara/temp swift://integration.sahara/temp1 After that just confirm that ``temp1`` has been created in our ``integration`` container. Limitations ----------- **Note:** Please note that container names should be a valid URI. sahara-8.0.0/doc/source/user/plugins.rst0000666000175100017510000000331613245514472020254 0ustar zuulzuul00000000000000Provisioning Plugins ==================== This page lists all available provisioning plugins. In general a plugin enables sahara to deploy a specific data processing framework (for example, Hadoop) or distribution, and allows configuration of topology and management/monitoring tools. * :doc:`vanilla-plugin` - deploys Vanilla Apache Hadoop * :doc:`ambari-plugin` - deploys Hortonworks Data Platform * :doc:`spark-plugin` - deploys Apache Spark with Cloudera HDFS * :doc:`storm-plugin` - deploys Apache Storm * :doc:`mapr-plugin` - deploys MapR plugin with MapR File System * :doc:`cdh-plugin` - deploys Cloudera Hadoop Managing plugins ---------------- Since the Newton release a project admin can configure plugins by specifying additional values for plugin's labels. To disable a plugin (Vanilla Apache Hadoop, for example), the admin can run the following command: .. sourcecode:: console cat update_configs.json { "plugin_labels": { "enabled": { "status": true } } } openstack dataprocessing plugin update vanilla update_configs.json Additionally, specific versions can be disabled by the following command: .. sourcecode:: console cat update_configs.json { "version_labels": { "2.7.1": { "enabled": { "status": true } } } } openstack dataprocessing plugin update vanilla update_configs.json Finally, to see all labels of a specific plugin and to see the current status of the plugin (is it stable or not, deprecation status) the following command can be executed from the CLI: .. sourcecode:: console openstack dataprocessing plugin show vanilla The same actions are available from UI respectively. sahara-8.0.0/doc/source/user/features.rst0000666000175100017510000002737213245514472020421 0ustar zuulzuul00000000000000Features Overview ================= This page highlights some of the most prominent features available in sahara. The guidance provided here is primarily focused on the runtime aspects of sahara. For discussions about configuring the sahara server processes please see the :doc:`../admin/configuration-guide` and :doc:`../admin/advanced-configuration-guide`. Anti-affinity ------------- One of the problems with running data processing applications on OpenStack is the inability to control where an instance is actually running. It is not always possible to ensure that two new virtual machines are started on different physical machines. As a result, any replication within the cluster is not reliable because all replicas may be co-located on one physical machine. To remedy this, sahara provides the anti-affinity feature to explicitly command all instances of the specified processes to spawn on different Compute nodes. This is especially useful for Hadoop data node processes to increase HDFS replica reliability. Starting with the Juno release, sahara can create server groups with the ``anti-affinity`` policy to enable this feature. Sahara creates one server group per cluster and assigns all instances with affected processes to this server group. Refer to the `Nova Anti-Affinity documentation`_ on how server group affinity filters work. This feature is supported by all plugins out of the box, and can be enabled during the cluster template creation. .. _Nova Anti-Affinity documentation: https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#servergroupantiaffinityfilter Block Storage support --------------------- OpenStack Block Storage (cinder) can be used as an alternative for ephemeral drives on instances. Using Block Storage volumes increases the reliability of data which is important for HDFS services. A user can set how many volumes will be attached to each instance in a node group and the size of each volume. All volumes are attached during cluster creation and scaling operations. If volumes are used for the HDFS storage it's important to make sure that the linear read-write operations as well as IOpS level are high enough to handle the workload. Volumes placed on the same compute host provide a higher level of performance. In some cases cinder volumes can be backed by a distributed storage like Ceph. In this type of installation it's important to make sure that the network latency and speed do not become a blocker for HDFS. Distributed storage solutions usually provide their own replication mechanism. HDFS replication should be disabled so that it does not generate redundant traffic across the cloud. Cluster scaling --------------- Cluster scaling allows users to change the number of running instances in a cluster without needing to recreate the cluster. Users may increase or decrease the number of instances in node groups or add new node groups to existing clusters. If a cluster fails to scale properly, all changes will be rolled back. Data locality ------------- For optimal performance, it is best for data processing applications to work on data local to the same rack, OpenStack Compute node, or virtual machine. Hadoop supports a data locality feature and can schedule jobs to task tracker nodes that are local for the input stream. In this manner the task tracker nodes can communicate directly with the local data nodes. Sahara supports topology configuration for HDFS and Object Storage data sources. For more information on configuring this option please see the :ref:`data_locality_configuration` documentation. Volume-to-instance locality --------------------------- Having an instance and an attached volume on the same physical host can be very helpful in order to achieve high-performance disk I/O operations. To achieve this, sahara provides access to the Block Storage volume instance locality functionality. For more information on using volume instance locality with sahara, please see the :ref:`volume_instance_locality_configuration` documentation. Distributed Mode ---------------- The :doc:`../install/installation-guide` suggests launching sahara in distributed mode with ``sahara-api`` and ``sahara-engine`` processes potentially running on several machines simultaneously. Running in distributed mode allows sahara to offload intensive tasks to the engine processes while keeping the API process free to handle requests. For an expanded discussion of configuring sahara to run in distributed mode please see the :ref:`distributed-mode-configuration` documentation. Hadoop HDFS and YARN High Availability -------------------------------------- Currently HDFS and YARN HA are supported with the HDP 2.4 plugin and CDH 5.7 plugins. Hadoop HDFS and YARN High Availability provide an architecture to ensure that HDFS or YARN will continue to work in the result of an active namenode or resourcemanager failure. They use 2 namenodes and 2 resourcemanagers in an active/passive state to provide this availability. In the HDP 2.4 plugin, the feature can be enabled through dashboard in the Cluster Template creation form. High availability is achieved by using a set of journalnodes, Zookeeper servers, and ZooKeeper Failover Controllers (ZKFC), as well as additional configuration changes to HDFS and other services that use HDFS. In the CDH 5.7 plugin, HA for HDFS and YARN is enabled through adding several HDFS_JOURNALNODE roles in the node group templates of cluster template. The HDFS HA is enabled when HDFS_JOURNALNODE roles are added and the roles setup meets below requirements: * HDFS_JOURNALNODE number is odd, and at least 3. * Zookeeper is enabled. * NameNode and SecondaryNameNode are on different physical hosts by setting anti-affinity. * Cluster has both ResourceManager and StandByResourceManager. In this case, the original SecondaryNameNode node will be used as the Standby NameNode. Networking support ------------------ Sahara supports neutron implementations of OpenStack Networking. Object Storage support ---------------------- Sahara can use OpenStack Object Storage (swift) to store job binaries and data sources utilized by its job executions and clusters. In order to leverage this support within Hadoop, including using Object Storage for data sources for EDP, Hadoop requires the application of a patch. For additional information about enabling this support, including patching Hadoop and configuring sahara, please refer to the :doc:`hadoop-swift` documentation. Shared Filesystem support ------------------------- Sahara can also use NFS shares through the OpenStack Shared Filesystem service (manila) to store job binaries and data sources. See :doc:`edp` for more information on this feature. Orchestration support --------------------- Sahara may use the `OpenStack Orchestration engine `_ (heat) to provision nodes for clusters. For more information about enabling Orchestration usage in sahara please see :ref:`orchestration-configuration`. DNS support ----------- Sahara can resolve hostnames of cluster instances by using DNS. For this Sahara uses designate. For additional details see :doc:`../admin/advanced-configuration-guide`. Kerberos support ---------------- You can protect your HDP or CDH cluster using MIT Kerberos security. To get more details about this, please, see documentation for the appropriate plugin. Plugin Capabilities ------------------- The following table provides a plugin capability matrix: +--------------------------+---------+----------+----------+-------+ | Feature/Plugin | Vanilla | HDP | Cloudera | Spark | +==========================+=========+==========+==========+=======+ | Neutron network | x | x | x | x | +--------------------------+---------+----------+----------+-------+ | Cluster Scaling | x | x | x | x | +--------------------------+---------+----------+----------+-------+ | Swift Integration | x | x | x | x | +--------------------------+---------+----------+----------+-------+ | Cinder Support | x | x | x | x | +--------------------------+---------+----------+----------+-------+ | Data Locality | x | x | x | x | +--------------------------+---------+----------+----------+-------+ | DNS | x | x | x | x | +--------------------------+---------+----------+----------+-------+ | Kerberos | \- | x | x | \- | +--------------------------+---------+----------+----------+-------+ | HDFS HA | \- | x | x | \- | +--------------------------+---------+----------+----------+-------+ | EDP | x | x | x | x | +--------------------------+---------+----------+----------+-------+ Security group management ------------------------- Security groups are sets of IP filter rules that are applied to an instance's networking. They are project specified, and project members can edit the default rules for their group and add new rules sets. All projects have a "default" security group, which is applied to instances that have no other security group defined. Unless changed, this security group denies all incoming traffic. Sahara allows you to control which security groups will be used for created instances. This can be done by providing the ``security_groups`` parameter for the node group or node group template. The default for this option is an empty list, which will result in the default project security group being used for the instances. Sahara may also create a security group for instances in the node group automatically. This security group will only contain open ports for required instance processes and the sahara engine. This option is useful for development and for when your installation is secured from outside environments. For production environments we recommend controlling the security group policy manually. Shared and protected resources support -------------------------------------- Sahara allows you to create resources that can be shared across projects and protected from modifications. To provide this feature all sahara objects that can be accessed through REST API have ``is_public`` and ``is_protected`` boolean fields. They can be initially created with enabled ``is_public`` and ``is_protected`` parameters or these parameters can be updated after creation. Both fields are set to ``False`` by default. If some object has its ``is_public`` field set to ``True``, it means that it's visible not only from the project in which it was created, but from any other projects too. If some object has its ``is_protected`` field set to ``True``, it means that it can not be modified (updated, scaled, canceled or deleted) unless this field is set to ``False``. Public objects created in one project can be used from other projects (for example, a cluster can be created from a public cluster template which is created in another project), but modification operations are possible only from the project in which object was created. Data source placeholders support -------------------------------- Sahara supports special strings that can be used in data source URLs. These strings will be replaced with appropriate values during job execution which allows the use of the same data source as an output multiple times. There are 2 types of string currently supported: * ``%JOB_EXEC_ID%`` - this string will be replaced with the job execution ID. * ``%RANDSTR(len)%`` - this string will be replaced with random string of lowercase letters of length ``len``. ``len`` must be less than 1024. After placeholders are replaced, the real URLs are stored in the ``data_source_urls`` field of the job execution object. This is used later to find objects created by a particular job run. sahara-8.0.0/doc/source/user/registering-image.rst0000666000175100017510000000207513245514472022176 0ustar zuulzuul00000000000000Registering an Image ==================== Sahara deploys a cluster of machines using images stored in Glance. Each plugin has its own requirements on the image contents (see specific plugin documentation for details). Two general requirements for an image are to have the cloud-init and the ssh-server packages installed. Sahara requires the images to be registered in the Sahara Image Registry. A registered image must have two properties set: * username - a name of the default cloud-init user. * tags - certain tags mark image to be suitable for certain plugins. The tags depend on the plugin used, you can find required tags in the plugin's documentations. The default username specified for these images is different for each distribution: +--------------+------------+ | OS | username | +==============+============+ | Ubuntu 12,14 | ubuntu | +--------------+------------+ | Fedora | fedora | +--------------+------------+ | CentOS 6.x | cloud-user | +--------------+------------+ | CentOS 7.x | centos | +--------------+------------+ sahara-8.0.0/doc/source/user/index.rst0000666000175100017510000000115513245514472017701 0ustar zuulzuul00000000000000========== User Guide ========== General concepts and guides =========================== .. toctree:: :maxdepth: 2 overview quickstart dashboard-user-guide features registering-image statuses sahara-on-ironic Plugins ======= .. toctree:: :maxdepth: 2 plugins vanilla-plugin ambari-plugin spark-plugin storm-plugin cdh-plugin mapr-plugin Elastic Data Processing ======================= .. toctree:: :maxdepth: 2 edp Guest Images ============ .. toctree:: :maxdepth: 2 guest-requirements hadoop-swift vanilla-imagebuilder cdh-imagebuilder sahara-8.0.0/doc/source/user/guest-requirements.rst0000666000175100017510000000337013245514472022443 0ustar zuulzuul00000000000000Requirements for Guests ======================= Sahara manages guests of various platforms (for example Ubuntu, Fedora, RHEL, and CentOS) with various versions of the Hadoop ecosystem projects installed. There are common requirements for all guests, and additional requirements based on the plugin that is used for cluster deployment. Common Requirements ------------------- * The operating system must be Linux * cloud-init must be installed * ssh-server must be installed + if a firewall is active it must allow connections on port 22 to enable ssh Vanilla Plugin Requirements --------------------------- If the Vanilla Plugin is used for cluster deployment the guest is required to have * ssh-client installed * Java (version >= 6) * Apache Hadoop installed * 'hadoop' user created See :doc:`hadoop-swift` for information on using Swift with your sahara cluster (for EDP support Swift integration is currently required). To support EDP, the following components must also be installed on the guest: * Oozie version 4 or higher * mysql * hive See :doc:`vanilla-imagebuilder` for instructions on building images for this plugin. Hortonworks Plugin Requirements ------------------------------- This plugin does not have any additional requirements. Currently, only the CentOS Linux and Ubuntu distributions are supported but other distributions will be supported in the future. To speed up provisioning, the HDP packages can be pre-installed on the image used. The packages' versions depend on the HDP version being used. Cloudera Plugin Requirements ---------------------------- Cloudera Plugin does not have any additional requirements, just build a CDH image to deploy the cluster. See :doc:`cdh-imagebuilder` for instructions on building images for this plugin. sahara-8.0.0/doc/source/user/mapr-plugin.rst0000666000175100017510000000750513245514472021032 0ustar zuulzuul00000000000000MapR Distribution Plugin ======================== The MapR Sahara plugin allows to provision MapR clusters on OpenStack in an easy way and do it, quickly, conveniently and simply. Operation --------- The MapR Plugin performs the following four primary functions during cluster creation: 1. MapR components deployment - the plugin manages the deployment of the required software to the target VMs 2. Services Installation - MapR services are installed according to provided roles list 3. Services Configuration - the plugin combines default settings with user provided settings 4. Services Start - the plugin starts appropriate services according to specified roles Images ------ The Sahara MapR plugin can make use of either minimal (operating system only) images or pre-populated MapR images. The base requirement for both is that the image is cloud-init enabled and contains a supported operating system (see http://maprdocs.mapr.com/home/InteropMatrix/r_os_matrix.html). The advantage of a pre-populated image is that provisioning time is reduced, as packages do not need to be downloaded which make up the majority of the time spent in the provisioning cycle. In addition, provisioning large clusters will put a burden on the network as packages for all nodes need to be downloaded from the package repository. For more information about MapR images, refer to ``_. There are VM images provided for use with the MapR Plugin, that can also be built using the tools available in sahara-image-elements: https://s3-us-west-2.amazonaws.com/sahara-images/index.html MapR plugin needs an image to be tagged in Sahara Image Registry with two tags: 'mapr' and '' (e.g. '5.2.0.mrv2'). The default username specified for these images is different for each distribution: +--------------+------------+ | OS | username | +==============+============+ | Ubuntu 14 | ubuntu | +--------------+------------+ | CentOS 6 | cloud-user | +--------------+------------+ | CentOS 7 | centos | +--------------+------------+ Hadoop Version Support ---------------------- The MapR plugin currently supports Hadoop 2.7.0 (5.2.0.mrv2). Cluster Validation ------------------ When the user creates or scales a Hadoop cluster using a mapr plugin, the cluster topology requested by the user is verified for consistency. Every MapR cluster must contain: * at least 1 *CLDB* process * exactly 1 *Webserver* process * odd number of *ZooKeeper* processes but not less than 1 * *FileServer* process on every node * at least 1 ephemeral drive (then you need to specify the ephemeral drive in the flavor not on the node group template creation) or 1 Cinder volume per instance Every Hadoop cluster must contain exactly 1 *Oozie* process Every MapReduce v1 cluster must contain: * at least 1 *JobTracker* process * at least 1 *TaskTracker* process Every MapReduce v2 cluster must contain: * exactly 1 *ResourceManager* process * exactly 1 *HistoryServer* process * at least 1 *NodeManager* process Every Spark cluster must contain: * exactly 1 *Spark Master* process * exactly 1 *Spark HistoryServer* process * at least 1 *Spark Slave* (worker) process HBase service is considered valid if: * cluster has at least 1 *HBase-Master* process * cluster has at least 1 *HBase-RegionServer* process Hive service is considered valid if: * cluster has exactly 1 *HiveMetastore* process * cluster has exactly 1 *HiveServer2* process Hue service is considered valid if: * cluster has exactly 1 *Hue* process * *Hue* process resides on the same node as *HttpFS* process HttpFS service is considered valid if cluster has exactly 1 *HttpFS* process Sqoop service is considered valid if cluster has exactly 1 *Sqoop2-Server* process The MapR Plugin --------------- For more information, please contact MapR. sahara-8.0.0/doc/source/user/statuses.rst0000666000175100017510000001222513245514472020445 0ustar zuulzuul00000000000000Sahara Cluster Statuses Overview ================================ All Sahara Cluster operations are performed in multiple steps. A Cluster object has a ``Status`` attribute which changes when Sahara finishes one step of operations and starts another one. Also a Cluster object has a ``Status description`` attribute which changes whenever Cluster errors occur. Sahara supports three types of Cluster operations: * Create a new Cluster * Scale/Shrink an existing Cluster * Delete an existing Cluster Creating a new Cluster ---------------------- 1. Validating ~~~~~~~~~~~~~ Before performing any operations with OpenStack environment, Sahara validates user input. There are two types of validations, that are done: * Check that a request contains all necessary fields and that the request does not violate any constraints like unique naming, etc. * Plugin check (optional). The provisioning Plugin may also perform any specific checks like a Cluster topology validation check. If any of the validations fails during creating, the Cluster object will still be kept in the database with an ``Error`` status. If any validations fails during scaling the ``Active`` Cluster, it will be kept with an ``Active`` status. In both cases status description will contain error messages about the reasons of failure. 2. InfraUpdating ~~~~~~~~~~~~~~~~ This status means that the Provisioning plugin is performing some infrastructure updates. 3. Spawning ~~~~~~~~~~~ Sahara sends requests to OpenStack for all resources to be created: * VMs * Volumes * Floating IPs (if Sahara is configured to use Floating IPs) It takes some time for OpenStack to schedule all the required VMs and Volumes, so sahara will wait until all of the VMs are in an ``Active`` state. 4. Waiting ~~~~~~~~~~ Sahara waits while VMs' operating systems boot up and all internal infrastructure components like networks and volumes are attached and ready to use. 5. Preparing ~~~~~~~~~~~~ Sahara prepares a Cluster for starting. This step includes generating the ``/etc/hosts`` file or changing ``/etc/resolv.conf`` file (if you use Designate service), so that all instances can access each other by a hostname. Also Sahara updates the ``authorized_keys`` file on each VM, so that VMs can communicate without passwords. 6. Configuring ~~~~~~~~~~~~~~ Sahara pushes service configurations to VMs. Both XML and JSON based configurations and environmental variables are set on this step. 7. Starting ~~~~~~~~~~~ Sahara is starting Hadoop services on Cluster's VMs. 8. Active ~~~~~~~~~ Active status means that a Cluster has started successfully and is ready to run EDP Jobs. Scaling/Shrinking an existing Cluster ------------------------------------- 1. Validating ~~~~~~~~~~~~~ Sahara checks the scale/shrink request for validity. The Plugin method called for performing Plugin specific checks is different from the validation method in creation. 2. Scaling ~~~~~~~~~~ Sahara performs database operations updating all affected existing Node Groups and creating new ones to join the existing Node Groups. 3. Adding Instances ~~~~~~~~~~~~~~~~~~~ Status is similar to ``Spawning`` in Cluster creation. Sahara adds required amount of VMs to the existing Node Groups and creates new Node Groups. 4. Configuring ~~~~~~~~~~~~~~ Status is similar to ``Configuring`` in Cluster creation. New instances are being configured in the same manner as already existing ones. The VMs in the existing Cluster are also updated with a new ``/etc/hosts`` file or ``/etc/resolv.conf`` file. 5. Decommissioning ~~~~~~~~~~~~~~~~~~ Sahara stops Hadoop services on VMs that will be deleted from a Cluster. Decommissioning a Data Node may take some time because Hadoop rearranges data replicas around the Cluster, so that no data will be lost after that Data Node is deleted. 6. Deleting Instances ~~~~~~~~~~~~~~~~~~~~~ Sahara sends requests to OpenStack to release unneeded resources: * VMs * Volumes * Floating IPs (if they are used) 7. Active ~~~~~~~~~ The same ``Active`` status as after Cluster creation. Deleting an existing Cluster ---------------------------- 1. Deleting ~~~~~~~~~~~ The only step, that releases all Cluster's resources and removes it from the database. 2. Force Deleting ~~~~~~~~~~~~~~~~~ In extreme cases the regular "Deleting" step may hang. Sahara APIv2 introduces the ability to force delete a Cluster. This prevents deleting from hanging but comes with the risk of orphaned resources. Error State ----------- If the Cluster creation fails, the Cluster will enter the ``Error`` state. This status means the Cluster may not be able to perform any operations normally. This cluster will stay in the database until it is manually deleted. The reason for failure may be found in the sahara logs. Also, the status description will contain information about the error. If an error occurs during the ``Adding Instances`` operation, Sahara will first try to rollback this operation. If a rollback is impossible or fails itself, then the Cluster will also go into an ``Error`` state. If a rollback was successful, Cluster will get into an ``Active`` state and status description will contain a short message about the reason of ``Adding Instances`` failure. sahara-8.0.0/doc/source/user/edp.rst0000666000175100017510000007345613245514472017357 0ustar zuulzuul00000000000000Elastic Data Processing (EDP) ============================= Overview -------- Sahara's Elastic Data Processing facility or :dfn:`EDP` allows the execution of jobs on clusters created from sahara. EDP supports: * Hive, Pig, MapReduce, MapReduce.Streaming, Java, and Shell job types on Hadoop clusters * Spark jobs on Spark standalone clusters, MapR (v5.0.0 - v5.2.0) clusters, Vanilla clusters (v2.7.1) and CDH clusters (v5.3.0 or higher). * storage of job binaries in the OpenStack Object Storage service (swift), the OpenStack Shared file systems service (manila), sahara's own database, or any S3-like object store * access to input and output data sources in + HDFS for all job types + swift for all types excluding Hive + manila (NFS shares only) for all types excluding Pig * configuration of jobs at submission time * execution of jobs on existing clusters or transient clusters Interfaces ---------- The EDP features can be used from the sahara web UI which is described in the :doc:`dashboard-user-guide`. The EDP features also can be used directly by a client through the `REST api `_ EDP Concepts ------------ Sahara EDP uses a collection of simple objects to define and execute jobs. These objects are stored in the sahara database when they are created, allowing them to be reused. This modular approach with database persistence allows code and data to be reused across multiple jobs. The essential components of a job are: * executable code to run * input and output data paths, as needed for the job * any additional configuration values needed for the job run These components are supplied through the objects described below. Job Binaries ++++++++++++ A :dfn:`Job Binary` object stores a URL to a single script or Jar file and any credentials needed to retrieve the file. The file itself may be stored in the sahara internal database (**only API v1.1**), in swift, or in manila. Files in the sahara database are stored as raw bytes in a :dfn:`Job Binary Internal` object. This object's sole purpose is to store a file for later retrieval. No extra credentials need to be supplied for files stored internally. Sahara requires credentials (username and password) to access files stored in swift unless swift proxy users are configured as described in :doc:`../admin/advanced-configuration-guide`. The swift service must be running in the same OpenStack installation referenced by sahara. Sahara requires the following credentials/configs to access files stored in an S3-like object store: ``accesskey``, ``secretkey``, ``endpoint``. These credentials are specified through the `extra` in the body of the request when creating a job binary or data source referencing S3. To reference a binary file stored in manila, create the job binary with the URL ``manila://{share_id}/{path}``. This assumes that you have already stored that file in the appropriate path on the share. The share will be automatically mounted to any cluster nodes which require access to the file, if it is not mounted already. There is a configurable limit on the size of a single job binary that may be retrieved by sahara. This limit is 5MB and may be set with the *job_binary_max_KB* setting in the :file:`sahara.conf` configuration file. Jobs ++++ A :dfn:`Job` object specifies the type of the job and lists all of the individual Job Binary objects that are required for execution. An individual Job Binary may be referenced by multiple Jobs. A Job object specifies a main binary and/or supporting libraries depending on its type: +-------------------------+-------------+-----------+ | Job type | Main binary | Libraries | +=========================+=============+===========+ | ``Hive`` | required | optional | +-------------------------+-------------+-----------+ | ``Pig`` | required | optional | +-------------------------+-------------+-----------+ | ``MapReduce`` | not used | required | +-------------------------+-------------+-----------+ | ``MapReduce.Streaming`` | not used | optional | +-------------------------+-------------+-----------+ | ``Java`` | not used | required | +-------------------------+-------------+-----------+ | ``Shell`` | required | optional | +-------------------------+-------------+-----------+ | ``Spark`` | required | optional | +-------------------------+-------------+-----------+ | ``Storm`` | required | not used | +-------------------------+-------------+-----------+ | ``Storm Pyelus`` | required | not used | +-------------------------+-------------+-----------+ Data Sources ++++++++++++ A :dfn:`Data Source` object stores a URL which designates the location of input or output data and any credentials needed to access the location. Sahara supports data sources in swift. The swift service must be running in the same OpenStack installation referenced by sahara. Sahara also supports data sources in HDFS. Any HDFS instance running on a sahara cluster in the same OpenStack installation is accessible without manual configuration. Other instances of HDFS may be used as well provided that the URL is resolvable from the node executing the job. Sahara supports data sources in manila as well. To reference a path on an NFS share as a data source, create the data source with the URL ``manila://{share_id}/{path}``. As in the case of job binaries, the specified share will be automatically mounted to your cluster's nodes as needed to access the data source. Some job types require the use of data source objects to specify input and output when a job is launched. For example, when running a Pig job the UI will prompt the user for input and output data source objects. Other job types like Java or Spark do not require the user to specify data sources. For these job types, data paths are passed as arguments. For convenience, sahara allows data source objects to be referenced by name or id. The section `Using Data Source References as Arguments`_ gives further details. Job Execution +++++++++++++ Job objects must be *launched* or *executed* in order for them to run on the cluster. During job launch, a user specifies execution details including data sources, configuration values, and program arguments. The relevant details will vary by job type. The launch will create a :dfn:`Job Execution` object in sahara which is used to monitor and manage the job. To execute Hadoop jobs, sahara generates an Oozie workflow and submits it to the Oozie server running on the cluster. Familiarity with Oozie is not necessary for using sahara but it may be beneficial to the user. A link to the Oozie web console can be found in the sahara web UI in the cluster details. For Spark jobs, sahara uses the *spark-submit* shell script and executes the Spark job from the master node in case of Spark cluster and from the Spark Job History server in other cases. Logs of spark jobs run by sahara can be found on this node under the */tmp/spark-edp* directory. .. _edp_workflow: General Workflow ---------------- The general workflow for defining and executing a job in sahara is essentially the same whether using the web UI or the REST API. 1. Launch a cluster from sahara if there is not one already available 2. Create all of the Job Binaries needed to run the job, stored in the sahara database, in swift, or in manila + When using the REST API and internal storage of job binaries, the Job Binary Internal objects must be created first + Once the Job Binary Internal objects are created, Job Binary objects may be created which refer to them by URL 3. Create a Job object which references the Job Binaries created in step 2 4. Create an input Data Source which points to the data you wish to process 5. Create an output Data Source which points to the location for output data 6. Create a Job Execution object specifying the cluster and Job object plus relevant data sources, configuration values, and program arguments + When using the web UI this is done with the :guilabel:`Launch On Existing Cluster` or :guilabel:`Launch on New Cluster` buttons on the Jobs tab + When using the REST API this is done via the */jobs//execute* method The workflow is simpler when using existing objects. For example, to construct a new job which uses existing binaries and input data a user may only need to perform steps 3, 5, and 6 above. Of course, to repeat the same job multiple times a user would need only step 6. Specifying Configuration Values, Parameters, and Arguments ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Jobs can be configured at launch. The job type determines the kinds of values that may be set: +--------------------------+---------------+------------+-----------+ | Job type | Configuration | Parameters | Arguments | | | Values | | | +==========================+===============+============+===========+ | ``Hive`` | Yes | Yes | No | +--------------------------+---------------+------------+-----------+ | ``Pig`` | Yes | Yes | Yes | +--------------------------+---------------+------------+-----------+ | ``MapReduce`` | Yes | No | No | +--------------------------+---------------+------------+-----------+ | ``MapReduce.Streaming`` | Yes | No | No | +--------------------------+---------------+------------+-----------+ | ``Java`` | Yes | No | Yes | +--------------------------+---------------+------------+-----------+ | ``Shell`` | Yes | Yes | Yes | +--------------------------+---------------+------------+-----------+ | ``Spark`` | Yes | No | Yes | +--------------------------+---------------+------------+-----------+ | ``Storm`` | Yes | No | Yes | +--------------------------+---------------+------------+-----------+ | ``Storm Pyelus`` | Yes | No | Yes | +--------------------------+---------------+------------+-----------+ * :dfn:`Configuration values` are key/value pairs. + The EDP configuration values have names beginning with *edp.* and are consumed by sahara + Other configuration values may be read at runtime by Hadoop jobs + Currently additional configuration values are not available to Spark jobs at runtime * :dfn:`Parameters` are key/value pairs. They supply values for the Hive and Pig parameter substitution mechanisms. In Shell jobs, they are passed as environment variables. * :dfn:`Arguments` are strings passed as command line arguments to a shell or main program These values can be set on the :guilabel:`Configure` tab during job launch through the web UI or through the *job_configs* parameter when using the */jobs//execute* REST method. In some cases sahara generates configuration values or parameters automatically. Values set explicitly by the user during launch will override those generated by sahara. Using Data Source References as Arguments +++++++++++++++++++++++++++++++++++++++++ Sometimes it's necessary or desirable to pass a data path as an argument to a job. In these cases, a user may simply type out the path as an argument when launching a job. If the path requires credentials, the user can manually add the credentials as configuration values. However, if a data source object has been created that contains the desired path and credentials there is no need to specify this information manually. As a convenience, sahara allows data source objects to be referenced by name or id in arguments, configuration values, or parameters. When the job is executed, sahara will replace the reference with the path stored in the data source object and will add any necessary credentials to the job configuration. Referencing an existing data source object is much faster than adding this information by hand. This is particularly useful for job types like Java or Spark that do not use data source objects directly. There are two job configuration parameters that enable data source references. They may be used with any job type and are set on the ``Configuration`` tab when the job is launched: * ``edp.substitute_data_source_for_name`` (default **False**) If set to **True**, causes sahara to look for data source object name references in configuration values, arguments, and parameters when a job is launched. Name references have the form **datasource://name_of_the_object**. For example, assume a user has a WordCount application that takes an input path as an argument. If there is a data source object named **my_input**, a user may simply set the **edp.substitute_data_source_for_name** configuration parameter to **True** and add **datasource://my_input** as an argument when launching the job. * ``edp.substitute_data_source_for_uuid`` (default **False**) If set to **True**, causes sahara to look for data source object ids in configuration values, arguments, and parameters when a job is launched. A data source object id is a uuid, so they are unique. The id of a data source object is available through the UI or the sahara command line client. A user may simply use the id as a value. Creating an Interface for Your Job ++++++++++++++++++++++++++++++++++ In order to better document your job for cluster operators (or for yourself in the future), sahara allows the addition of an interface (or method signature) to your job template. A sample interface for the Teragen Hadoop example might be: +---------+---------+-----------+-------------+----------+--------------------+ | Name | Mapping | Location | Value | Required | Default | | | Type | | Type | | | +=========+=========+===========+=============+==========+====================+ | Example | args | 0 | string | false | teragen | | Class | | | | | | +---------+---------+-----------+-------------+----------+--------------------+ | Rows | args | 1 | number | true | unset | +---------+---------+-----------+-------------+----------+--------------------+ | Output | args | 2 | data_source | false | hdfs://ip:port/path| | Path | | | | | | +---------+---------+-----------+-------------+----------+--------------------+ | Mapper | configs | mapred. | number | false | unset | | Count | | map.tasks | | | | +---------+---------+-----------+-------------+----------+--------------------+ A "Description" field may also be added to each interface argument. To create such an interface via the REST API, provide an "interface" argument, the value of which consists of a list of JSON objects, as below: .. sourcecode:: json [ { "name": "Example Class", "description": "Indicates which example job class should be used.", "mapping_type": "args", "location": "0", "value_type": "string", "required": false, "default": "teragen" }, ] Creating this interface would allow you to specify a configuration for any execution of the job template by passing an "interface" map similar to: .. sourcecode:: json { "Rows": "1000000", "Mapper Count": "3", "Output Path": "hdfs://mycluster:8020/user/myuser/teragen-output" } The specified arguments would be automatically placed into the args, configs, and params for the job, according to the mapping type and location fields of each interface argument. The final ``job_configs`` map would be: .. sourcecode:: json { "job_configs": { "configs": { "mapred.map.tasks": "3" }, "args": [ "teragen", "1000000", "hdfs://mycluster:8020/user/myuser/teragen-output" ] } } Rules for specifying an interface are as follows: - Mapping Type must be one of ``configs``, ``params``, or ``args``. Only types supported for your job type are allowed (see above.) - Location must be a string for ``configs`` and ``params``, and an integer for ``args``. The set of ``args`` locations must be an unbroken series of integers starting from 0. - Value Type must be one of ``string``, ``number``, or ``data_source``. Data sources may be passed as UUIDs or as valid paths (see above.) All values should be sent as JSON strings. (Note that booleans and null values are serialized differently in different languages. Please specify them as a string representation of the appropriate constants for your data processing engine.) - ``args`` that are not required must be given a default value. The additional one-time complexity of specifying an interface on your template allows a simpler repeated execution path, and also allows us to generate a customized form for your job in the Horizon UI. This may be particularly useful in cases in which an operator who is not a data processing job developer will be running and administering the jobs. Generation of Swift Properties for Data Sources +++++++++++++++++++++++++++++++++++++++++++++++ If swift proxy users are not configured (see :doc:`../admin/advanced-configuration-guide`) and a job is run with data source objects containing swift paths, sahara will automatically generate swift username and password configuration values based on the credentials in the data sources. If the input and output data sources are both in swift, it is expected that they specify the same credentials. The swift credentials may be set explicitly with the following configuration values: +------------------------------------+ | Name | +====================================+ | fs.swift.service.sahara.username | +------------------------------------+ | fs.swift.service.sahara.password | +------------------------------------+ Setting the swift credentials explicitly is required when passing literal swift paths as arguments instead of using data source references. When possible, use data source references as described in `Using Data Source References as Arguments`_. Additional Details for Hive jobs ++++++++++++++++++++++++++++++++ Sahara will automatically generate values for the ``INPUT`` and ``OUTPUT`` parameters required by Hive based on the specified data sources. Additional Details for Pig jobs +++++++++++++++++++++++++++++++ Sahara will automatically generate values for the ``INPUT`` and ``OUTPUT`` parameters required by Pig based on the specified data sources. For Pig jobs, ``arguments`` should be thought of as command line arguments separated by spaces and passed to the ``pig`` shell. ``Parameters`` are a shorthand and are actually translated to the arguments ``-param name=value`` Additional Details for MapReduce jobs +++++++++++++++++++++++++++++++++++++ **Important!** If the job type is MapReduce, the mapper and reducer classes *must* be specified as configuration values. Note that the UI will not prompt the user for these required values; they must be added manually with the ``Configure`` tab. Make sure to add these values with the correct names: +-----------------------------+----------------------------------------+ | Name | Example Value | +=============================+========================================+ | mapred.mapper.new-api | true | +-----------------------------+----------------------------------------+ | mapred.reducer.new-api | true | +-----------------------------+----------------------------------------+ | mapreduce.job.map.class | org.apache.oozie.example.SampleMapper | +-----------------------------+----------------------------------------+ | mapreduce.job.reduce.class | org.apache.oozie.example.SampleReducer | +-----------------------------+----------------------------------------+ Additional Details for MapReduce.Streaming jobs +++++++++++++++++++++++++++++++++++++++++++++++ **Important!** If the job type is MapReduce.Streaming, the streaming mapper and reducer classes *must* be specified. In this case, the UI *will* prompt the user to enter mapper and reducer values on the form and will take care of adding them to the job configuration with the appropriate names. If using the python client, however, be certain to add these values to the job configuration manually with the correct names: +-------------------------+---------------+ | Name | Example Value | +=========================+===============+ | edp.streaming.mapper | /bin/cat | +-------------------------+---------------+ | edp.streaming.reducer | /usr/bin/wc | +-------------------------+---------------+ Additional Details for Java jobs ++++++++++++++++++++++++++++++++ Data Source objects are not used directly with Java job types. Instead, any input or output paths must be specified as arguments at job launch either explicitly or by reference as described in `Using Data Source References as Arguments`_. Using data source references is the recommended way to pass paths to Java jobs. If configuration values are specified, they must be added to the job's Hadoop configuration at runtime. There are two methods of doing this. The simplest way is to use the **edp.java.adapt_for_oozie** option described below. The other method is to use the code from `this example `_ to explicitly load the values. The following special configuration values are read by sahara and affect how Java jobs are run: * ``edp.java.main_class`` (required) Specifies the full name of the class containing ``main(String[] args)`` A Java job will execute the **main** method of the specified main class. Any arguments set during job launch will be passed to the program through the **args** array. * ``oozie.libpath`` (optional) Specifies configuration values for the Oozie share libs, these libs can be shared by different workflows * ``edp.java.java_opts`` (optional) Specifies configuration values for the JVM * ``edp.java.adapt_for_oozie`` (optional) Specifies that sahara should perform special handling of configuration values and exit conditions. The default is **False**. If this configuration value is set to **True**, sahara will modify the job's Hadoop configuration before invoking the specified **main** method. Any configuration values specified during job launch (excluding those beginning with **edp.**) will be automatically set in the job's Hadoop configuration and will be available through standard methods. Secondly, setting this option to **True** ensures that Oozie will handle program exit conditions correctly. At this time, the following special configuration value only applies when running jobs on a cluster generated by the Cloudera plugin with the **Enable Hbase Common Lib** cluster config set to **True** (the default value): * ``edp.hbase_common_lib`` (optional) Specifies that a common Hbase lib generated by sahara in HDFS be added to the **oozie.libpath**. This for use when an Hbase application is driven from a Java job. Default is **False**. The **edp-wordcount** example bundled with sahara shows how to use configuration values, arguments, and swift data paths in a Java job type. Note that the example does not use the **edp.java.adapt_for_oozie** option but includes the code to load the configuration values explicitly. Additional Details for Shell jobs +++++++++++++++++++++++++++++++++ A shell job will execute the script specified as ``main``, and will place any files specified as ``libs`` in the same working directory (on both the filesystem and in HDFS). Command line arguments may be passed to the script through the ``args`` array, and any ``params`` values will be passed as environment variables. Data Source objects are not used directly with Shell job types but data source references may be used as described in `Using Data Source References as Arguments`_. The **edp-shell** example bundled with sahara contains a script which will output the executing user to a file specified by the first command line argument. Additional Details for Spark jobs +++++++++++++++++++++++++++++++++ Data Source objects are not used directly with Spark job types. Instead, any input or output paths must be specified as arguments at job launch either explicitly or by reference as described in `Using Data Source References as Arguments`_. Using data source references is the recommended way to pass paths to Spark jobs. Spark jobs use some special configuration values: * ``edp.java.main_class`` (required) Specifies the full name of the class containing the Java or Scala main method: + ``main(String[] args)`` for Java + ``main(args: Array[String]`` for Scala A Spark job will execute the **main** method of the specified main class. Any arguments set during job launch will be passed to the program through the **args** array. * ``edp.spark.adapt_for_swift`` (optional) If set to **True**, instructs sahara to modify the job's Hadoop configuration so that swift paths may be accessed. Without this configuration value, swift paths will not be accessible to Spark jobs. The default is **False**. * ``edp.spark.driver.classpath`` (optional) If set to empty string sahara will use default classpath for the cluster during job execution. Otherwise this will override default value for the cluster for particular job execution. The **edp-spark** example bundled with sahara contains a Spark program for estimating Pi. Special Sahara URLs ------------------- Sahara uses custom URLs to refer to objects stored in swift, in manila, in the sahara internal database, or in S3-like storage. These URLs are usually not meant to be used outside of sahara. Sahara swift URLs passed to running jobs as input or output sources include a ".sahara" suffix on the container, for example: ``swift://container.sahara/object`` You may notice these swift URLs in job logs, however, you do not need to add the suffix to the containers yourself. sahara will add the suffix if necessary, so when using the UI or the python client you may write the above URL simply as: ``swift://container/object`` Sahara internal database URLs have the form: ``internal-db://sahara-generated-uuid`` This indicates a file object in the sahara database which has the given uuid as a key. Manila NFS filesystem reference URLS take the form: ``manila://share-uuid/path`` This format should be used when referring to a job binary or a data source stored in a manila NFS share. For job binaries only, S3 urls take the form: ``s3://bucket/path/to/object`` EDP Requirements ================ The OpenStack installation and the cluster launched from sahara must meet the following minimum requirements in order for EDP to function: OpenStack Services ------------------ When a Hadoop job is executed, binaries are first uploaded to a cluster node and then moved from the node local filesystem to HDFS. Therefore, there must be an instance of HDFS available to the nodes in the sahara cluster. If the swift service *is not* running in the OpenStack installation: + Job binaries may only be stored in the sahara internal database + Data sources require a long-running HDFS If the swift service *is* running in the OpenStack installation: + Job binaries may be stored in swift or the sahara internal database + Data sources may be in swift or a long-running HDFS Cluster Processes ----------------- Requirements for EDP support depend on the EDP job type and plugin used for the cluster. For example a Vanilla sahara cluster must run at least one instance of these processes to support EDP: * For Hadoop version 1: + jobtracker + namenode + oozie + tasktracker + datanode * For Hadoop version 2: + namenode + datanode + resourcemanager + nodemanager + historyserver + oozie + spark history server EDP Technical Considerations ============================ There are several things in EDP which require attention in order to work properly. They are listed on this page. Transient Clusters ------------------ EDP allows running jobs on transient clusters. In this case the cluster is created specifically for the job and is shut down automatically once the job is finished. Two config parameters control the behaviour of periodic clusters: * periodic_enable - if set to 'false', sahara will do nothing to a transient cluster once the job it was created for is completed. If it is set to 'true', then the behaviour depends on the value of the next parameter. * use_identity_api_v3 - set it to 'false' if your OpenStack installation does not provide keystone API v3. In that case sahara will not terminate unneeded clusters. Instead it will set their state to 'AwaitingTermination' meaning that they could be manually deleted by a user. If the parameter is set to 'true', sahara will itself terminate the cluster. The limitation is caused by lack of 'trusts' feature in Keystone API older than v3. If both parameters are set to 'true', sahara works with transient clusters in the following manner: 1. When a user requests for a job to be executed on a transient cluster, sahara creates such a cluster. 2. Sahara drops the user's credentials once the cluster is created but prior to that it creates a trust allowing it to operate with the cluster instances in the future without user credentials. 3. Once a cluster is not needed, sahara terminates its instances using the stored trust. sahara drops the trust after that. sahara-8.0.0/doc/source/config-generator.conf0000666000175100017510000000061613245514472021163 0ustar zuulzuul00000000000000[DEFAULT] wrap_width = 79 namespace = sahara.config namespace = keystonemiddleware.auth_token namespace = oslo.concurrency namespace = oslo.db namespace = oslo.log namespace = oslo.messaging namespace = oslo.middleware.cors namespace = oslo.middleware.http_proxy_to_wsgi namespace = oslo.policy namespace = oslo.service.periodic_task namespace = oslo.service.sslutils namespace = oslo.service.wsgi sahara-8.0.0/doc/source/conf.py0000666000175100017510000002113213245514472016356 0ustar zuulzuul00000000000000# -*- coding: utf-8 -*- # Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import subprocess import sys import warnings # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) sys.path.insert(0, os.path.abspath('../../sahara')) sys.path.append(os.path.abspath('..')) sys.path.append(os.path.abspath('../bin')) # -- General configuration ----------------------------------------------------- on_rtd = os.environ.get('READTHEDOCS', None) == 'True' # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.viewcode', 'sphinxcontrib.httpdomain', 'oslo_config.sphinxconfiggen', 'oslo_config.sphinxext', 'openstackdocstheme'] # openstackdocstheme options repository_name = 'openstack/sahara' bug_project = 'sahara' bug_tag = 'doc' html_last_updated_fmt = '%Y-%m-%d %H:%M' config_generator_config_file = 'config-generator.conf' config_sample_basename = 'sahara' # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = u'Sahara' copyright = u'2014, OpenStack Foundation' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # Version info from sahara.version import version_info as s_version release = s_version.release_string() # The short X.Y version. version = s_version.version_string() # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = [] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. if on_rtd: html_theme_path = ['.'] html_theme = '_theme_rtd' html_theme = 'openstackdocs' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". html_title = 'Sahara' # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". # html_static_path = ['_static'] # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. html_sidebars = { 'index': ['sidebarlinks.html', 'localtoc.html', 'searchbox.html', 'sourcelink.html'], '**': ['localtoc.html', 'relations.html', 'searchbox.html', 'sourcelink.html'] } # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'SaharaDoc' # -- Options for LaTeX output -------------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. #'preamble': '', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'saharadoc.tex', u'Sahara', u'OpenStack Foundation', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output -------------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'sahara', u'Sahara', [u'OpenStack Foundation'], 1) ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------------ # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'Sahara', u'Sahara', u'OpenStack Foundation', 'Sahara', 'Sahara', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' sahara-8.0.0/doc/source/reference/0000775000175100017510000000000013245515026017010 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/reference/edp-spi.rst0000666000175100017510000001502513245514472021114 0ustar zuulzuul00000000000000Elastic Data Processing (EDP) SPI ================================= The EDP job engine objects provide methods for creating, monitoring, and terminating jobs on Sahara clusters. Provisioning plugins that support EDP must return an EDP job engine object from the :ref:`get_edp_engine` method described in :doc:`plugin-spi`. Sahara provides subclasses of the base job engine interface that support EDP on clusters running Oozie, Spark, and/or Storm. These are described below. .. _edp_spi_job_types: Job Types --------- Some of the methods below test job type. Sahara supports the following string values for job types: * Hive * Java * Pig * MapReduce * MapReduce.Streaming * Spark * Shell * Storm .. note:: Constants for job types are defined in *sahara.utils.edp*. Job Status Values ------------------------ Several of the methods below return a job status value. A job status value is a dictionary of the form: {'status': *job_status_value*} where *job_status_value* is one of the following string values: * DONEWITHERROR * FAILED * TOBEKILLED * KILLED * PENDING * RUNNING * SUCCEEDED Note, constants for job status are defined in *sahara.utils.edp* EDP Job Engine Interface ------------------------ The sahara.service.edp.base_engine.JobEngine class is an abstract class with the following interface: cancel_job(job_execution) ~~~~~~~~~~~~~~~~~~~~~~~~~ Stops the running job whose id is stored in the job_execution object. *Returns*: None if the operation was unsuccessful or an updated job status value. get_job_status(job_execution) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns the current status of the job whose id is stored in the job_execution object. *Returns*: a job status value. run_job(job_execution) ~~~~~~~~~~~~~~~~~~~~~~ Starts the job described by the job_execution object *Returns*: a tuple of the form (job_id, job_status_value, job_extra_info). * *job_id* is required and must be a string that allows the EDP engine to uniquely identify the job. * *job_status_value* may be None or a job status value * *job_extra_info* may be None or optionally a dictionary that the EDP engine uses to store extra information on the job_execution_object. validate_job_execution(cluster, job, data) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Checks whether or not the job can run on the cluster with the specified data. Data contains values passed to the */jobs//execute* REST API method during job launch. If the job cannot run for any reason, including job configuration, cluster configuration, or invalid data, this method should raise an exception. *Returns*: None get_possible_job_config(job_type) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns hints used by the Sahara UI to prompt users for values when configuring and launching a job. Note that no hints are required. See :doc:`../user/edp` for more information on how configuration values, parameters, and arguments are used by different job types. *Returns*: a dictionary of the following form, containing hints for configs, parameters, and arguments for the job type: {'job_config': {'configs': [], 'params': {}, 'args': []}} * *args* is a list of strings * *params* contains simple key/value pairs * each item in *configs* is a dictionary with entries for 'name' (required), 'value', and 'description' get_supported_job_types() ~~~~~~~~~~~~~~~~~~~~~~~~~ This method returns the job types that the engine supports. Not all engines will support all job types. *Returns*: a list of job types supported by the engine. Oozie Job Engine Interface -------------------------- The sahara.service.edp.oozie.engine.OozieJobEngine class is derived from JobEngine. It provides implementations for all of the methods in the base interface but adds a few more abstract methods. Note that the *validate_job_execution(cluster, job, data)* method does basic checks on the job configuration but probably should be overloaded to include additional checks on the cluster configuration. For example, the job engines for plugins that support Oozie add checks to make sure that the Oozie service is up and running. get_hdfs_user() ~~~~~~~~~~~~~~~ Oozie uses HDFS to distribute job files. This method gives the name of the account that is used on the data nodes to access HDFS (such as 'hadoop' or 'hdfs'). The Oozie job engine expects that HDFS contains a directory for this user under */user/*. *Returns*: a string giving the username for the account used to access HDFS on the cluster. create_hdfs_dir(remote, dir_name) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The remote object *remote* references a node in the cluster. This method creates the HDFS directory *dir_name* under the user specified by *get_hdfs_user()* in the HDFS accessible from the specified node. For example, if the HDFS user is 'hadoop' and the dir_name is 'test' this method would create '/user/hadoop/test'. The reason that this method is broken out in the interface as an abstract method is that different versions of Hadoop treat path creation differently. *Returns*: None get_oozie_server_uri(cluster) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns the full URI for the Oozie server, for example *http://my_oozie_host:11000/oozie*. This URI is used by an Oozie client to send commands and queries to the Oozie server. *Returns*: a string giving the Oozie server URI. get_oozie_server(self, cluster) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns the node instance for the host in the cluster running the Oozie server. *Returns*: a node instance. get_name_node_uri(self, cluster) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns the full URI for the Hadoop NameNode, for example *http://master_node:8020*. *Returns*: a string giving the NameNode URI. get_resource_manager_uri(self, cluster) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns the full URI for the Hadoop JobTracker for Hadoop version 1 or the Hadoop ResourceManager for Hadoop version 2. *Returns*: a string giving the JobTracker or ResourceManager URI. Spark Job Engine ---------------- The sahara.service.edp.spark.engine.SparkJobEngine class provides a full EDP implementation for Spark standalone clusters. .. note:: The *validate_job_execution(cluster, job, data)* method does basic checks on the job configuration but probably should be overloaded to include additional checks on the cluster configuration. For example, the job engine returned by the Spark plugin checks that the Spark version is >= 1.0.0 to ensure that *spark-submit* is available. get_driver_classpath(self) ~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns driver class path. *Returns*: a string of the following format ' --driver-class-path *class_path_value*'. sahara-8.0.0/doc/source/reference/restapi.rst0000666000175100017510000000763613245514472021233 0ustar zuulzuul00000000000000Sahara REST API v1.1 ******************** 1 General API information ========================= This section contains base info about the sahara REST API design. 1.1 Authentication and Authorization ------------------------------------ The sahara API uses the OpenStack Identity service as the default authentication service. When the Identity service is enabled, users who submit requests to the sahara service must provide an authentication token in the ``X-Auth-Token`` request header. A user can obtain the token by authenticating to the Identity service endpoint. For more information about the Identity service, please see the `keystone project developer documentation `_ With each request, a user must specify the keystone project in the url path, for example: '/v1.1/{project_id}/clusters'. Sahara will perform the requested operation in the specified project using the provided credentials. Therefore, clusters may be created and managed only within projects to which the user has access. 1.2 Request / Response Types ---------------------------- The sahara API supports the JSON data serialization format. This means that for requests that contain a body, the ``Content-Type`` header must be set to the MIME type value ``application/json``. Also, clients should accept JSON serialized responses by specifying the ``Accept`` header with the MIME type value ``application/json`` or adding the ``.json`` extension to the resource name. The default response format is ``application/json`` if the client does not specify an ``Accept`` header or append the ``.json`` extension in the URL path. Example: .. sourcecode:: text GET /v1.1/{project_id}/clusters.json or .. sourcecode:: text GET /v1.1/{project_id}/clusters Accept: application/json 1.3 Navigation by response -------------------------- Sahara API supports delivering response data by pages. User can pass two parameters in API GET requests which return an array of objects. The parameters are: ``limit`` - maximum number of objects in response data. This parameter must be a positive integer number. ``marker`` - ID of the last element on the list which won't be in response. Example: Get 15 clusters after cluster with id=d62ad147-5c10-418c-a21a-3a6597044f29: .. sourcecode:: text GET /v1.1/{project_id}/clusters?limit=15&marker=d62ad147-5c10-418c-a21a-3a6597044f29 For convenience, response contains markers of previous and following pages which are named 'prev' and 'next' fields. Also there is ``sort_by`` parameter for sorting objects. Sahara API supports ascending and descending sorting. Examples: Sort clusters by name: .. sourcecode:: text GET /v1.1/{project_id}/clusters?sort_by=name Sort clusters by date of creation in descending order: .. sourcecode:: text GET /v1.1/{project_id}/clusters?sort_by=-created_at 1.4 Faults ---------- The sahara API returns an error response if a failure occurs while processing a request. Sahara uses only standard HTTP error codes. 4xx errors indicate problems in the particular request being sent from the client and 5xx errors indicate server-side problems. The response body will contain richer information about the cause of the error. An error response follows the format illustrated by the following example: .. sourcecode:: http HTTP/1.1 400 BAD REQUEST Content-type: application/json Content-length: 126 { "error_name": "CLUSTER_NAME_ALREADY_EXISTS", "error_message": "Cluster with name 'test-cluster' already exists", "error_code": 400 } The ``error_code`` attribute is an HTTP response code. The ``error_name`` attribute indicates the generic error type without any concrete ids or names, etc. The last attribute, ``error_message``, contains a human readable error description. 2 API ===== - `Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing) `_ sahara-8.0.0/doc/source/reference/plugins.rst0000666000175100017510000000227313245514472021235 0ustar zuulzuul00000000000000Pluggable Provisioning Mechanism ================================ Sahara can be integrated with 3rd party management tools like Apache Ambari and Cloudera Management Console. The integration is achieved using the plugin mechanism. In short, responsibilities are divided between the Sahara core and a plugin as follows. Sahara interacts with the user and uses Heat to provision OpenStack resources (VMs, baremetal servers, security groups, etc.) The plugin installs and configures a Hadoop cluster on the provisioned instances. Optionally, a plugin can deploy management and monitoring tools for the cluster. Sahara provides plugins with utility methods to work with provisioned instances. A plugin must extend the `sahara.plugins.provisioning:ProvisioningPluginBase` class and implement all the required methods. Read :doc:`plugin-spi` for details. The `instance` objects provided by Sahara have a `remote` property which can be used to interact with instances. The `remote` is a context manager so you can use it in `with instance.remote:` statements. The list of available commands can be found in `sahara.utils.remote.InstanceInteropHelper`. See the source code of the Vanilla plugin for usage examples. sahara-8.0.0/doc/source/reference/index.rst0000666000175100017510000000034413245514472020660 0ustar zuulzuul00000000000000===================== Programming Reference ===================== Plugins and EDP =============== .. toctree:: :maxdepth: 2 plugins plugin-spi edp-spi REST API ======== .. toctree:: :maxdepth: 2 restapi sahara-8.0.0/doc/source/reference/plugin-spi.rst0000666000175100017510000004443013245514472021644 0ustar zuulzuul00000000000000Plugin SPI ========== Plugin interface ---------------- get_versions() ~~~~~~~~~~~~~~ Returns all available versions of the plugin. Depending on the plugin, this version may map directly to the HDFS version, or it may not; check your plugin's documentation. It is responsibility of the plugin to make sure that all required images for each hadoop version are available, as well as configs and whatever else that plugin needs to create the Hadoop cluster. *Returns*: list of strings representing plugin versions *Example return value*: [“1.2.1â€, “2.3.0â€, “2.4.1â€] get_configs( hadoop_version ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lists all configs supported by the plugin with descriptions, defaults, and targets for which this config is applicable. *Returns*: list of configs *Example return value*: ((“JobTracker heap sizeâ€, "JobTracker heap size, in MB", "int", “512â€, `“mapreduceâ€`, "node", True, 1)) get_node_processes( hadoop_version ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns all supported services and node processes for a given Hadoop version. Each node process belongs to a single service and that relationship is reflected in the returned dict object. See example for details. *Returns*: dictionary having entries (service -> list of processes) *Example return value*: {"mapreduce": ["tasktracker", "jobtracker"], "hdfs": ["datanode", "namenode"]} get_required_image_tags( hadoop_version ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lists tags that should be added to OpenStack Image via Image Registry. Tags are used to filter Images by plugin and hadoop version. *Returns*: list of tags *Example return value*: ["tag1", "some_other_tag", ...] validate( cluster ) ~~~~~~~~~~~~~~~~~~~ Validates a given cluster object. Raises a *SaharaException* with a meaningful message in the case of validation failure. *Returns*: None *Example exception*: validate_scaling( cluster, existing, additional ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To be improved. Validates a given cluster before scaling operation. *Returns*: list of validation_errors update_infra( cluster ) ~~~~~~~~~~~~~~~~~~~~~~~ This method is no longer used now that Sahara utilizes Heat for OpenStack resource provisioning, and is not currently utilized by any plugin. *Returns*: None configure_cluster( cluster ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configures cluster on the VMs provisioned by sahara. In this function the plugin should perform all actions like adjusting OS, installing required packages (including Hadoop, if needed), configuring Hadoop, etc. *Returns*: None start_cluster( cluster ) ~~~~~~~~~~~~~~~~~~~~~~~~ Start already configured cluster. This method is guaranteed to be called only on a cluster which was already prepared with configure_cluster(...) call. *Returns*: None scale_cluster( cluster, instances ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Scale an existing cluster with additional instances. The instances argument is a list of ready-to-configure instances. Plugin should do all configuration operations in this method and start all services on those instances. *Returns*: None .. _get_edp_engine: get_edp_engine( cluster, job_type ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns an EDP job engine object that supports the specified job_type on the given cluster, or None if there is no support. The EDP job engine object returned must implement the interface described in :doc:`edp-spi`. The job_type is a String matching one of the job types listed in :ref:`edp_spi_job_types`. *Returns*: an EDP job engine object or None decommission_nodes( cluster, instances ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Scale cluster down by removing a list of instances. The plugin should stop services on the provided list of instances. The plugin also may need to update some configurations on other instances when nodes are removed; if so, this method must perform that reconfiguration. *Returns*: None on_terminate_cluster( cluster ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When user terminates cluster, sahara simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing the plugin to do some clean-up. *Returns*: None get_open_ports( node_group ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When user requests sahara to automatically create a security group for the node group (``auto_security_group`` property set to True), sahara will call this plugin method to get a list of ports that need to be opened. *Returns*: list of ports to be open in auto security group for the given node group get_edp_job_types( versions ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Optional method, which provides the ability to see all supported job types for specified plugin versions. *Returns*: dict with supported job types for specified versions of plugin recommend_configs( self, cluster, scaling=False ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Optional method, which provides recommendations for cluster configuration before creating/scaling operation. get_image_arguments( self, hadoop_version ): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Optional method, which gets the argument set taken by the plugin's image generator, or NotImplemented if the plugin does not provide image generation support. See :doc:`../contributor/image-gen`. *Returns*: A sequence with items of type sahara.plugins.images.ImageArgument. pack_image( self, hadoop_version, remote, test_only=False, ... ): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Optional method which packs an image for registration in Glance and use by Sahara. This method is called from the image generation CLI rather than from the Sahara api or engine service. See :doc:`../contributor/image-gen`. *Returns*: None (modifies the image pointed to by the remote in-place.) validate_images( self, cluster, test_only=False, image_arguments=None ): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Validates the image to be used to create a cluster, to ensure that it meets the specifications of the plugin. See :doc:`../contributor/image-gen`. *Returns*: None; may raise a sahara.plugins.exceptions.ImageValidationError Object Model ============ Here is a description of all the objects involved in the API. Notes: - clusters and node_groups have 'extra' fields allowing the plugin to persist any supplementary info about the cluster. - node_process is just a process that runs on some node in cluster. Example list of node processes: 1. jobtracker 2. namenode 3. tasktracker 4. datanode - Each plugin may have different names for the same processes. Config ------ An object, describing one configuration entry +-------------------+--------+------------------------------------------------+ | Property | Type | Description | +===================+========+================================================+ | name | string | Config name. | +-------------------+--------+------------------------------------------------+ | description | string | A hint for user, what this config is used for. | +-------------------+--------+------------------------------------------------+ | config_type | enum | possible values are: 'string', 'integer', | | | | 'boolean', 'enum'. | +-------------------+--------+------------------------------------------------+ | config_values | list | List of possible values, if config_type is | | | | enum. | +-------------------+--------+------------------------------------------------+ | default_value | string | Default value for config. | +-------------------+--------+------------------------------------------------+ | applicable_target | string | The target could be either a service returned | | | | by get_node_processes(...) call | | | | in form of 'service:', or | | | | 'general'. | +-------------------+--------+------------------------------------------------+ | scope | enum | Could be either 'node' or 'cluster'. | +-------------------+--------+------------------------------------------------+ | is_optional | bool | If is_optional is False and no default_value | | | | is specified, user must provide a value. | +-------------------+--------+------------------------------------------------+ | priority | int | 1 or 2. A Hint for UI. Configs with priority | | | | *1* are always displayed. | | | | Priority *2* means user should click a button | | | | to see the config. | +-------------------+--------+------------------------------------------------+ User Input ---------- Value provided by user for a specific config. +----------+--------+--------------------------------------------------------+ | Property | Type | Description | +==========+========+========================================================+ | config | config | A config object for which this user_input is provided. | +----------+--------+--------------------------------------------------------+ | value | ... | Value for the config. Type depends on Config type. | +----------+--------+--------------------------------------------------------+ Instance -------- An instance created for cluster. +---------------+---------+---------------------------------------------------+ | Property | Type | Description | +===============+=========+===================================================+ | instance_id | string | Unique instance identifier. | +---------------+---------+---------------------------------------------------+ | instance_name | string | OpenStack instance name. | +---------------+---------+---------------------------------------------------+ | internal_ip | string | IP to communicate with other instances. | +---------------+---------+---------------------------------------------------+ | management_ip | string | IP of instance, accessible outside of internal | | | | network. | +---------------+---------+---------------------------------------------------+ | volumes | list | List of volumes attached to instance. Empty if | | | | ephemeral drive is used. | +---------------+---------+---------------------------------------------------+ | nova_info | object | Nova instance object. | +---------------+---------+---------------------------------------------------+ | username | string | Username, that sahara uses for establishing | | | | remote connections to instance. | +---------------+---------+---------------------------------------------------+ | hostname | string | Same as instance_name. | +---------------+---------+---------------------------------------------------+ | fqdn | string | Fully qualified domain name for this instance. | +---------------+---------+---------------------------------------------------+ | remote | helpers | Object with helpers for performing remote | | | | operations. | +---------------+---------+---------------------------------------------------+ Node Group ---------- Group of instances. +----------------------+--------+---------------------------------------------+ | Property | Type | Description | +======================+========+=============================================+ | name | string | Name of this Node Group in Cluster. | +----------------------+--------+---------------------------------------------+ | flavor_id | string | OpenStack Flavor used to boot instances. | +----------------------+--------+---------------------------------------------+ | image_id | string | Image id used to boot instances. | +----------------------+--------+---------------------------------------------+ | node_processes | list | List of processes running on each instance. | +----------------------+--------+---------------------------------------------+ | node_configs | dict | Configs dictionary, applied to instances. | +----------------------+--------+---------------------------------------------+ | volumes_per_node | int | Number of volumes mounted to each instance. | | | | 0 means use ephemeral drive. | +----------------------+--------+---------------------------------------------+ | volumes_size | int | Size of each volume (GB). | +----------------------+--------+---------------------------------------------+ | volumes_mount_prefix | string | Prefix added to mount path of each volume. | +----------------------+--------+---------------------------------------------+ | floating_ip_pool | string | Floating IP Pool name. All instances in the | | | | Node Group will have Floating IPs assigned | | | | from this pool. | +----------------------+--------+---------------------------------------------+ | count | int | Number of instances in this Node Group. | +----------------------+--------+---------------------------------------------+ | username | string | Username used by sahara to establish remote | | | | connections to instances. | +----------------------+--------+---------------------------------------------+ | configuration | dict | Merged dictionary of node configurations | | | | and cluster configurations. | +----------------------+--------+---------------------------------------------+ | storage_paths | list | List of directories where storage should be | | | | placed. | +----------------------+--------+---------------------------------------------+ Cluster ------- Contains all relevant info about cluster. This object is is provided to the plugin for both cluster creation and scaling. The “Cluster Lifecycle†section below further specifies which fields are filled at which moment. +----------------------------+--------+---------------------------------------+ | Property | Type | Description | +============================+========+=======================================+ | name | string | Cluster name. | +----------------------------+--------+---------------------------------------+ | project_id | string | OpenStack Project id where this | | | | Cluster is available. | +----------------------------+--------+---------------------------------------+ | plugin_name | string | Plugin name. | +----------------------------+--------+---------------------------------------+ | hadoop_version | string | Hadoop version running on instances. | +----------------------------+--------+---------------------------------------+ | default_image_id | string | OpenStack image used to boot | | | | instances. | +----------------------------+--------+---------------------------------------+ | node_groups | list | List of Node Groups. | +----------------------------+--------+---------------------------------------+ | cluster_configs | dict | Dictionary of Cluster scoped | | | | configurations. | +----------------------------+--------+---------------------------------------+ | cluster_template_id | string | Cluster Template used for Node Groups | | | | and Configurations. | +----------------------------+--------+---------------------------------------+ | user_keypair_id | string | OpenStack keypair added to instances | | | | to make them accessible for user. | +----------------------------+--------+---------------------------------------+ | neutron_management_network | string | Neutron network ID. Instances will | | | | get fixed IPs in this network. | +----------------------------+--------+---------------------------------------+ | anti_affinity | list | List of processes that will be run on | | | | different hosts. | +----------------------------+--------+---------------------------------------+ | description | string | Cluster Description. | +----------------------------+--------+---------------------------------------+ | info | dict | Dictionary for additional information.| +----------------------------+--------+---------------------------------------+ Validation Error ---------------- Describes what is wrong with one of the values provided by user. +---------------+--------+-----------------------------------------------+ | Property | Type | Description | +===============+========+===============================================+ | config | config | A config object that is not valid. | +---------------+--------+-----------------------------------------------+ | error_message | string | Message that describes what exactly is wrong. | +---------------+--------+-----------------------------------------------+ sahara-8.0.0/doc/source/contributor/0000775000175100017510000000000013245515026017424 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/contributor/log-guidelines.rst0000666000175100017510000000242013245514472023071 0ustar zuulzuul00000000000000 Log Guidelines ============== Levels Guidelines ----------------- During the Kilo release cycle the sahara community defined the following log levels: * Debug: Shows everything and is likely not suitable for normal production operation due to the sheer size of logs generated (e.g. scripts executions, process execution, etc.). * Info: Usually indicates successful service start/stop, versions and such non-error related data. This should include largely positive units of work that are accomplished (e.g. service setup and configuration, cluster start, job execution information). * Warning: Indicates that there might be a systemic issue; potential predictive failure notice (e.g. job execution failed). * Error: An error has occurred and the administrator should research the error information (e.g. cluster failed to start, plugin violations of operation). * Critical: An error has occurred and the system might be unstable, anything that eliminates part of sahara's intended functionalities; immediately get administrator assistance (e.g. failed to access keystone/database, failed to load plugin). Formatting Guidelines --------------------- Sahara uses string formatting defined in `PEP 3101`_ for logs. .. _PEP 3101: https://www.python.org/dev/peps/pep-3101/ sahara-8.0.0/doc/source/contributor/apiv2.rst0000666000175100017510000001001113245514472021176 0ustar zuulzuul00000000000000API Version 2 Development ========================= The sahara project is currently in the process of creating a new RESTful application programming interface (API). This interface is by-default enabled, although it remains experimental. This document defines the steps necessary to enable and communicate with the new API. This API has a few fundamental changes from the previous APIs and they should be noted before proceeding with development work. .. warning:: This API is currently marked as experimental. It is not supported by the sahara python client. These instructions are included purely for developers who wish to help participate in the development effort. Enabling the experimental API ----------------------------- There are a few changes to the WSGI pipeline that must be made to enable the new v2 API. These changes will leave the 1.0 and 1.1 API versions in place and will not adjust their communication parameters. To begin, uncomment, or add, the following sections in your api-paste.ini file: .. sourcecode:: ini [app:sahara_apiv2] paste.app_factory = sahara.api.middleware.sahara_middleware:RouterV2.factory [filter:auth_validator_v2] paste.filter_factory = sahara.api.middleware.auth_valid:AuthValidatorV2.factory These lines define a new authentication filter for the v2 API, and define the application that will handle the new calls. With these new entries in the paste configuration, we can now enable them with the following changes to the api-paste.ini file: .. sourcecode:: ini [pipeline:sahara] pipeline = cors request_id acl auth_validator_v2 sahara_api [composite:sahara_api] use = egg:Paste#urlmap /: sahara_apiv2 There are 2 significant changes occurring here; changing the authentication validator in the pipeline, and changing the root "/" application to the new v2 handler. At this point the sahara API server should be configured to accept requests on the new v2 endpoints. Communicating with the v2 API ----------------------------- The v2 API makes at least one major change from the previous versions, removing the OpenStack project identifier from the URL. Now users of the API do not provide their project ID explictly; instead we fully trust keystonemiddeware to provide it in the WSGI environment based on the given user token. For example, in previous versions of the API, a call to get the list of clusters for project "12345678-1234-1234-1234-123456789ABC" would have been made as follows:: GET /v1.1/12345678-1234-1234-1234-123456789ABC/clusters X-Auth-Token: {valid auth token} This call would now be made to the following URL:: GET /v2/clusters X-Auth-Token: {valid auth token} Using a tool like `HTTPie `_, the same request could be made like this:: $ httpie http://{sahara service ip:port}/v2/clusters \ X-Auth-Token:{valid auth token} Following the implementation progress ------------------------------------- As the creation of this API will be under regular change until it moves out of the experimental phase, a wiki page has been established to help track the progress. https://wiki.openstack.org/wiki/Sahara/api-v2 This page will help to coordinate the various reviews, specs, and work items that are a continuing facet of this work. The API service layer --------------------- When contributing to the version 2 API, it will be necessary to add code that modifies the data and behavior of HTTP calls as they are sent to and from the processing engine and data abstraction layers. Most frequently in the sahara codebase, these interactions are handled in the modules of the ``sahara.service.api`` package. This package contains code for all versions of the API and follows a namespace mapping that is similar to the routing functions of ``sahara.api`` Although these modules are not the definitive end of all answers to API related code questions, they are a solid starting point when examining the extent of new work. Furthermore, they serve as a central point to begin API debugging efforts when the need arises. sahara-8.0.0/doc/source/contributor/launchpad.rst0000666000175100017510000000357413245514472022134 0ustar zuulzuul00000000000000Project hosting =============== `Launchpad`_ hosts the Sahara project. The Sahara project homepage on Launchpad is http://launchpad.net/sahara. Launchpad credentials --------------------- Creating a login on Launchpad is important even if you don't use the Launchpad site itself, since Launchpad credentials are used for logging in on several OpenStack-related sites. These sites include: * `Wiki`_ * Gerrit (see :doc:`gerrit`) * Jenkins (see :doc:`jenkins`) Mailing list ------------ The mailing list email is ``openstack-dev@lists.openstack.org``; use the subject prefix ``[sahara]`` to address the team. To participate in the mailing list subscribe to the list at http://lists.openstack.org/cgi-bin/mailman/listinfo Bug tracking ------------ Report Sahara bugs at https://bugs.launchpad.net/sahara Feature requests (Blueprints) ----------------------------- Sahara uses specs to track feature requests. Blueprints are at https://blueprints.launchpad.net/sahara. They provide a high-level summary of proposed changes and track associated commits. Sahara also uses specs for in-depth descriptions and discussions of blueprints. Specs follow a defined format and are submitted as change requests to the openstack/sahara-specs repository. Every blueprint should have an associated spec that is agreed on and merged to the sahara-specs repository before it is approved, unless the whole team agrees that the implementation path for the feature described in the blueprint is completely understood. Technical support ----------------- Sahara uses `Ask OpenStack`_ to track Sahara technical support questions. Questions related to Sahara should be tagged with 'sahara'. .. _Launchpad: http://launchpad.net .. _Wiki: http://wiki.openstack.org/sahara .. _Sahara Drivers: https://launchpad.net/~sahara-drivers .. _Sahara Bug Team: https://launchpad.net/~sahara-bugs .. _Ask OpenStack: https://ask.openstack.org sahara-8.0.0/doc/source/contributor/testing.rst0000666000175100017510000000202513245514472021640 0ustar zuulzuul00000000000000Sahara Testing ============== We have a bunch of different tests for Sahara. Unit Tests ++++++++++ In most Sahara sub-repositories we have a directory that contains Python unit tests, located at `_package_/tests/unit` or `_package_/tests`. Scenario integration tests ++++++++++++++++++++++++++ New scenario integration tests were implemented for Sahara. They are available in the sahara-tests repository (https://git.openstack.org/cgit/openstack/sahara-tests). Tempest tests +++++++++++++ Sahara has a Tempest plugin in the sahara-tests repository covering all major API features. Additional tests ++++++++++++++++ Additional tests reside in the sahara-tests repository (as above): * REST API tests checking to ensure that the Sahara REST API works. The only parts that are not tested are cluster creation and EDP. * CLI tests check read-only operations using the Sahara CLI. For more information about these tests, please read `Tempest Integration of Sahara `_. sahara-8.0.0/doc/source/contributor/development-environment.rst0000666000175100017510000000672313245514472025060 0ustar zuulzuul00000000000000Setting Up a Development Environment ==================================== This page describes how to setup a Sahara development environment by either installing it as a part of DevStack or pointing a local running instance at an external OpenStack. You should be able to debug and test your changes without having to deploy Sahara. Setup a Local Environment with Sahara inside DevStack ----------------------------------------------------- See :doc:`the main article `. Setup a Local Environment with an external OpenStack ---------------------------------------------------- 1. Install prerequisites On OS X Systems: .. sourcecode:: console # we actually need pip, which is part of python package $ brew install python mysql postgresql rabbitmq $ pip install virtualenv tox On Ubuntu: .. sourcecode:: console $ sudo apt-get update $ sudo apt-get install git-core python-dev python-virtualenv gcc libpq-dev libmysqlclient-dev python-pip rabbitmq-server $ sudo pip install tox On Red Hat and related distributions (CentOS/Fedora/RHEL/Scientific Linux): .. sourcecode:: console $ sudo yum install git-core python-devel python-virtualenv gcc python-pip mariadb-devel postgresql-devel erlang $ sudo pip install tox $ sudo wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.2/rabbitmq-server-3.2.2-1.noarch.rpm $ sudo rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc $ sudo yum install rabbitmq-server-3.2.2-1.noarch.rpm On openSUSE-based distributions (SLES 12, openSUSE, Factory or Tumbleweed): .. sourcecode:: console $ sudo zypper in gcc git libmysqlclient-devel postgresql-devel python-devel python-pip python-tox python-virtualenv 2. Grab the code .. sourcecode:: console $ git clone git://git.openstack.org/openstack/sahara.git $ cd sahara 3. Generate Sahara sample using tox .. sourcecode:: console tox -e genconfig 4. Create config file from the sample .. sourcecode:: console $ cp ./etc/sahara/sahara.conf.sample ./etc/sahara/sahara.conf 5. Look through the sahara.conf and modify parameter values as needed For details see :doc:`Sahara Configuration Guide <../admin/configuration-guide>` 6. Create database schema .. sourcecode:: console $ tox -e venv -- sahara-db-manage --config-file etc/sahara/sahara.conf upgrade head 7. To start Sahara API and Engine processes call .. sourcecode:: console $ tox -e venv -- sahara-api --config-file etc/sahara/sahara.conf --debug $ tox -e venv -- sahara-engine --config-file etc/sahara/sahara.conf --debug Setup local OpenStack dashboard with Sahara plugin -------------------------------------------------- .. toctree:: :maxdepth: 1 dashboard-dev-environment-guide Tips and tricks for dev environment ----------------------------------- 1. Pip speedup Add the following lines to ~/.pip/pip.conf .. sourcecode:: cfg [global] download-cache = /home//.pip/cache index-url = Note that the ``~/.pip/cache`` folder should be created manually. 2. Git hook for fast checks Just add the following lines to .git/hooks/pre-commit and do chmod +x for it. .. sourcecode:: console #!/bin/sh # Run fast checks (PEP8 style check and PyFlakes fast static analysis) tox -epep8 You can add also other checks for pre-push, for example pylint (see below) and tests (tox -epy27). 3. Running static analysis (PyLint) Just run the following command .. sourcecode:: console tox -e pylint sahara-8.0.0/doc/source/contributor/jenkins.rst0000666000175100017510000000316213245514472021627 0ustar zuulzuul00000000000000Continuous Integration with Jenkins =================================== Each change made to Sahara core code is tested with unit and integration tests and style checks using flake8. Unit tests and style checks are performed on public `OpenStack Jenkins `_ managed by `Zuul `_. Unit tests are checked using python 2.7. The result of those checks and Unit tests are represented as a vote of +1 or -1 in the *Verify* column in code reviews from the *Jenkins* user. Integration tests check CRUD operations for the Image Registry, Templates, and Clusters. Also a test job is launched on a created Cluster to verify Hadoop work. All integration tests are launched by `Jenkins `_ on the internal Mirantis OpenStack Lab. Jenkins keeps a pool of VMs to run tests in parallel. Even with the pool of VMs integration testing may take a while. Jenkins is controlled for the most part by Zuul which determines what jobs are run when. Zuul status is available at this address: `Zuul Status `_. For more information see: `Sahara Hadoop Cluster CI `_. The integration tests result is represented as a vote of +1 or -1 in the *Verify* column in a code review from the *Sahara Hadoop Cluster CI* user. You can put *sahara-ci-recheck* in comment, if you want to recheck sahara-ci jobs. Also, you can put *recheck* in comment, if you want to recheck both Jenkins and sahara-ci jobs. Finally, you can put *reverify* in a comment, if you only want to recheck Jenkins jobs. sahara-8.0.0/doc/source/contributor/adding-database-migrations.rst0000666000175100017510000001023613245514472025330 0ustar zuulzuul00000000000000Adding Database Migrations ========================== The migrations in ``sahara/db/migration/alembic_migrations/versions`` contain the changes needed to migrate between Sahara database revisions. A migration occurs by executing a script that details the changes needed to upgrade or downgrade the database. The migration scripts are ordered so that multiple scripts can run sequentially. The scripts are executed by Sahara's migration wrapper which uses the Alembic library to manage the migration. Sahara supports migration from Icehouse or later. Any code modifications that change the structure of the database require a migration script so that previously existing databases will continue to function when the new code is released. This page gives a brief overview of how to add the migration. Generate a New Migration Script +++++++++++++++++++++++++++++++ New migration scripts can be generated using the ``sahara-db-manage`` command. To generate a migration stub to be filled in by the developer:: $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" To autogenerate a migration script that reflects the current structure of the database:: $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" --autogenerate Each of these commands will create a file of the form ``revision_description`` where ``revision`` is a string generated by Alembic and ``description`` is based on the text passed with the ``-m`` option. Follow the Sahara Naming Convention +++++++++++++++++++++++++++++++++++ By convention Sahara uses 3-digit revision numbers, and this scheme differs from the strings generated by Alembic. Consequently, it's necessary to rename the generated script and modify the revision identifiers in the script. Open the new script and look for the variable ``down_revision``. The value should be a 3-digit numeric string, and it identifies the current revision number of the database. Set the ``revision`` value to the ``down_revision`` value + 1. For example, the lines:: # revision identifiers, used by Alembic. revision = '507eb70202af' down_revision = '006' will become:: # revision identifiers, used by Alembic. revision = '007' down_revision = '006' Modify any comments in the file to match the changes and rename the file to match the new revision number:: $ mv 507eb70202af_my_new_revision.py 007_my_new_revision.py Add Alembic Operations to the Script ++++++++++++++++++++++++++++++++++++ The migration script contains method ``upgrade()``. Sahara has not supported downgrades since the Kilo release. Fill in this method with the appropriate Alembic operations to perform upgrades. In the above example, an upgrade will move from revision '006' to revision '007'. Command Summary for sahara-db-manage ++++++++++++++++++++++++++++++++++++ You can upgrade to the latest database version via:: $ sahara-db-manage --config-file /path/to/sahara.conf upgrade head To check the current database version:: $ sahara-db-manage --config-file /path/to/sahara.conf current To create a script to run the migration offline:: $ sahara-db-manage --config-file /path/to/sahara.conf upgrade head --sql To run the offline migration between specific migration versions:: $ sahara-db-manage --config-file /path/to/sahara.conf upgrade : --sql To upgrade the database incrementally:: $ sahara-db-manage --config-file /path/to/sahara.conf upgrade --delta <# of revs> To create a new revision:: $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" --autogenerate To create a blank file:: $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" This command does not perform any migrations, it only sets the revision. Revision may be any existing revision. Use this command carefully:: $ sahara-db-manage --config-file /path/to/sahara.conf stamp To verify that the timeline does branch, you can run this command:: $ sahara-db-manage --config-file /path/to/sahara.conf check_migration If the migration path does branch, you can find the branch point via:: $ sahara-db-manage --config-file /path/to/sahara.conf history sahara-8.0.0/doc/source/contributor/gerrit.rst0000666000175100017510000000101513245514472021455 0ustar zuulzuul00000000000000Code Reviews with Gerrit ======================== Sahara uses the `Gerrit`_ tool to review proposed code changes. The review site is https://review.openstack.org. Gerrit is a complete replacement for Github pull requests. `All Github pull requests to the Sahara repository will be ignored`. See `Development Workflow`_ for information about how to get started using Gerrit. .. _Gerrit: http://code.google.com/p/gerrit .. _Development Workflow: https://docs.openstack.org/infra/manual/developers.html#development-workflow sahara-8.0.0/doc/source/contributor/dashboard-dev-environment-guide.rst0000666000175100017510000000756213245514472026336 0ustar zuulzuul00000000000000Sahara UI Dev Environment Setup =============================== This page describes how to setup Horizon for developing Sahara by either installing it as part of DevStack with Sahara or installing it in an isolated environment and running from the command line. Install as a part of DevStack ----------------------------- See the `DevStack guide `_ for more information on installing and configuring DevStack with Sahara. Sahara UI can be installed as a DevStack plugin by adding the following line to your ``local.conf`` file .. sourcecode:: bash # Enable sahara-dashboard enable_plugin sahara-dashboard git://git.openstack.org/openstack/sahara-dashboard Isolated Dashboard for Sahara ----------------------------- These installation steps serve two purposes: 1. Setup a dev environment 2. Setup an isolated Dashboard for Sahara **Note** The host where you are going to perform installation has to be able to connect to all OpenStack endpoints. You can list all available endpoints using the following command: .. sourcecode:: console $ openstack endpoint list You can list the registered services with this command: .. sourcecode:: console $ openstack service list Sahara service should be present in keystone service list with service type *data-processing* 1. Install prerequisites .. sourcecode:: console $ sudo apt-get update $ sudo apt-get install git-core python-dev gcc python-setuptools \ python-virtualenv node-less libssl-dev libffi-dev libxslt-dev .. On Ubuntu 12.10 and higher you have to install the following lib as well: .. sourcecode:: console $ sudo apt-get install nodejs-legacy .. 2. Checkout Horizon from git and switch to your version of OpenStack Here is an example: .. sourcecode:: console $ git clone https://git.openstack.org/cgit/openstack/horizon/ {HORIZON_DIR} .. Then install the virtual environment: .. sourcecode:: console $ python {HORIZON_DIR}/tools/install_venv.py .. 3. Create a ``local_settings.py`` file .. sourcecode:: console $ cp {HORIZON_DIR}/openstack_dashboard/local/local_settings.py.example \ {HORIZON_DIR}/openstack_dashboard/local/local_settings.py .. 4. Modify ``{HORIZON_DIR}/openstack_dashboard/local/local_settings.py`` Set the proper values for host and url variables: .. sourcecode:: python OPENSTACK_HOST = "ip of your controller" .. If you wish to disable floating IP options during node group template creation, add the following parameter: .. sourcecode:: python SAHARA_FLOATING_IP_DISABLED = True .. 5. Clone sahara-dashboard repository and checkout the desired branch .. sourcecode:: console $ git clone https://git.openstack.org/cgit/openstack/sahara-dashboard/ \ {SAHARA_DASHBOARD_DIR} .. 6. Copy plugin-enabling files from sahara-dashboard repository to horizon .. sourcecode:: console $ cp -a {SAHARA_DASHBOARD_DIR}/sahara_dashboard/enabled/* {HORIZON_DIR}/openstack_dashboard/local/enabled/ .. 7. Install sahara-dashboard project into your horizon virtualenv in editable mode .. sourcecode:: console $ . {HORIZON_DIR}/.venv/bin/activate $ pip install -e {SAHARA_DASHBOARD_DIR} .. 8. Start Horizon .. sourcecode:: console $ . {HORIZON_DIR}/.venv/bin/activate $ python {HORIZON_DIR}/manage.py runserver 0.0.0.0:8080 .. This will start Horizon in debug mode. That means the logs will be written to console and if any exceptions happen, you will see the stack-trace rendered as a web-page. Debug mode can be disabled by changing ``DEBUG=True`` to ``False`` in ``local_settings.py``. In that case Horizon should be started slightly differently, otherwise it will not serve static files: .. sourcecode:: console $ . {HORIZON_DIR}/.venv/bin/activate $ python {HORIZON_DIR}/manage.py runserver --insecure 0.0.0.0:8080 .. .. note:: It is not recommended to use Horizon in this mode for production. sahara-8.0.0/doc/source/contributor/index.rst0000666000175100017510000000074713245514472021303 0ustar zuulzuul00000000000000===================== Developer Information ===================== Programming HowTos and Tutorials ================================ .. toctree:: :maxdepth: 2 development-guidelines development-environment devstack dashboard-dev-environment-guide how-to-participate how-to-build-oozie adding-database-migrations testing log-guidelines apiv2 image-gen Other Resources =============== .. toctree:: :maxdepth: 2 launchpad gerrit jenkins sahara-8.0.0/doc/source/contributor/how-to-participate.rst0000666000175100017510000000457013245514472023712 0ustar zuulzuul00000000000000How to Participate ================== Getting started --------------- * Make sure that your local git is properly configured by executing ``git config --list``. If not, configure ``user.name``, ``user.email`` * Create account on `Launchpad `_ (if you don't have one) * Subscribe to `OpenStack general mail-list `_ * Subscribe to `OpenStack development mail-list `_ * Create `OpenStack profile `_ * Login to `OpenStack Gerrit `_ with your Launchpad id * Sign `OpenStack Individual Contributor License Agreement `_ * Make sure that your email is listed in `identities `_ * Subscribe to code-reviews. Go to your settings on https://review.openstack.org * Go to ``watched projects`` * Add ``openstack/sahara``, ``openstack/sahara-extra``, ``openstack/python-saharaclient``, and ``openstack/sahara-image-elements`` How to stay in touch with the community --------------------------------------- * If you have something to discuss use `OpenStack development mail-list `_. Prefix the mail subject with ``[Sahara]`` * Join ``#openstack-sahara`` IRC channel on `freenode `_ * Attend Sahara team meetings * Weekly on Thursdays at 1400 UTC and 1800 UTC (on alternate weeks) * IRC channel: ``#openstack-meeting-alt`` (1800UTC) or ``#openstack-meeting-3`` (1400UTC) * See agenda at https://wiki.openstack.org/wiki/Meetings/SaharaAgenda How to post your first patch for review --------------------------------------- * Checkout Sahara code from `its repository `_ * Carefully read https://docs.openstack.org/infra/manual/developers.html#development-workflow * Pay special attention to https://docs.openstack.org/infra/manual/developers.html#committing-a-change * Apply and commit your changes * Make sure that your code passes ``PEP8`` checks and unit-tests. See :doc:`development-guidelines` * Post your patch for review * Monitor the status of your patch review on https://review.openstack.org/#/ sahara-8.0.0/doc/source/contributor/devstack.rst0000666000175100017510000001340313245514472021771 0ustar zuulzuul00000000000000Setup DevStack ============== DevStack can be installed on Fedora, Ubuntu, and CentOS. For supported versions see `DevStack documentation `_ We recommend that you install DevStack in a VM, rather than on your main system. That way you may avoid contamination of your system. You may find hypervisor and VM requirements in the next section. If you still want to install DevStack on your baremetal system, just skip the next section and read further. Start VM and set up OS ---------------------- In order to run DevStack in a local VM, you need to start by installing a guest with Ubuntu 14.04 server. Download an image file from `Ubuntu's web site `_ and create a new guest from it. Virtualization solution must support nested virtualization. Without nested virtualization VMs running inside the DevStack will be extremely slow lacking hardware acceleration, i.e. you will run QEMU VMs without KVM. On Linux QEMU/KVM supports nested virtualization, on Mac OS - VMware Fusion. VMware Fusion requires adjustments to run VM with fixed IP. You may find instructions which can help :ref:`below `. Start a new VM with Ubuntu Server 14.04. Recommended settings: - Processor - at least 2 cores - Memory - at least 8GB - Hard Drive - at least 60GB When allocating CPUs and RAM to the DevStack, assess how big clusters you want to run. A single Hadoop VM needs at least 1 cpu and 1G of RAM to run. While it is possible for several VMs to share a single cpu core, remember that they can't share the RAM. After you installed the VM, connect to it via SSH and proceed with the instructions below. Install DevStack ---------------- The instructions assume that you've decided to install DevStack into Ubuntu 14.04 system. **Note:** Make sure to use bash, as other shells are not fully compatible and may cause hard to debug problems. 1. Clone DevStack: .. sourcecode:: console $ sudo apt-get install git-core $ git clone https://git.openstack.org/openstack-dev/devstack.git 2. Create the file ``local.conf`` in devstack directory with the following content: .. sourcecode:: bash [[local|localrc]] ADMIN_PASSWORD=nova MYSQL_PASSWORD=nova RABBIT_PASSWORD=nova SERVICE_PASSWORD=$ADMIN_PASSWORD SERVICE_TOKEN=nova # Enable Swift enable_service s-proxy s-object s-container s-account SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5 SWIFT_REPLICAS=1 SWIFT_DATA_DIR=$DEST/data # Force checkout prerequisites # FORCE_PREREQ=1 # keystone is now configured by default to use PKI as the token format # which produces huge tokens. # set UUID as keystone token format which is much shorter and easier to # work with. KEYSTONE_TOKEN_FORMAT=UUID # Change the FLOATING_RANGE to whatever IPs VM is working in. # In NAT mode it is the subnet VMware Fusion provides, in bridged mode # it is your local network. But only use the top end of the network by # using a /27 and starting at the 224 octet. FLOATING_RANGE=192.168.55.224/27 # Set ``OFFLINE`` to ``True`` to configure ``stack.sh`` to run cleanly # without Internet access. ``stack.sh`` must have been previously run # with Internet access to install prerequisites and fetch repositories. # OFFLINE=True # Enable sahara enable_plugin sahara git://git.openstack.org/openstack/sahara # Enable heat enable_plugin heat git://git.openstack.org/openstack/heat In cases where you need to specify a git refspec (branch, tag, or commit hash) for the sahara in-tree devstack plugin (or sahara repo), it should be appended to the git repo URL as follows: .. sourcecode:: bash enable_plugin sahara git://git.openstack.org/openstack/sahara 3. Sahara can send notifications to Ceilometer, if Ceilometer is enabled. If you want to enable Ceilometer add the following lines to the ``local.conf`` file: .. sourcecode:: bash enable_plugin ceilometer git://git.openstack.org/openstack/ceilometer 4. Start DevStack: .. sourcecode:: console $ ./stack.sh 5. Once the previous step is finished Devstack will print a Horizon URL. Navigate to this URL and login with login "admin" and password from ``local.conf``. 6. Congratulations! You have OpenStack running in your VM and you're ready to launch VMs inside that VM. :) Managing sahara in DevStack --------------------------- If you install DevStack with sahara included you can rejoin screen with the ``screen -c stack-screenrc`` command and switch to the ``sahara`` tab. Here you can manage the sahara service as other OpenStack services. Sahara source code is located at ``$DEST/sahara`` which is usually ``/opt/stack/sahara``. .. _fusion-fixed-ip: Setting fixed IP address for VMware Fusion VM --------------------------------------------- 1. Open file ``/Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf`` 2. There is a block named "subnet". It might look like this: .. sourcecode:: text subnet 192.168.55.0 netmask 255.255.255.0 { range 192.168.55.128 192.168.55.254; 3. You need to pick an IP address outside of that range. For example - ``192.168.55.20`` 4. Copy VM MAC address from VM settings->Network->Advanced 5. Append the following block to file ``dhcpd.conf`` (don't forget to replace ``VM_HOSTNAME`` and ``VM_MAC_ADDRESS`` with actual values): .. sourcecode:: text host VM_HOSTNAME { hardware ethernet VM_MAC_ADDRESS; fixed-address 192.168.55.20; } 6. Now quit all the VMware Fusion applications and restart vmnet: .. sourcecode:: console $ sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --stop $ sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --start 7. Now start your VM; it should have new fixed IP address. sahara-8.0.0/doc/source/contributor/how-to-build-oozie.rst0000666000175100017510000000350313245514472023622 0ustar zuulzuul00000000000000How to build Oozie ================== .. note:: Apache does not make Oozie builds, so it has to be built manually. Download -------- * Download tarball from `Apache mirror `_ * Unpack it with .. sourcecode:: console $ tar -xzvf oozie-4.0.1.tar.gz Hadoop Versions --------------- To build Oozie the following command can be used: .. sourcecode:: console $ {oozie_dir}/bin/mkdistro.sh -DskipTests By default it builds against Hadoop 1.1.1. To built it with Hadoop version 2.x: * The hadoop-2 version should be changed in pom.xml. This can be done manually or with the following command (you should replace 2.x.x with your hadoop version): .. sourcecode:: console $ find . -name pom.xml | xargs sed -ri 's/2.3.0/2.x.x/' * The build command should be launched with the ``-P hadoop-2`` flag JDK Versions ------------ By default, the build configuration enforces that JDK 1.6.* is being used. There are 2 build properties that can be used to change the JDK version requirements: * ``javaVersion`` specifies the version of the JDK used to compile (default 1.6). * ``targetJavaVersion`` specifies the version of the generated bytecode (default 1.6). For example, to specify JDK version 1.7, the build command should contain the ``-D javaVersion=1.7 -D tagetJavaVersion=1.7`` flags. Build ----- To build Oozie with Hadoop 2.6.0 and JDK version 1.7, the following command can be used: .. sourcecode:: console $ {oozie_dir}/bin/mkdistro.sh assembly:single -P hadoop-2 -D javaVersion=1.7 -D targetJavaVersion=1.7 -D skipTests Also, the pig version can be passed as a maven property with the flag ``-D pig.version=x.x.x``. You can find similar instructions to build oozie.tar.gz here: http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Building_Oozie sahara-8.0.0/doc/source/contributor/development-guidelines.rst0000666000175100017510000002064513245514472024643 0ustar zuulzuul00000000000000Development Guidelines ====================== Coding Guidelines ----------------- For all the Python code in Sahara we have a rule - it should pass `PEP 8`_. All Bash code should pass `bashate`_. To check your code against PEP 8 and bashate run: .. sourcecode:: console $ tox -e pep8 .. note:: For more details on coding guidelines see file ``HACKING.rst`` in the root of Sahara repo. Static analysis --------------- The static analysis checks are optional in Sahara, but they are still very useful. The gate job will inform you if the number of static analysis warnings has increased after your change. We recommend to always check the static warnings. To run check first commit your change, then execute the following command: .. sourcecode:: console $ tox -e pylint Modification of Upstream Files ------------------------------ We never modify upstream files in Sahara. Any changes in upstream files should be made in the upstream project and then merged back in to Sahara. This includes whitespace changes, comments, and typos. Any change requests containing upstream file modifications are almost certain to receive lots of negative reviews. Be warned. Examples of upstream files are default xml configuration files used to configure Hadoop, or code imported from the OpenStack Oslo project. The xml files will usually be found in ``resource`` directories with an accompanying ``README`` file that identifies where the files came from. For example: .. sourcecode:: console $ pwd /home/me/sahara/sahara/plugins/vanilla/v2_7_1/resources $ ls core-default.xml hdfs-default.xml oozie-default.xml README.rst create_oozie_db.sql mapred-default.xml post_conf.template yarn-default.xml .. Testing Guidelines ------------------ Sahara has a suite of tests that are run on all submitted code, and it is recommended that developers execute the tests themselves to catch regressions early. Developers are also expected to keep the test suite up-to-date with any submitted code changes. Unit tests are located at ``sahara/tests/unit``. Sahara's suite of unit tests can be executed in an isolated environment with `Tox`_. To execute the unit tests run the following from the root of Sahara repo: .. sourcecode:: console $ tox -e py27 Documentation Guidelines ------------------------ All Sahara docs are written using Sphinx / RST and located in the main repo in the ``doc`` directory. You can add or edit pages here to update the https://docs.openstack.org/sahara/latest/ site. The documentation in docstrings should follow the `PEP 257`_ conventions (as mentioned in the `PEP 8`_ guidelines). More specifically: 1. Triple quotes should be used for all docstrings. 2. If the docstring is simple and fits on one line, then just use one line. 3. For docstrings that take multiple lines, there should be a newline after the opening quotes, and before the closing quotes. 4. `Sphinx`_ is used to build documentation, so use the restructured text markup to designate parameters, return values, etc. Run the following command to build docs locally. .. sourcecode:: console $ tox -e docs After it you can access generated docs in ``doc/build/`` directory, for example, main page - ``doc/build/html/index.html``. To make the doc generation process faster you can use: .. sourcecode:: console $ SPHINX_DEBUG=1 tox -e docs To avoid sahara reinstallation to virtual env each time you want to rebuild docs you can use the following command (it can be executed only after running ``tox -e docs`` first time): .. sourcecode:: console $ SPHINX_DEBUG=1 .tox/docs/bin/python setup.py build_sphinx .. note:: For more details on documentation guidelines see HACKING.rst in the root of the Sahara repo. .. _PEP 8: http://www.python.org/dev/peps/pep-0008/ .. _bashate: https://git.openstack.org/cgit/openstack-dev/bashate .. _PEP 257: http://www.python.org/dev/peps/pep-0257/ .. _Tox: http://tox.testrun.org/ .. _Sphinx: http://sphinx.pocoo.org/markup/index.html Event log Guidelines -------------------- Currently Sahara keeps useful information about provisioning for each cluster. Cluster provisioning can be represented as a linear series of provisioning steps, which are executed one after another. Each step may consist of several events. The number of events depends on the step and the number of instances in the cluster. Also each event can contain information about its cluster, instance, and node group. In case of errors, events contain useful information for identifying the error. Additionally, each exception in sahara contains a unique identifier that allows the user to find extra information about that error in the sahara logs. You can see an example of provisioning progress information here: https://developer.openstack.org/api-ref/data-processing/#event-log This means that if you add some important phase for cluster provisioning to the sahara code, it's recommended to add a new provisioning step for this phase. This will allow users to use event log for handling errors during this phase. Sahara already has special utils for operating provisioning steps and events in the module ``sahara/utils/cluster_progress_ops.py``. .. note:: It's strictly recommended not to use ``conductor`` event log ops directly to assign events and operate provisioning steps. .. note:: You should not start a new provisioning step until the previous step has successfully completed. .. note:: It's strictly recommended to use ``event_wrapper`` for event handling. OpenStack client usage guidelines --------------------------------- The sahara project uses several OpenStack clients internally. These clients are all wrapped by utility functions which make using them more convenient. When developing sahara, if you need to use an OpenStack client you should check the ``sahara.utils.openstack`` package for the appropriate one. When developing new OpenStack client interactions in sahara, it is important to understand the ``sahara.service.sessions`` package and the usage of the keystone ``Session`` and auth plugin objects (for example, ``Token`` and ``Password``). Sahara is migrating all clients to use this authentication methodology, where available. For more information on using sessions with keystone, please see https://docs.openstack.org/keystoneauth/latest/using-sessions.html Storing sensitive information ----------------------------- During the course of development, there is often cause to store sensitive information (for example, login credentials) in the records for a cluster, job, or some other record. Storing secret information this way is **not** safe. To mitigate the risk of storing this information, sahara provides access to the OpenStack Key Manager service (implemented by the `barbican project `_) through the `castellan library `_. To utilize the external key manager, the functions in ``sahara.service.castellan.utils`` are provided as wrappers around the castellan library. These functions allow a developer to store, retrieve, and delete secrets from the manager. Secrets that are managed through the key manager have an identifier associated with them. These identifiers are considered safe to store in the database. The following are some examples of working with secrets in the sahara codebase. These examples are considered basic, any developer wishing to learn more about the advanced features of storing secrets should look to the code and docstrings contained in the ``sahara.service.castellan`` module. **Storing a secret** .. sourcecode:: python from sahara.service.castellan import utils as key_manager password = 'SooperSecretPassword' identifier = key_manager.store_secret(password) **Retrieving a secret** .. sourcecode:: python from sahara.service.castellan import utils as key_manager password = key_manager.get_secret(identifier) **Deleting a secret** .. sourcecode:: python from sahara.service.castellan import utils as key_manager key_manager.delete_secret(identifier) When storing secrets through this interface it is important to remember that if an external key manager is being used, each stored secret creates an entry in an external service. When you are finished using the secret it is good practice to delete it, as not doing so may leave artifacts in those external services. For more information on configuring sahara to use the OpenStack Key Manager service, see :ref:`external_key_manager_usage`. sahara-8.0.0/doc/source/contributor/image-gen.rst0000666000175100017510000003632713245514472022030 0ustar zuulzuul00000000000000Image Generation ================ As of Newton, Sahara supports the creation of image generation and image validation tooling as part of the plugin. If implemented properly, this feature will enable your plugin to: * Validate that images passed to it for use in cluster provisioning meet its specifications. * Provision images from "clean" (OS-only) images. * Pack pre-populated images for registration in Glance and use by Sahara. All of these features can use the same image declaration, meaning that logic for these three use cases can be maintained in one place. This guide will explain how to enable this feature for your plugin, as well as how to write or modify the image generation manifests that this feature uses. Image Generation CLI -------------------- The key user-facing interface to this feature is the CLI script ``sahara-image-pack``. This script will be installed with all other Sahara binaries. Interface ~~~~~~~~~ The script ``sahara-image-pack`` takes the following primary arguments: :: --config-file PATH Path to a config file to use. Multiple config files can be specified, with values in later files taking precedence. Defaults to None. --image IMAGE The path to an image to modify. This image will be modified in-place: be sure to target a copy if you wish to maintain a clean master image. --root-filesystem ROOT_FS The filesystem to mount as the root volume on the image. Novalue is required if only one filesystem is detected. --test-only If this flag is set, no changes will be made to the image; instead, the script will fail if discrepancies are found between the image and the intended state. After these arguments, the script takes ``PLUGIN`` and ``VERSION`` arguments. These arguments will allow any plugin and version combination which supports the image packing feature. Plugins may require their own arguments at specific versions; use the ``--help`` feature with ``PLUGIN`` and ``VERSION`` to see the appropriate argument structure. a plausible command-line invocation would be: :: sahara-image-pack --image CentOS.qcow2 \ --config-file etc/sahara/sahara.conf \ cdh 5.7.0 [cdh 5.7.0 specific arguments, if any] This script will modify the target image in-place. Please copy your image if you want a backup or if you wish to create multiple images from a single base image. This CLI will automatically populate the set of available plugins and versions from the plugin set loaded in Sahara, and will show any plugin for which the image packing feature is available. The next sections of this guide will first describe how to modify an image packing specification for one of the plugins, and second, how to enable the image packing feature for new or existing plugins. Dev notes on the CLI itself ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The script depends on a python library which is not packaged in pip, but is available through yum, dnf, and apt. If you have installed Sahara through yum, dnf, or apt, you should have appropriate dependencies, but if you wish to use the script but are working with Sahara from source, run whichever of the following is appropriate to your OS: :: sudo yum install libguestfs python-libguestfs libguestfs-tools sudo dnf install libguestfs python-libguestfs libguestfs-tools sudo apt-get install libguestfs python-libguestfs libguestfs-tools If you are using tox to create virtual environments for your Sahara work, please use the ``images`` environment to run sahara-image-pack. This environment is configured to use system site packages, and will thus be able to find its dependency on python-libguestfs. The Image Manifest ------------------ As you'll read in the next section, Sahara's image packing tools allow plugin authors to use any toolchain they choose. However, Sahara does provide a built-in image packing framework which is uniquely suited to OpenStack use cases, as it is designed to run the same logic while pre-packing an image or while preparing an instance to launch a cluster after it is spawned in OpenStack. By convention, the image specification, and all the scripts that it calls, should be located in the plugin's resources directory under a subdirectory named "images". A sample specification is below; the example is reasonably silly in practice, and is only designed to highlight the use of the currently available validator types. We'll go through each piece of this specification, but the full sample is presented for context. :: arguments: java-distro: description: The java distribution. default: openjdk required: false choices: - oracle-java - openjdk validators: - os_case: - redhat: - package: nfs-utils - debian: - package: nfs-common - argument_case: argument_name: java-distro cases: openjdk: - any: - all: - package: java-1.8.0-openjdk-devel - argument_set: argument_name: java-version value: 1.8.0 - all: - package: java-1.7.0-openjdk-devel - argument_set: argument_name: java-version value: 1.7.0 oracle-java: - script: install_oracle_java.sh - script: setup_java.sh - package: - hadoop - hadoop-libhdfs - hadoop-native - hadoop-pipes - hadoop-sbin - hadoop-lzo - lzo - lzo-devel - hadoop-lzo-native The Arguments Section --------------------- First, the image specification should describe any arguments that may be used to adjust properties of the image: :: arguments: # The section header - java-distro: # The friendly name of the argument, and the name of the variable passed to scripts description: The java distribution. # A friendly description to be used in help text default: openjdk # A default value for the argument required: false # Whether or not the argument is required choices: # The argument value must match an element of this list - oracle-java - openjdk Specifications may contain any number of arguments, as declared above, by adding more members to the list under the ``arguments`` key. The Validators Section ---------------------- This is where the logical flow of the image packing and validation process is declared. A tiny example validator list is specified below. :: validators: - package: nfs-utils - script: setup_java.sh This is fairly straightforward: this specification will install the nfs-utils package (or check that it's present) and then run the ``setup_java.sh`` script. All validators may be run in two modes: reconcile mode and test-only mode (reconcile == false). If validators are run in reconcile mode, any image or instance state which is not already true will be updated, if possible. If validators are run in test-only mode, they will only test the image or instance, and will raise an error if this fails. We'll now go over the types of validators that are currently available in Sahara. This framework is made to easily allow new validators to be created and old ones to be extended: if there's something you need, please do file a wishlist bug or write and propose your own! Action validators ----------------- These validators take specific, concrete actions to assess or modify your image or instance. The Package Validator ~~~~~~~~~~~~~~~~~~~~~ This validator type will install a package on the image, or validate that a package is installed on the image. It can take several formats, as below: :: validators: - package: hadoop - package: - hadoop-libhdfs - nfs-utils: version: 1.3.3-8 As you can see, a package declaration can consist of: * The package name as a string * A list of packages, any of which may be: * The package name as a string * A dict with the package name as a key and a version property The Script Validator ~~~~~~~~~~~~~~~~~~~~ This validator will run a script on the image. It can take several formats as well: :: validators: - script: simple_script.sh # Runs this file - script: set_java_home: # The name of a script file arguments: # Only the named environment arguments are passed, for clarity - jdk-home - jre-home output: OUTPUT_VAR - script: store_nfs_version: # Because inline is set, this is just a friendly name - inline: rpm -q nfs-utils # Runs this text directly, rather than reading a file - output: nfs-version # Places the stdout of this script into an argument # for future scripts to consume; if none exists, the # argument is created Two variables are always available to scripts run under this framework: * ``distro``: The distro of the image, in case you want to switch on distro within your script (rather than by using the os_case validator). * ``test_only``: If this value equates to boolean false, then the script should attempt to change the image or instance if it does not already meet the specification. If this equates to boolean true, the script should exit with a failure code if the image or instance does not already meet the specification. Flow Control Validators ----------------------- These validators are used to build more complex logic into your specifications explicitly in the yaml layer, rather than by deferring too much logic to scripts. The OS Case Validator ~~~~~~~~~~~~~~~~~~~~~ This validator runs different logic depending on which distribution of Linux is being used in the guest. :: validators: - os_case: # The contents are expressed as a list, not a dict, to preserve order - fedora: # Only the first match runs, so put distros before families - package: nfs_utils # The content of each case is a list of validators - redhat: # Red Hat distros include fedora, centos, and rhel - package: nfs-utils - debian: # The major supported Debian distro in Sahara is ubuntu - package: nfs-common The Argument Case Validator ~~~~~~~~~~~~~~~~~~~~~~~~~~~ This validator runs different logic depending on the value of an argument. :: validators: - argument_case: argument_name: java-distro # The name of the argument cases: # The cases are expressed as a dict, as only one can equal the argument's value openjdk: - script: setup-openjdk # The content of each case is a list of validators oracle-java: - script: setup-oracle-java The All Validator ~~~~~~~~~~~~~~~~~ This validator runs all the validators within it, as one logical block. If any validators within it fail to validate or modify the image or instance, it will fail. :: validators: - all: - package: nfs-utils - script: setup-nfs.sh The Any Validator ~~~~~~~~~~~~~~~~~ This validator attempts to run each validator within it, until one succeeds, and will report success if any do. If this is run in reconcile mode, it will first try each validator in test-only mode, and will succeed without making changes if any succeed (in the case below, if openjdk 1.7.0 were already installed, the validator would succeed and would not install 1.8.0.) :: validators: - any: # This validator will try to install openjdk-1.8.0, but it will settle for 1.7.0 if that fails - package: java-1.8.0-openjdk-devel - package: java-1.7.0-openjdk-devel The Argument Set Validator ~~~~~~~~~~~~~~~~~~~~~~~~~~ You may find that you wish to store state in one place in the specification for use in another. In this case, you can use this validator to set an argument for future use. :: validators: - argument_set: argument_name: java-version value: 1.7.0 SPI Methods ----------- In order to make this feature available for your plugin, you must implement the following optional plugin SPI methods. When implementing these, you may choose to use your own framework of choice (Packer for image packing, etc.) By doing so, you can ignore the entire framework and specification language described above. However, you may wish to instead use the abstraction we've provided (its ability to keep logic in one place for both image packing and cluster validation is useful in the OpenStack context.) We will, of course, focus on that framework here. :: def get_image_arguments(self, hadoop_version): """Gets the argument set taken by the plugin's image generator""" def pack_image(self, hadoop_version, remote, test_only=False, image_arguments=None): """Packs an image for registration in Glance and use by Sahara""" def validate_images(self, cluster, test_only=False, image_arguments=None): """Validates the image to be used by a cluster""" The validate_images method is called after Heat provisioning of your cluster, but before cluster configuration. If the test_only keyword of this method is set to True, the method should only test the instances without modification. If it is set to False, the method should make any necessary changes (this can be used to allow clusters to be spun up from clean, OS-only images.) This method is expected to use an ssh remote to communicate with instances, as per normal in Sahara. The pack_image method can be used to modify an image file (it is called by the CLI above). This method expects an ImageRemote, which is essentially a libguestfs handle to the disk image file, allowing commands to be run on the image directly (though it could be any concretion that allows commands to be run against the image.) By this means, the validators described above can execute the same logic in the image packing, instance validation, and instance preparation cases with the same degree of interactivity and logical control. In order to future-proof this document against possible changes, the doctext of these methods will not be reproduced here, but they are documented very fully in the sahara.plugins.provisioning abstraction. These abstractions can be found in the module sahara.plugins.images. You will find that the framework has been built with extensibility and abstraction in mind: you can overwrite validator types, add your own without modifying any core sahara modules, declare hierarchies of resource locations for shared resources, and more. These features are documented in the sahara.plugins.images module itself (which has copious doctext,) and we encourage you to explore and ask questions of the community if you are curious or wish to build your own image generation tooling. sahara-8.0.0/doc/source/images/0000775000175100017510000000000013245515026016317 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/images/hadoop-cluster-example.jpg0000666000175100017510000011235013245514472023413 0ustar zuulzuul00000000000000ÿØÿàJFIFHHÿÛC       ÿÛC ÿÀhX"ÿÄ ÿÄc !1"AQ2Waq–#‘ÒÓÔ3BRTVX”•³Ñ$8Ubu’“¡ 47tv±²%&(56DFScrs‚´µCGfw„£ÁðÿÄÿÄ*!1AQRa"2B‘#3b‚ÿÚ ?ýS¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥C3í_Ó],Õ¨¹½¯b)œðo¬S·.>ý·ýµÃ"Öm0ı{fk’çV›uŠñÃÐ'È|%™<’Vž öîMÚ• •¬ú] gS%çV–qI*fî·€Œµ”íõ’u›K®X4½KÚdbÐVQ"ì‡ŽÑ !JúŠ’?-Ú• ƵŸK³,né˜b¹Ý¦ée²óøBlgšÅÕÍ^Í’wû+>³év¥·=ì<´ß[µ!.MT7Âà ;Uî߉üÔzT×=%Ô»”‹.¨{ôøß ðµ¶ß žD{· ~Zãk¾f9S˜F-¨Ö[ù•:—-ñß y%­úƒù»þÊ õ*^4~VntÖ>¢Ù\ÊD•Ã6¤È@%Hãï$þCK¾¼i‡1FžÞ5Ë$qÖ˜M±Ù>§ضŸz¹ ¾Ú ý*˜ë¶‘iíù¬c7Ô;5šë!´:ÔIrRHIÜH#òW×;ÖÍ*Ó âj}hÇ߸6§b·9ðÙu ¨('4¨Na¬ú]§Ö»]ï5Îí6h´ÛäKx!G«toª¤ŸÊ){Ö}.Ç0ûv _3»L,ríÃÐno>ÃüÒTŽ*öî#ì ›R¡*Öm0F59YÕ¤bŠWw/F*çÃn_ûݾÚZuŸK¯ØlíC³gV™xͰ¬Lº4ð1Ø(¨)^Í‚‡ç ›R¡8–³ivweºäxvwi»Û,ˆ.\%Ex-¸É+%gÙꤟ°WZô«SܜΟg¶‹ó–ÆÒì´Â|8YB·âUî‰üÔŠT ×]#ÔkÓØæ ¨vkÝÎ;jyذ丄$€¥îùkMxÒ“.s°ê%š~FÛ2«kŸ k~¢xûÓÄïöPOéPëÆ7›6^¢ÙFQé'Á%ñéb7ãü-ˆ¬ä:ñ£øž\Þ’j-–ݺã-7n~@KêSÛt€Où܆ßh ŸR Y¾¹é&›]ذgš‡f±Üe4f4É q²JB€÷nûE}siÒÝ2ôŸç–‹ÂhS°ý5ðŽº¶å>ð9ÏA8¥BrgÒì&ÁlÊrìêÓi´^‚M¾d§‚[”Ž` ûwOz\µ›K¬ØL-HºçvˆØÅÅin-ÕǶŽò•ˈJ½»ñWæ4jT%gÒÙ8"õ>>wi^(ÚŠwFJ‚øUõ/·ÛK³i~Q‰Ü³¬{:´Ü1ûG?N¸°øS1ø$)\•ìØ-Ú•Àu7Õc÷=Ì-y 8¯˜¯½B]KN„…pVÝÒ­”“±öRŠ)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J Ñõf2¼A3m~Îá‰Ã\|©q’¦ÓÓP:„|þŽþ®þ]ö« Û|GšK/Gem£æ¡m‚öP1—å#Ä xŽ?Õ‡9xS¾Žwôá5 õ<¿s'ÕüµbÐu¾gÑÌfKCɈüžTLˆaQ‘”²¯6Ãc‰üžUÙ¥Y»|F›[MFe_ÎJ[õçYjXàˆñÚk—epl þݼë±J«Ø1”W# (ö% „’?%e»t]ë³”9ß×K`þ}ë³J¯Á°ºþ’"0Ü«ŸHrßß¿eVèKw®¸¬©Ý÷æ[·û|ë³J«ÖØRȊënKh·ÛYzÝ NÆLV])m…mùë³J»Ð"HBPôv\K0-° Ÿ³*­ñÒX\fTÒ>j `¤}ƒÊ»4 ëÏ£ìôõ}1ÇóyQøm²¨èŒÊ[Wš`$þO*ìÒƒ®Ôl!m±–ÒçÏJ[+íΰͺnFg¸·C}aÇâ²â€Ø¶@üµÅøQ$© “—xöO6¶Võátñ¿«šy©)²qŒ#rÓk¸Z®y5Šå6%íÙ-%Ç›â1E\7Z¹ž"¾w­_ÕûDz%e–Ix-§I^íº$™/ÇrÚTµ¸ämˆB¥-ÆÀ”„–¸vރݮ[á¼ÚZv+KB>jT€@ûò¯œ˜öæ"äµ1ÐFÉq)ÙçØW‡¼Y^¾™,_ÿoèSIðˆ˜žë¢•Kü²¼1}2X¿;¿¡O–W†/¦Kçwô)¤øN±åtR©–W†/¦Kçwô)òÊðÅôÉbüîþ…4Ÿ°º)T¿Ë+ÃÓ%‹ó»úùexbúd±~wBšOƒX]ª_å•á‹é’ÅùÝý |²¼1}2X¿;¿¡M'Á¬y]ª¯ñKáã0¼ÇDZý`ÆdÜå«„hŠš–}^æÒç³õ'z´RâT@ÒQ¬K)J$¥)@¨³è¾#®Ø‹XVjõÅ»{7—D˜/%§:ÑÜ7¹RT8òÆÝÅOiAçK÷€ÝÈ$êT¹-ßX{UfÅ}\y¤¡Æë$1»g‚T笠yn|¶¯–{à;Hõ ̯W ³=¶ÃÔ£§!³ZïHbß9æiõ·Ò*+HíË=ÊM@'ÿÂqk•ª­¾OÓ—26ñ‹^Eð£ úTƒ 2]S¶É<¸¯ÖÝ@ý¦ücÝgjöw§ðtŠrñÝ1“¾W”.èÒ#@‡èª‘× ‘ÍjÙ~Èß~ýƒo›ø"Ó¼ÝÙAÍDÔ‹4+µªž÷nµ_PÔ;Ìx͆Ûô†–ÒÀQ@!E®žü•ï5·¸x>Ò™Z…‹ê¶nGf{ÆšÃÑk·OKvû…¤©-Öҥ:Ø =‚“¿m÷Ú«Ì3ÇDËåß¹fZ!Åp]L¸&ÙŠd’g0÷]÷ è Ñë0ÛÔ;«Ì{;×vwŽ l-Êu£ðJ™Æs·0uÀ“Íõ¦Sqý /†ÉºÇmûm½³J¼iÖ‘dö¬‚Á›j Â:‰ Øl7Kùz×hCÛóK ¥ QqKÛÙÜSýÑŒG@tæÝ¥ø3×löÇu•Oy.¼Tóªu|””¤YgnÞUEgþ52¬w9Ôì7ÐÆRÖ”z<›äö.ÌGoÑŽ*BV9)À ½@áî;èÝ8Ïl:¥‚X5[ª´ävön0ËÉ p6âB€X€¡¾Än{ƒA$¥)AÀ§êÌsÔáÁbî›ËC$ä±›èŒðÛ—oÜz_7·o}O*€Y°kfažÍÅo ™t¹^Z~þÉt+Ñ%ˆŒ¥-°ã»)i[wùÛÔòJRJRJRJRJRJRJRJRJRJRJRJRJRJRJRJRJRJRJRJRJRJRJR‚2|LksXi°ï“+]Í7N’6´6cóß—î„/m¶í¿@¼P^¯9~‹Ø®²m‡=œúnÓ"¸[}»DVº’›icºéSMst‡FÇjŸ&^Ÿüw7PÜü88ªÝLŽ+ãðW¥¤o¿úÛ¶ßëÚ«Muÿ­!ÿFdŸÙÄ®˜«¼D©’f)3 ®7†bxuž>?‹cÐ-–è©àÔxì%)Þ}ª'Ì“¹>Òkiè±þl×ú‚¾µÓ¼ÜÓe³Î¼®$™IÙ%ˆ­õt!%\[Oï”vØi VÞ‘ mfg›‹RlÏ~Ö̈.MŠÚ~2V‚ëH^ü¤é â­‰ö;yVTý™íJzšó+Üb¤uVÒ”­ae)RÐ Û`T|Åyb랣%Ô[æ{‰_î00{ÄŒzÓ}¿Ãim®#5×)%²r#n,~æ;”ÍjšÈY›¨Û®g©5³‰mÉ`Ø2&Ô¦Ÿ¸EmÊe+x#×ÃÁÆôQó·!\§$DôtŠ|½/|ÕMÆ.XòMHÃm7(Å!ès¯X}²¤…M­aCt¡¸ò Ô¢/Á“c56¢È!´ºÓ­qZB†éRT;Aìª 1/™×Ãýû9ÇÙùf[×’¸`+¬m.î\}SËnÇÈì=•£Ê/™UŽQ‘\²ü¡ HÎ\Ç¢GEÅ6ë| ¥+O7úK1Тž°IW¬” ¤_¼Âw5åôï¢Æþ.×ú‚ºñ_³N\–á= BáÚòN!¨Ù-ÒXZ›¨9;Es!j·ä™,ÊlDiRÊV÷¥´„õûä~˜æUª–LË,¸°ÕÌZn®Y7c¤üûÿÀ°”ËSÚ Ù ©)PBRxõPR®ü'%|#Ò·—®½7ñfÿÔôXßÅÚÿPW1ìëU/–oCµg“Ò&³dnäôk“÷1&¿rŽÓªää6Qe¥>uqâJ@$ûÏlkT[P2hˆÒZô‰Žõ_waó–²%iÚ¦–‹ô…mY§Ykò¬#Íì²1̳ƒt¶ËIC¬>È ûˆ>iPö(G˜ ×ÏÂíþ÷ y†ŽdWYWGôþàÃ6Ù²œê>ý¦K!ØÕžë[`8ÑYî Ø'rMH*#¡Cÿ½_ÿFcÙʯ>ÛHÜÕèØí;ú= JR³E)J)JÉ\—Iµ]ØxsÀ¬š·ÂÖ¢1p¶â× (&Û ¡-.¹)W¤î—b$© Ý'‘Nÿ7¿½0¿ ûI­—[ý÷ákn´8צ@º&#"ã¸×S‘çÉ.(ï²vÿ}^;W*!á¾ó;tÌÔqw(Óý,¹¢éŒX“cn,޳[ú7¤ÉJÉp3Û`9{ví¶‹"ÿƒó7»Xrý=¶k× `¹&f¬á‹J±Ô:ûSW%)§$uASC†Àá$J½³J/Ãμçþ!¼H c7ºiíƒ1r×mz\ÜhÊv†¨¹Ç Q² ¥\ƒ±fénŸÙt§Oqí7Ç:¦ÙÛ™·ES§u© ¤'’ðŽÄŸ¶¥T R” ‚`8uÌ3ÛÕ§"EÆVIyj}Â2Tƒè&#-ˆI$†Ð¿[cë{¶©Ý@t÷OœÃó-@È—wfbrûÛ7D2†ÊLPˆl±ÓQÜò'¢U¿oåSê)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J`iùÖæî¦â¿ÃÁEÇLNJãðW¥¡JsŽÜwëqòßêöÕi®î!¿ZB\ZR> É;“°ýÎ%YcÅÕ­íæÿ„÷ÄœQv¡kê#¼#-™~gFÿ7¾ÞuLø¥Âq]@×-"Çó+#[r dOïóKqx«Õ öÜ×L:ú‘£ž^T•é1¿Œµþ¸§¤Fþ0×úâªÏ’¯‡¯¢ËGçwôéòUðõôYhüîþlÌÛá‘ôüþ–—¤EÛoHgoýñOH‹üaŸÏVü•|=}Z?;¿§O’¯‡¯¢ËGçwôê>¯û"kçô´ý&7ñ–¿×HŠ|ä3þ¸ª·ä«áëè²ÑùÝý:|•|=}Z?;¿§S­¼Gìú|Ïéi⑱}“ÿÎ+>“÷2Zÿh*¬ù*øzú,´~wNŸ%__E–ÎïéÓ[xÙôùŸÒÒëÅìû=ÿÏŸIüe¯õÅUŸ%__E–ÎïéÓä«áëè²ÑùÝý:kÙ3Yï?¥§é1¿Œµþ¸¨ž„- ñ«Å J‡Á˜ßpw¹Ë¨ÇÉWÃ×Ñe£ó»úuÝðµ„☹jî?†Ùµ[“xGd¨§š›•ÈúÄžû òí{ÞŸ7§ešÅù=EJR³E)J)J)J)J)J)J ïN0kæ)›ê-úë)—båwÖnVô6áRše˜eIP#Õ<ÚQØoØŠ±*¹Ó[kgÎ5"㓺ú­—‹ë2lIrWU)Š˜L!A)ÜôÇU=»÷öÕ@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)@¥)ANÉÖöõKá„QŠ.ÁðX…KCýn[ùnž;mùj¶×OñŠÒôfIýœJ±ŸÝU¯ê àÖðõãÆ>êëuÕ5óÛm¸ñNÞ{ïUωåþ gÚ[ª×ÆÇg¸M±]_?2. ! ¾áòKaæ[B”{¨$Šé†b2Fªeû'D¢ڨó±ôÃ/ëiÖ¬7¡Ä(¥HP޲#ȃí©@RT¤B†àƒ¾â¾S"EŸøc¶üy-©—šq;¥Ä(l¤‘í+ncXäÅÞˆžo$Xóì—¥zudv•ßm·y//…ÎÊ,“VS!cr·#»Å.v$¤²¿œ®Ûx‹Õ˜ê‘h·¿.}–Ûw·H™d\6Qé78ÑÅ¿H[®2S#t©ÀÚ÷G‘Üíéix¦1<ÛLì~ß Ùù V:è¡M–”Üz ¶¢’˜;VžÕ¤Z_cYð =Ò”Ì$$”¥Ä8”ï·[M¨ P6Ûjåéß˯«Nð§¦êþ¯bI®d,~ñÅ—7‹Èb%©Æœ‡¡&R^ /(6´€•l£¸'´-Ö-SÍôŽèÆCÛdL¯Mâ,Ø­µè 2b¤!…&K®HAKäuJÅHî=p‘ëW1lm×—!Û:äÔ\Öµ2’U-Cç·Ï HHWžÀ ÔÁÒ}3¶|$-øŠ7Âû á¨-§Ò_0•ì;Ž~¶Þ[÷¤ã¼÷=ZxyÏ$ÖlÏÊíØöFµKÊðLvö‡ËÊR#]£­ësp®K ýêp¹·Ì[o°Ø™ÛVuBèPôfE÷›u»?nèdm[ˆðÚ’Ì÷Û‘úÄ)Ýíä#×‡Ñ¸í²¯ Ž-]ç‹¥ÖÃd±Øgã¥kôg.4IЭ†éò;Vš6’éŒ;¬V.bjÑ5á"D4Âl6ë£n+PÛº†Ãcæ6R1Þ;ž­;ÃÎ8æ°jÅÉM•l\®‰›“^f\˜€™L»è÷52†›CÒšé0 Ýj Y@(wÞ½?ƒ_æeXe%¸@D)WK|yoGCÉy--h RR´’•€Ob mZ¹z=¥S­lôóvÝ×Pé²·ë)NÛ^ß¶¥¬²ÌvQ;Hi¦’„!!)J@Ø`öU±ÒÕê¥ïKts¨Ž…ŒV¯£1¿ìåÔ±jJV¥¥#rIØï5ðÀ ómOÕ¨•–ùs‰d´¿±ã-›s*C¶|”ÚŸuä¥C±$n6¯>Ù1Ò^õ‡¡éJV[H¥)@¥)@¥)@¥)@¥)@¥)A]i²óÕgŽ2ä͔ߙü/€aú ú[w)ëu|ý»Õ‹UÖ›dyï8Ô‹nJè¶ÙoÌIJ)q:Aq‡R½‡TuVç­ßc¸ß¶ÕbÐ)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)JP)\Kˆb®õ‚ëi•¹Øoí>ê ôbÙqñÖj?‚éÃ\µ½$íéæj éoÿ«sÛê©•ÿ³e©–‚Ýám¸²¨òâIh8ÓÍ(l¤)'±US’gâÓ{Ê/ÙA‹§¸®·ï{H3 y–—§A. p¢“Üv“µHq}uÀ3¶Åۿ¿cÑâ;.éz‡9ƒÔ„­(•,8•-%dl“Ǧ®\{n„xGbÌ= ÖýEÆ­ýÂØÔèóŒŸbT¦œq(Ä•<†Ã`2¯ ùRRUò›ÔŽßû;oìµ)ÂõbѪöɱðMÄd]-—?$Ù¥&â„[D’R§‚Ýa$ù!d‘È'¿ÛÔM!ÔŒ¢õmÅu^K&t³*ÉêÛíGe¾iZÐÒ=du@Z·;ñO–Ýí¿o*zuð€Éð÷t…6¶oŠÜñ‰w~‰Ç-IrGò_M&6ëâžç`vu«{D²g­W ‡_µrêí¶áðs±QÝŸ—‡‚üT%m¤(«˜%$$ñ$ö«cÔ½ ™|ƒ¦ø¾kŒL½ÙR¸Q-lÍméQƒ(â´$V R’·aÞ¹d:÷¥–[¼ÌJ[hºåÑTZk‹pŽ›„™w !-#š½DTú–òztð¨¯Z¨K’Õ7Ä>£3â$X²bLJ*I˜ûÝ-œa¸^«)ä…9lQW’j7tðï¨x½êœƒÆ6£Ì“Õ2ã>,e…ú\VÂytÛkJ] (Ù$8µA#¹5}MÕ=(ÃsÉ–‹VmÐò »Âlw³AQ–¤%-¡–|ÓÔÜÜòQÜWÛ5Ö<O³86\ÏR± iVñ…r”šë¥a-­¾J éì±$í±ó§©o'§O Œh¶CÑ2û•x‹ÔÌ~Û…é†Õþ‡VÛl:cޣބ,ƒæI>L9Vçÿ HØ™lý–¥:‘ªÚu£o/#ÌsMœÈíR®‘Ùh6—ÚÞŽ‡”„„¥.õ<‰!¨ß`{9vi¥¸e±ìöã¨ÖŒiܲqm÷yó‚£9Áµ­•¶ÚÖVÁ¢·!¶þÊz–òztðˆ|—r¯ç7©ìí¿²Ó仕9½Hÿgmý–¦ë&)XlY]÷*²/¹B%YjîL± É`¡(m*ïÕürI)OH‚w"¶v½kÒ{ÆuFߟYÜÄÂÖÙ¼)ðˆÁHp¶ V­¶Ù`§í§©o'§O Ù~âÞ‡¡gºÙ¨™=¡_»ZÞœÄ6$§Ú‡U¦ÜZ‘O0ÜÁ"®û†ÓZaجVøð-ÖöS,XÍÚe¤’„¤vÛ ÐÙ5sL2Lb~k`Ïlw ¯Ÿ¦Ü£ÌBãÇâ¥sX;'`A;ûéˆêî—çЮ+=±ß"ÚR9è3ò# Ýe'Õì’{ûVfmÕ1X¯Dº•Â5‹Jµ*\˜ Ør0ÚHjÝ5©¦ÉØ)A$ì7í½|±mlÒ,âúæ1‡jF=y»²‡r)í¼òR‚ÉJNû@?mBɵ*oÖÍ"»eÊÀ­šÉÈÐó‘Õjjsj”o~héïËq±Ü}T—­ºE/NŸÍÔŒ}Œ•o¢2mKœØ”]X(éïË‘l>º µ*’ëf‘a¹ x–W¨øý¦ôèl·dæÚ}Aò6A;žGËß_LÛYt£M§1lÏõÁË”Ï]–.3›an7¹O A#pFÿUÊ•Ì5wL4ùˆ³œöÉbfè’¸NO˜†S! J ˆå°R|½â—ý\Ó [·åù&{c¶Ù.Ü}á&bbO$•'‚ÉÙ[¤ÛÙA.¥Ddêî—ÃÂÙÔYy표^J‚»®bE’¢›ñ;¨çæ)W4Âã†IÔX9íüb”‰væ ÄiAA$)Íø‚ ’<üÈ —R¢8ö®é†[ܲ¼c=²],Ö~^Ÿ:$Ä:Än(殢ÁÙ;'¹ßÙXõwKõ ¹Ï`ÙõŽü‹bBæ*Ô<#¤ï±_êTùû¾• Ã5ŸIõâýŸÔKþtVºïG·ÍCÎ6ÞàrPI$ ÈÃÖÍ"˲W0ìcQñû¥ñ¢è]¾,æÜ’Ù!ÍÐþ®ÇuKN3»ÖY›j.?sˆÃ1±;ë6Ø+m* u¥Âaò¥îv'›ª†À{ªÃªóMu&6y–ê–-¶; á×ôÙŒ–\çéNz+N­Jì6RTáAŸ›V”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥”¥•À:·òûk¡xÈìXôÝ/×X¶èm¸ÛK~S©m´­jBJ”vÝJRR´(6T­:òÎ^%ØY‡9R¢CD£iaÄ­J B^>¡^è;¤À)'±Ó‡?.º1ŽÜ±Å¶7/ñ·¨w ʈ‚ˆm¢×&Öà{¦º‚xò “°!$®+QHmçQ¹X¥Þël½ÚnùdÞÎQrÖñèr-ñø6Hq$•´-\ÎÇg6Ûa[Y¿T«s _úXŒô~|ú{ÿŸ¶ƒ¤Œï‘ Ís¶ÎøN úP‰ M½¥ÉiK)Z¹)m‚ÓP+VÉa¾ä Ä›ÆW*=þ5›n<è7kzå!)‰=e¤¬/vŠÜCaj(<’ºI‰ßFŽÌV;-´ÚÉCi Hû¯­”A¾»{‡=wv[€˜‹nU½º’ IK‰tì Ç»òíµuXÀì†××vzuå6Û‡ÂqŸ¸È.¼‰Õ8…rnU²GHÙRJPtÕh¶,È.@Ž¿KHäÒOX ¾Þ¶ß]q‹c³ÁeÈðm‘c´÷îˆi”¡+í·p~Ý«½JŒ+¢ÚVmÖȱK»–YJ9äýb%†ËotÈi‡Õ Œ°„(ƒæ7zïÖ²ñÙ,«ƒít ˤ¤Á„—–_R¥Ñ¿šŠP¢ÕAðzÙa·Lôˆv»cw7’㌫¦†Ýu[zÛ(^Þ翟zÔØpË<ˬLúýˆÁ–?mn4ÂÓ½t²wæP•ì¢NÎq #êí]ëEŠT³ñ—óJ¾[Ý–˜’âÆ# ÃŽ’‚²T’[ …ìv*Iöl*EA¯“a²J’'Ê´ÃzBv!×J—¸ò<ˆß·²£×;-qÏ`M¹àÞrE±öÚ¼»3ž£eQʉÜ)Jâ 8žÈ=Ƕ`|ÙQ÷Ÿ1…ðµ¶Š­ï/àBç'e·þ üfÈßoWñûíA²™c³\Ãfãj‰+¤‘×a â°nÞB£’±(6g£Ä±b®1.—e˹úd¢DP¦HSÍ!iXóm¤tÓÁ (‘¶ÇyŠ|‡ÙY Õ5m°Ü`"" Á‘'ñm¥´- RHì6Øly¨ï][¦ ‰^#À‹q±Dv=®XŸž-5 %i à6Iìâ†Äßʺ­Úޏ­ÙX°Ùñ1¥¿5”²¦–Ó…aÀ¶øÚQÝõ/q¹$GzßZnP/6¸—{T¶eBšÊ$F}•òC­,rJ’} ‚4eYîøížô-VøwïM˜—bZÔÓ0Yi…Ð¦Š’“Ï`½Ô ;ñ÷VÊÖö,Åòv)k·³c—)¶ ©¦ÔÓªq(õø„,îÛ›¤@Ûp†ò*ëÜ Å¹Á‘nœÂ)¥2ók¥hPÙI?QŠ„ –Ø¥.Ûj‰K*,2”ä±Áe‚ù•Õ;êZaYÌnt"b¦Õ"ÊÍŽâåºÍfˆ¸i´°Ó}QÅ ku•§¦x„¨Èîm¾òy¶¸0\Ëí"$Û…ÉVöY·uf rqa•­a´”!)RŠ€JIØ“æCn݆ÈÔ³pjÓ ŠŠúéaÎGÌòÛ}ë ÇìnMøEË<%Kä×TtG–ÛïØw®Úd²µ©´8•) â´‚7IÛ}»±¾€ïAГÙ&È&ÚaȶuÖµ¼»‘¿jÌÛ –ä´¹qµD”´Ž!O²•<öÜzïÒƒ¥2Ëi¸Óp¶Å”ýÌ<ÊWÃìÜv¬H±Ù¥Æj»\G£³·M—J†Ãd‘°ì}•Þ¥EVK:á&Ú»dUCIÝ1Ë)é¾ý“¶ÞtnÇgj­Í["¢"þs e!³ïÝ;l|‡æ®õ(:1¬–˜q܉Û†ߪÓl¥(^ãcȱíY‡e´[‚žÙ0pl¾‹)G!î;½wiA¯ƒ`²[S¶ÛL8Ž(qRØa ¨q ÔØâI3"Z!³![îël!+;ù÷~þÚØRƒâÄHÑJÌvßUeÅðH”|Ôvó'ß_jRJRJRJRJRJRJRJRJRJUo¨>!´£L2ñ\ºÿ%«³‘“3Ñ!Û%Mq ©E)ZÄv×À•Ëmö;yPY¨&k­XŸZm7¼¢mŘw´ó†¨ö™r”¤ñJ·RiJo²‡ÏÛî4¾ë^aVÝA¼N¸7e»ôýÖí2Ýu\ÒTžL¡²ê;ó’6ò=è't¨#š×€7§£T6ãø¿*Yµ¯¿à× FµÍ¸®Çk+]rÓ-§’PVÌ-°ê»(|Ôý”ºTÖ¬=±ÝòÕgµ‡M¹>«khvO¥Z%Ãâ•o·»HçóOdîGox žÒ«Ì^´ÛR/ã¸ÂæüèÍ)ç"Í2*R@;8óIA;‘Øëƒ_t×%Í\ÓÛEÂè»ÛN¾ÂšvË5–¹µË˜- Ñù§cËcìÞ‚Å¥WK×Í5o=h¹÷A2„>À³K=R¯Òém±·=¾ºÎI¯šm‰æMà7¹÷Fï.­†ÐÛVY¯4TöÜ?ÛJh|á¹*Øwßm‰J®µ_tÓLoqqìÆçq:cIu”G³Ì”•%J zì´´ƒºObwí½u3ÏZO§..>Axžü–÷ l´Ëžê…ðZXm|RâJ¶$+í ´+“ü!ùê³ÌuóÁZ°^oöë²1܆¤Gº3÷Ô‡vBÊã6ÚžIR¥r) 6'rkfêÌXš]cÔŒ‡%rÕ ]Í SÖ»¹^“ªç u´_k›iJà8«}qAn­ÖÛIZÜJR‘¹$쮄œ‚ÉíÂýÞw+›OH‡O$=!¦xuVÚ7ÝIGU¾D ‡4ïæ*¸ü9Ò«žê|ë•î~)}lCœÄÈV€„ƒHn›ê +‘åŶü¼»ÖÇÕ]1¾àòóLqÙŸ㌭•­Ë4¦]a¶ÛIR[eƃªB~jNû{M„峦ڜ¹Ø1‹´Õ³t6åÅÁ„ïIèºúCüy´âTžÎ ‚®C~۩˸\ã Å·®+bÝ%´©r ó)Ä(p)OâÊ@=ý`}•Á5«ÔkuÎå‰Í¹HbÌ€©fE¢\E$¨Ž)y´©}’¯šûë†ë–êœùVÜ2uÊCðšKï VyÒNÀ…>ÒŽçÈh7¬c3«»®Cq•:ÎÊ댫ÑÙœâ›ZÝe>©ïº’Ÿ$“Ú»–¼[³±-‹už3 ϘåÂJRŽÎÈq@­Õoûâ@;ýUÂõóMµ$wÆ'ݹ0‡[r,³b %µ«gi(=Èì¹öoµ-zû¦—œåzso¸Ý|m÷c)¥Ùf¡ž£{òu4ÛÕ;[fôiÉ>UÈ*»¯ÚimϦÒîDß•%¨¤Ùf­ž£€ŽºZ-mëüöÓL¿_4×ÊZÃrI÷Fn”„1e›!²;#wZiMŽã¾êííÚ‚Å¥Wú…®šu¥·(¶œÊuÎ<™Œ-¶i“Qȧº˜iiIÜăõW×=Ö¼M#[eåó®Û»¥K‹èÖ™rÊ€ 'aµ”|áó¶ÿuî•Éõ¯Oñ N×›_fÜ[´ÞJG´Ë}ÅsAZy4Ûjq¨?9#o/:KÖ¼ŸÇÔùn#’´¡·“i–§‰+(Ç õ‡¬î¯Ê‚wJ‚[õ¯º`56ÛŠ¬T¤<ê­2Ðø)RRvŽ¦Ãªî¡Ü ûý”ŵ«Ì±{¶ca›qrÕd ôÇ´ËŽâx£š¸´ëiq~©ýêNç·Ÿj Ý* €kfŸêkW± × µ¡+’dÚeÄâûq¶Ž4üÝëY‹x…À3áv‹‚"õx¸Z ;8Â6‰PÖøG“m®Km¶V¢@îw; Íë!¿FÇ-¦é*4Ù Zd¢G$»»Ž%†ÛJ”@*¶v× U®çL÷nו\š“0ȆÚã6ß¡4P#uì ¥sW­ë‘äW²éÆaŸÜ-[ÞL»Ìæ‡-sís‰Q¹õ:k[A ¢TBˆY à·Ûs¹¯zm+=V™±>èo蔸e¥Yf¥žª*r×Gm’}n{Ûâ‚Ä Yªîó¯ºiaÍѧw;ÑżÓ¦ì³]k›€º-ë Ï-‡·jgù¦ºu³‹åW £ !æÓË6Se+QH%ÆZRtžÄî=´¼mûV‘äÎ9LU'ŠìOBt.ê]Hy—9£fŽ<ŠT7Q<€±Þ£š‰®{¥²!ÃÌç\£9piO1è¶ysJH’ÃK ;‘ØìkMë—éýïÈò¬‹!Œ»­±Ó4[lé1žd©µi–”àõ@+Ù@)CßAk§æ²³PKîµ`ÎlÔ Äëƒv[·LDu»L·]W4•'“(lºŽÀüä¼Ö¼½= ©·Áâ €÷Á2úû—:ñ~Ÿ[ç™õùPN””­%*ƒæ F¯çbÏܲ‰·9oXb[ÛJ-Píe÷q Q[¨é:áRT„ð`O´íª³k^Á®k›q]ŽÖV$ºå¦[O$ ­˜[aÕvPù©;û+~´àõŠñ‘cSn/A²!Kš§í2ã- ¯ÔmÖÒ·=TŸ˜»Ïµå¥õì#~ãjçTþ­FSp½OÁ¥åÙäL„©Q$[_ŠÝµ¥€ÏQ¡)¶¿8V­É w·¸.½i¾£_ŸÆ±Y÷7§Fio8™6i±•–óHA;‘Øè,:ãÁ>꯱ýyÓ|£3³\.K½Gqöœië4ÆäÑ!`<ãA¥m±ÛewöoV;w4Û– Qoƒ®Ár¿-J¹Àe£ ¼”!´ºz‰RT Ûh@ädì÷TÜŠ-ÚJ^·GU•˜)uê×-rB•ÍŽ T”£©°ê»¨w>ÿeî•ŵ«Ì±{¶ca›qrÕd ôÇ´ËŽâx£š¸´ëiq~©ýêNç·Ÿj`Ù§úšÕÅìBuÆBmhJä™6™q8ƒ¾ÜCí£ŸÍ?7z Ý*½ÓíxÓOºÉ²áóîoËŠÁêeY¦CH@PI!o´„¨î¡ØkŽ-¯šm™eÎàö ÷G.ì—’¶ß²ÍŽÐ-øçKgmŽÞ·fô%*»¯šm/=Vš1>èo锸e¥Yf¥žª*Zém°>·=mqKξ饇7FÜî7D_óL›²Íu®nP:èh´¬7<¶ݨ,JUwœkæšéÖBÎ/•\.Œ\$4‡›L{,ÙM”­E —iHÒ{¸ö×ÛPõÃOt¶T(y”Û”wn )æ[<¹IIîXia'r;ú•Íu«ÓëM¦÷”M¸³öžpÕÓ.R”ž)VêC-)MöPùà{}Æ—ÝkÀ1¬*Û¨7‰×ì·~Ÿ¢:ݦ[®«šJ“É”6]G`~rFÞG½î•sZðôôj‚¦ÜŠ‚ßËëî\éÿÅú}oþg×åK6µàüá¨Ö¹·ØíebK®Ze´òJ*Ù…¶Weš“¿²‚wJ‚aÚÕ€g¶;¾GŒÍ¸½Æ•.jŸ´ËŒ´€‚³Å·[JÜõR{ wŸj㧺ݧڢìö°é·'ÕmmÉô«D¸|R­öã×iþiìÈíïÚUy‚kÖ›jEñüw¸\ߥ<âdY¦E@JHgi('r;½q°kîšä¹«š{h¸]{iרSNÙf²×6¹så´?4ìyl}›ÐX´ªézù¦­ç£M>è/æP‡Ñøigª@ uú]-¶#Öç·×YÉ5óM±<ɼ÷>èÝåÕ°ÚjË5択ۇã›iMœ7%[ûí± ±)Uîy¯o¦·¨øþ]p¹±6S)}´F³L–’‚¢JÙiitžÄƒSÈÒØ–€¶ÜFà¸ö‘ ûR” TÛ‚[¡k=ÿQ[¿!Ù·k¾Öí´%<˜m‡_Z]'}öQuC¸ÛÔìjyP ^žO·ëvAª ¸0¸—«ºÐÜP“Ômqžâ–O–Ä>~© ŸqNûì+ à‘É@+•k2{²,ÝÖú¶ÊÓm„ü²‘æCh*ÛýÔî£x‹Àt÷!NÜ[ÖM”S!V<~Þ¹’YiGd¸ñ6ÊO³¨´“ì¢ß*{м4j‘ü wíUðÝf´ªÑ—ÜJdd9»(ɯÓOuÊ›)Ån¯>( Kh½BY’_c»(´ã¤)|G%«a¾ÉÌŸ`­ lu݉´¼7Úæ-1XG~U¯æÏª_ìmßµV~T÷¡åáŸTØÛ¿j­%®öe“KÅ-˜¶^ÄÛsÈbr¦XÝa¨Ž-”¼„ºµvG&Ö•ýáSñ&1io ѸRÂÇGžæ¯9uRv»Äé¢;ò§½6}Qÿcnýª³ò§½yüš5Gý»öªØ5”YžÊœÃ[yj¹µ»šêz:ÜSiPW‘<®Õ øôÇ¿ F!™ü*SÖ 6º}¯K¯Ï˧˷/*px»Ù<^Ok±vñU¨îÚ®(°x_ÏØ¹!| *h‚¦½u„IäŸß †ç°ï_QâW2ø|\΃j§ÁÞ…Ð6Ñ ×ïÏ~¿SÒ¹ïÇÕã¿oMÜ’ÃIRÝ}¤%%EJb|‡ÿ÷¾²§ZB…¸§7â «o=‡¶£ƒÇäâïá\ÂÖım´[.>õbïðâT7Sd–Ÿ`;ÕvYYX P ß°5¿GŠ+«jZÑáSÒ§ëPmGËsþÞ¶˜®Qh̬1r[ ËvÎe¥¸‚‚x-HVàùwI­˜}¢ß\<Ùh÷æ ~ÝéÁãòqwð÷ë/Ã^©!#̈Öõïì%n~ÁSm0×Ý9ÕYòìvIS­×ûsaÙ¶+Ì`Üc 9–\›{:•#s·-ûWG¸öUKâ%„ã¸å·Y-(K7ý?¹E¹F’²Üˆ§Ü¸«#¹iÆV°SåºP|À5L›$Eu¬óZ›\Í¢-X*€6?UgŠ}¸Gp:ÊO’†ã쯥xæ8§Ü)°÷ Í(1Å>áN)÷ Í(1Å>áM‡¸ViAŽ)þüÔâ=³J qO¸Sˆ÷ Í(0R“æ‘ù©°÷ Íjï¹›DGow(ð‘:[0#—Ç«%Õqm¤û Ì™±[,.C(zBŠYmK .¨$¨„ûNÀvß` i,–gî2­¹–GfE¿!DÂy–'­öYBÖ•­²R¿Y õÊóUÎÝbvâû7 ¾žmÆÝ2K¶Ù EP1™YRQ±p©AÂÒ¸­I U°µH€  ÿwú©Å?Áš³J qáN)÷ Í(86;åZ¥ü=ðür̈Ïè®zCjB½)R9'‚’­ø„qç¸ Êv>u¶WÍ?eF¤G¶î§1™n\l†¯ @èGh¸ÙTu+–áJ!*‰ìƒÜ{BHØvž)÷ 'æ²³AŽ)÷ qO¸ViALjíêÍQÈ–‰¼ø°±ë9~Ûr¸L›t‘"äµ** wš¾EaNž¥( ÜvT–¸¸Ú]B›XÝ*ïÖm©,Z…¦Ø½ˆríùœÙñäÈܨ¶†!¸úÙIÛºß~ƬGNÉß`{Õ?œgv½ÌôÿnËŽÚð«”{“*{£ªÒbÅ\Ž¢6üZZé¶´¨l}÷Øm¨òØ›™qmÖ_B]iÄÒ´(n•<Á‚¢WÌ'%Ó;UëO,bÑ`}ɉ‹¢–ºjD§Péâ’@äâ\WŸ~[ÔçŠ} ú$½=sLí+ÒĬcÉž„—¥;ÖýÓÖýשçù;mSš l=œSåÄ~jÍ(:·e¶ïû]Úßl)M©—ãHi.4ê6)RP#̵tWaq7Ã{sš€˜F u@‡¿.AÞ˜?÷¼·ù½¶­Å(#®™ ¹› ³&µ®]ÒçÍ©r-M +ˆh¬©JYä„+ï¹* Vêßv´ÝPò­·³gÜŠùaĸ}e¶®;ñZOb“Ü:í©{o¿jÒݱ[dër °ä«b5©î.ÚñŠãŽ¥äº®JFÄ…©;,ž’ wÐnÇä8§Ü*96å”Ù~º½ ›µ¾4d¿l·ÛØ)œê’‚\mJqÀÚÔ¥ÃæÜ|ëa#µ;u½9†.òa*à‹zÜH‘ÐJ…¹ÀºR·’GmÔýè6|SîØ{…f”âŸàÍN#Ü+4 ÆÉþüÔÙ>ï÷Um¨Þ!t»K.±ñìªüó—¹lúK6›e¾EÂj™ßn¡f:¤#~Ü”wí¾õùcéGù P~å\ÿSS¥§²&Ñ Ódï¿÷Sdû¿ÝTgËJÈZƒ÷&çúš|±ô§ü…¨_rn©¦í¼#z<¯=“îÿu6O»ýÕF|±ô£ü…¨?rn©§ËJÈZ…÷&çúšnÛÁ½WžÉ÷º›$y÷UòÇÒŸò ýɹþ¦Ÿ,})ÿ!jÜ›Ÿêi»oôy^['ø#óVvOžßîª3å¥?ä-Bû“sýM>XúQþBÔ¹7?ÔÓvÞ èò¼öO»ýÔâŸbGæª3å¥?ä-Bû“sýMw,¾.ôNïx‡b™x¼ØeÜ_DXX'[š}õÚ^y¤·ÉG°IP$öÓvc¬&&%sñO¸Uy¦ø5óÍõýu”˱r»ë7zp©M2ˆL2¤¨êžm(ì7ìEX(QP&«Í5°f¶|ãR.9;¯ªÙx¾³&Ä—%uR˜©„ÂÏLuPáãÛ¿mBV5)JWVœ/$‡¯&{&KFÅtÇm–Ø­ÔV™ ?%n’ÞÛRë{w;uXµ]ZmyÃZñ’^'9+ðJF=lbÞ•?»"j’_)oU\Îçnûu‹QNÿ“l¯ý;ûÔš£:›ßMò°=¶Iߨ.‚‘ÐOùÀ?èÍ·ÿ¦n§•çñ' ÖmÂ-WMVÇ£L‰ÛØ}—%¦ÜK IðELþT¾þ˜1¿écû«r·®ì1mKoO&¦å†æÑ•®ó­ØãSÝÊz ²Fyò„Oãga…$”-*H.6´üäÝ”ù n‡eŬ¤9§·¶l÷ãR#G‡o´Æ&DS8>¿ƒù­‡ž´nHu\Ö ­Ðš¾¾TÞ>˜1¿éb‡Å/‡sç«øÑÿùcûª–­'»¥ozÇEDõfK–käŒB;vëµ®®,Ò[b5ñ–îo¸ˆî£¨°Ë‰eľK…€èJ (A£¼¯]Q˜ú‘iVðwX©=¤ÀÙH;ïÿºµŸ*oÿÎ oúX¬|©|;ý/ã_ÒÇ÷TÄR:J³7·e-Anö ÇŽ‘Å}Ãl–¼æ EŸ(\—ÓKaH’êXÙHꎺƒŠG¥ %F»˜”åÖXv–uGäåw‘hÇâÚg91€›‘@%o‡Œñpu¹2VÞͪÞ>)|;žÇX1¿ülV>T¾öÛãþ–?º«¹müž\=Õû^z±½ûœÌ‚ *fGQž½²3.¬›JÏO§!…n“ä¥õ§nn1OÓ.¹WÜzñ$å/\¥Ú¯à0Ê›<†R÷¡Åݦҩ g°*%H y›wåMáãËãƒþ–?º±ò¥ðïæuþ–?º¦+HbI½æ4Ñi¥)@ H#°Ø*¬ñIÿ §üÁ?Ú¢³ò¥ðïô¿ÿKÝU׈ú!”h¾SãÚaŸr ƌ̥º²ê6JG´ÔÞÕÜžjã­¢ñÉî¨ñFøiÿª¾Õñ…ÞÿfŸú«íX­‚”¥”¥”¥”¥”®¥ÎSС?.4We¸ËKq1ÙãÍå$n’¢ Ûa¹¿r(:Ùþ&9f•y›kìÅHRÛ‡r^P$U¶ÁR =æ¸[-w8ó.R¦ÞܚĹv$u°„m†Ð’€@Ý{©*^êïëíäu¬p_™1d³w†íÎÝZe<’Ü5K>¢ Hww8-IQ‚@; Îü<¨3JRJRJRƒÈý•yð3Qÿ [hªÞòþ!¾rv[á ŸÆløöõß¾Õ W‘í¿jÒ<™Ç)ФãÑ]‰èN…ÝK©2ç4lÂQÇ‘J†ê' v;ÐnÓä>ÊÍa?4}•š)J)Ju“â–œûÇ/˜è¸\²s˜µK-!B Ûˆ·Q*îžm¥Hõ{ö=«mt›'‘?#½]äÊ´8"F‹5µO9Ò²…/vœXQZ7Ül€‚|·­VqpÀ"ê(·-ûüÉ““Ž>¢#¾˜‹Sä@²;ƒçì©ÃÀìvÛm#Dí8&=¦–›>›Þuǘr_¢KS¡Ââ•)Õ;ë7ÙÕ8Ÿ.ÛTîª-ÃíÚ9ƒZñœb÷yÍí—Óå™q‡Y€—VâÜ;¥@%„-*I +W5žÝÎÖà ù Ò” R” R” WÁq[¢AmP’€½½`’A#=·óWÞ”˜XÅÓ…c³ãwûŸoy^›ðš™&C+!)yKÜ/šy+—ª’6ïÛ'5k…p¹eq\ÇaĹ {r..²–äóZÓˆ)YÙ.-Ä¡!\U˶Þ[Éqµp[ ¸ž.6•=ˆßë Èu*!#}Ís­Üi,ȼß,2‰{ºBDDÈyÇd0Ò› é,F+ Ž*qE\x•ö Wa¶bÍÈ£O¶Ú&[šÚá)s.Íði¤HGäTWëòZ†Û€A;í¸y»AšnäsÌÞjíêûš^Q6Z‡®¶âÉ\híoüÚi J|‡i$ÚµSøl—~› àëf™д¹?ÜU±[x#\ucfŸä±J¥]¹k×Àê5¬ZeV@?ïtBÀ×£òëwõÏÛ}Ç•s›â“ ´KŸÿßmª‰l›vi.ú*žyˆ«B]äÊ^.0OQ$’ß«¹;lv¶üGTnOeÏJ¢/ž'-6¦ñl‚åÛ5–ãs™o”S—-H‡Õ`Cr;®6ñ[ŠBB‰*Ý; «½_5‚>Y…G²åЬ³Iˈ«tërd[xAqòâ_ââ¹4A°äv'nñêW©¹n릕Sµ®¨cáÈIÂrk©Æ'¦É2â˘&xçÄ­àl…¬„$¹o°?O‰,#‡ðW Èo—i ¸õípLWŒˆN¥·Ö§:Ý%ŽK@G¨¬‡‘Úwëäôì·©Tž#â>ÏrɲŒnðÌI³N~RÙL~“–›;pc¿éR²Ú‹Ž¸€¹vBˆøGñw€H·Iš› éO¢$9ñ!0¸r$L&S1›RRÓê ¬8û\áJ€X;{›õòzv^uÔv9É/+ç:àe!KWÖ¢7?Y5öÓeçªÎ5eÉš-)¿3ø:_ Ãô9ô¶îSÖêùûw­G„ñfÓOú5 û1[}6Èó;Þq©Ü•‡Qm²ß™‰dRât‚â* (¥{¨ê­Ï[¾Çq¿m«¼¶ã¢Å¥)D•\ÚggnkÎIožÜ¿Á±ëc¶å)3”ôÿí¹WÎã~Ãj±ªº´åùLÍxÉ0yQÀÇ­Øõ²á ÏG#”§ž’—GSÉ[%¶½_gå ±j3©ßòm•ÿ¡'`º“TgS¿äÛ+ÿBNþÁtFƒ[mËÑJ“[I=$îO£#ê©ßÁvßò|oö)þê†è'ü‡àôfÛÿÓ7SÊÞ¬Fì1-?T´·ùØv+i~û“IµZ­Ñ¸õ¥L-´Ó{%«`7$Ë]<_&ӜݧÞÃo¸õñ”ù·ÈfGHŸ °‚Jwí¾Ûìjâ|¤2ß,:ëqîÖy†ÚSŠ "áKÄÕ!}6‚Є—±ˆú¢¿ÒJÒy;rB¶òPÜn>ºìümÿ'ÆÿbŸî¯¢ï–ÚX±å7¸˜]Ã!É%|1>dø.Ë|¢/RC ©ån‚ñopéOr²>³Ò¹9Í5Æ%e²Œ›ÃÖ¨ÎM}QÖÂt nµ6°•!G±))I«°ò¥¢óÑ[RkÝ¿ø.ÛþOþÅ?ÝUoŠ|´ 3q¸1Ò¤ÀI4GãQåڭʪüRÿ‹þiÿ0Oö¨©É¹&/¾¦‡Ú#=ÿü4ÿÕ_jøÃ ÄdýZ꯵a¶ŠR” R” R” Rµ—ì†ËŽÃjuîæÌ’ÌT8긅<ë‰m´}ªZ’‘õšÌ›„X®´Ä‰ 4¹ é2àIq|J¸¤ê;vÀ}Õ¤³[$ÞÝ·e9M»eú R£!˜×Èi¦q;÷ℬ©-4¯Y¤îó'éÉ>|…JÌcYnA¹.UÆ¢QàP‚Jʶ{ŠœIZ8‚FÛ¼€<… €êÍ)@¥)@¥)@¥)A…|ÓßÙQ™ZyoeËÍ3k¦a¡NúŒ‡äâÀ>¢OžG–ÊVÃΤÇÈíZ—÷áøå—íâÍè®uÚRéFG$ðR¿Ž<÷¹NƃlŸš>ÊÍa?4}•š)J)JsPp <ŵG™9XÛak×ÌE‡Áq;0V}owjœ¹¶ÝÎÕÎ1ŒNñ¨ öù0®Ö9“ž³Ãê¡>œã‘Û©ØÕŵ)^¯–ÝûTåÝŠFþ[ÐAt;O\ÒÝ.´à]™¹ªÞäÕ™M6P‡:ÒÞ{²I>]N'¿r máB‘‡È·Øì–u?d}SdÍ›&æµ»Õ/¨ç%8•©Ç?|ØH@=Áoºk¥l/$–Ì«Œf­çYqKBƒ³u;{%Ä´TöLv%°äi,¡ÖACˆZB’¤‘±0EX}™,·!‡Pãn¤- BJ’Fà‚<Å}*(ã¬` ¸ü·¬Çii¶”z?¡–ÉNݶm %°ØHlG»j”!ÖÜHZ•ADPs¥)@¥)@¥)@¥)@¥)Aå_û~ ä›6ͲA°òÿÆoÕ¥TΜdÖ,Ês%Ï®ñ,W„dw;Ý´ÜK ÜíÓ$*Co°µ•íÔSkH<’¤ÆÄc|aàËœúÍÒ­¬NºK#=f2[“æö m=:€©RDÓdU‹¤•Ød½ÕçÛ¿-ÆÛïåU¥Âņ¨j‡›ÞÚU®Ý6Ùor3cºÓr:eN8¶Ú }íÚFî8O-ŽàòVöÆü¹Çÿ¬Øý*|aàËœúÍÒ«ÌRz©´+xðßs$yW•M&tÆŒ(ñc¼·¡ú)Ù–”6x®ieGr£°ÚlÞ›Gq¼-W\‚år—…<§ãËP]–³Èüž! ÅÂInFæ¶_xòçþ³cô©ñ‡€.qÿë6?J£vLÚz¡ÙØ/Ö©¶áy–Â¥åG. [ ÈlJ-ôújeÄ”8ÖÛ•B¶P;€F¶×áÂ;Ân/¨7ëUã«qS·&YŠVëSVÚÞk¦Z餴•6RQÜÇj°þ0ðåÎ?ýfÇéSãþ\ãÿÖl~•N”7®±á§Žó’Ó:èä©2_TÉO<zlWൠø¬ŽN¶´0…’¢T¾6ß ¶h¶ï‚®9}Ú|v½¨i1â±Ñj$¦¤ +¤Úz«R™m*q{ž#¶Ä’l/Œ<ùsÿY±úTøÃÀ?—8ÿõ›¥Q»DïÝ ®óÿ\?æ®ÿØ5­øÃÀ?—8ÿõ›¥Q=GÖ¼ÅHg¾CÈ2«.D³Y-R&eÂRÒRÛhCd’¢9-[!#r¢©µ«<Õ­fg¢Èð‘þ,ÚiÿF¡f+s§Ýë,͵¹Äa˜Øõ›l¶•:Òá0ùR÷;ÍÕ ÆÃ`=ÕÙм.~hî‚]\BçØìpáK(;£®†’ >ÐËo«jùéö  Ã3ÏñßÚ‡ø!zjÙ×Bù|á²ÿ5 †Äuxí¹ìšÂmGDú”¥U}kÔ;•Ç\r-/r dÁ²ãöÛ»RUÕ[’^…%]øñ„‘°ßrjÁ¨%³j! nNùÜÕ2m“zîÄh¶=’)mé[ö’ØHò…s¥+Äö¥(¥(¥|’m†\um¤„mºˆCrçë êd7¸øåšmöcž…Èq¸±—!å¥#rÚRÕ°ì 5ð·[î"íqŸ2ôì¸3:*‡ qЄÃàŸ[emÉEJõ½o-¶WRÉUíèy¤”Þ­OLµ´Ë–Y£„eu Z*GXoÀ©+PØl ó©%‘Ä{+4¥”¥”¥”¥”¥óOÙQ©í‡;)Ìf[—[$!«ÂP:Ú.6UJå¸RˆJ€â{ ÷Ù)ò?eGÞ| ÆÂÖÚ*·¼¿oœ–ßøH'ñ›#~=½_Æwïµ?4}•šÂ|‡ÙY R” R” çl›Pp ¹ëÚ";‰K%¨¥ ™…ø‹d¤ÁBÊ»åS¥ù¶«ü÷O.YV¤iÖgtfca³gÊ”Ë\ÞLˆka! · X'}»µOÜòü´ÿ‡ûk‹é5–ɨ>îAÙª”·äúBÊW1åµ»›ž_‹S{wì6¶Ú¬:¯4yëšMe^§ £$.ÍôÁ0÷L{¥Èß¹töú¶«ƒæûM>ÊÙy´¸…”• ‡¸ŠL¸JÅåκd7—¥À¸M‹Ý=µkTE9Å­”¦Â”¤©ÃÈ­@w; êSJ*1ÉŦ±„Ñ{¹HY¾“ˆ·!HS*; ¥Ã¦ÕÉi¶äI’ ´…miJP)JP)JP)JPhrŒ Ía¢eŠZ/±š<ÍÊrP•{ÀX ‹Zü‰`Çë6¿¡V5(ÏÉÏ@~„p_»ñBŸ'=úÁ~ïÅý ±©CEsòsСîü_ЧÉÏ@~„p_»ñB¬jPÑ\üœôèGû¿ô)òsСîü_Ы”4W?'=úÁ~ïÅý |œôèGû¿ô*Æ¥ ÏÉÏ@~„p_»ñB·x¶–i¦ )ɸfŸã¶)'ƒŽÛmŒÇR‡¸”$*WJk0hâ„åPl.Çr,Ã<³Ù±äÛæc·– Ü¤„ Ï*.¥ÒSÜì‡ÕßÔ÷Tî ˜ ן˜g±1+Râ]`^Zc!yMñ¦Œ­ ‘ä*i;ì>o—¶‰NéJP*mɱI:Ï~ÄbØzY¾lËŸM#¯ןKLò‘â¦Ü;°çÚ§•¶ËÓõë5þ¾Ó™·c·9rR¾+€§dé~$…¥þÀoß½ò”¥”¥”¥”¥”¥”¥”¥•Zk¶AÒÑm±Û,r\»!RÛ²Øb<–ÜQç\P)b;|“ÍÕ· Q Üå>/n?á‚祖0çq[§Oéà—º­s#ÚBþá])Š÷ûaÎùiOº^“¥y¯á¿ßËm-û·;öº|9â÷ùm¥¿vç~×Wá²øWˆÅåéJWšþñ{ü¶Òß»sk§Ã~/ˆí›ioݹߵӆËàâ1yz2Tèú"L¶Y/¸h8àOQÃä”ïæNǰ÷V†Ûf‘y›#ÊìLB»ZdMn=ÇP–QB\VÁ *[iBŠT“À’>f…œ¿w5GUÏ'ÒY†„Ê‹×Ŧ,°òA qÊâ­¥qv)JP)JP)JPa^G¶ý«Hòg¦*“Ev'¡:u.¤<˜ѳ GE*¨ž@Øï[µ|ÓßÙQ™ZyoeËÍ3k¦a¡NúŒ‡äâÀ>¢OžG–ÊVÃ΂LŸš>ÊÍa?4}•š)J)J çPq¢ù©šm’Yd%»^7:àýÝAA[oBq¦€G’ÿ¤—Xn@Û}÷öUy¨PsÉ:™¦²ñ¥Ë“® ÈCO„6YT'ÇQ;î±Ö(Û`v=êÃs~#‰ï½áÿ$̲í&³d€ÃŒß¤»92äOFP˜òݽ†ß‹J=üýµaÕ} ™åëSt¦Ï›d˜‰>{³Pë,%IBCRÞe;{¥´“ßÌš°h¬w§z Óîóúê7j‚ö í¿³Ú”1¸°ßqÉò®kyÖ IKgªTµ…8y©~¯ ½£i/zë\íðnöÙv›¤6eCšÃ‘ä0òBÛu¥¤¥HROb’ 0h;q·RÚÒ´);‚ r¨ª®,á>–õê]†Ï‰Á<ôTË…JmH^û!(õ˜K`wß>Ê”‚Hj Ò±Þè3JÇzw Í+š)J)J)J)J)J)J@ðtù¬Ã>suk»»ziY Rœ!}€9v¢ù½»û÷©å@ð³æ9íǾ‰÷Åå©W¸ýT«Ð¥l¶–öÓ»Hm[þvôÊR” ÛmøzÏ~¹À¸­y“Ö+{W(¥j)n^|ÇXNÜA+/ Á'°ì;Tò vÌ[‹¬÷üÆ6@]¿Ü,VèíDÄf¦žâ=qÍN87=ÞF‚yJRJRJRJRJRJRJRƒÌ°ö»x¥Õ[œáÕ‘c·ØlЮýÎG\—w7Ý^þ ÷U…Uí‡üeu¯ÿ‹ŽÿöÑVlì¼±C#jˆœ²ªuW9Õ\C&Æí˜½·“oÉÇ\÷d¥æ],:ê–°€RSø’ÝûÖÕÍbÆlwH˜¾U&CwH‹m&®[–Æn¥% zIl¡%EiÛ’·ӾĊÛfØ?á…׸‹ˆŠq‹âoz|ºû0ë\<ý_Ý·ß¿—×UÆ_áÖõ–çr‰Ye½èë¾[ïqU6¯Ë†ˆÊdú+/ÛiE¥«t¶»‹Üêó½è¬nNš¶×ϘÆn¬Iç¹ Æq•Ë·ITw&³n´òÐâR]ýéï°;lw¯†£ë6C—©:TÕŠóe¶¼cÈrâ$³éuÐÏ(ÊJx¸ØR®7Iãê“çQw<&H¹"ùë“Z[·^"ˆ²bÛ­+ŠÝÃü9‰%ùŒ‡K {‹ G&Þýe“¾üDÙÝpiUçHãd…gå%Vn£k·D6ðŠO/](RT”ÜPP;ñÜ×ù'ªméÇGyZ½nÄ'³Šjl¶ZÉ 5Ý$&Ën™&‚–Î9ÓWM#¦ JžÛyŠ3âM‹>D‰—ˆŽÛ½ ª»$Æe¼™kR"–XSAÇzŠBÂx}S¾Ûìåºa7!¾dwûvQ&Ñ.û‹Œu‡ã ‡a¬8òÃéPP;‚ïØú¾`ùV¶ w{MÖNQ'±Á½h‘ЭˆÈ•É >§_ê¦S©Q.%CÕ ú½ß_dǧ¢PߛޤŠ× 9EùÜuˬļԧmæI¶Èô5LmnFLŽ%:”¥GˆWšTžêTùáÊû—d,gÙ>zÓ¹}¶ Û³Ü"ÛƒHµJô ê (+$² JB”¥- X*õ†Ýx^#ÃÊ_ºúf6¨]å_C«±.høZ”ا CIuÅ)$6%<¼É'sLkcÔoS,Ê1EÎvÚöÝå[߉ÖIHPZòRT‚6PíçR2 A‚6 ŽÄVÆ0«"e FÉmoëðãÔé4”rÛs¶ûoçí­ív¯m\§I!ðvL\(ÆÙ$AÆó¼†Ù¾]™Œ%©ÖÚHö%Þ {}U á ÿ'õÿܼƒûDUõX9>ùmSí‚”¥Ub”¥”¥‘Úµ. ïÃñË/ÛÅ›Ñ\ë´¤+ÒŒŽIà¤+~!yî'rm•óOÙQ©í‡;)Ìf[—[$!«ÂP:Ú.6UJå¸RˆJ€â{ ÷Ð’§æ²³XOÍefJRJR‚ºÔ®s S4Ú6Ô¥X§Í¸£ SLsm,¦ÔÇQ[¬±Ün{U†¿!ßÛUÞ æ™&=©škŒZc4»fM:áèâ˜+ShfŽ·Å@ìÜJFçÏʬ7>mCõãGL-9×ÀÍZ¾rb=·9¥¾Œ§Y܆ûôùy~ûò×SY5’”Ŵ‹a›‘d¹…IJXá)(vcˆHS‹S‹õi´žKq]’ìI÷4O.Çs­3µe8¦<‹®c“Ä¡ å:ÚÎÈõG%¡Jíü.ý÷ª³U€_Šì.a¬ø¶Â»ð*™(vüSù‡º¯Ž±{DJ™-5¬Ì9 ^ñLàê#Fð&R®á·ré Z~¢Sb~ÊÏÆßŠ¢->û×'öJ™R´¸ˆ´÷ï\¯Ù+^)þˆ´÷ï\ŸÙ*gJpxŽ/"ö©x ÙeýÓ§;•å2H;ÇcßÞºvÝNñyMÁÛ†œéäÆdHêChdrôVx tÉOSÖ W"õ¶ò¬*SƒÄqyß¿ÿDZ}÷®Oì”øÚñQôE§¿zå~ÉS*SƒÄqyß¿ÿDZ}÷®Wì”øÛñOôE§ßzå~ÉS*SƒÄqyÏï­úëѼäŽå åÒ£öaV.k Z§ñ éÊÆ²lfb`^¬Ò_Cæ:ÖÚ\mÆÝGªë.![¡`vP!$5£Î¡Ú*6ñ)ª­BWaÇP÷¯y£·dù+†Ñ³Ó7ªôlùí’Û¶z&•åY¯ ØR” R” R” R” R” TÁmø¦cžßáßÑ=ìªòÕÆLt¥ ÂZ"2ÈhìNû¥¤«¸Ö©å@4óO'Ṧ ärî Hk1½³uŽÛi!L!Ø`¥dù’Y'·°Š ý)J@í˜Výh¿êRo-ºåâÅn´ªFËdGzBäïÜ+¬FÛ~óΧ•^Zôþëo×<‹S]VëÆ?m´3%]T;é Z”6Û‰'mŽý‡JRJRJRJRJRJRJRƒÆ7üÛ0Åé,Ë·c.˜&;akɲAØn=†·¸öYŠeÑÝ—Šd¶›Ó /¤ë¶éÉBüÉýF¦#^äòüUÇÇ«ÿ6L»úÞ×ûE>85_ù²eßÖö¿Ú*ÜØ{©°÷TÅ'Ê7£Âà–á:ëƒçWšE¢\D¿:üÜBÜŒ²â7mJl”’=é$}u誡|!äþ£ýZ•hо«ÿt¶iöÁJV9 öß½Ufi\y~ôåQ¬\©Xä)ÊšÁ¤‡Èý•yð3Qÿ [hªÞòþ!¾rv[á ŸÆløöõß¾Õ¿'±í¿jÒ<™ÿ…1œv+±= л©u!æ\æ˜J8ò)PÝDòÇzk’Þ'È}•šâª;Vy k’Í+©ÊšÁ¤³JãÈ{ö¬î Ø‘Ï5v)¨Úy†Æ·Ç~>g6|I-D-„dž·ÒPc¹@a©ë‡`;oÞ ÙÎsoÆõÅ%X‘2FW2lhòÊ’ %1o)@Iä”v#Χ.¹ÐB4NëÞôÊÕsÓ+ZíØÛ®Lã-¾™B“)ÔºxòVÛº—çß}ûyUYª¿ãa…Ð[Ïÿ[ ­=cO£i•©-}nã)rg¡­jqD¨Êt½ÝÏ[÷n§ŸäíµUš©ßņ·}°[ÎÿÓaWmŸû!Ç?õÊkTÏŠüVËvѬ›#”Ì‘s²ÚŸv†&<™Q)ÜŽš€'°ó®jê]mVÛåºE¢ñ‰°¥ ¶üwÐÛ‰>a@ö"¶o]è˜dVt˜•ª9mÇDX‘‰i+QØnÑb‘’Ê„¨.nì •:ë²C +¦±È¸¥’“Å;'¿NvºçRoŠnæ˜Vünè–`Äyˆ*’ËŽ¿nKÁ³1§Š˜’_Û¬¥% ²H5xdºuæS˜¹åx…¦í*3e–™©-“¹GqÝ;÷Øöß½uÑ¥:j‹ËY 0[ ¹2ÊXnW¡£¨”%°ÒFû{ù„¼«”ã¶¿Lèé[×Oª5•¡×¥ën‹ ÛVÃxÖ?h1­Î¾ówmÉRn.ºÛƒ“on²ØüÅs%[¥?=-MóÒÝcNƒ ó¨öø†õs¼ Ëv‡%)À…<=u-²;?Iß`+Ñpq,bØãNÛ¬0c-‹{v¦ÔÓ)IL4|Æ;~ñ>Äù çÇ`±iË –¬)¶!¶BS ="ÖÍóME½„Š˜Ç={“xíÑæ›ÏŠ<ï»ë°¡_,s­÷é:6§"2³µ¸Ùe÷.>’²ÕÑJw;¤í¶ÿ]gÔmOÓÙX„«…ÎË™Ý!_›žÍ»C‘_q·-3ÅöK‹Þ8[AIPQRƒjIN÷Í»IôÎÓrrñmÀìq§:·\SíÂ@Y.…;íØ(-[Øò;ù×ÒÅ¥úuŒF=…Yíý ^šÑ(¦¦ƒ€Ø†Ö´rT@ìj&—žëFJÇe~ñ©ìL˜Þ%Ù~‰ŽXm·™ScAlDºI[„]–ߢ³ÅP²—=mù|Ý¥¤Vsš¿‘ß296¦ío­VØQ¢­/¶#¾¤ut¸R²@Û`;o¿rîFét·mÏIÓû ‹´’aAoñ¸\Ù#nÃšŠ€ò’*IoµÛ­-ºÍ² 1PûîIq- $-×Tµ¼Ê”I'ÚMZ+hfU›VcH‡hyÔ;EÆgTÿèî9ÿnuLGC´[aâgT÷?ú;ŽÚ\6ÏêvÙ?±èšV+5”Ó)JP)JP)JP)JP)JP*ºÓ\/$ÆsH½Þä´ä<–üÍÂÖ”>¥–ØL!I#dm¬ì7ìAöÕ‹UÖ›ZóˆÆ£ÊÊœ”«\ûó/X¯óBb,%a´îx²]ìvï¹öÐX´¥(]Zqlº.¼d™Œ·ÉÆîí² &ý$™l½%O–þ®éu¿[nû}UbÕuiŸ'^2ISÕ7ð=xí±à§AéÁé>‘ÅîÀ³¹Û¸ÛÏnÁbÒ” R” R” R” R” R” R” ó-‹·‰mkIìJñÕïݶÿœÌj¨ö²éž}jÏ[ÖÝ#´³z¸=»^GŽ9%•vŠÒ”¦aÕ„Éh­Àá ZVRT¨p׸±¿vÒ=T-޼>[Å öŽl¥m«íJˆ5§³g§§™Òa›´a¼äވ䴫Î24ÛXæaùî‘·‹Ù¢Ú³KýòGá—¢Ä òÜp©1CD©ÐÛ›•ËbNÝŒ÷åcú5Õ/¸÷ÕÓådòø¶Õ¸÷Õ×yÉŽÝmq%zU^;£Z¡#Rí×yvè¢ÕkÍ è)©‘™Š»ymÆ“³g¬äŽ+Ôë„*Û}À¨…Ï@5ºv›~¦ÇmŒ¡…«ipnQYI’žºK’]S ul¨-µ¡¶ÊH%a[o¸¼‡ˆ+ |Zê—Ü{êë(Ñ®¨öÿô=ÇõuOá÷'L½7Uö]áÛ3¾aÙ‹ö)pmÙ”‰òdX¥)¦Ý"Û,ˆ¯>c½%çÅIm~iÁ:!¨qíW,,Xm/"ýv²ÝÎJ¹€=lLFᇠ”sZ’b/¦R®?ŽÜñÙAV/Ê Éômª?qî?«§Ê Éômª_qî?«¨þ›ÉÓ7µQݼ;êEÿ·`H²Ù­XïöÂtM ~zæ´RÓ½0€´…”ç#¸Q;n;›+D´ß$ÅòKÎS‘ÚåÛ—6Ý ÚÛR.ìÍRÃ*q\€a–›JÔâ‚AQïÄ‘±ù@ØöÛâ×T~ãÜWYù@Ù>µKî=Çõuh¶λȚå˜Òj´kÝ…Uÿ(+'Ñ®¨ýǸþ®¸;ªz™¶«&èÖ_"ï tÑpÈínZ-°7íÖyoñ[>}6¥+m»o½ZvŒqÏU# ç–‰—„=ÿu Á±Cº•‘¨Ç‹éIüÊJ‡Ú _UÑ}1¤z{lÂÛžåÆKéwƒ‰ \é︧¤È#ÙÍ×­½€ßmêuXÖëLÃZ±¤DRøƒỖ³Z1ì ˜Îæ9ÄYìÆJ Ù‰øµ8ü×R;­ ´…/†ã’¸'qËz¶«Ï:ëßÄN$÷Ý’(`=(£·bGå5lU‹Þ"Q’wi3º|:ãwDúf}˜æÙeÝßZLéy,ØèZÏŸN4wPË)÷%Øió¬ü˜ô“Ù "ûÓsý¢­Z¦vYzŸsn:š·¹)km:J9pRJPRé'æÓLqÏDk“Ï5‡òdÒOâYÞ›Ÿíù2i'ñ,‹ïMÏöŠ®n*ïÖÏÜ¥[­÷›#°ïr#z¶kB ¡KBS1ð”B”Ò8 ÁPóç«Úµ©z`þ)rÊŒ ÃÑ®Æbíø«¯¡Éq×j¸«¢ür¥¨²‡Y Ò ÙP@R6¨×ƒù;ÊÃù2i'ñ,‹ïMÏöŠ|™4“ø–E÷¦çûEA²OÙ½¦å2 žÉõø=f·Ý.+¶Ú¦Ëbæ¹)S8ï·º# 6ŸUÇyrQîÕ“¤Ù¶gžIÈ®—¨Öˆ–k}Þu¢Òé’¥F}M©ÇT£Ä@ìA;÷ØZ#¹D"g-yÌµŽøvÇí)3´ï4ÍqÛ^¼Yñ²9²›JÇ—V,‡VËèßÍ+O FûÕ±áïS¯ºb»YsxñcæuÅVkâb‚–d,!.3-¤žémæ–—I‰ãØv+¦v«}‹w&ÓC©w¨¥ÊunI.)iÿåÛΠž pœÍŒ¿Ö|ļ†n5uªëceÔ5"uºQej\u,„õš[ PA -*ZwßNôK·é¶™ÚpËUù»ÔXKZ'!) t»)×T6I#Õ.ùþ÷òTä¤4ïS[M'XVÕ‹F’òÈ×T³š3«HX줜:JŠO´nÁûA#Üiñì×Ðî­}Ì—ýÕêm¿%6>úõq¹nË?Í}ê×ÜÉÝOf¾‡ukîd¿î¯Sl}ôØûéÆäø8J|¼³ñì×Ðî­}Ì—ýÔøökèwV¾æKþêõ6ÇßM¾œnOƒ„§ËË?Í}ê×ÜÉÝOf¾‡ukîd¿î¯Sl}ôØûéÆäø8J|¼³ñì×Ðî­}Ì—ýÔøökèwV¾æKþêõ6ÇßM¾œnOƒ„§ËË ×em´mÕ§?1¿Á æ¯`ä¾)i TóÃÆ™[çåš«¨–´Y¯Ù£ñPÅœ<—•l¶Ål¦;.¸ƒÅO(­×Ä”¤¬$ÇsuïçßꬰòÚ¹eÚo–7e× ãç Ò”®.Å)J)J)J)J)JW:k;;•œêCcrÓj‹}e¼x¾ÈB 3 ‚¾š€õ“Ö.÷;÷ÜUUÖ›eùNGœj=šÿ7¿3С·Î: °ê#óÿãƒqö{(,ZR” ®­7ŒÕíxÉ,S[1HØí²Lª8 ™«zH|%ͽcÅ,îûn=õbÕujÍòÚï‘éìˆÍ %«¶\ã:PZ¤>ô„8’½ö %¤v¶ÿ]‹JRJRJRJRJRJRJR\R„§Èm\©A©µf”Ú›ViA©µf”Ú›ViA‚ãÓNû÷ßí®t R” WžuÓüb´‡ý’g½ ^[ñQcÈr qÒ+~1˜ÉÆ&˜9Äøñ’°Ü]ÑÁÐSß·}·í]0Ι!Ï/Ù+2µù?fʬ³1ÜŠÜÔûlöË2c:7C¨>i?UVŸzÓü毟wm¿«§ÅV´ÿ9»çÝÛoêëgYðÈÒ±ù&f–iös57³‹r"˜KSª^ÎÇ**è¸Är$„¬ $ ÍuÑ]-bìÍí¬2f0é«uñ CA”9×àlæÛz|UkOó›¾}ݶþ®ŸZÓüæïŸwm¿«¨çíNºþIå¯Ãì²*ÓÅŠê-Y‚ÚBk~›'¿t§s¶û‘¹Ø÷5Î&ŠAfÀÄKfÛÅ’QgHhI,–HG~ß‹R‘öPŠ­iþswÏ»¶ßÕÓâ«ZœÝóîí·õu<ý¨Ö=É Ò{mÀ]"`öàú=%-ó ZD„­/¶„(”¡µ‡É '}öÜ ìX4Mqw£É±â0ã¿H–ÃÄ­Çt2¶RB–IâuÄ%;ñHZ¶zŠüUkOó›¾}ݶþ®ŸZÓüæïŸwm¿«¨ÿ”ïG¹¿{Bt…ôÁmÌÛÓ·4#Çi!Il2— ‰iH‹¥jQJ S¿`*Yf°Yñæ$F²[Ú†Ô©.Ìy-‚ÞuEn,ýjQ$ÕiñU­?Înù÷vÛúº|UkOó›¾}ݶþ®¦&cñDé?’Û¨Ž…ŒV¯£1¿ìåÔKâ¯ZœÕóîí·õußð±dÈqýqÕÛ~O™IÉæˆ8êÌùŒ²‚Ü­‘Á Û¿}·ï^]²gÓèõlšEùJèÎ-XÝBÀ'ä·UÆ¿À™9xìt¹ÄIyQ—ÒGË‹%jó^ß*œ¹ä>Úƒg8–;}Ô "ºä …qÇfN~Ù­ÏqØ‹iÄ€¯XðBŠý_w~Õ:pr¼ë1¢‚h^žOÒ­.´`w;ƒ¤Ûœ˜µ¾ÂJP¾´·žß°t´žÕyáÿ É4÷I¬ØŽ]%§îМ§ÜmòòHvcΣe õHòíåì«JRJRJRJRJRJRJRJRJRJRJRJRUöjË2Íu›3 áׯmq\h«“Í®•/s·.OÛnÀUƒPL<–æî>Í•—‰Þš¶ºúT ˜¥ÃeðáØ ˆ„÷'æþJ Ý)J@-š‡*á­™—ªØÊ³Xm×tK =GU!éRÛ`Ðwýñ©ýA-¹µ¦n³_´ý¬y \mv+}Íë˜ãÊCO¼úÉíËdT{’=!A;ªcZ¼Oc:%™Ø0»î1|¸ÈÈm«Äwà¶…4ÛVÈk”òT y) ì†Û¨nE\õ®¸cÖK´†¥Ý-e¼Âi—އÚO’ H OeØŽÇz?_¼naðqM?ŸŒ`™^Q”ê]¡»&+jЇ'¦“Ë®ù*àÓc¸äIƒ·`HšèˆÌw]b_Úk¿b·ìNXƒ}±_¢ˆò ºSÉ$ìHRÈP?ÿ›ÑÎG‡X߈kö't‘¦²ðÀצY­‹”›†å-ära„•!¥$¥ ¥;nT>£× SÕ_žõ2í¦ºA“YqÖ/6ö¡ËB]bé”Y´Ên)B]@â G¬¢´’6(P{’IŽÜá¿p¶ßíÒâÆå×}‰M¸Û\Fç’’HNÀ÷®)Êq…±"R2;Z™†”.C‚ce,¥ctöH#¸'ÌWæ‹ò¨Îú3M2òAvJAÙJH>Ѿêõºš ÈÁ¼*hEÿÃWÕu™i¹jbÞ¶J¸cƒ“â¶´ºó -Ç@ $ìP~¢D¸A¸Åne¶tyL:96ë.%Ä,{Áƒù+ÎWzkaÒlÿU®ö ätiÞVîq´©-zcÓãIüPçÄ¥AÝÁ$®Ýª=ÿµë6%›ÌµÜ¥¿‹Ý2geY˜8ãöh,'€ª ¼ãžŒµ¤î7Üî¢óm̧ÿÂ"t¼c²]Óœ³!¶juÍãJŠ·¡Ã‚ÒÕ·Íñë$÷ íåAëkO‹Œÿ7H-ö;UÒkÚÉ ùö Øoü †Y:¹·`øž;úÀй"ä)“޴ŽÀ~|`KÑ›’…:Øÿ9îŸ1æ+ó;Âæêý»YµÉÐÓ¢x•ÿÀåHmHmÉ3ç<û6â†Ä†ÔS¸>\{«C¤˜Å…ùzaÒ}:ËlÚé`ÊâÉÔ‹Œ»dÆݽ./á6C€6ê<çÕõFÄíAú=«:Ý…i.Ÿds¸3qg„ôç Á’Ò伆ˆ% *O}ü½µ#ÅsKWe³]¢Ki—oVÈ÷Vaºê„²ëa`©ï؉·¿7¬šhwÂK™€Êw:&ÊíÖÉEyR}RPC?¼YRŽéO~õÓÓÌZ°¼V°{Ý®ãvÕŒ×N`]ñLª3…®Þ#!.ÙÙQ0ã JPž%Dï·dÐ~ÂÈñë“R·_­Ò›ˆH¶e!id>dWò×Ëð·,È2kOJ"P·×é­ñi*¤¬òÙ "|ëò×BñóÃO—*8‰¤·;vClµ`Ó­‘ž”¦Ôפ>®´ä»ßt6J=Ç•nï0L{Àþ“ßE–ñ_nRí|‚èÞ6åí‡m—øü/¬-Q“ÖPì•pÙ# ÷ž²øÄôw²æ’¢Ê¿Ûï·ø8ôeZÛÀ?)eYW †ÇbNäÞ»:®]w°_r ŽçlfÃÎÇnà”%n?A.-¡À=äm]pd9åû%/¬Vi[z1”äéÚŸ–êæcˆãZŒœv;nµ¿¯‚#ËCIõJÀY‹’¤û{ƒQÆ|YÆ´àVL·*ÅÐÚŸ¶‰·5¦ç:󌺘¼´¹$…2µ$Ä nTv«!Ñx·¼¶ë˜[óì²Á*ù,;ƒV©,6‡›`,7Ýl­h;8®éP÷ùÖ†õágO®QeZm—+Í’Ó>ËÃ* ÙȬ†€qÖÖâ?tW. ¾ß¾üf—ìí½Nîœ_Ø e¯ø‹(fÝ•?d·ºåÒ,8á†ØCœßyå„ û{T\ÇhÔO÷[ŽudÉaCli¬Ì~4»±XB¤[~cñÓ$¸‚R¦ã!+ ”„¬9¾É5?¹xxǦdd°ò›ô fæíՀߣ<ÜwžŽ–à‡™XõÒ„Ô J“º A ÷ðíÁpÆ$CŠ'\#K³ªÇ!›ƒ‰yÅSï¼R¡Är%RŸ4ì>ºˆ¦Iå&¸ãšˆø€¿\ Í„Î%7*»À•y—16õ°Âc[cܤFcbµãªK HùÜIÞÿÓ/¶È—m1ÆçL²*ûcµÈ¿,³ÑB¥¹k@eJæGFJVe¬¾ÄÅ—Ã!ŠYâYð¼«(°¦¢åpq»K( ^}(Úä¢åõ¿Ð'—V;ncòJT¤sHÜ'¾Ä€n:®ð­²`—ˆÓmñvëh›e¡ÇZa‡ŽêƒaÇ@îZÂÙ vÚÄ®•­¿(RÛ¿‰Q ÿ­^ÿFcÙË©p]D<=)7MpÖL‚ç §l¶N¨î•JÇ@>ôúK`a¯6ÙËѲGÖ²³­?ü(Ô-?Ì>j/à|ÉÒ½HÝRúñÇÇž|¼aS§Féûê¿Ïð;ÖM©:q—[æ0Ô,JmÂLæ–¥º—á­”6$)@Èì*Á_íí¬¦’¾Ð ^sfÒk5·R’æDÓ³ŒµI|<áJ¦<¦·X'ÅöïØl=•aÕyáþvyrÒk,ÝLD´dŽ;;Ó¶.„‰¹ âƒ{vî6>ÝêàR” R” R” R” R” R” WÊKÞ×¾š¾)“°ßaõ×Ö°Fà}‘áÿÂ1§÷ ,+Ô-8Ìnn9zÉ’ØaŽI‹l’äwÁ<öåɲG³b7ØÔ“4ñá¤x<í"…>5ÒHÖç[e-ÁbIh4¹;¨l ž õCT4wYÛÎ,÷+ÅãaøoZ3ˆqN[§»=N3¸’Ym°¤ïºW°4¦x^¼ãyΨê˜ÚàNiÝ90ÛºÜ_é¦"Ñp6…o¾éHõ·jžÆÉqÙ¶×/0ïö×íìïÔ”Ô¦ÔÊ6óÝ`ñ}µùnÃ5o(ðg}Õ‹þ'»7RuIVsh·°ê'¿¡ÀӑЄìâ’ÙW7â£ìïXÈñë5þÝâÿá ¾Û4†f˜*Š›T¨‘®’ÙJâÇu!eH`8•§ÏÏÌꈌ+ͰL!ÒåÖ^{{UŠ+–÷Zu¸†º›¿ënSîÜýUa«#Ç‘Lõ_­Â45IxÊGM•0µo²OÔkó_Y¼8\츆EpjeÎøˆ®¼äiÏ[K’ä¹ ¨HØí¶Ç½F2» øxpÒŒYZ~þ?lÀóyVýR‰&Ï2åDä ›¹Je¥%Ù‘ÜW' ‚ˆÝiì(?T~±ú#† z,¢Ãþ’Ž›¤ù«}”OÕ]Ws<>¹e³fʰeÝ\“ÍkøU-®K,¿Y J¶p’R;m²vÔ¶Iå¸Ü}TäÈò ýUùÅ.—ê~Q§¾µç)ÔŒR~’Þ/×gæßÍÙËEɦÕèÒY‹N8xŽ˜>kßaÛo¦žkôü¦ëàǵjÄ›¥ÎëaÊâ³xS®ÉRbÔ…¥sJöwÜAú2VÈîvØûví\€A€6#Ý_xæ¬_ØÐöu^Ûâ_1¬E{^,ö@·š“Ë€G£.ܼ…¶µ«™ßæ„‚6¿^˜RÖËkqÁ°©e+ò['Ý:¯JVŸlh®µÌîÚ™¦×¼}÷Qf²N¸=|BetÒ¶œ†ãlòF㫳ªOmŽÇ½XN|Ú¯59éÔÍ5^0fü™× ʀh³èNt: ÈëpÛomXno°ÛßT]_è_”çšMeʳF7‰ŽÍL„åƒsm¿P÷‹B>ß?mXuÐ}C¹j®•Ùó»´Ðå\]š…±¨¶Ì·™N܉=ÒÐ'¿™5? R” R” R” R” R” R” R” ÆÀùÖBAW?%r®•íÙlYç?od=)¨î-†Ï’Ü %#ò(;!ÆOpGª…l¤÷ÛÏÝ_’þ¯M•§ºSœ+Äžfþ«åº‘_*üài–=5iS&ÿ‹¶€–@Û`C…'q°ö(ö¨ÜñE­víJÎ.W½-ɲ¸ž=ð«‹¶ ¦)#œbT'’JÜ$ÆÃÖ;‡èˆS>‘ù+?ŠòâŸÍ_šú9Ÿ?aÔ7=.ñ•j÷Rùü`Ynw³qe¦Œpãò ?ù¢˜s’R;n·~äè,>"2<2i}ŠF°ÜUž½­lÛ®qÕypÜ×\žJÙy<º ið}#lÄq«-™xí›¶Áµ¹Ô*„ÄT!…u +ݰO"Ný»ïÞ¢WMJÍàg(Åbè¦Q:Ö© 2oìÉ„"% Ûw ðwdîwõ7ìvÞ™¦¤çÎHÍŽÇ¢™FM ÆÛZ®$Âm„)J ¤¥ç½Ò'díß¶ô OM°, ‰1pœ.Çafbú’[·@n:^W½aröù×NÍ£ú]ŽKj~?§8ŶKU1§¢Ú™imÈP)S‰RRQÇ}·ÐÔMCÍ0ÉÑ"ãz;‘æMHeN8ý®D6ÐÂØ!A÷PI#¿`E}3¬ÿ1Å-ö¹– #Ȳ·§¤ªDktˆ® ⓳…çP Ü‘ê“Ý&‚áÿÂV¡˜ôksñm™5ê Ò}Ê%ö]¥¦æGô§‹…´+u)!<ˆ(}ƒz¼ÀØP\ƒ?Ì,Øu¯$·iCx¸ÎáéH²"&T.I$õã©lñ#cÅG¹íX?ËÚÓæó´“"zðµœm/ÄR œw+.†vÖùþ__j å* kϳøì¶^’d0.±´³>üC2Pv(R]-y¹,|ÓõV1ÿ1ȱ۵æó¤™;2Üc['?oÏÙ€mM:¤'sêúÊþª å*§Úƒ™fþéEˆz#i[ Ȇç¥Ët£ êö#a¿-‡q_,Qó\¶õ"ÙhÆO‰Fe’â'\¤Âq§T†^Z· “ܰ4*ºÆõ+8½æNãW-Éì¶Ô8ú{—&¢­-ïÁA-¼§@^ÃoSq¸ßj'R³…gÇ:)“¦ÓéF?á“ Ðú`ÖáÖêñß¶Ü7ïåAbÒ«›þ¥g6ŒÕ¼^Ý¢y=ÞØ·YB¯±¥BLT¥{rQBÞK»#s¿«¿nÛ×<ëR3lRøÅªÃ¢Ù>UÖâîÙ0›i¥AAKÏ!{€ì’;ÐXt¨ êc‡;¼sH2LÁ2Û[.Øü6ÄRÙ ëºÉÜíÇ#õW<Ë>Ìq«¢ëeÒL‡$•qHT›| ô ÐC¥×R‚A%>¢•Ý>îôºTñ¨³·åp´"¹Ýfñö$DL¸œÜ­kt4xì7â³æ6¬·Ÿæ Óã˜/HòÝÂøþ ™=4ާùõz>^·Ïòúè't¨‹?ËîØUÏ(Ÿ¤™ªç©Ð±I~"¥LâSÓR-Dì9,wzΟf9E¦íp¾é&C‹È·§”xw rqâ¥qh²êÒ;€Ÿ\§º‡×A;¥@4óPó<ÊTÖ2MÉ0äEi.4íÒD7S%DPŽƒ« €7ï°ï_RsŒ§"zÏ~ÑLŸ„ÓkZnW PœeÅ%@„²ò—º$n¶è,ZUugÔ¬Þå›/¢™E²Ø—žh_Ÿ“ Q”Åa(x»²öz›÷I:“œ3žŒE­Ê´™MÇü"L˜"B€%â‚ðwŠw ú›ö; •\åZ•œØ2æ±ÛNŠd÷û{…«Ì)0‘Èåº]y.z›öO}»o\õQ³l>ëÝŽhÎK—°û!ÕͶI†Ól«‘u ß``FƂå@óí@Ì1­ÎcÚE‘eªš…)ô[$ClÃ#m’¾»¨}ÏÍßæšeYþ_`Åí7ÛF’äWùדi†üDHº9¢u-ªx©]ü»w žR w û1‡€DËãi&C.í!IK˜ãr" ¬¥Tµ: ežÊe"gù„?‘˜=¤y{»+RQ­ø†k ,$¬:Y‚UÝ~B‚yJãþc|Äî¹×HòÂ?F³Ë‘r&ñ@Pé©·TØÜ’‘ÉC¸÷Vp<ÿ.ËcÜŸÈ4“#Ä—)S \Ÿˆâ¦HGAÕ€FÀzÄ|áA;¥WÚy¨¹®eq“ #Ѽ—e†:¨•t“ ÆÞW :i <µrØ•wvó¯–#©YÆG•;`¼è¦OŽ@l:Sw&ã¯Ù %§”ç¯æ7OÛµJ®`ê^s/k–Þ¤[2 Éñ(̲\D딘N4ê‚€àËËVä{€6‚Ã¥WXÞ¥g¼ÉÜj墙=–Ú‡BoräÂTU¥½ø(%·”è Ømên7íDêVp¬øâ'E2tÚ}(Çü"2azLzÜ:Ý^;öÛ†ýü¨,ZUsÔ¬æÑš·‹Û´O'»Ûë(Uö4¨IŠ”¯nJ([ÉwdnwõwíÛzçjFmŠ_µXt['Ê¢ºÒ]ÂÛ&m4¢H()yä/p=’Gz ¸<Ùu¥¶SÈm¸;ö‚j æ8s°Ç4ƒ$Ì-µ¸òíÃlE ¾»¨ÜÎÜwò?Us̳ìDZZ.¶]$ÈrIW…I·À‘@ÝD:]u($Sê)]ÓîïAçø~îò³eÏ4× Y ‚×’5”®cѹM’Ê”¦.àÆ<„0 ú¼ëÔ¶œ[°»=ë%Žßo]ÒJ¦M\hèmRd+ç:áHÖv¨÷¨ÍãP3f oÊáiEsºÌ(ãìHˆ™q9¹ZÖèhñØoÅgÌmYo?̧Ç0^‘ä)»…ñü2"ziN;óêô|½oŸåõÐl±Ý/ÓÌFï6ý‹`ö EÊ㿥˃ni‡ŸÜîy­ ÷ïö×[âwK~•|:o‹ü#:KsdÊø)ž«Ï¶®HqKã¹ZUÜ+ÌþuÕ±gù}Û ¹åô“#µ\àõ:)/ÄT©œR zjC¥¡È‡%ŽàïYÂsìÇ(´Ý®Ý$Èqyôòá"#ŽN¥f÷,Ùx´íÊ-–ļóBüü˜Jˆ¤ + CÅÝ—°ÛÔß¸Ü IÔœáœôb-h¦Pý¤Ên?ádÁÂ/ƒ¼S¹Ôß±ØPXµÀ0&±Ç>ȼ·5Yu髚ØB60Ê!²ÇMGs¹! ¯gή®U©Y̓.k´è¦O·¸Y ¼Â“ ‘ÌŽ[¥×’穹ßd÷Û¶õôÓ=»áù¦¡ä—±ß˜^Ù¹ÄC\¹2Ú!0ÁJ÷É¥ݶ4)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)J)JÿÙsahara-8.0.0/doc/source/images/sahara-architecture.svg0000666000175100017510000115000413245514472022766 0ustar zuulzuul00000000000000 image/svg+xml sahara-8.0.0/doc/source/images/openstack-interop.png0000666000175100017510000011060313245514472022501 0ustar zuulzuul00000000000000‰PNG  IHDRÀÐ_w´€IDATxÚìÝtÕõçÿOrC4j”Q£Ótd8¢RER~±ƒJ´è gÒ9ù Lå(¶ÖͶ8£ô`Å5+¨¨”EVlQ±â•-Ez°‡ŽÌ”N™J§´K»Ì 3Ŷ߽¯o>Ÿä“/ß{s“Üäþ{>ÎyHrsÿåÞÏý¾¾ŸÆÄ«JV}²¦&«Ù«ÖdµÅÔlûó©ö÷êy  ëmìÚX€©µQ:к/Y«“µ1Y[“µ/YG’d±ØëÝ™¬.{[ºÍÉZdìù³ Tí´±¼6¶#Y3“ÕÀŸ»‰öÀG!s=8:4Ѓ«D¢"8kÂÙa]rñåÁe—^̸þ³Á¬™·žPú¾~îêþÓ¯?áh¢üLØ>þÌ7}O8n6œp¤³è Ld®™z6?}ÝgÂùZÅ4Ün$†óÝuÇW‚O6NŠÛ.DCYD (Á66Ø~D;ôRï°†Kk´9qÞðL^n í»€jcËÊÊÞ¡Í¿“w|ážž­ë’Aø·¦{oå^²'í+Ùe¼W4÷ŒFó¯ÔCäݳsØæ„ünc“ïÕ¿¦ÍÿU¤ý l§Ÿ¬3Ýs´@Ð|§î lìØ“Â3u6œƒ¡ü.õGVÞÎA@Ke'kQBoh´z„ðr °Í± ‰DE¸7­>ô9ø)¼ |Ö„³ý½.9Hò¤M”'þ/mla¾ísm='+**Þ6¬Ê@Á©7Þª£ OZu˜ƒÂ^ÕT½•••ÇíßµËþä M$oÐÆ×Êü44ödÆaÓ½u(êõý;0›{[;ûKQmúö;ÁS®ö÷¦7á66QžøÚØâ\ŒP«ò»aÑ/'ÿ­å%@~ÒIî LÛê<ýøZjŠ´Yòd0®údzƒÚX*˵åõwƒK.¾Üõ2ÝÛ€ µ­Ã²GWqðRbiZqÖô®½!YÕ¼-€,¶±ÝsBicK°V=õBpjÍink«ÿcX‰€œšéæ¢]uåuÁ[›wqÀRÂ+E»ƒ´dm5Ì[²ÓÆ&*ŽÐÆ2ÚÆ¶©¬¬üWá­ÀÈ›]VVö‘>gͼ5ì ä@…²jkÏt!xO²êx›ƒocËËËÑÆR*-t¦“ ¦{^ðB0#k¡ ¿þÊÁ åJ=TçŸw Áû Ãõ€Aµ±.üÒÆRþ”ÍÿCpyâw:IÂ[€á×fÃM¸]%T´¶mÙݳ‚i²$k"o€6–ÊNÖÖWz}Ø“$ÓxË0|¦ºžßŽ»ïç`„J;\Ï ÁÍÂX@m¬ëù¥¥Ò• Á£Fúa¤ âÑîGÜö¹6@¨~kÇÛ{üáÐÚ+¸‚·º­¬¬ü5m,•iOðµ×LÛ×1cÆþ°æYU“HŒÚkìJ¤,ÆBeZo¾¶Ã_k5o% ¾S5æ§´±Ô@GÚ|²qRؾŽ{Ò÷9É@–”••½¨Xõæé—j õüs¯ª‡â¸ Ám¼£€¾ÆŒó*m,5Ø…ÝIÆÑ£ÇüwÞM ÝŒîÖÑá67pPƒ©ïït½ÀFÏ¢Xm,•¥Z³jƒ¿(Vo)¯¦¢bÔ/ ²PY(7_-YykÝm옪1ÿLKekQ¬êêqš®ÄPh£¼<ñ„>P5Lj9iT6†ê«>™¡Ð€U=¶z%m,•Ú¹íÇAÝø a7nÜWyw0p“ôAšHT¯|ë»`PY©‡Xæï\ÅÛ ´±´±TvêéÇ×ö5u4ùo-o1 ¼<ñœ>Ho™ÕÊ•Õr«–&kï4”ªêêêhc©l×S®¶«BW?È» €ÌÕ%ʿӇ(‹²PÙ®G–<é0sÕPºmlw/m,•ÕZõÔ aûZU5öß £lÈØb}€~úºÏp@Ae½4×ñœ³ëÝ\à9¼ÝPjÆŒ9éë´±Ôp²I$Fý)ï6úW‘HTÖ‡§öoå`‚Žú‹¯.q½À[yË¡ÔÚØQ£Fý†6–îQ6'WŸòO¼ÝèßT}pž{ÎyHPÃV»v¼$CÀ1‚Y¬´±•¥zoçÏ‚±cOr'Ùw€~ܧÍÛ>×–·î‹:îøÂ=iç@éçÚW3×[‹è@„²ô‹µ†A£„”—W|-ßÛØl×¶-»ÃöxÆõŸ .»ôŠà¦nGhëžl\ÿ“=¶ùþõíx{O0¯ýΰi™~c°åõwKªM¾êÊë\ûÊ–s¤3jTå}hê€"_?ØÏšpvøÁ÷³e® ·©­=3xiÝ9Ý“Q«»¦ ê¥^_¾ë^w€¶šwJEÍ)§¾—ïml6KÓõFž<î”°ýÖ¿îëgW¾8äÛ˜5óÖðú\ÈõO°é³@·©6Ymε×L/‰ç]'ŠõøO?ýŒ¼ëH­º¬¬ì#H Q-´ì‡ß\¯¬ºfÕ†ð>€S×k/¿åð~Þz(•6¶¼¼üX¾·±Ù<¨«6Ymb´½V0ÖχÚ¬¹ÔÚcÜNÕ¶¨·Ù¿\Ü÷еôhzWƒ)LÖæ¤ ›òúƒ=.çSø%g^Éà£6×ðömlq•Úe=^õFÆý\âõó¯ýùҬ߶®W=Ã¥€U§ÖœæN2²Î)´èÃRs‡ )»ð[7~BÊ𫆶¹óÂ…gô»çŸwA8?Ìõh®°·†-Çý¾æÍ½­½g>›æï¹û¡ëÔ×ú¾~®ž· ….£.÷3Í?»ëޝ44övݦ?tO¥ßÑõè±é`Ù=¾öÏß~BÏ‘æºé±¹û£Ç¦à]sÝÎ:ëìßÙ´Þ~ -®ê|ø™´'Õî)ü¾ò­ïomÞ¶{šì_æéÇ׆߆d]·¾¯ßÓÐf×ΪÝÔÿþ¬ÿ»Ÿ;ìZÿ×ZÅþüë³À¶¯¼õˆ§Å2N8kžÏØ…_}л~õs]NAURúW__rñå=AQ ³˜È<27¼ÎØ^ e…[ ÝS€ÖÁ–».…Zý\sÞÜ$º¬žO…TýL·gìþŸ:ÐS¨Öué@íÍ×vôé©ÐïŽ=ºg.± Õ~H×õúíÁû;ÃÛs-׋€õW—^ò©ìZ3o?ÐÆW©}RàT{¤“wý­Ë “‰jçâæ÷F{n5ÇW'£s€Õþº¯u}ú¿k](ÖÿKa *úi_Hm‘>,oŸ÷Å‚À.üêÿM]’2ì©wT—Qÿ}…W}_ÃðôµcÑ×ê¡õ/ç®ætgb†ôé9ó‡_Ç v‹’D{Ct ¦ë×€Uþ¼9=>=vf÷=h].Ú›¡>“fèa¾”V…5¬ ÚØ¢-µoj]›¦@¬…¨tP½·qmµ2S¿«ßÑÉ@w²R£`ôµ.Ÿj,Ãèž“ºÉjå­@¼ûLÌ[uØõºjx²¾VlÜåu¥Ë¥š#åÏÇS‚zTýmŒtõØú ‹(p§ë͈ Àú…׸¡Éî1(dû=À©hÜ×:0Œ»œÂ²ê—ÏKõfÛ´vÞ~ -Þ}iuRNíœ[ÚµáþsáÚMœôÊSÛîŸT/¯¾v+H€i_(‰¬!¾ê P¹ïE·ÔP 4Þ¼¯héû~ªzftyÍIó´ü-KÜÁ–±ÃéJÕíÛ¸¬ÛRŽ{Lºœ€—ê@Íݶþ¯Þý_=‡Ä ¤‡ Í§à5ÒÆ-ÒäFª¸wjÜ"YjŸÕŽ+,»žs;ý®õCN¿=”a/`R*¨9ÀþbPn±©?|NDÆ› –ª¢—wCëÔóëh¹RÐöÂ2vÁ+w€uÿR| &gòØtóùoù‡péÃ5ÐÆe©=K·ß±NâéD?BGáWí¸þ¯µÜ‚a:yèN ºéö&3€LÌЇ¥ægÚ6H*7LΟK«àš.x¦:hpAÚŸgbìþ¯Å«ÜP3wÙ¸¬¡Õ©zdTuyõŒd€uÀøŠ²Ï:Ç`V)ml‘•[00ÝŠôºŒÚôèÊÑ:©©öW+8»“„jËÝZ q£rÀ}K'lû:‰·ñ¦;¿µ°[¡Y?sM.ÐêÀÉ_eÙï}ˆŒºÅ±\(uáÝâRº®è¢S ËþW\v×ç/l催·ëõÈ4«4 \!º˜Œu?5ç.Ÿÿ–Õ'û­aŸJÐÆe¹‘-n ¹¸¡Ðj§ü“xn+µË~ûëÚT­Ù Ÿû#€2 Àú½R ÀÞâcu¼õˆW£KµÍB Àn‘*õ8è ê›k¿Ógx´z`Õ{ Ç¦¬tjìjÒÑ@ªëÐó]`Êõ ëºt@¦ëR°vsYÝJÓúž±ó”Yòdx:8s[©÷BßÓÁÛIáz XÇlºN=nw" Ÿ÷ºtÏQ²òÖmlñ•Û‚Î…~­ü¬vS{ûjÄŽkg£ j±k7ø'8uyc·‘‹kû ÀúìPû«“œ©öŒ/–rSd’Ÿ‡yÛÞÎ|Né°JYÆÎÉÕ˜¾§.+wp¥=Ó-·Â«[¼Å•⢠۸`ë?Ÿ §š³½ºÏ™ Õ‹`w⛨ó±Üß)YëxÛ¡TŒUùƒ|oc³Y:i¨íÎÜ–u~éD;Q×6è÷üï»ýÕ£m[¦Ø-r¨Ò¾ÄÅü¼ë³M³¢¢âeÞu¤·$ß4Ü×?ÐIuö[å“S‚¶Ô#«^aÿgÑÒ’Ûû7Õ¶îºÔóê‚vtH¶z6t¹èméûê-v½Èq÷ß_ÌË_Õ:î±û÷G­z—>5åª +”‚6¶$Jm¤Ú&õ¾jÔMºXµ_jç¢íª¾Ö÷£í›kýµôu\û­‘'jwãÚ×b*·µ^²æó– ½f}hú{á–Zé J‹UåûüÙBß´¢bÔG†ùi ¥¨¬†¹áâɪç-@zUeee˜ÈâO¥PêÐ0f778nh•r %ko9”Z[^^þ›Rlc©‘)·}"‘xŸ·™éÔ‡g©õ€jNž±óy£sr©ìÖ„ÅðgÐÆ2Ê„†ÒJÙ¶}]È[ €ÌLLÖѸ­ƒŠ½4g,ÝÜ`*{½É:¬ Þn(Å6¶¬¬ì£Rlc©á-·]T¢<ñëä¿U¼ÕÈÜc÷Âå ‚Êf5|ü÷\ï/½(ù66Õ¹5˜rû''k1o1¦Étï!È<5j8æþ2ôN€6–6–ÊZi»?½¦ÊÊÊt’‘Å„Õú0Õ|"{ ŠìöUcÆŒýaî/@Keµ4uG»Øöõ>ÞZ NM²öëµãîû9È †T—_våQ{p¶‘·ÐÝÆ–••ýœ6–ji(½éîýýam†d†±+#¿öò[hPƒ*­ªmz‡>OämÐÆRÙ©gW¾è¯öVoâ-ÀÐ…‹µœÞá*ÉpPÝ_9Qž8nú ÐÆRY-­"^[{& _eÕÉÚ£ØK.¾cn™víõÇwnû10E¾ËÜÛÚý³ýÉšÆ[þ6öÓ×}& -©^ßìߟ‘5ä™&c{*jkÏ<úäcÏq S„µê©‚³&œí‡ßɪååŒ\{æ™uÇhcK¢×—“‹ä¹>=S¯¸öø›¯íàÀ¦J=N³fÞê˜IÖ^ò@îÚØ«®¼. -¾^_N.Pxzz**++ßuÇWÂÅ’8È)ÜžëÆOðÌ6$«ž—9@Ke½×W ŸÍá%@áéÓS¡ULz`YðÃ]¿à §€¹ºöšéþÙÞdµðÒhc©¡ªùò]÷ú½¾ú;v&«†—6…­§§ÂØ-“8HËßÒßå‘%OŸlœä_íùÜ‘¬*^Î@~·±áü®-¯¿´ÍŒ{’߯nMÖ$^ÊõTÌLÖv÷öÄsq–_½w߯Ì¢ÁW=l½XËÉÆü*íã<ãúφÛYymì.ûwElŠéžCÛ;ãŒñÝñ…{‚m[vs”ƒÒ:1½‡L÷ÐJ†âÞÆžyFÝ1ÚØÜ®¡pÙ¥Wøí+Á€Õ¬å¦{5á Qž8®…@ž]ù"M#PÏ?÷j¸Ÿh¤7â Á(®6¶¬¬ìðM¾×icG¦´ ™zß/hhŒßí_ °µÈt¯|$œÞV5e{ì÷öj˜ó¤ ›¢e]¦{ÕQæø´±Ô ×OЉÅyíwö™JRVVöAòßÉšÌKø¾Ú“µÛgZŒ‰µÁ÷B¨·GCœµ0N$ôª·wi²yé´±´±ƒ[;A‹ÞtÃÍÑ=|û­º¹ãí=Ey ¦Ñ·Ÿ ç™éñÆ »Sí1ÝsÎfz ¦M».µ6v׎÷ƒ—Ö½.`ÕþùÛƒk¯™7’æ#Ó½˜ÕB/Iöàc±‰ÌisU7~BŸ6 ]+„a}º¯½üV°ìÑU=AW‹ÖÄ=F{@öÓä¿«“Õjسmì€O&ž5áìtmìLw/ïLÜ^'&&k¶éÞÆg½=`;šê íŠ)W·}®-\ùÉÇž Ö¬ÚlyýÝpQ“á>øÒí¨4tY·« «ywš·«m3bæ•õTyyù¿'ÿÝlº°ROM=z¹hcmoh^µ± àÑ6ÖºéN&Ú°«žï6ø+ðrR õ`LÊô Í/õhA-t2kæ­=uˬÖ0¬Æ•°Üåô{*Z]WŠ¡ÊéÂ>²½mØÕchàO ŸÛØd»õwÒÆ~¨ûšüÿ/ìNäO Šù M=¨Kì€Ù©áĉDÅþH ¦å‰-//ÿ‰½Ýõö~Ì7Ý«³r (ÛØd»÷O#ÕÆ&Û󌴱ڪ™6 5 «OV“=pR@m³µ(Y_KÖVÓÝûáW›W­öw'qàncB t°vÄ0O †«U¯.S@òÀJ{pvO [»ˆ§ ·4ÏÍíG|À°ç. W»‹§ ·ÜÐ/<Êìd±—[gƒ¹¾Þoª»»í÷÷Ù¯÷Ûë\g/Ó’¬íö{ìešLüà‘ÛÝ`Ÿ—ýÞó0Å>/‡ìýÒó±×^W—º»bnÓ=6]çAï6ŽØû_ÏÁ€ÀðYd?ä[#ß_`¿÷°¤½Óôíí\¹ý:0÷x¡Smèܹ]ýî ÓÛã[eoCߟ˜æ¾Gpµ½þ}‘ÛæÝ†±·È»nco{£½\£÷ýèmÖÙ°»+òL²¹‹ƒ3 €á3ņݨ†È‡¶ðìÈï6EnÇ︡ÖË#!ÓàÚ¡¾yx¶ýz~Ìívx}NÌcð_sšÜa¿×’æ±Õsp9×lØ€¢VoC§æ²®4ÝC‹‡#7ÄÜ®;«í×ö{~¹¹Ås¼Û=sÚ€ÝýkÊðùšlï‡B­†BÌ ¯OóØ6¥ þ`È‚:ÓÛ“ªrÃŒ;‡)×÷€×™ÞyÁ©jvŠÛJ^l2ëjzçüº9¾ æ›3À]xl`È2Ü4×ן×:ixOðJÓwþn:Ù ÀnØô”˜ëÓkõ\ר ½Z0«:æñ¥ À.Ü×fáïF€RˆÚž&Dº€Ugú®®ì(Ébn·_/Œ¹O‹m`¯†Üb¿^”"àï÷~gIÌeÖe€Ýüæö ²LÁ.ºª±zAØ€µ<沓í×êÉtórý Új¿×áºLp½Ã¦ïbQúÿQF+€«ì×[ííUÅà {݇#!¶Ã»œëïòîƒÌ7½C¢§E°nS½ÇÕö±´Ï¡9·5ÒÎüÝÀ0@í6|±amý‡ ƒ›#Áö¨ ^û½€·>DëíïºPX3€,MÞõð¸nÃßf(ÓlLïÖHýysÌí6ÚësÏ-lµÎ ¼ëLßý}ØËºÇw·w}[½Ût«KOMñØöDr§Ù° 4Ek’ ¼÷Ù æöÁm1'îÜ`Cž.;ÛéæÄÅ›ìu)`WÛPÛfúΛ5öë6sâêËê¥czWIžc¿çk1ñ‹F5Øëô÷ó­±×±À>Þº nw‘}l&æv£÷«Â^ß´Ècs·Ù”æ±ÍŽyl`ÈÚXÎ @½I¿è]•½LMoÓ]gu?/šrS7ˆß«6½ÓXhcÀòHª•óæax»ëlËãçESMºñ{qÓ[hcÀÀyKk)t€ÀÁ@.öT&›‘YY(•¼×ôöäFkSä}\aƒÚ>Ówõè:ûýÃ1m@‡÷=mñv(€lXTHžè]vŽý¾¿×{—ýÞï{Mïþð ‘ zÐôÝ¿¼#à ÀëMßýÒ](v¡¾)r»züÞÜ»í÷—Ú¯§Ù¯ýð\m¯ïˆÉ¬s Ù°0 Xµm 5ÔGCp6'ap€=yÜ)ÁYÎÎ=ç¼à²K¯ë¦nfͼ5¶ÜeT—\|yø»cÇž”éíéƒnC²–ØÁ&‚1@0¨œI¹÷q ‘3c®k½¬³ß†ë¨ŽH^j¿ž–æ:'Ù ì÷¢¦Ef[LÀtöÚ€›i®³×µ1ær‘ûänwqÌewÙ@îÂ}Üåì1M/MÈNØm¶awMEyÅû ´M]Üö¹¶`QÇÁC, –=º*X³jCðüs¯[^7¬÷vþ,ø‡÷~•µúá®_„×ûÒº7‚gW¾ÞnÇÝ÷‡÷ãŠ)Wuã' 4ó!€¤Àƒ]«ÞO…Ì6ܹ\›æº'G°zx¦ø¼nõ.ÛjÒ/žuÔ ­m^pNªk3 À-öò S\ö€=þðo7.Ì/·?«·'Üç½öguÇÉ|‚*Û/)++Û‘¬âB£z]']ØöÊ*è®zê…`Û–ÝY µÙ.…îMß~'xúñµá}VOó ©B±†Z­³œµ¼,0€!à6´¹ÏÙÃ6,îöp½éÛ# Î~í2©{d›½Ëö·Âò¾˜\s¹Õi~–îºÚÒ\v_䲓Ӵ‰õÞó°Ò;qàžKÚL€)¶áìJÞý 8zôèž «U…Gõ´æsÐhíÚñ~Ø;ÝO(Ö"šû3‰— @À Àò{ÔàÙ‘¹ÚœØ7\¹ÉdÞìe«éín¹\…é;_Øý^cÌe×ÙŸÕd€3é޹ݸ¾Âþ¬.ÅñÛ"{»ºÌ^ššæðv&ïÏ£aOÃ…o™Õ†]…Ãb »™ÖŽ·÷?¡ž0aâñÈs´ß~ 1ìýiKqðÓlz·Úptre¹=xÒÁ˜z@b~·Þþl³­Å1F®×Cql5€ä(»ù»³c.»;,Ý\Ûh° ÀKLfs€ÍÀçÏ Êû½ÀšI®³á:nð¤Èsèn·#æ²{ímp7šG¦M¡Ý€xjèWDCo"Q.(u×_ ^ùÖwK2ðöWêõ~ðþÎ`ÆõŸ ÆT9æ=G쇑>,'òCDª!zÑ!mÍö@iŸ=HÓÊ¡‡mMŠœ¸Ò÷Ù˸Ëi(\SävwÚƒ&÷Zm'ÈQváuIär‹¼6ʵ‡ ì×^Vˆ=`N\ZŸÁ©VöWcv½Åq«@5'®½ßû^…½/qÁ8]×kì/ªåVLïŠÏþêÓ“bÚÃEÞcÓ×+"'›Ì{€› «@(r:Ù‘ ½?5‘Ū4Ü÷‘%O†½„ÜÍ%~ò±çÂço̘±~ïðQûa7™—€7ÚPëoaí!¨¶Gû"{õöû{"·«ßUÏo•m eïl0P|¸Æ¶QnJ‘>+]Ϧ ‰S¼Àé¾·×ôîÙ»ÞŒÜ>À{¼ëtË×dð¼Dpµ×dà­¦w¾ôºHØ]ï…ô.ïrëMf xÒÆ(Zš{²Þ_ÄJ[Ík¿3œïªU“ ³Ù Õ••~Ö‡ÒL^†à ð&{°=yÒ`z‡Ù»¤¸žÜE‘HwÀUˆsÖ98òŸÚ£©i~^g/ãLQ^hƒó Û–©}›h/m¯fØËiÁ§96L¶™§†„'ú½ë“"VØŸ­°—í0ñÓGÜ\ÜV{ÛËMæ½¥Ñìßîr{»‹bCƒ½í:{ì°ÂVªã=7öú:í×´±JR•ýpÙç±pÏ\…4ëðÖÎm?Ò:÷œóý•³uy¾a®08}žyÍh>Ú´ÈAœ‚·Úþ¾_®G`~$"Îä2ØfðÃã0m, cðuócÂy½ê•dNonJ{·|ú†ãååån¾ðûaÖJà¸lì—†Øö°^7î̾[)u¿éÝ>#Z À0âXŸíêÁÖðåõ´±0|jlÖ|5·WÜó}OÞR)ýÚæÎ FªtAXÃ\5¬©—oIàÅ&õ~’êõÕPæ%ö`êýÞRû;MÞ.F&»ùÅj³[hc ût0<߇Á÷ÜsÎ ‡ß–ê¶E…„Gù+H¯6é÷DqàC1ßßlN\kEÌåÖx—›mÿ¿8ær ím5€`Ðêløègó?™6²@‹N¸3áöEšßË¢V…„½³4œu/뢵ÒôÝ۱†Õè¶Z+ÚsÐ`z‡;»ßu+“ú«–ê`í°½l@  謤ë N­9-è|øBeï+<íÚëýU£é .N ¸nŠÂ!Ó»gïòHn0½‹×ðþÈô]iµÑôn%rÀôîó«ý¤ À€8͆}€€vã wn™~#{÷I©÷~ü™û­¡7¸˜ÕØ÷°8دݖþÞ¼ê½czWvnKqR¤Â¾NÜåæ˜Wo1}÷Ê$  œÝö&AÝø Á²GW‹p/á/üé]Ǽ£é F)#” {ìÙÏwÖÌ[Ã}f ŒÅ[Ú²jÂÇÎúÐÐ 0 Äµ NØëûôãk ˆ%RZÅû®ŸéÏ Ö"JÕ¼%@@±Ñܾž…®æÞÖN¯o‰Öƒ÷wU£«>²¯…­„`€PLp´J0fÌØãZ‰ XÚõÚËo§ŸVëækë«:Þ& † Ù° 4€< `³S}µ§Ÿqüùç^%Ra½ùÚŽ þÜ»¼×toƒ€´± 6ü†ûzN˜0ñø¦o¿Cð£úÔ¶-»ƒ ?y‘ ÁÚûµ· 88ÐÆ(4ö¼] Ñyõ Çt|TªÅ±þðâËÜâX:aRËÛœhc -xµQÐø3?Fø¥ú--ˆæ ‡Þj_CgÚXyOÛÛÕÕãŽ1ì™Ê´¶¼þ®¿0Ö:ÞFàà @ ß-PãSYYÉ‚WÔ€ë¥uocªÆ¸¼·88ÐÆÈWõÉ:¬ÆG{½è¨ÁTçÃÏöì°}Mœhcä.5<×^3 G ©Z¦ßèBpo+ppÈ@³a`#(ú|Ê)5Çv¼½‡G ©ôÒrðöùBÛÖ„CŸ—=ºŠGe¥z`™ë>”¬Þf  oê.»ô ‚•Õjºè‚ïæm0r­Êt÷ÐkVm ´QY­'{Îà†½A@Ži«šàÂßÿƒã6j8ê¼ú†ã64´ñv¹´_tê©#¬QÃ<xo7€)4V0Ìšt0W{úôþRÃVïíüYPYYézëxÛ   jhfͼ• F k]uåu®˜aÐàà @ 'ºÔÐt>üLA†ª/ßuo¸rõ¶-»S^¦m¯n=ãúφE°Í¼u<àðÞvàà @ `¤U'ëH¢À†û@NÁ®Ðp& a.èú=¾£G.¹øò>X5:ô¹¿E°¨õ»þJÑ.ÀF{{ v„ý¬ÛÝñöž>—U Ö},Ô¿×-³ZÝߣ·ÀI‹Œ×CYÈX[)ÈÇ•‚ª€6Õ£w}Ÿ¾î3áex]ŽëqM€Õ­]iÍöç¿ÖŸÑPnýßñ¸ËÁ ö#ª$çëÿ-ÓoŒ½¬†ƒ»ës8îùI€5¬Y[é£{ëºRíG¬ ®Ÿ»9Èú¿n]L˜9À  WZt wՕו\VOoÜeÕ“<جÀ«ï)Ç­î¬^áþ°zÝ}Œ»ÝBÀn1±dMâí0FÒdcW.¥¬aÈê¡»¬®RPݵãý`]^‹méwý•¤£áO o¥zíæükÖ"`6<Ôñö#©&YGµ¨’\©๷µ‡_?ýøÚ¶FR€uû"g€¸ë=V«?§ºšk­Ëhåiÿû¯½üV¸°•"¢°{>“uˆ·À€͆U  ³íqa°˜°zYÕ ¬Ð©9¿ÚŠèËwÝ]ÖÊË ¤ ÀwÝñ•ž}„5„Z—–nS k¹-t¹'{.è¸ûþ°WX·ûÒº7Š:kë'Öó¶@ g zE 1Tµþö0ˆjÝT—ÑVDqóu&<õø†µ0–?wWשßÓmD¯S+4»UšÝ}HWîþ)k+ ÿv5[½ÏÑ€w»î¶ ño¥ç×~¨Íçmδ±raªšTsbK¡¢{íŽTE÷.æÒ0qõ°ÛµÞvàà @ *’uÐx+ST¶KýíÚ>Þràà @ —–˜4[QT÷ÿ]ÈÛ œhcäR]YYÙ‡jp¢sQ)j¨õ굧 Ãoò5öo¦{åq€ƒ3m,€œZaR¬ÁŽÊ¨4wüŠ)W»¡Ï?âÀ¸`U$kZ²V&ë  |•®¶&k¾aº(Puååå¿ÒMËôY‹Ê¨n™ÕêÂï/“ÿ6ð6*HíÉ:à‡›SjNýà^ôo×þ7¾íOþìwöůTiWkû‚`Ƭ[{Aã…GFª<æ½^%kAÀ 5VC“’!ø7:¨i›;€G¥­/ßu¯ ¿šîa‘(,““µÝ™uö ¾üçÇ^}ûoƒ]?;LQ)kÛž_>ö`ò§®ê9i’üìø í€A`97Íîá̽­ž`*møµÕÊÛ¦àh.çQýýN­=ãÇžx–`G ªV~ó;AýÇÏ…`õ /âí€  ÐÌv!˜áÐT´4:À ¿ y»œ÷÷ûãywSoAŽJ½ûþ¡àOnÿR¨è^ ~̘±/›îyå@P0¦ºáЗ\|9 cQá‰oίNÌæmRpÛS÷_ÿ7•Õzb͆`ìI'…mDòßïÅ@P`&»…±&]ؼùÚ‚`‰ÖŽ·÷×^3Ý…ß¿©% ¿‰ÄñGžZK`£†¥6t}?8ýŒña[qRuõw =ÁÀ Lc¢<ñ¾§±cO :~†@XbõôãkƒSkNóÃ/ Ýžzc·7úÒ½Ô¨ Á£«ª¾ÁÛoØiîÚ,\NVh+´\véÞ÷}¢ýLXm«¥Ÿß[n/×Ο `ù¬Ú~ †4·}®-xoçχE^ö>kæ­þ|ß]†­Ž U—þ†×¶Ü@@£F¤þçko•£«\ÛÁˆ‘áÓ”¬}&{[Èl°³©ýnmµÇ~Ýæ}NlNó{“½Ë­æOWjì Øf¯¦Ù׌+÷šÓ¿s¼ËÕÛb*€œš‘(Oü5TŸlœ¼öò[Å"­5«6gM8Û¿+ û|*H'U;öößýœpFXiž¹^{£FUþ‹=Fö¹ð™­<ÃkóS™c/Ó¹e¢ÕåSõFw&ë¸(TØâêñ_`ÿ¶«+**Þ®¬¬üûªªª>{Ëg£’ÇŸÿ¡ëM¶'?S5f»n+ùýuöµ´Ô¾'s¬`¸Îæ­£DE0¯ýNÈ*¢RÏ~ûçoïùÀ±CžÛxÙ´°÷÷ϾøUB5âuñ¥ŸríI1o¬·•NmŠ@Xm·¿ÞíT "×dx;þíí·a6Õüí6èÖEîÃJûoªáÍºÞ à‚ReC¥BîòD"ñF2àþÓ`ìÉãN O¬Ÿ{ÎyÁe—^ÖM7ÜŽ4‹+w•cÕïj*^&·5zôè÷“Ç/¯$ÿ¿Äž°i"ÈÊYb׬FéÉÇž#@x½ò­ïçŸwÿ!²5Y¼Ô Z8ä°zÜÉÇéý¥rQõÊ–°=©¨õ)ž¡Œw'Ëïá: }ël-°AÑß2n¢ GMßaÃѶ¶Î†Ä#Þåô;ë½à¹ÔUwöy¿ß`¯7ˆÜN&ÓX–ØËÏŒùYw?¢!¼Í†Ü¸aÐS½ë$çØ]™|Ïîv[bƕ¨HUXí¸ûþà¡–Ë]Ž Smyýݰ²=eN;Rèz_Z÷FðìÊÃÛÕíkzÞS®êÆOHŒ++Gÿľÿü`ÌB}Dg”׸†E«³RtaöúþÙÿ¿0ìÑ7ݽ¾ÿfX ¤X„Ûµ¶/ ŒQùÐ ÜV,ï){­áŸ2¼ÝôëÜeÃè„ÕcÚlº{‚ÛÝì…FÙzûû: ße/×a/7Ã^—ëiu¡Òõº.ñžß‰6òng¾ýÞÓÿ‚Yõö:×¥ÿ½?q¸ÓăÖâW{½ë&çž^gSôK~îÿuyyùoM|ojŸ «…1@óý¸fÓ·ß ï뢎žæ ã‡V'*Û×z«ÉÎbrJÄìDyâ_mC6’áü¯Û~Üñ…{‚ÚÚ3ýƒí|•ú»>½n#AŒÊYÝ÷ðÝ'׉7 üýäz?·F¾_mìN/¶'Í·Ô~?ºRò${½.ØN±—[söÛ  ŸÍÞ÷\(žù]·UgµËøh¯ýn{*R`×ÓÛ Zíã!ç–Fh4†dàýM$†ë»Ì¸þ³Á—ïº7Ù§Yl‹{>ÿÜ«iCq"‘Ø–üw‘}_@Z M+ÜD5¤޲mËnÂfž•þ&msç…óq¼Fÿב<…—tÁS/P0fìØãßÛ{ Få¬þ÷÷ÿ1l[ʉÿ0…=ÜÐ-þ4'ægõ^Xt¸.r­š| Åu«Ý= U1·¡žá}ýàƒörqöDtÕïµo2ñó¹Û"—ƒžfÞ@ÎYè]”<>û[3/WAð‘%O;ÞÞS’ÇDzÜÚâSǬ‘Å?;ÕO ÂÍ0Ì!ÐÏòÅɆöß:£°EpîKõÎëobúnm4Ó;àÙïýleÌ ‡ˆƒÉŸºŠFå¼Î®?ßµ+…Ü«rŸÉlÅe€£2YÀÇ…hlkØò:Žz—I€«3¼þ¸­Žü Ûiwb?8: z¥È À9½ZëC‹]jž®æÓr|Ô·4¼ûÁû;ÞpÑ-ÛÁ³Ñ¾/'òG “ Æ/\ãqÕ•×…ó1hpG~q«O_÷™ž9¾¶öÚƒ•Š˜¿›æ’¹…Wt´Ð°HD! Joº¥•Få¼®žv½k{f–xÞcúî­*÷ØËï´RíðÓ;¿8U®1½#yÒÝN&–Û ;ÑôcÞ”ª­ñ®×õ·ÛÇ£ß[Dvz®çh>¯‰ÌáÕâPöKgÄÀçk(¸zÉý0lÓɩɼìÄчf«RCL´B©·ɽ|õ¡9û»3Eð;{¼Éû=ÍûšÆË¹ hñœàOnÿŒÊyéDŒ)ü…°ZMê!ÐóíÉÊ4xo$¼ú´ ´qãÚšs¹=¦ÿ!Їl›êv2]çÁÙ…¦wõæÙ`÷X7{¿7‘:À“¡î¾èº–¦¸L\nò>‹ö€ï3ųMÖHß.ÿó^«7«3)f#_Ú‹¸eú=ÓÌlO¼^Óì¨ V­ýP^— Ãosu-> Eò}¯¹\­X¨ßÕ{îÏI±¥¤•ö̶çíN4½ûPº^†ý€)ª”°±'ýE©´x ?,zI=ÔYpƒ ¢î÷÷˜¾ûêªí^nz×bp—™mÃä>Ó»˜X•é;]ÅðToïæÈçÃnÓÿüå¨Ó;t»1ÅefÛŸG{¶·Úï×F>GöEÂôB° À ¾Z[EŒc£üÙaà N×~hºWnॠ• Ú–'ŸšÈŠ…sok{:Ku¨´®Ð¼éK.¾<º˜•;3¿Æ†Ñ‘X¬jª= ¼^ŠF^Â`Š*‘ì‡ÙÁ®[m?ݪ .“éíä{oÔ:?¿‘59t  ù½Zà’à™ÿAØôŽx¨á¥  ?>¥a¶ÛMd£öI6…X=Äú(¶!?¯½üV¸pÅíó¾öòFæóú½ËMîö¦SÐÖиC¦wžñx0E•PFö>ï÷†ŒÆYl{Ã…­´}‘A³ð‚°Ýe/k™Òvm§°Ñ}ø5fÌØãêUC£¡Áß\û‚XaZ㪧^{vu6Wsx"g ý:dÏ·›üZ„J,+Lïp>Íokå%K¦(02¤9Él'Ó× 4ܵ×LÞÚ¼‹`Y û «3ÃÐ `ˆgDõA©¹BšC´É†®B£Î–ª·Xó‰Õ›ª"´5zX‡ûƒD\žz¨u› º_ûó¥aH¿ìÒ+â¬òKó¾´òærx'ÀßE¿·~;4`Š"¢`´Þ?ŽÑ‰}‚dá—yÕ6 †Þ`Y¤žÈæd-0Ý=’ `‡Ò„̰N?­ö˜æ+”jjõĺRh¾ã ÷œP‚¤Ÿ+Xë÷TjÔT1ósû+…÷Íö>/°¡±WWnõNHµ«X†µéo£…nÚƒø|¯p±”ÉŸº* Ù¬{îÿz°ìك׿÷#ÂE²cj2ý“¡×·¨wîб¥w¬Ho0€aQoϲu؆f³ šC»/“<ÄÒmì²·»ÒÞ-ô¡9OÕE|{‰é­çXó…+ ð±è>k.úþa~l]=íúà¯^ÙBÈ£ÀÀàu¸­!éõ-þÒè@·¾KòïþsCo0€q+ejµÅf[f}g²ÞMÖol¨ó{Ø:ìe\ͱ¿§ë`1î•¡ýí9´rôÔ»ÿ=«]ÿØYAËM7­í ²Þ£Zh¥ç@Á·rtU÷âtÁÏ_¼ûþ!ÂE2WëNÒë[:¥]M4’Ð;FZYÄ# (ë@Í-ºÕÉS2hÚžiŸ×Èk1¯‰y~Ÿu2$Ê]ÿñß žX³SoÿÝσ?½ó¿„ØØÞàm{~ÉsC€þÕ•••ýÈÐë[Ò¥MÆŽ=Éõ¿C òÚnú.¬uÔä×ÊË…Hóš5”øˆéÝ»¸Ãäç|g½Âù³_ú)]õ?^z38ýŒñ=!˜ž`Šœ±‰ý´EêZ‘Áõh ‰Õöú¦Úÿ«Zúù½åöríý\NíõÒÁ=Ó½Wý>[Óø³›F;ô5Ð:$ôú–viqÖÚÚ3]þ;3´½¿`@t °Ûœ8Ïq=OMÖèDÂ:ï¹Ý›ÁÚHÓB^ágõp^2« ]ßï ÁÍsB€ûµÁ†Ïþ.˜ôÓGt"Qk-ì±_·ymìæ4¿×dúnÍ’N— ¯ƒáÚü.„Küï®Å®~¥çD[8²¯/¥zóµáÉÓ½JôOxŸnMö !|Û§ðeÝ´ÈÉÍjȇ;V^QñžîÓ_>úÁe=Á­ba,ŠÜ¯L‚ç {¹t½À³íe:"ø éÁ”jM Mí9’áýhÂgá>{²Éç°¬¬ì=çW]y]8”ðG¹ÒÉmßiCð¯òå¸@q©³ú™Ô«Ünçi6Þ§-…ÜJÜ:ÓBc¹œÿR¯û2îäSÆ;ÈÒœ`c‡Bó|PE€Ý‚‰™´Sµö²UCÀj'÷Û0›j%ý6èÖEðJûoªáÍºÞ ÞþL4©‡s+w¥y|õ&ÿׅȆ×óÛ2ýÆà‡»~Aè£bÇÒÈ‚ÿÁ°¨*€,ªµýmó2›§jDND¬ôžs”ÍÉÑ}Ñí×¶Ü@hÂÂXnuhö ¦Š(WÛvê¨wÂNóg;͉Cƒ5\y§×¦±Ó…æ)¦waÀÃöÿé>k–ØËÎLÑ~ê>ùSu\n³íiÜ0è©Þuf€5Œy«÷õRûµ¦°ì5}§µ¸9¾³íc;jŸƒ}öz\ð]b¿ßö·éç\[ðJá†ðK¥«Û~Ü3Ú.ŒUÁ¡"€lÑüŠ=öƒ÷˜‰ß×Fgäè p»÷üëàjÒßœ‡ë…>ÿ×ßü}Î{¯Õû«çqÙ³/þ¨b ÀnšÌrð´·ù/ûÁò¨ ‚ZO ÙtM>bCq… ­m^;§ÿ§êXozWкÛþlFŠÜiâ‡AwÚûXo7xµ ¯n䎂ô"û½C6ì7ØûpÐ~Ö¶™Þ56x©Åþ~—é;”»XTØ¿sjnyTµåõw{ÆJñÞ€!Õßdâ{òôä„¿ ·Ü4÷­¦°æ¿N˜xNʺ|êµÁ‚{þΩµÏpºßsåÿÎ÷ö ﯾï^»šƒ«¬yjmŸËfrÝ*Ý¡<~í¬ûqÏý_'üQÅ€§Ùû¾$òý¦˜¼ÓÀº˜vÍ…Rg C»lØŒ½ÖZ û#'kýìzzÛ#L¡tñ°~oA\júöVG‡@·˜Þ!ÚÑ ¸Ý>ÎbZW'Â0£PC¸£2­—ֽѳEǤ²}fvCLøý· ]8Q×sás™o†¿G>+xå7¿¾Ž>qáEÁM·´ö©OÿÑg{VKVHõ{k'êªðûú7]ù·¥0­ßùý‹þ0ø“Û¿Þ÷›ç¶5§~_AÛ¿~¿\hÖýô¿¯›°S¨=é8b¹½ï )‚¡ ÀµizkªìÏ6 2ω ÐM)žÓ¶Èe£Ã ] oÌB®Ëà~F°{>ãFø´™ôó– ½ú(‘¨žîUB5àê|ø™îÛå‰ÿ0lÉ K–ÚÛC6`¹ùH<5ya’é»B·zW†sUî¬àT×£^[\zâÙp¦·óÄš á嬣?ûßßÿÇ0àª78Õ\\Jt‡#쀩" À›Lß^^¥Ùô¿®ÄÎA`·ÕÑæhÏbÌAq4/5}‡A«çu—ýÿPpTs8ÝóÙlŠgµð*»ˆQÐþùÛ sÔ K‹¦™îùÀÍa!€¡Z`z‡Ùº=uF|7gÙòÎÓ»hŒjž!r#€Uš«Ëø½­ ÀýØ/Ýûà !›L€l£½ïà 6ÆÕìA`Yn?¯&šÞaÌ›b. À®§x¡÷{‹À#s‚ýÜsÎ ÞÛù3‚5èÚñöžàäq§¸6t‡„k†é^Ûù _åéÙtÓ=gÍí[©ÞŽ,ÿ½F,/ZòØ —hÖ^źü¾­-vá«èc0E· ó䘟íòÂÜÄ~¤‚hͰdÝêͳ3À¢¯6{¿71‡8“!ÐmþyÕ ¡Ïz,kVm ÄQC®‡Xæ†BÿÚ0EÀ è âpe.5:`óçmkuÑ–B ÀZ$Kó€5<ù…MÛNÀZÉ9Ui¥g(uýÇ/ü wþãù ƒÎo¼Ðç2`Šœ•ÏŒ¸¹½Ó¼vÈÙmOÒÕG.;Û^nŰhµŠÒ(˜T{Ç…ÈÅö¤¯z_·FÚÓ‘À3bž wòy»éíå.daÈŸqýg oTÖªé¢K\{s7‡‚BªyàüÒbz·±rÃëûùþΚf5;ù”VXÖ÷ô3í•«^`ÿ÷\NWÑûÖõƒŸ†s€ÝÞ»®þÓ^ü×Åÿ-í–H`Šœ1·^D—} +lÐ= ‚Í6À)ôµÛ¯ï¶—; v‡LïjÌ“3¼óMïþÁKS\&.7™ør€wsµmËŠÝz‹ ü³©¶¼¼ü·z,¯|ë»7*kõäcÏu÷'*Rü€TÛ³óî †Æ£ðéoØa${@ªƒ§ª˜Ë6ØÀ\=RØ_ZT=¾ ªº~-Rý=€õóT•*¬j¸³z5§¸á¿ß„µÒtª¡Ðù€_}ûo÷~~~Ïs¥Õ¬µ…SC¸³Qê×ãßÐõ}‚*¸?š{·Ë:õÄζŸ%‡cBàÎÈÉ+…»Æ˜0ë¦tdº÷­Nâíµ÷¡1ÅefÛŸG‡Goµß¯œÞ—&L;ëLߞ㥑@ìL‰¹í­æÄÞó {‡½çh¿}N ]ø9rÕ•×Ú¨¬—ö’6Å1MÀ%·ðÆÃü‰b£±ÖxR:›¹Œë±X7R8z=ÿóµ·‚±'öûCŸ;¸¿`yõ´ëÃëS Ì·¬ÀéVÂNU:Y aÝÄÝVPq+iS`¯}©Oñ³ö3%¶'ÐþŸ7é?›ëMáyîy<¶w޹¿Ô°Î5ªò4ú³Ò|0¬ð\̦˜¾=/êuÑB+êý=ê}~.°J½´úÙøuÂ\Ýàk[n{NSý\ÁQ!R{çSv'ü°« ªÇ¯žr·‡±+ÍsÖPo08GÜüÝèÞ´nXñš^xŸAÁYÎ&¬QÃRZQ|ôèÑ®=­ã- •Ó;gj OGIô((à4½Û\ýÀôí]×FX£†­4¼Þ0 @gîšøí"P¼4ìP Õh+ŠcæÄ!¶šGW‹¬ž_×ËùÄš ƒÀZäÊØy¾ò ¿ÚI?¿çþ¯çMvû›4C³ýl>sô1€ À#h¢mO4ÜyŸýw…)ž¡»È‚Êʪïéu®ÅŠŠ9€]rñåa/wºaÞºŒŠÀšýZÔñ€=êyÞu¢¦šÞFX2¾t½bRÏ3]—‹¬zè‰g{†þº9®.GWŽŽ–ˆr½©êÕïh- uVÓÜ_·Ú´†I§Z :Ø B-<õàcßè¹|ª O&y Z{ÿ&Á®ïuSø5v¨wªÇªŸ1|xjÓ·ßé^'£²ò_yÛðiΧÛîhOGI¿>2é·šŸí¬®fÿòѧÒ^îæ¹íáå†õµVpÖ×ý•ßkìæ+è*Ì)«wÙÝ~ºmôs].nA®‘ÀæÜßuiµì+š§‡U3Õåž^·1|>uÀ(ÐPk=/鞇hÖsûé?úlø»ú™®ïOïü/ýÎAV¯þ}?Þž†¦»û •Àu¿R=Vp«_»ÛwA[=æÑž}ÝŽî£zÇýÛÑudûd%¬5Y‹ìçH&´•U0馢`.«æÞÖNÎAZsšûÔòV`lcàö‡ÝhØî¨”­î'üª>4½ó³€éíK€]Ï·J=Õéz&¥ÞóþV“6vKª¸ëðŒY·†•ê÷O?c|ÊëÿñÒ›',Üe2ØÃÙ- Ò–L:áÿ޵¿e“Oëïvt"… Ù}ÞûjWaXû—ÄöG ¶uã'Ÿlœ>?qC¡S`õ?»òÅp5ãçŸ{5øá®_ôYàiËëï;·ý8övßÚ¼+ضewŸïéòr®ëÓý(öÞ÷˜íy«PØÝê}`Uó””üüöµ.4üľVÀÀՃéB§±Ã½ÜóÁúÍßÔm©—Ô]—¬z‰*Uš#í/¸ Ý}qÁS—ÿÒ½†=Ê~­ùÕ~ˆŽÛnÊ­h­¬Ç¢ßÕí«7W·©…ÊÜuD{Ûý¬žcý«ž]Ý®®×õz«'Ü…l}_½Òþíh>µ¿²vt”R2ÃZ(˜5óÖ’ÀªW¾õÝ@C¾ã†BÇàïï NwJŸçóÜsÎ ƒ° ÀcÇž;wøÍ×vôéqV¾ìÒ+NøûèúŸ~|mÑÿ ¼ÇÞÌ[€Û VßY>ÜiöÀE#™ç€Gh$õ¨š˜^Võxj®o&òÕ#ê~Wû§ê!v·¥ÞÓTX¥0ÝsX=Ô~s´XÛ]€NuŸ5gÛý¾‚kª¬ÒðfXµë!WÀu—Qð»…^w™tÛcå8wÙ÷Eå{uÅ„ßci°þ nŸ÷Å’ Àúÿ_¸'v(t4w>üLx¹¦‹. ^Z÷FvÕ¬žd…^ÍkÕån™Õ½K‚zvýës·£ßuPá[׫c•þ¯ëÒuûßà¦nv¯ÁVï€Ò¶Øônw4‰§£äÝÁ޶2Ù`º÷ÖI“ V½þ½…sV£C~ýR`MµM’›Ã¬[]Gº9ºnÛ)³Â¶€uŸú °n2WÆ­žY¨rÜÖOîqFÛò°¶}JuêÝVOw• ûéîOŽ0EkmÓ¿ j¥€5„9n(t4kÈ®zg£Ã›hõ»3®ÿløõ7×v·‰wÝñ•>—SO±nGÿ×u(H·þöî›®G¿*]låN˜÷'PBÚLïvG-<ðFdZÚ7øyðÈ`W ® ²ZÍ:®WX¥­‡2¤W½®ýà¸áÍþBUî÷û[ج¿`š.køt¶VµNµ÷3=À5¤àhi§ ­5ÑäF•ZVÅ …ö/£0¶7Ü{} ¶ Ç~XV¹¯5DZ¿¯íRÝ'݆†>k²¸äb+ µ7ì ”´f|£+ú¢´Í´¯‰ãÄhÞøDÃèÀqasÑ’ÇžUî¬zPû[5Z=­ú}…eÍÿÕ0`[ÿo*0j˜sªëý_ó÷` [ÖýTOñ_ùËpµf?ا Àšsœéów;þYy€™ŒBü»È S:«‹žˆ/¥9ÀqC”ÝPhÿ2.À¦:9àæ³º¯¿|×½}†;ßö¹¶0`ïx{OÏeôõ+Û¬çé€G½¿»LúÕŸµ|KÜxx°‚¢pRˆÔÿ3¹.\¿ןë«¥ôü¿«B°VSVpÖbmQ”ÉàÁ`-Ä¥!Ú&2d[{ùþçÛÚÂÞlÍ-jVÏvtˆ¸V}Ös£ùÁº­H]C  À(ļÙÜt¦»ÈS)àèPèÚÚ3{.£Àªï_1åê~GÁYßèüïëD‚³ÒGz^ý•Yòdì·]P\ùè³€ý@ÝÞ(:ty(Xs}Ýe´íRªÛQ w¯00òÆ;ùGz¯zê…’ ÀþPhÿ2 m ² ¨šÓÛq÷ý=+6kßßè>Ânè³~]íY×¥¹¾ê9V€~èeáeô=74X½ÑÅúüíÏ—†ñÔSN{ƒwP:–Ú)íãÚÈÓQ’lتíç²n5è_›þWÏYÖ*È ¿®Wsýæï¥ À©Â~GáX‹G¹¡Âù€Uþžº~PO7Ø] mŽëUM·€•ž=¯Ã€õ“b{$¿4+Ee«ÜœiÛž6ð–Šß Ó;—³•§£è釿_5ñz´ZgE†—ÍIÖ|X?´j%õ䯅ºþÂëAv=’ù€ýÇë—[¸JATóy£ >i¡§tC£õœ=ôijáãUpÖ|\wþuEçÎu°¿õB°¶`Ò}ÐßR÷Ya_庆ª4kx·žw9p·£®ÛÕ ÝŽ{¬:q@F\U¢<n—TÌCp©Ü•[l옱¿âí?-Bq˜¨’âVà\9Ò·9ÒXáMÉÍcuQCg€˜]©VÎ׬R€Ëd;H5TZ7º ´_ …ú]¿õ¾‡ŸÈê*Ðêá® öêõnm_ÐsŸüCYÚ/=— þþ‚aî5@FÎèÑc7%²0•»ýËËG}‘wPÜ4tõ€=xZÍÓQtRÍá­OÖ"3²C|F<¿°i[øa¦€äÏ[UPR0Ž À U b®êüýa5¬8ÁŠëÙThSO®†k¸²©B­zOõ¸4<¼¿ëPϯzEÝ‚`êIÖª×þ‚Yw~å/Ãû£Ëú¿¯Ëéû®÷=Õu]F¥¿WÜÐëþ¡–ä¨A×_¸'l?«ªªÔ!TÅ[ (N³Mï^¿³y: þo©<Ü·zû ³Z,ÉØE¯À(8mzÝkßVÄ¢SÚNKCŸmûÙÌ[ (NÚûõˆ}£ßÍÓ‘÷4„yÙ­ŠŠ"»@Ý–'Zn›­–L¦À@a5jô;zí_rñåÁwý‚PGe\;·ý¸gèseeÕZÞM@qÒv7n»£<¡ÞôU/†ºG,kޝ‚lÛüh+]NÛ߸@ìo™”É6KúèþµÃYZ‰ÚD†nS`0JÔÄQ£*ÿŰ705€ÒÉ’+¦\mçýŽ~ßtÈh‘¤=öi£a»£|“nv«)ž G|¤b,íÉ«ç1ºç.E&£T?C‰ŠptÛ]w|…€Gõ[·Ìj ÛÌQ£Fý‹í Pdv·Úƒ£]œåÊ;Ûíßfj <Öp¾–¡eðuvýùá·¶gâù À@È­ot>ü !JY_¾ëÞðu’(Oü®D޽€’´ÆônwTÇÓ‘w:M÷^½3Jà±jNspúã -ƒ,…^=‡Ú‰çƒ"}, ƒM¢"xdÉ“„=*eøµÕÊ[(N‹MïÒI<9¡“ZplNŠŸ—ZüÞð ý7^ ¸ ¢®žv}øÁ=yÊ(±“ÊáûaQÇ„>ª§ÚæÎóÃïBÞ*@qj3½ÛµðtäÌLûwØÎSÒžÅÁ¹ç7ûÞÞƒ„—”V´Ös7îäS‚®ü”ç„"ñº £=^ ,xåæü–••}dØ(ZÍ6øê ?Ÿ§cDT¥ùþj„Ñý|h.z0í37}ôîû‡0”V™NTtïUøàcßà9¡À@?'[mØùíÝ ´Tõ'ú}ÀP1Ò 6FŒ´ŠŠFÓ(±ÑAÅA:8Á ½‘$ÄÁ„i=š`$–¥•f W2Ž[½QC·,5F»M‡‰NšîØ I¼^;7YËÜezH‚°oýöÙ7EžçUUŸÏZ¿¥çœzNÕ®ªïþýYøyååŸ ƒMX/=·)=çOÎëœóÛ¹Pšð *æY¾ZæÏÑãb‘±u•z+±ÀØÞš\"{m¬<þ-±àÔ¥áIç¾°ÓJ5=éê\ÔÇ+õËüƒÐK¥ËËK@ë±ÊöÖüñ`ïÄQ±/uúÞ÷ýÑ[s¯ûÔï⃼º)Ûë÷¤SÏØ|êèÕ½†• rµ~~ÇÇ!Kï¹cpØàµñ¥Ÿd{BïŸí­í¯$¶:‚~':ˆÑµU€ç· t×Ѓ]7tØ¡ß|ð°4xhh»­bHÇACþßAƒÿï¡ÃFlØ1øÅÁC†?ÖÖ6àÿ¬ü|y¦§$]ëmÆÇhzþ8Õ ŒéÏ´_'uVUjKw?Ç¥"ø^|ùœô©¿û'AO À°ÿïáËŠ!ѱ’!ÑY¬^—Ž}Tù}t¥Ï²Ð·†æ]¯;†ü·!‡ ÿ»¡ûó¡xðÐaéᣎNsLzüIggõþ?½2ýÀWguæ?´ãÿ£ŠËDwâÙu+÷½W÷U É?m8艤s8u¬d<¹ _L—^Hé™c#æå,Ê?À×CmˆçÄ”3Îî‘ðù³Ï§w~õáô…Mÿ*à)ºÇÔÖÖÖŽcåè±ãÓ¯®zXhlŠó®¹~Çg×|Èó\OyèýôÓ"ìvt yè !‡ütOv §¥Óg}4ýŸ¼=½î¦{Ó·|-]|Ç“éÍ_YŸ®xè‡YÝÿ7¿J|î7ÝVk6ü:»Ý[î{.ýìŠ'²ûýó¿ý“¦œ“Ž8|Ìn:Smnkk¬*·7èãÿ¶Ø¾g¡§6¹YðÀ” uõùì®â³ÌÅ^–-”$DÖo=úÐߦnj?vÇçÓ–––ï$]غQtDcˆì²ÁC†½ÚÒÚº­VhŒ®ëø‰'g]ÙºŸùÒ£éÝþ¸[CmwW„îÛü~zãÎ~çè49þ}] ­Ž…Ö&›‹¬³7ÄèòþOe`%CÛÞÚÚúF’Ï þüÒÙv9e}u}c›«ÊgÏ¢ëûÿUþ;ÏSzÖÔø°Q µ/µ¶¶ý¾ ìØt££á1:­ý9èîkýÕӿȺӻ Å•¥ïæÁrRœÀ(æZ[(X ÀÐøbŠŠÏ,1,úÎ/¯.ûy½ü²à;rä{ÊŸ9¨³Æ Ô•˜Ã»bÀ€Ž]æîÆpái^“…݇v÷¶V>þ³ôãK×dC¨cžqùïÓÚÚöÿ$shg&}7‡8^»Z ;ÎF¿ÝÓX ÀаŠij±îDlcø­–––ÿU|^9~â$«E÷Ãzá™W³Ì¢cŸ¼;Üù&và€ÖÊêÐC.bA©Ë?ú¹ôó«¿Û”wO]ï.º;=cúe;-ºÕÒ¹/ß㕺.é½Õ“c®òÖÄð`%C3‰‘]15+æþÆ–Hï”>Ï=_:)?+ÿz§ lhtßÖ·Ÿx)½|öœtРAÕÁw–§6t¯Qqf°½}ÐIÕbU1Ü÷c7¯ÎºBî¾Í%¾á ßÈþ~;…áÎm bÞð”~LçåøOo`%CÚ–¼ÛÝýUÒõ"¤¯åŸ÷j]]q¹Ñ£Ü¾háÒlè­@Ú»‹[}ðÜ vÌñÍ?3þKÒ¹º³ÑzÐfT®‡‹½â¢b(ïEsfó]cÕda¶{ÃpÌ—.½m8€3zñb8#éºÃCžÌAV W¾nAñj¥z(uqwS¿Nö¼Rpìq~ò<8pàö«>47}â›Ï ¨=¼—ï™S?Pýx½,øB÷Š¡/ ÚÚÚ_/l±gn„4µgë¾o½‘-¤5zì{Ëgøþ)é½/s…'ä×+±é9=/Ûú?ò3˜êóš9ûÊâõsŽC²…7ßÙMøŸMÛ‡ÛW©•Ï&¿-ncÒ “ÓÅ‹nMŸ]¿Qhí†úúšÇÒkç}2[ˆ¬ô8ʼn‡!8ÕSº9ø¶¶¶í˜ÛÃ,¢+iNoßTìE|Ú´Kv wiiiýEÞiÛÛÎíÚ<˜ õô¦‡Å™èôâËç`ªÏkÊg§94!³b7xî~ÞæÈü3É[åÛ;å¤Ó³m”ì'¼o[Ýû•³¹½‡Qýø¼™–åi Ý'–½_R¾1·7†9÷÷=y›¥âq8ÿòù逃¶wáì¬k¬  YŜɞÆô±X/=ý¬s0ÕçuÔ¸cŠ޶qƒN®Ô¶á·;ÖiÏßbQ­£ÛZÛ¶ŸqúÙÙ|aäw­8ApÛ­wfózRý¸ÄV”«»q@·‹꺖–l£ìì€;bÌ1ÙðÛfݶ¨n‚ðÎó„»ë ĸx.v4xÛ÷~ò–¦ú¬þæþ%-(4ýƒf]Úÿ‘Ôîü®ë¡ûŒ“òÑÞX¾¿ÃFŒÜ6{Ö•é—–ÝÓ´ÝáXÁ9ND—¼¼˜U^o'û÷Îz¡gœUÞã-¶/Šù½µª¯ Ü>`ÇøqÆu¦§5},û°sïÚÇ1Õgµäö»{úÃ=Ô‹K“ÎNb±H u.Äzµ—NÅÖŽó+µ>ÉÏ*êØ ³!¿Ÿ»iYúÈÚ§n{¥èzßùåÕéõ×Þuy«æóµ)éìœÏtÂzΨüìRvà<ì°ôãKוu¼¯ð)g_T~!Õ‡{šÓGb$B:gÞ|ALõY3ãÂâõÐç4«ê®ïãy “òÐÕóIc=’9ùçз«Ãà€¶0ñÄw®¾j^6‡8‚zá™Wû}Ðßqõ_~#ëìF‡;öK.ïÑ[Uq"bmþú4ÎSzÞ¼òpçX\ÉÞ½Qѽ9jìöD7˜¾{W§C† }gÃÿü©0¦z½Ömø‡´­½½ØOÝ‚14£Z]ßjýáDy{ƯH:×3‰¾¹Vh<¨ã mǽ÷}ÛfžIÖMy³±5PtX{zÕéªýÌSßËöàûŒ ûç7/Oç^}mzÚ©gÖZ°ª\[’ÎíŠîÊï$OOè=ÃËgG>&]pË×ÇÜKø’/J[[ÛŠ ¬L_ˆa§ºÀª¯»¿+Š4™Ýu}ëí3kló3?ŽÏ'U«LתaÃýÝÑcÇÿþä?>uË3f½Ø¢"4âcŸÞ¥æ]s}öóÖf£ÆŒ>*«ós÷T±Rs ó^™ÿîqBØfè#1×÷§Åú ®Îö™·b˪#ÆŒ× ¦¯Äâ'[ ¸í±ç~ ”©^«ÿöÈ·‹Å¯~“èþÒ\ö¦ë[ïFåŸg&«#GØÜtÎeÞ¼7!ù+îcc~¿qÿ‹ò¿{¼çÙjú‘%ù0°¬ë{㛤bï3¦_V~á^åš^Ï·ô¨ñ¸õ…Mÿ*œ©¯§þîŸÒÃ?¢x½[ì¤I4J×·;E×u\Ò¹Ú´¼æä'"´.©Q‹òŸuE~½ ùߨ1¯bÇBWç;N×·I룋îN;Ú±·ÜóB0½$žgÙŠÐgŸ;c»m‘TOVœdy߉'¯sO&¶¡94C×`¯?xÆ€,øÄâH‚`s×mkþ>6bGg$† Ho—tΉÊB°N°ê©=ÿèäSËÛ‰X÷€F§ë PÁ&V›ËÏÍ_Y/ª¬þbí+éè±ï-Þ,_«ÔD‡ ½à¬"ÿ‡ï˜¬º³¾öijåaÏ^×hº¾Uá7Î~§‡::½ýÁï ~j§ºûѧã'ž\^­p‚Æ^0.ɇCÇÂX±:´-’ÔÔÿõݦ_>'Ûî(=‹EiÌÓ£‘éúT‰aÏ/Æ‹btù"è|ª«Å±Ž;ñŒòpAé­×¨Xkk<÷† =8 Â÷®}<5?XíMýÝk¿LïüêÃé¿¿jn:hPG±Úóo“ÎýCÍù¥‘éúTiÏÏf+= ¿jO ¢•†C?ïÃ#½(¶‹X—”¶–8hðàí§ŸuNÖÑûÏŸüLúÜp“RY}äúÿ’=/N:õŒt`zóŠ)±Ï¹Q,42]_€.dÛ rÈvÞÕÞÖŠ‡~X^k­Ãˆ^6¥R·$ùÐh¥ö²â„]lqd®/N×  ó“l^Ý í¼RûZ·Ü÷\:¨cÈöü vÉ>2®RÓóxn–(•ì¼7ç´Äêõ4]_€=|h|;^ c¯WNíO}|éšâMöíü9ô>]_€=Ø/’'y §¨N›vI‚78¬ Wéúì…lèóÐCFl_ùøÏ„8u@Ï¡ÁC‡o¼ó^Ð+t}öBœ)̆>/¸åkœê–ºî¦{ÓÒðp‡ôèg9]_€½ ƒ¤ÇŸt¶à¦ºµ&œpZñF¼Ða=B×`tä/–éâ;žÚT·Ö _øF€ßLì ÝI×`?ÄV5éø‰' lªGjôØ÷oÌÎH@÷ÐõØO¯Ç‹gtê„5ÕÃs7:Üà€èú€ÉñâyÈ¡‡oÔTOÕýó«tÀÀŽâz”Ãö‹®/ÀZ/¢¸àjAMõhxúy†AÀþÑõè&â…ôãK×4MÛSà…ÀœèþúŸ¼½xÓ~Àa{M× › ­Ô–ÖÖ¶í÷}ë XîѺýÁï—WƒvO× ›eóÇNø£íÍÄྫƒ‡V¼‰tø@—t}zÀôxa4åœTÞs^ñÐÓ¹7|9ý‘›²Uï|äŸk^ÿîGœ~tÑÝÙå>vóêtåã?ÛéçÑmûˆÿ.]ùLzùG?—Þü•õͶÒD‡ìB× ÍMš°Ó¹?øŠëþkÚÖÖž­d|ø¨£³ŸÇ×ÿéÆ;vºn|]¬v\\®ã !é‚[¾¶ËíG@ŽÛ(.+%7úßþø“Î.ÞЧ9ü`'º¾=,[ú¢9 àÝà¿XûJöõûÿôÊ!5º¿G{bv‹ùÓŸùÒ£ÙåbµãèÝàãN<# ºŸ_ýÝn?®ûÏ®xb§€ÜÈÃü}ŽÃ2º¾½dI’w"à®pñutË—‹áË1Ô¹Àt£“[½ X„àÀþÊ·ÝÐf›<íÂkŠ7÷y?ÐõèM‹ã÷’/€w€#Ð:, ±'yA6Ì9ºÂåëDg8®]á¸~u8|LzĘcvºýfë¼GÅß4±0èúô¦AvOxæ• vêöF‡·ôF•…Ýè d•ÖUUÏn¶l0èúô•I>gµÙð„Nëòç_º¦æçbîo¬Ùc¸s\.æïÆJÏE°Ž0ÜU5{>rüûŠ<Éá@“ÑõècSâxüÄ“›*„ÅV<åÅ«ºš§‹ZÅ×±xU /¶**¶.ŠËÅåãëa#ŽÈV~^³á×»Üflu["5{Ž¿Qþ¦?Êá@Ñõè†Wj뀃¶ÿÕÓ¿hša4É÷?®µÑݹ¾1g·Xñ¹èWÏ•¾ñ‹ïdcNoR£ss†ãûç_>¿©p1L¼¥¥å×=š„®/@?ób¼ G˜k– Á6Âo’ïåÃr£  ]Å÷â¿ÑÝ-.Ýbîj\/‚ìiÓ.Ù±'p ŽËÅI„bžp ±>oöuÙðòø:ºÎEǹYpq" ÿ N× ʶBаÖLa,BmtvϘ~Y~c…æ·µ†:¡9:»hcñ«â²Eø-ßîܾœ].‚u\.®Wî°ßþà÷³…ÇnøÂ7šêo' ò×9ìh`º¾ýØYñâ|ĘñÛ›m>ªêÝE‡½Rv4(]_€~®½x¡.ûUª;+ºÝù‡Í9®/@Y/Ö§œ}‘°¦zzÿß7Œ®/@‰-i¶Ä wlù#°©î¬¥+Ÿ)¯þ<Üá@ƒÐõ¨c+ãÅ;gÚTwVŒ,È?¬p˜Ð t}ê\,L´5¶ÒVÝÙýçT<·gŨº¾ ä®x1m~bÕ^Nè~˱ÿqþa¹Ã €:§ë Ð`†&«ô¦³?r“§¨â9”tÎýýiå¿/ê”®/@›/î†B«©xîäCŸÓü9õH×  dC¡sLz÷£?èÔ>Õ}ßz£<ôù~‡uH× ‰ÄPèñ‚?á„ÓÒ¿zú‚Ú«Š¹ã“¦œS|XØ”?— žèú4¡ØøÍxñ?mÚ%ÅR{UÓ.¼¦¿ñÜ™à0 Žèú4¹I•z;Þο|¾€§v[W\÷_‹ [*u–À:¢ë @fzÒ¹‡kzÞìët‚ÕžÂoÔ‡ uB×€]\Z„`áUuÅè€Ò‡êèó®/5ÅÖl8ôq'žaa,•)ÍùÝš€þN×€½2%ÉÆ?ñäô/Ö¾"6i­|ügéIg^}phiiùð @Ðõ`ŸL¬ÔkñÆ1¨cÈö/]#6YÝøÅ‡Óƒ‡Vœ5Q¼ ¿Óõ`¿ÅÞ®+‹7‘é³>šÞÿ7¿¼bØû.¸º<ß7öжÕý®/Ýbf¥^7”±8iÛmkþ^PlÐZ|Ç“éᣎ.‡ß8Òá Óõ Û ¯ÔýñÆÒÚÚ¶ý¢9 -Õ@ý™W.(ßòìÌ9ý®/=jG7ø°#ŽÚvþ!@Öy}~õwÓÑcß[¿Ï'sÀ ¿Òõ ×D7øâM'V ¶Rt}v}g䦬£Ÿ¼{æ|ž§7ýœ®/}ûÔÖÖž-œ$÷ÿºï[odÁwØ¡ï)‚ošŸÐé) @?¦ë @¿x3Š…’¶$ùüàX-úîG,lö³ŠÇäüË秃‡R¾±ÂótOcú9]_ú•Q•º%SJÛ ÜaKG¸ï+ƒèÎØQ½µÑ,O[ú9]_úµØ;xA’/”uâiÜ~ãN×løµ@ÚË‹[röEÙðô҇גγæížªôsº¾ÔXs’ÎNc¾FŽ»ý?ÌÿBºòñŸ ¨=¼—ï¤)ç”CoÔË‚/uB×€º6£RŠ7²mÎäÇn^m/ánª›¿²>½™ß3zü¶Ò†­•Z[©©ž‚Ô ]_Æä¤óŒîÖ"¤Å\á÷ÿé•iì'Ûò³{¿…Q +Ÿvá5éÐCF”µŠz³R‹“ÎyÙPt}hè7¹ÝÉ·‹7»ƒzgê¹³ßù袻ÓýPЭª:~ÝM÷fózu ©½q¶|UÒ¹¢³aÎÔ]_šF{ÚîªÔær¨uä¾sÞìë²Ng³•ŽœcÞôq'ž‘m1UzãäAìß;Kè éúÐôb˜t ß}±ö"ü?î¤íˆ£C+7ÚªÒ·­ùûtÁ-_K/ùð¢¬Ë;rÔÑ¿¯ ¼Q›ò“3+Õáé@Òõ€*qx^ÒyFxKu0°cë1ÇOÙû ÇÐà¥+Ÿ©‹¦ï~ôÇég¾ôhÖÙýy>öÄla°a·øP°6ÿ;Œó” ÎéúÀ^ˆn甤ó ñòJ=™t.ö´KhÔ1ä÷G¿÷·ž1ý²¬›«MÇÖ@Ña½ó‘îñyº19:ÔqŸtÿÓw¤Ò?éìtÈÁÃßé"è¦yÈíŠîÊï$; D×PœIžV©ù•Z™t~k7!3«ÁC‡m9jì¿?îä_Ÿ6íß¿] ËoŸuÁœmÑ‘Ð<û#7íR3¯\ul#XG˜:|ÔÑYµµµ§{ºÏdוš×ç¿óü<ÜÒ @£¾Wëú@—tΓ]T©ûó°{ÇÚÍ{’°â>6æ÷»*ÿ=âÌwÌqêá Ièú@?24Ë’ÎNò´üÍynZ—Ô¨E¥ËD]‘_/nc¤?)èúÐøt}hhº¾4<]_š®/ O×€†¦ë @ÃÓõ ¡éúÐðt}hhº¾4<]_š®/ O×€†¦ë @ÃÓõ ¡éúÐðt}hhº¾4<]_š®/ O×€†¦ë @ÃÓõ ¡éúÐðt}hhº¾4<]_ZèúNÍkw¦Ujr7ßoÜæDO€Æ×_º¾›óÚø7tóýÆmÞïiиúÛ\_€n×çú Àt›þ¼Âóþàá•Zÿ»6äÿ¯‡Ö¸þ¬J­­ÔúJ-Î/S+ÌŸ'+uK¥&tñ;]Q©òû^U©™5~¿%IçÜå™ùï·>¿ÍQž’ݯ¿¯ð¼?xR¥Þ¬Ô–<¬F¸}-¿ÜÆJµ—.»2ÿþkùåâz¯ÖÀSó¿ÓÖ<ü®Ëoÿí<@:ò ×ß”‡àÍù×”.7.ÿÞóùm®ËïKþ;X€  ›Ô˾¾›óyÿnª:?žÉIU·Uü{§ä_Ï(Ó"-ØûK¡vs~‚ ¼Úô‘¥ß¯èÚ.˯»¤t¹¸í»òïϯ À~§—.;9ÿÝ7xЏzÚ×ws)¤ï®6TýûæÕ¸­¹ùe‹Àù@þuõãÉUxfþõâ.þ–ñ³Eù×ñw}­Æå:òŸmª Àkk\¶èJOðTØ?õÒõ­Àû»V #ŽáÉ1÷wU)LÓnnû­R^œ_oZÓ",™ì~ñ¬ÇóŸ—ðüÝõ+PºÿáÉ»óçTýžåï•õ*Oc€Ýk„®ïàPtz_Ï¿ÿZþ·¸%Ùu5çø{mÉkSþßçóë–päWK·»©ôÿgUý>ËóŸ½_çí<<—ç—ƒúÖäÝÎqñoê© P[#u}«ý¥{¸L„üêù¹3òÀa4¶D*:´ÑmYuÙ£ K—mÏï³:ØßÛ]–ßVG¿Sl¥´(¿Í…ù}$5p,°5)ÿﲿU!±‘º¾Í €؃Fíú Àì ë+44]ßÆÐž‡àáþ»Òõ ¡éúÐðt}hhº¾{65éÜo€:¥ë»w6Wjƒ?@ýÑõÝ7Ë+µÈŸ ¾èúÐÐt}÷ß‚dçsóнuo©Ôúüo;#ÿyü]£kæï¯Ô”·9½Rwå×]Ÿÿ­Ëªº™Iç|ä%Içþ¾eÓ*µ*¿ß*u…‡h6º¾¦zpüÿÆJmªÔëù×oçßù•z³R¯VêùJm­Ô–JM*]U~ÙMyP]Ÿ_fk)D'ùu~™ÿìÉü~¶æ×KóÀ[Xžos~›/ç_Ghn÷N×·çpšÙöRX-þÎ «N>Ä÷–å_OÍ¿¾¿ê>&—£ÂÆIDAT¸£…0]ßæ!yC|«Åstk?zn@Ý‹aÌÑ™ZÄ+a÷·µBã Aƒv º÷~åÁ,€vg¨íîŠüä_'û]?wÓ²ôòÙsÒã'NJÛÚÚwù÷åÿîØ£õº¤³ÃÖÛt}›×æü¤Gt~‡æ{Ñý]åÏ&†ZÆÊXyöírŒp!qæù—¤7~êÏÒ{îX“…Èþt÷µ¢ãÅW_5/ ö5Bñ¦J-K:ç_¶÷ðã¡ëÛÜâÔkÉ®'žÖ&æõÀ~‡Þ˜‡¸1©1/7æá~iÙ=éKÏmj¨°»·µñ¥Ÿ¤«ÿòé¼k®ÏN$;w‡“‡‘¦£ºñ1Ñõ¥'Yb(þœJÍò<€n ½ÇŒ?6 {1O7† 7cèÝ]½üÂÒ·ß—Í)>â=°½êï«ú.ÉËþÒõ8@±`N,¨³!©šÃ‹C-^tkúí'^r÷±¾¾æ±tîÕצ£Ž]=L5N.D×no‡Iëú qIçœÕ7“ÒÐæX´*VeŽ!¾‚l†áø»ÇBbÃwóéú€iIgqG‹`ÞÆ+°öì*Ó±Âô9r^ì9\ “Þ’tî•<¡êqÒõ8€à»Ó0çX¼)²2§·÷ëÙõÓkç}2=lÄÈm¥Çd]þ8%y(Öõ8à{öûÏM¿ºêaA´Ÿt…o»õÎôè±Ç¼“ìúÐß žý´bå‰ÇP^AúõJÍôT¨-Vu¾%éœ[š<ôí±}Ñ Ï¼*dÖQ®Ú[ø¤s5hrÑ-Ü\§Xl)æš •õYŸ_º"6lx1G8Vƒ¾ÔShv±•ÎŽUƒc;£˜W*DÖÅÊÜW_5/mmm-‚p<Σ<å€ftV¢ëÛðõÄ7ŸMOü£“~›ØhR‹*µ5ÑõmšZ´pi¹l` áÅ‚H;VxÖõm®zdíÓé˜1Gý>üßN:·Lh81ÿsS’¯ð¬ëÛœµñ¥Ÿ¤_xYy¥èe  ‘LL:÷†M¬®¯Jÿüæåi[k[±wðý•jw˜õ.»z3‚Î)'n__µÓ¾Áƒ:ÞÉCp êpêÕÔ¤s®gzöûÏ͆¿ ~ª\__óX:ìáE~^êÑ„$ïüÎ8ï¢ô7¾!ð©šõä_'=tøˆ­y^— Ô‘áI¾àU {~ÕÞ„àÒpè•! D÷.†²f ^½üÂ<µ×áK c-v(ýÝŠ0#G¾'}æ©ï vjŸêÎ/¯.o‘4ÓáôW±èÕÖ¶¶ö¬›'Щý©ë¯½¡À±uÖp‡Ðßt$ù¼ßy×\/È©ý®˜3þÞ Ÿ¼»G0@¿²<ËÑcǧ¯¼üsANð¢XííŠE± …úØò(ÛÆæÕë8Õ-µháÒòPh[#ýÂ]Tfž‰à¦ºu(ôè?óÛ<Ïu˜}mdKKKR}èo7Õ­uÛ­w]àM5 ¯-‰€röûÏØTt‡ ;ôwyžåpúJ{KK˯sUÖâE·]àçr@_‰}Ó1£ÔTÕÆ—~’¶¶¶nËC°}€>‘ ¾êCs5Õ£uòŸº%ÀW8쀾CRÓ{îXÓ0Aë¿?ø­tö¬+ÓcÆ›¶µµ§ƒ JŸ8)½þÚÒ—žÛ´Óe_xæÕô´SÏÌjR{¶nüÔŸàïwؽmhKKË;cˆj#,¶tõUó²¡7õН/Ÿ=' ÀñýC‡HYûôŽë<óÔ÷²ïâcŸR{¸žøæ³å=zÕ”$“N˜Ü+Blü{¢£Ý꟯¸ý¾¬#|ôØñYX€{¿:hðïó€€>0#ií"ÈávwÝìy×\ŸÞbÈ÷îpÜNt‹cuìZººû—zååŸwy¹—_øQúõ5eÝЮ.×ý˜c(w#tæËõ£Æó€'8ü€Þ47ÂHÌ—­÷`õ©O|6 ²~óòÝ^îÙõÓ;¿¼: ¢]à»<÷‚,Pça-«É'ž’>ù×ßÙq¹ø»ÅêÙ¦GŽ|ÏŽËÅ0ë¸ê¹Æ3λh§ÛuÄèôÞ¯<¸Óå-\šrð°—‰¡Ü1Œ»èX7ÐBXÓ~@oZa$‡ª÷`uæÔdqOÚZãêCÂ#x~î¦eY§6ººEç8î§€ãrX#€G§øóKW¤ƒɾ_ü.^01 ¿p#D¯þËod‹tÅåŠP]„øsþ伬û÷}í¼Ofß»øÂË"ÿ»ógmO¬ ô%IƒÌNlü[j MŽ[]ÅjÐÕ8Bgtp#xVßVã·å×Ð[ÝÅïÇœãbîq|ß/_.†9' â÷‰0|ÊI§ïÒí-ö*/ÞU¯ ’åxžÃ€»1·ºŠaß{³Vҫѱ-ßG€cXuùòÑÝï[+—«Õ.‚x ™NºÂs†ãg±P½?NÑÉ΃9? 75ÌàèÎÆ¿¥˜Û[^t*ÂmQs¯¾v8®a3†"{ —Ãsu®þ]b(t9ǪԵ.׏]V#c¿ßXMºÖ>ÆÅ¿kOÝíþ^ß~â¥b8÷/z@¯8pÐj¾z¬X¬ªXd)Âjá—1ï7†!6:¹Æjàb«è$Çþ¾fcxr„Ób¯ß" îm.oÓa6n7:½q{QÅâX±V\&†`G:.WÜGüþõ¾pü›ò¼Ö‘ô…eIÞ…l„}f‹úÁs/ȶ,ÊWöÿñ½•åËF§8VŽ¡ÏÅ÷"ˆa·Ìq½/-»'»ìWW=¼£[_Wßtvãû1¬¹ø^„æ¸|l±TèE/*®süÄI;î;‚wÑ!®÷Ç%þ½ù¿k®Ãè Ó“¼ãÙ(¸z›¡b«¡}­˜sÛ]×½ù}â~÷÷÷îCìê8¨XúH‡Ð:Z[[Á$æ¹6bV}_Ñ=ÏÃïF‡ЗV$ ²Ï¬êŸUÖ}à èKZ[[·Å¼ÔZ«'+u ó§óðûf¥:n@Ÿjoo,BJ¬˜,´©î¬XÁ:À‹i@05ÉWKŽ­7ÕUì\©·’Î}§ú^{{û7“|ÏYáMhÅ*Ú£Ž]à…Ž0 ?ÕÖÖþv–Ûn½SˆST±¨Z~_Œó+/ ¿™¡åƒ‡YKuÇÐç-•šè°ú¥öööç"¼œrÒéé?n|C SûTOþõwÒC‡0ô¨ G¶µµÿ2±7°ÚÇzá™Wӣǎ/Â¡Ï@˜ÚÖÚö¿#È|êŸîÔkãK?ÉF äá÷ÕJ uõâÒ<̤+n¿OÈS]V •ŸqÞEEø}³R£>@½Y¡¦­­=ýÒ²{„=U³ó[„ß–––ßTþ;ÉaÔ«IÞ ^¼èV¡OÕöœ‡ß³.@½[P„àO|ìӟʼ:fü±åa϶;Æü–––w"ð\>{NúÊË?›´þûƒß*¯ö¼©R@£¹´µµõw|&09}æ©ï „MTqÒcÞ5×gsÂóðûrbÁ+ M®„àD:äàaé=w¬›¤ë[òµ²R ÑµWjY1$zîÕ×ÝÀ[];ï“;º¾•Çüוÿ^ášÍÔöÖöE0Š9¡_]õ°ÐØ@õèC›?qR¹ëCžÇyÚÍjh¥î*BÒÅ^–¾ôÜ&²Î»¾±Úw©ëûÛÊoI:;ÿMozkkëI>7øóKWdAJ ¬¯zdíÓÕ]ßõ‰-Žv1¼Rá)†EßùåÕ‚eT _?sêvß–––MÌõØ£K+êa*:ŠV‹îŸCWÜ~ßNß|¸óòüd{iV¥ž/ÂUìAØÐè¾ßÏwñ¢[³}Uð]‘Ø×à€L«Ôº"l}äöE —¦/¿ð#´+þÞ±¸ÕÈ‘ï|zØäJÝ_ìöé]jÞ5×g?¿øÂ˲톢ƌ>*«ós÷Toæ¿k„Ýèت€ý0ªR3+µ0™1§xC¥^M:;Ç{ ÉXqóûŒ9»‹+uiÒ¹à—UšèuÑuW© •š–WÌ;ž[©EIçêÔÕµ(ÿyQWä׋۰0@ùÿðwª«‹qXIEND®B`‚sahara-8.0.0/doc/source/index.rst0000666000175100017510000000165413245514472016727 0ustar zuulzuul00000000000000Welcome to Sahara! ================== The sahara project aims to provide users with a simple means to provision data processing frameworks (such as Apache Hadoop, Apache Spark and Apache Storm) on OpenStack. This is accomplished by specifying configuration parameters such as the framework version, cluster topology, node hardware details and more. Overview -------- .. toctree:: :maxdepth: 2 intro/index Installation ------------ .. toctree:: :maxdepth: 2 install/index Configuration ------------- .. toctree:: :maxdepth: 2 configuration/index User Guide ---------- .. toctree:: :maxdepth: 2 user/index Operator Documentation ---------------------- .. toctree:: :maxdepth: 2 admin/index Contributor Documentation ------------------------- .. toctree:: :maxdepth: 2 contributor/index Programming Reference --------------------- .. toctree:: :maxdepth: 2 reference/index sahara-8.0.0/doc/source/install/0000775000175100017510000000000013245515026016520 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/install/dashboard-guide.rst0000666000175100017510000000510513245514472022303 0ustar zuulzuul00000000000000Sahara Dashboard Configuration Guide ==================================== After installing the Sahara dashboard, there are a few extra configurations that can be made. Dashboard configurations are applied through Horizon's local_settings.py file. The sample configuration file is available `from the Horizon repository. `_ 1. Networking ------------- Depending on the Networking backend (Neutron) used in the cloud, Sahara panels will determine automatically which input fields should be displayed. If you wish to disable floating IP options during node group template creation, add the following parameter: Example: .. sourcecode:: python SAHARA_FLOATING_IP_DISABLED = True .. 2. Different endpoint --------------------- Sahara UI panels normally use ``data-processing`` endpoint from Keystone to talk to Sahara service. In some cases it may be useful to switch to another endpoint, for example use locally installed Sahara instead of the one on the OpenStack controller. To switch the UI to another endpoint the endpoint should be registered in the first place. Local endpoint example: .. sourcecode:: console $ openstack service create --name sahara_local --description \ "Sahara Data Processing (local installation)" \ data_processing_local $ openstack endpoint create --region RegionOne \ --publicurl http://127.0.0.1:8386/v1.1/%\(project_id\)s \ --adminurl http://127.0.0.1:8386/v1.1/%\(project_id\)s \ --internalurl http://127.0.0.1:8386/v1.1/%\(project_id\)s \ data_processing_local .. Then the endpoint name should be changed in ``sahara.py`` under the module of `sahara-dashboard/sahara_dashboard/api/sahara.py `__. .. sourcecode:: python # "type" of Sahara service registered in keystone SAHARA_SERVICE = 'data_processing_local' 3. Hiding health check info --------------------------- Sahara UI panels normally contain some information about cluster health. If the relevant functionality has been disabled in the Sahara service, then operators may prefer to not have any references to health at all in the UI, since there would not be any usable health information in that case. The visibility of health check info can be toggled via the ``SAHARA_VERIFICATION_DISABLED`` parameter, whose default value is False, meaning that the health check info will be visible. Example: .. sourcecode:: python SAHARA_VERIFICATION_DISABLED = True .. sahara-8.0.0/doc/source/install/installation-guide.rst0000666000175100017510000002030713245514472023056 0ustar zuulzuul00000000000000Sahara Installation Guide ========================= We recommend installing sahara in a way that will keep your system in a consistent state. We suggest the following options: * Install via `Fuel `_ * Install via `Kolla `_ * Install via `RDO `_ * Install into a virtual environment To install with Fuel -------------------- 1. Start by following the `MOS Quickstart `_ to install and setup OpenStack. 2. Enable the sahara service during installation. To install with Kolla --------------------- 1. Start by following the `Kolla Quickstart `_ to install and setup OpenStack. 2. Enable the sahara service during installation. To install with RDO ------------------- 1. Start by following the `RDO Quickstart `_ to install and setup OpenStack. 2. Install sahara: .. sourcecode:: console # yum install openstack-sahara .. 3. Configure sahara as needed. The configuration file is located in ``/etc/sahara/sahara.conf``. For details see :doc:`Sahara Configuration Guide <../admin/configuration-guide>` 4. Create the database schema: .. sourcecode:: console # sahara-db-manage --config-file /etc/sahara/sahara.conf upgrade head .. 5. Go through :ref:`common_installation_steps` and make any necessary changes. 6. Start the sahara-api and sahara-engine services: .. sourcecode:: console # systemctl start openstack-sahara-api # systemctl start openstack-sahara-engine .. 7. *(Optional)* Enable sahara services to start on boot .. sourcecode:: console # systemctl enable openstack-sahara-api # systemctl enable openstack-sahara-engine .. To install into a virtual environment ------------------------------------- 1. First you need to install a number of packages with your OS package manager. The list of packages depends on the OS you use. For Ubuntu run: .. sourcecode:: console $ sudo apt-get install python-setuptools python-virtualenv python-dev .. For Fedora: .. sourcecode:: console $ sudo yum install gcc python-setuptools python-virtualenv python-devel .. For CentOS: .. sourcecode:: console $ sudo yum install gcc python-setuptools python-devel $ sudo easy_install pip $ sudo pip install virtualenv 2. Setup a virtual environment for sahara: .. sourcecode:: console $ virtualenv sahara-venv .. This will install a python virtual environment into ``sahara-venv`` directory in your current working directory. This command does not require super user privileges and can be executed in any directory where the current user has write permissions. 3. You can get a sahara archive from ``_ and install it using pip: .. sourcecode:: console $ sahara-venv/bin/pip install 'http://tarballs.openstack.org/sahara/sahara-master.tar.gz' .. Note that ``sahara-master.tar.gz`` contains the latest changes and might not be stable at the moment. We recommend browsing ``_ and selecting the latest stable release. For installation just execute (where replace the 'release' word with release name, e.g. 'mitaka'): .. sourcecode:: console $ sahara-venv/bin/pip install 'http://tarballs.openstack.org/sahara/sahara-stable-release.tar.gz' .. For example, you can get Sahara Mitaka release by executing: .. sourcecode:: console $ sahara-venv/bin/pip install 'http://tarballs.openstack.org/sahara/sahara-stable-mitaka.tar.gz' .. 4. After installation you should create a configuration file from the sample file located in ``sahara-venv/share/sahara/sahara.conf.sample-basic``: .. sourcecode:: console $ mkdir sahara-venv/etc $ cp sahara-venv/share/sahara/sahara.conf.sample-basic sahara-venv/etc/sahara.conf .. Make any necessary changes to ``sahara-venv/etc/sahara.conf``. For details see :doc:`Sahara Configuration Guide <../admin/configuration-guide>` .. _common_installation_steps: Common installation steps ------------------------- The steps below are common to both the RDO and virtual environment installations of sahara. 1. If you use sahara with a MySQL database, then for storing big job binaries in the sahara internal database you must configure the size of the maximum allowed packet. Edit the ``my.cnf`` file and change the ``max_allowed_packet`` parameter as follows: .. sourcecode:: ini ... [mysqld] ... max_allowed_packet = 256M .. Then restart the mysql server to ensure these changes are active. 2. Create the database schema: .. sourcecode:: console $ sahara-venv/bin/sahara-db-manage --config-file sahara-venv/etc/sahara.conf upgrade head .. 3. Start sahara services from different terminals: .. sourcecode:: console # first terminal $ sahara-venv/bin/sahara-api --config-file sahara-venv/etc/sahara.conf # second terminal $ sahara-venv/bin/sahara-engine --config-file sahara-venv/etc/sahara.conf .. .. _register-sahara-label: 4. For sahara to be accessible in the OpenStack Dashboard and for python-saharaclient to work properly you must register sahara in the Identity service catalog. For example: .. sourcecode:: console $ openstack service create --name sahara --description \ "Sahara Data Processing" data-processing $ openstack endpoint create --region RegionOne \ --publicurl http://10.0.0.2:8386/v1.1/%\(project_id\)s \ --adminurl http://10.0.0.2:8386/v1.1/%\(project_id\)s \ --internalurl http://10.0.0.2:8386/v1.1/%\(project_id\)s \ data-processing .. note:: You have to install the openstack-client package in order to execute ``openstack`` command. .. 5. For more information on configuring sahara with the OpenStack Dashboard please see :doc:`dashboard-guide`. Optional installation of default templates ------------------------------------------ Sahara bundles default templates that define simple clusters for the supported plugins. These templates may optionally be added to the sahara database using a simple CLI included with sahara. The default template CLI is described in detail in a *README* file included with the sahara sources at ``/db/templates/README.rst`` but it is summarized here. Flavor id values must be specified for the default templates included with sahara. The recommended configuration values below correspond to the *m1.medium* and *m1.large* flavors in a default OpenStack installation (if these flavors have been edited, their corresponding values will be different). Values for flavor_id should be added to ``/etc/sahara/sahara.conf`` or another configuration file in the sections shown here: .. sourcecode:: ini [DEFAULT] # Use m1.medium for {flavor_id} unless specified in another section flavor_id = 2 [cdh-5-default-namenode] # Use m1.large for {flavor_id} in the cdh-5-default-namenode template flavor_id = 4 [cdh-530-default-namenode] # Use m1.large for {flavor_id} in the cdh-530-default-namenode template flavor_id = 4 The above configuration values are included in a sample configuration file at ``/plugins/default_templates/template.conf`` The command to install all of the default templates is as follows, where ``$PROJECT_ID`` should be a valid project id and the above configuration values have been set in ``myconfig``: .. sourcecode:: console $ sahara-templates --config-file /etc/sahara/sahara.conf --config-file myconfig update -t $PROJECT_ID Help is available from the ``sahara-templates`` command: .. sourcecode:: console $ sahara-templates --help $ sahara-templates update --help Notes: ------ Ensure that your operating system is not blocking the sahara port (default: 8386). You may need to configure iptables in CentOS and other Linux distributions to allow this access. To get the list of all possible options run: .. sourcecode:: console $ sahara-venv/bin/python sahara-venv/bin/sahara-api --help $ sahara-venv/bin/python sahara-venv/bin/sahara-engine --help .. Further, consider reading :doc:`../intro/overview` for general sahara concepts and :doc:`../user/plugins` for specific plugin features/requirements. sahara-8.0.0/doc/source/install/index.rst0000666000175100017510000000020113245514472020360 0ustar zuulzuul00000000000000================== Installation Guide ================== .. toctree:: :maxdepth: 2 installation-guide dashboard-guide sahara-8.0.0/doc/source/admin/0000775000175100017510000000000013245515026016142 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/admin/advanced-configuration-guide.rst0000666000175100017510000006573313245514472024425 0ustar zuulzuul00000000000000Sahara Advanced Configuration Guide =================================== This guide addresses specific aspects of Sahara configuration that pertain to advanced usage. It is divided into sections about various features that can be utilized, and their related configurations. .. _custom_network_topologies: Custom network topologies ------------------------- Sahara accesses instances at several stages of cluster spawning through SSH and HTTP. Floating IPs and network namespaces will be automatically used for access when present. When floating IPs are not assigned to instances and namespaces are not being used, sahara will need an alternative method to reach them. The ``proxy_command`` parameter of the configuration file can be used to give sahara a command to access instances. This command is run on the sahara host and must open a netcat socket to the instance destination port. The ``{host}`` and ``{port}`` keywords should be used to describe the destination, they will be substituted at runtime. Other keywords that can be used are: ``{tenant_id}``, ``{network_id}`` and ``{router_id}``. Additionally, if ``proxy_command_use_internal_ip`` is set to ``True``, then the internal IP will be subsituted for ``{host}`` in the command. Otherwise (if ``False``, by default) the management IP will be used: this corresponds to floating IP if present in the relevant node group, else the internal IP. The option is ignored if ``proxy_command`` is not also set. For example, the following parameter in the sahara configuration file would be used if instances are accessed through a relay machine: .. sourcecode:: cfg [DEFAULT] proxy_command='ssh relay-machine-{tenant_id} nc {host} {port}' Whereas the following shows an example of accessing instances though a custom network namespace: .. sourcecode:: cfg [DEFAULT] proxy_command='ip netns exec ns_for_{network_id} nc {host} {port}' .. _dns_hostname_resolution: DNS Hostname Resolution ----------------------- Sahara can resolve hostnames of cluster instances by using DNS. For this Sahara uses Designate. With this feature, for each instance of the cluster Sahara will create two ``A`` records (for internal and external ips) under one hostname and one ``PTR`` record. Also all links in the Sahara dashboard will be displayed as hostnames instead of just ip addresses. You should configure DNS server with Designate. Designate service should be properly installed and registered in Keystone catalog. The detailed instructions about Designate configuration can be found here: `Designate manual installation`_ and here: `Configuring OpenStack Networking with Designate`_. Also if you use devstack you can just enable Designate plugin: `Designate devstack`_. When Designate is configured you should create domain(s) for hostname resolution. This can be done by using the Designate dashboard or by CLI. Also you have to create ``in-addr.arpa.`` domain for reverse hostname resolution because some plugins (e.g. ``HDP``) determine hostname by ip. Sahara also should be properly configured. In ``sahara.conf`` you must specify two config properties: .. sourcecode:: cfg [DEFAULT] # Use Designate for internal and external hostnames resolution: use_designate=true # IP addresses of Designate nameservers: nameservers=1.1.1.1,2.2.2.2 An OpenStack operator should properly configure the network. It must enable DHCP and specify DNS server ip addresses (e.g. 1.1.1.1 and 2.2.2.2) in ``DNS Name Servers`` field in the ``Subnet Details``. If the subnet already exists and changing it or creating new one is impossible then Sahara will manually change ``/etc/resolv.conf`` file on every instance of the cluster (if ``nameservers`` list has been specified in ``sahara.conf``). In this case, though, Sahara cannot guarantee that these changes will not be overwritten by DHCP or other services of the existing network. Sahara has a health check for track this situation (and if it occurs the health status will be red). In order to resolve hostnames from your local machine you should properly change your ``/etc/resolv.conf`` file by adding appropriate ip addresses of DNS servers (e.g. 1.1.1.1 and 2.2.2.2). Also the VMs with DNS servers should be available from your local machine. .. _Designate manual installation: https://docs.openstack.org/designate/latest/install/index.html .. _Configuring OpenStack Networking with Designate: https://docs.openstack.org/neutron/latest/admin/config-dns-int.html#configuring-openstack-networking-for-integration-with-an-external-dns-service .. _Designate devstack: https://docs.openstack.org/designate/latest/contributor/devstack.html .. _data_locality_configuration: Data-locality configuration --------------------------- Hadoop provides the data-locality feature to enable task tracker and data nodes the capability of spawning on the same rack, Compute node, or virtual machine. Sahara exposes this functionality to the user through a few configuration parameters and user defined topology files. To enable data-locality, set the ``enable_data_locality`` parameter to ``true`` in the sahara configuration file .. sourcecode:: cfg [DEFAULT] enable_data_locality=true With data locality enabled, you must now specify the topology files for the Compute and Object Storage services. These files are specified in the sahara configuration file as follows: .. sourcecode:: cfg [DEFAULT] compute_topology_file=/etc/sahara/compute.topology swift_topology_file=/etc/sahara/swift.topology The ``compute_topology_file`` should contain mappings between Compute nodes and racks in the following format: .. sourcecode:: cfg compute1 /rack1 compute2 /rack2 compute3 /rack2 Note that the Compute node names must be exactly the same as configured in OpenStack (``host`` column in admin list for instances). The ``swift_topology_file`` should contain mappings between Object Storage nodes and racks in the following format: .. sourcecode:: cfg node1 /rack1 node2 /rack2 node3 /rack2 Note that the Object Storage node names must be exactly the same as configured in the object ring. Also, you should ensure that instances with the task tracker process have direct access to the Object Storage nodes. Hadoop versions after 1.2.0 support four-layer topology (for more detail please see `HADOOP-8468 JIRA issue`_). To enable this feature set the ``enable_hypervisor_awareness`` parameter to ``true`` in the configuration file. In this case sahara will add the Compute node ID as a second level of topology for virtual machines. .. _HADOOP-8468 JIRA issue: https://issues.apache.org/jira/browse/HADOOP-8468 .. _distributed-mode-configuration: Distributed mode configuration ------------------------------ Sahara can be configured to run in a distributed mode that creates a separation between the API and engine processes. This allows the API process to remain relatively free to handle requests while offloading intensive tasks to the engine processes. The ``sahara-api`` application works as a front-end and serves user requests. It offloads 'heavy' tasks to the ``sahara-engine`` process via RPC mechanisms. While the ``sahara-engine`` process could be loaded with tasks, ``sahara-api`` stays free and hence may quickly respond to user queries. If sahara runs on several hosts, the API requests could be balanced between several ``sahara-api`` hosts using a load balancer. It is not required to balance load between different ``sahara-engine`` hosts as this will be automatically done via the message broker. If a single host becomes unavailable, other hosts will continue serving user requests. Hence, a better scalability is achieved and some fault tolerance as well. Note that distributed mode is not a true high availability. While the failure of a single host does not affect the work of the others, all of the operations running on the failed host will stop. For example, if a cluster scaling is interrupted, the cluster will be stuck in a half-scaled state. The cluster might continue working, but it will be impossible to scale it further or run jobs on it via EDP. To run sahara in distributed mode pick several hosts on which you want to run sahara services and follow these steps: * On each host install and configure sahara using the `installation guide <../install/installation-guide.html>`_ except: * Do not run ``sahara-db-manage`` or launch sahara with ``sahara-all`` * Ensure that each configuration file provides a database connection string to a single database for all hosts. * Run ``sahara-db-manage`` as described in the installation guide, but only on a single (arbitrarily picked) host. * The ``sahara-api`` and ``sahara-engine`` processes use oslo.messaging to communicate with each other. You will need to configure it properly on each host (see below). * Run ``sahara-api`` and ``sahara-engine`` on the desired hosts. You may run both processes on the same or separate hosts as long as they are configured to use the same message broker and database. To configure oslo.messaging, first you will need to choose a message broker driver. Currently there are two drivers provided: RabbitMQ or ZeroMQ. For the RabbitMQ drivers please see the :ref:`notification-configuration` documentation for an explanation of common configuration options. For an expanded view of all the options provided by each message broker driver in oslo.messaging please refer to the options available in the respective source trees: * For Rabbit MQ see * rabbit_opts variable in `impl_rabbit.py `_ * amqp_opts variable in `amqp.py `_ * For Zmq see * zmq_opts variable in `impl_zmq.py `_ * matchmaker_opts variable in `matchmaker.py `_ * matchmaker_redis_opts variable in `matchmaker_redis.py `_ * matchmaker_opts variable in `matchmaker_ring.py `_ These options will also be present in the generated sample configuration file. For instructions on creating the configuration file please see the :doc:`configuration-guide`. .. _distributed-periodic-tasks: Distributed periodic tasks configuration ---------------------------------------- If sahara is configured to run in distributed mode (see :ref:`distributed-mode-configuration`), periodic tasks can also be launched in distributed mode. In this case tasks will be split across all ``sahara-engine`` processes. This will reduce overall load. Distributed periodic tasks are based on Hash Ring implementation and the Tooz library that provides group membership support for a set of backends. In order to use periodic tasks distribution, the following steps are required: * One of the `supported backends `_ should be configured and started. * Backend URL should be set in the sahara configuration file with the ``periodic_coordinator_backend_url`` parameter. For example, if the ZooKeeper backend is being used: .. sourcecode:: cfg [DEFAULT] periodic_coordinator_backend_url=kazoo://IP:PORT * Tooz extras should be installed. When using Zookeeper as coordination backend, ``kazoo`` library should be installed. It can be done with pip: .. sourcecode:: console pip install tooz[zookeeper] * Periodic tasks can be performed in parallel. Number of threads to run periodic tasks on a single engine can be set with ``periodic_workers_number`` parameter (only 1 thread will be launched by default). Example: .. sourcecode:: cfg [DEFAULT] periodic_workers_number=2 * ``coordinator_heartbeat_interval`` can be set to change the interval between heartbeat execution (1 second by default). Heartbeats are needed to make sure that connection to the coordination backend is active. Example: .. sourcecode:: cfg [DEFAULT] coordinator_heartbeat_interval=2 * ``hash_ring_replicas_count`` can be set to change the number of replicas for each engine on a Hash Ring. Each replica is a point on a Hash Ring that belongs to a particular engine. A larger number of replicas leads to better task distribution across the set of engines. (40 by default). Example: .. sourcecode:: cfg [DEFAULT] hash_ring_replicas_count=100 .. _external_key_manager_usage: External key manager usage -------------------------- Sahara generates and stores several passwords during the course of operation. To harden sahara's usage of passwords it can be instructed to use an external key manager for storage and retrieval of these secrets. To enable this feature there must first be an OpenStack Key Manager service deployed within the stack. With a Key Manager service deployed on the stack, sahara must be configured to enable the external storage of secrets. Sahara uses the `castellan `_ library to interface with the OpenStack Key Manager service. This library provides configurable access to a key manager. To configure sahara to use barbican as the key manager, edit the sahara configuration file as follows: .. sourcecode:: cfg [DEFAULT] use_barbican_key_manager=true Enabling the ``use_barbican_key_manager`` option will configure castellan to use barbican as its key management implementation. By default it will attempt to find barbican in the Identity service's service catalog. For added control of the barbican server location, optional configuration values may be added to specify the URL for the barbican API server. .. sourcecode:: cfg [castellan] barbican_api_endpoint=http://{barbican controller IP:PORT}/ barbican_api_version=v1 The specific values for the barbican endpoint will be dictated by the IP address of the controller for your installation. With all of these values configured and the Key Manager service deployed, sahara will begin storing its secrets in the external manager. Indirect instance access through proxy nodes -------------------------------------------- .. warning:: The indirect VMs access feature is in alpha state. We do not recommend using it in a production environment. Sahara needs to access instances through SSH during cluster setup. This access can be obtained a number of different ways (see :ref:`floating_ip_management`,:ref:`custom_network_topologies`).Sometimes it is impossible to provide access to all nodes (because of limited numbers of floating IPs or security policies). In these cases access can be gained using other nodes of the cluster as proxy gateways. To enable this set ``is_proxy_gateway=true`` for the node group you want to use as proxy. Sahara will communicate with all other cluster instances through the instances of this node group. Note, if ``use_floating_ips=true`` and the cluster contains a node group with ``is_proxy_gateway=true``, the requirement to have ``floating_ip_pool`` specified is applied only to the proxy node group. Other instances will be accessed through proxy instances using the standard private network. Note, the Cloudera Hadoop plugin doesn't support access to Cloudera manager through a proxy node. This means that for CDH clusters only nodes with the Cloudera manager can be designated as proxy gateway nodes. Multi region deployment ----------------------- Sahara supports multi region deployment. To enable this option each instance of sahara should have the ``os_region_name=`` parameter set in the configuration file. The following example demonstrates configuring sahara to use the ``RegionOne`` region: .. sourcecode:: cfg [DEFAULT] os_region_name=RegionOne .. _non-root-users: Non-root users -------------- In cases where a proxy command is being used to access cluster instances (for example, when using namespaces or when specifying a custom proxy command), rootwrap functionality is provided to allow users other than ``root`` access to the needed operating system facilities. To use rootwrap the following configuration parameter is required to be set: .. sourcecode:: cfg [DEFAULT] use_rootwrap=true Assuming you elect to leverage the default rootwrap command (``sahara-rootwrap``), you will need to perform the following additional setup steps: * Copy the provided sudoers configuration file from the local project file ``etc/sudoers.d/sahara-rootwrap`` to the system specific location, usually ``/etc/sudoers.d``. This file is setup to allow a user named ``sahara`` access to the rootwrap script. It contains the following: .. sourcecode:: cfg sahara ALL = (root) NOPASSWD: /usr/bin/sahara-rootwrap /etc/sahara/rootwrap.conf * When using devstack to deploy sahara, please pay attention that you need to change user in script from ``sahara`` to ``stack``. * Copy the provided rootwrap configuration file from the local project file ``etc/sahara/rootwrap.conf`` to the system specific location, usually ``/etc/sahara``. This file contains the default configuration for rootwrap. * Copy the provided rootwrap filters file from the local project file ``etc/sahara/rootwrap.d/sahara.filters`` to the location specified in the rootwrap configuration file, usually ``/etc/sahara/rootwrap.d``. This file contains the filters that will allow the ``sahara`` user to access the ``ip netns exec``, ``nc``, and ``kill`` commands through the rootwrap (depending on ``proxy_command`` you may need to set additional filters). It should look similar to the followings: .. sourcecode:: cfg [Filters] ip: IpNetnsExecFilter, ip, root nc: CommandFilter, nc, root kill: CommandFilter, kill, root If you wish to use a rootwrap command other than ``sahara-rootwrap`` you can set the following parameter in your sahara configuration file: .. sourcecode:: cfg [DEFAULT] rootwrap_command='sudo sahara-rootwrap /etc/sahara/rootwrap.conf' For more information on rootwrap please refer to the `official Rootwrap documentation `_ Object Storage access using proxy users --------------------------------------- To improve security for clusters accessing files in Object Storage, sahara can be configured to use proxy users and delegated trusts for access. This behavior has been implemented to reduce the need for storing and distributing user credentials. The use of proxy users involves creating an Identity domain that will be designated as the home for these users. Proxy users will be created on demand by sahara and will only exist during a job execution which requires Object Storage access. The domain created for the proxy users must be backed by a driver that allows sahara's admin user to create new user accounts. This new domain should contain no roles, to limit the potential access of a proxy user. Once the domain has been created, sahara must be configured to use it by adding the domain name and any potential delegated roles that must be used for Object Storage access to the sahara configuration file. With the domain enabled in sahara, users will no longer be required to enter credentials for their data sources and job binaries referenced in Object Storage. Detailed instructions ^^^^^^^^^^^^^^^^^^^^^ First a domain must be created in the Identity service to hold proxy users created by sahara. This domain must have an identity backend driver that allows for sahara to create new users. The default SQL engine is sufficient but if your keystone identity is backed by LDAP or similar then domain specific configurations should be used to ensure sahara's access. Please see the `Keystone documentation`_ for more information. .. _Keystone documentation: https://docs.openstack.org/keystone/latest/configuration.html#domain-specific-drivers With the domain created, sahara's configuration file should be updated to include the new domain name and any potential roles that will be needed. For this example let's assume that the name of the proxy domain is ``sahara_proxy`` and the roles needed by proxy users will be ``Member`` and ``SwiftUser``. .. sourcecode:: cfg [DEFAULT] use_domain_for_proxy_users=true proxy_user_domain_name=sahara_proxy proxy_user_role_names=Member,SwiftUser A note on the use of roles. In the context of the proxy user, any roles specified here are roles intended to be delegated to the proxy user from the user with access to Object Storage. More specifically, any roles that are required for Object Storage access by the project owning the object store must be delegated to the proxy user for authentication to be successful. Finally, the stack administrator must ensure that images registered with sahara have the latest version of the Hadoop swift filesystem plugin installed. The sources for this plugin can be found in the `sahara extra repository`_. For more information on images or swift integration see the sahara documentation sections :ref:`diskimage-builder-label` and :ref:`swift-integration-label`. .. _Sahara extra repository: http://git.openstack.org/cgit/openstack/sahara-extra .. _volume_instance_locality_configuration: Volume instance locality configuration -------------------------------------- The Block Storage service provides the ability to define volume instance locality to ensure that instance volumes are created on the same host as the hypervisor. The ``InstanceLocalityFilter`` provides the mechanism for the selection of a storage provider located on the same physical host as an instance. To enable this functionality for instances of a specific node group, the ``volume_local_to_instance`` field in the node group template should be set to ``true`` and some extra configurations are needed: * The cinder-volume service should be launched on every physical host and at least one physical host should run both cinder-scheduler and cinder-volume services. * ``InstanceLocalityFilter`` should be added to the list of default filters (``scheduler_default_filters`` in cinder) for the Block Storage configuration. * The Extended Server Attributes extension needs to be active in the Compute service (this is true by default in nova), so that the ``OS-EXT-SRV-ATTR:host`` property is returned when requesting instance info. * The user making the call needs to have sufficient rights for the property to be returned by the Compute service. This can be done by: * by changing nova's ``policy.json`` to allow the user access to the ``extended_server_attributes`` option. * by designating an account with privileged rights in the cinder configuration: .. sourcecode:: cfg os_privileged_user_name = os_privileged_user_password = os_privileged_user_tenant = It should be noted that in a situation when the host has no space for volume creation, the created volume will have an ``Error`` state and can not be used. Autoconfiguration for templates ------------------------------- :doc:`configs-recommendations` NTP service configuration ------------------------- By default sahara will enable the NTP service on all cluster instances if the NTP package is included in the image (the sahara disk image builder will include NTP in all images it generates). The default NTP server will be ``pool.ntp.org``; this can be overridden using the ``default_ntp_server`` setting in the ``DEFAULT`` section of the sahara configuration file. If you are creating cluster templates using the sahara UI and would like to specify a different NTP server for a particular cluster template, use the ``URL of NTP server`` setting in the ``General Parameters`` section when you create the template. If you would like to disable NTP for a particular cluster template, deselect the ``Enable NTP service`` checkbox in the ``General Parameters`` section when you create the template. If you are creating clusters using the sahara CLI, you can specify another NTP server or disable NTP service using the examples below. If you want to enable configuring the NTP service, you should specify the following configs for the cluster: .. sourcecode:: json { "cluster_configs": { "general": { "URL of NTP server": "your_server.net" } } } If you want to disable configuring NTP service, you should specify following configs for the cluster: .. sourcecode:: json { "cluster_configs": { "general": { "Enable NTP service": false } } } CORS (Cross Origin Resource Sharing) Configuration -------------------------------------------------- Sahara provides direct API access to user-agents (browsers) via the HTTP CORS protocol. Detailed documentation, as well as troubleshooting examples, may be found in the OpenStack `Administrator Guide`_. To get started quickly, use the example configuration block below, replacing the :code:`allowed origin` field with the host(s) from which your API expects access. .. sourcecode:: cfg [cors] allowed_origin=https://we.example.com:443 max_age=3600 allow_credentials=true [cors.additional_domain_1] allowed_origin=https://additional_domain_1.example.com:443 [cors.additional_domain_2] allowed_origin=https://additional_domain_2.example.com:443 For more information on Cross Origin Resource Sharing, please review the `W3C CORS specification`_. .. _Administrator Guide: https://docs.openstack.org/oslo.middleware/latest/admin/cross-project-cors.html .. _W3C CORS specification: http://www.w3.org/TR/cors/ Cleanup time for incomplete clusters ------------------------------------ Sahara provides maximal time (in hours) for clusters allowed to be in states other than "Active", "Deleting" or "Error". If a cluster is not in "Active", "Deleting" or "Error" state and last update of it was longer than ``cleanup_time_for_incomplete_clusters`` hours ago then it will be deleted automatically. You can enable this feature by adding appropriate config property in the ``DEFAULT`` section (by default it set up to ``0`` value which means that automatic clean up is disabled). For example, if you want cluster to be deleted after 3 hours if it didn't leave "Starting" state then you should specify: .. sourcecode:: cfg [DEFAULT] cleanup_time_for_incomplete_clusters = 3 Security Group Rules Configuration ---------------------------------- When auto_security_group is used, the amount of created security group rules may be bigger than the default values configured in ``neutron.conf``. Then the default limit should be raised up to some bigger value which is proportional to the number of cluster node groups. You can change it in ``neutron.conf`` file: .. sourcecode:: cfg [quotas] quota_security_group = 1000 quota_security_group_rule = 10000 Or you can execute openstack CLI command: .. sourcecode:: console openstack quota set --secgroups 1000 --secgroup-rules 10000 $PROJECT_ID sahara-8.0.0/doc/source/admin/configuration-guide.rst0000666000175100017510000001556613245514472022661 0ustar zuulzuul00000000000000Sahara Configuration Guide ========================== This guide covers the steps for a basic configuration of sahara. It will help you to configure the service in the most simple manner. Basic configuration ------------------- Sahara is packaged with a basic sample configuration file: ``sahara.conf.sample-basic``. This file contains all the essential parameters that are required for sahara. We recommend creating your configuration file based on this basic example. If a more thorough configuration is needed we recommend using the ``tox`` tool to create a full configuration file by executing the following command: .. sourcecode:: cfg $ tox -e genconfig Running this command will create a file named ``sahara.conf.sample`` in the ``etc/sahara`` directory of the project. After creating a configuration file by either copying the basic example or generating one, edit the ``connection`` parameter in the ``[database]`` section. The URL provided here should point to an empty database. For example, the connection string for a MySQL database will be: .. sourcecode:: cfg connection=mysql://username:password@host:port/database Next you will configure the Identity service parameters in the ``[keystone_authtoken]`` section. The ``auth_uri`` parameter should point to the public Identity API endpoint. The ``identity_uri`` should point to the admin Identity API endpoint. For example: .. sourcecode:: cfg auth_uri=http://127.0.0.1:5000/v2.0/ identity_uri=http://127.0.0.1:35357/ Specify the ``username``, ``password`` and ``project_name``. These parameters must specify an Identity user who has the ``admin`` role in the given project. These credentials allow sahara to authenticate and authorize its users. Next you will configure the default Networking service. If using neutron for networking the following parameter should be set in the ``[DEFAULT]`` section: With these parameters set, sahara is ready to run. By default the sahara's log level is set to INFO. If you wish to increase the logging levels for troubleshooting, set ``debug`` to ``true`` in the ``[DEFAULT]`` section of the configuration file. Networking configuration ------------------------ By default sahara is configured to use the neutron. Additionally, if the cluster supports network namespaces the ``use_namespaces`` property can be used to enable their usage. .. sourcecode:: cfg [DEFAULT] use_namespaces=True .. note:: If a user other than ``root`` will be running the Sahara server instance and namespaces are used, some additional configuration is required, please see :ref:`non-root-users` for more information. .. _floating_ip_management: Floating IP management ++++++++++++++++++++++ During cluster setup sahara must access instances through a secure shell (SSH). To establish this connection it may use either the fixed or floating IP address of an instance. By default sahara is configured to use floating IP addresses for access. This is controlled by the ``use_floating_ips`` configuration parameter. With this setup the user has two options for ensuring that the instances in the node groups templates that requires floating IPs gain a floating IP address: * The user may specify a floating IP address pool for each node group that requires floating IPs directly. From Newton changes were made to allow the coexistence of clusters using floating IPs and clusters using fixed IPs. If ``use_floating_ips`` is True it means that the floating IPs can be used by Sahara to spawn clusters. But, differently from previous versions, this does not mean that all instances in the cluster must have floating IPs and that all clusters must use floating IPs. It is possible in a single Sahara deploy to have clusters setup using fixed IPs, clusters using floating IPs and cluster that use both. If not using floating IP addresses (``use_floating_ips=False``) sahara will use fixed IP addresses for instance management. When using neutron for the Networking service the user will be able to choose the fixed IP network for all instances in a cluster. .. _notification-configuration: Notifications configuration --------------------------- Sahara can be configured to send notifications to the OpenStack Telemetry module. To enable this functionality the following parameter ``enable`` should be set in the ``[oslo_messaging_notifications]`` section of the configuration file: .. sourcecode:: cfg [oslo_messaging_notifications] enable = true And the following parameter ``driver`` should be set in the ``[oslo_messaging_notifications]`` section of the configuration file: .. sourcecode:: cfg [oslo_messaging_notifications] driver = messaging By default sahara is configured to use RabbitMQ as its message broker. If you are using RabbitMQ as the message broker, then you should set the following parameter in the ``[DEFAULT]`` section: .. sourcecode:: cfg rpc_backend = rabbit You may also need to specify the connection parameters for your RabbitMQ installation. The following example shows the default values in the ``[oslo_messaging_rabbit]`` section which may need adjustment: .. sourcecode:: cfg rabbit_host=localhost rabbit_port=5672 rabbit_hosts=$rabbit_host:$rabbit_port rabbit_userid=guest rabbit_password=guest rabbit_virtual_host=/ .. .. _orchestration-configuration: Orchestration configuration --------------------------- By default sahara is configured to use the heat engine for instance creation. The heat engine uses the OpenStack Orchestration service to provision instances. This engine makes calls directly to the services required for instance provisioning. .. _policy-configuration-label: Policy configuration -------------------- Sahara's public API calls may be restricted to certain sets of users by using a policy configuration file. The location of the policy file(s) is controlled by the ``policy_file`` and ``policy_dirs`` parameters in the ``[oslo_policy]`` section. By default sahara will search for a ``policy.json`` file in the same directory as the ``sahara.conf`` configuration file. Examples ++++++++ Example 1. Allow all method to all users (default policy). .. sourcecode:: json { "default": "" } Example 2. Disallow image registry manipulations to non-admin users. .. sourcecode:: json { "default": "", "data-processing:images:register": "role:admin", "data-processing:images:unregister": "role:admin", "data-processing:images:add_tags": "role:admin", "data-processing:images:remove_tags": "role:admin" } API configuration ----------------- Sahara uses the ``api-paste.ini`` file to configure the data processing API service. For middleware injection sahara uses pastedeploy library. The location of the api-paste file is controlled by the ``api_paste_config`` parameter in the ``[default]`` section. By default sahara will search for a ``api-paste.ini`` file in the same directory as the configuration file. sahara-8.0.0/doc/source/admin/upgrade-guide.rst0000666000175100017510000001427113245514472021431 0ustar zuulzuul00000000000000Sahara Upgrade Guide ==================== This page contains details about upgrading sahara between releases such as configuration file updates, database migrations, and architectural changes. Icehouse -> Juno ---------------- Main binary renamed to sahara-all +++++++++++++++++++++++++++++++++ The All-In-One sahara binary has been renamed from ``sahara-api`` to ``sahara-all``. The new name should be used in all cases where the All-In-One sahara is desired. Authentication middleware changes +++++++++++++++++++++++++++++++++ The custom auth_token middleware has been deprecated in favor of the keystone middleware. This change requires an update to the sahara configuration file. To update your configuration file you should replace the following parameters from the ``[DEFAULT]`` section with the new parameters in the ``[keystone_authtoken]`` section: +-----------------------+--------------------+ | Old parameter name | New parameter name | +=======================+====================+ | os_admin_username | admin_user | +-----------------------+--------------------+ | os_admin_password | admin_password | +-----------------------+--------------------+ | os_admin_tenant_name | admin_tenant_name | +-----------------------+--------------------+ Additionally, the parameters ``os_auth_protocol``, ``os_auth_host``, and ``os_auth_port`` have been combined to create the ``auth_uri`` and ``identity_uri`` parameters. These new parameters should be full URIs to the keystone public and admin endpoints, respectively. For more information about these configuration parameters please see the :doc:`../admin/configuration-guide`. Database package changes ++++++++++++++++++++++++ The oslo based code from sahara.openstack.common.db has been replaced by the usage of the oslo.db package. This change does not require any update to sahara's configuration file. Additionally, the usage of SQLite databases has been deprecated. Please use MySQL or PostgreSQL databases for sahara. SQLite has been deprecated because it does not, and is not going to, support the ``ALTER COLUMN`` and ``DROP COLUMN`` commands required for migrations between versions. For more information please see http://www.sqlite.org/omitted.html Sahara integration into OpenStack Dashboard +++++++++++++++++++++++++++++++++++++++++++ The sahara dashboard package has been deprecated in the Juno release. The functionality of the dashboard has been fully incorporated into the OpenStack Dashboard. The sahara interface is available under the "Project" -> "Data Processing" tab. The Data processing service endpoints must be registered in the Identity service catalog for the Dashboard to properly recognize and display those user interface components. For more details on this process please see :ref:`registering Sahara in installation guide `. The `sahara-dashboard `_ project is now used solely to host sahara user interface integration tests. Virtual machine user name changes +++++++++++++++++++++++++++++++++ The HEAT infrastructure engine has been updated to use the same rules for instance user names as the direct engine. In previous releases the user name for instances created by sahara using HEAT was always 'ec2-user'. As of Juno, the user name is taken from the image registry as described in the :doc:`../user/registering-image` document. This change breaks backward compatibility for clusters created using the HEAT infrastructure engine prior to the Juno release. Clusters will continue to operate, but we do not recommended using the scaling operations with them. Anti affinity implementation changed ++++++++++++++++++++++++++++++++++++ Starting with the Juno release the anti affinity feature is implemented using server groups. From the user perspective there will be no noticeable changes with this feature. Internally this change has introduced the following behavior: 1) Server group objects will be created for any clusters with anti affinity enabled. 2) Affected instances on the same host will not be allowed even if they do not have common processes. Prior to Juno, instances with differing processes were allowed on the same host. The new implementation guarantees that all affected instances will be on different hosts regardless of their processes. The new anti affinity implementation will only be applied for new clusters. Clusters created with previous versions will continue to operate under the older implementation, this applies to scaling operations on these clusters as well. Juno -> Kilo ------------ Sahara requires policy configuration ++++++++++++++++++++++++++++++++++++ Sahara now requires a policy configuration file. The ``policy.json`` file should be placed in the same directory as the sahara configuration file or specified using the ``policy_file`` parameter. For more details about the policy file please see the :ref:`policy section in the configuration guide `. Kilo -> Liberty --------------- Direct engine deprecation +++++++++++++++++++++++++ In the Liberty release the direct infrastructure engine has been deprecated and the heat infrastructure engine is now default. This means, that it is preferable to use heat engine instead now. In the Liberty release you can continue to operate clusters with the direct engine (create, delete, scale). Using heat engine only the delete operation is available on clusters that were created by the direct engine. After the Liberty release the direct engine will be removed, this means that you will only be able to delete clusters created with the direct engine. Policy namespace changed (policy.json) ++++++++++++++++++++++++++++++++++++++ The "data-processing:" namespace has been added to the beginning of the all Sahara's policy based actions, so, you need to update the policy.json file by prepending all actions with "data-processing:". Liberty -> Mitaka ----------------- Direct engine is removed. Mitaka -> Newton ---------------- Sahara CLI command is deprecated, please use OpenStack Client. .. note:: Since Mitaka release sahara actively uses release notes so you can see all required upgrade actions here: https://docs.openstack.org/releasenotes/sahara/ sahara-8.0.0/doc/source/admin/configs-recommendations.rst0000666000175100017510000000360213245514472023520 0ustar zuulzuul00000000000000:orphan: Autoconfiguring templates ========================= During the Liberty development cycle sahara implemented a tool that recommends and applies configuration values for cluster templates and node group templates. These recommendations are based on the number of specific instances and on flavors of the cluster node groups. Currently the following plugins support this feature: * CDH; * Ambari; * Spark; * the Vanilla Apache Hadoop plugin. By default this feature is enabled for all cluster templates and node group templates. If you want to disable this feature for a particular cluster or node group template you should set the ``use_autoconfig`` field to ``false``. .. NOTE Also, if you manually set configs from the list below, the recommended configs will not be applied. The following describes the settings for which sahara can recommend autoconfiguration: The Cloudera, Spark and Vanilla Apache Hadoop plugin support configuring ``dfs.replication`` (``dfs_replication`` for Cloudera plugin) which is calculated as a minimum from the amount of ``datanode`` (``HDFS_DATANODE`` for Cloudera plugin) instances in the cluster and the default value for ``dfs.replication``. The Vanilla Apache Hadoop plugin and Cloudera plugin support autoconfiguration of basic YARN and MapReduce configs. These autoconfigurations are based on the following documentation: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html The Ambari plugin has its own strategies on configuration recommendations. You can choose one of ``ALWAYS_APPLY``, ``NEVER_APPLY``, and ``ONLY_STACK_DEFAULTS_APPLY``. By default the Ambari plugin follows the ``NEVER_APPLY`` strategy. You can get more information about strategies in Ambari's official documentation: https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-ClusterCreationTemplateStructure sahara-8.0.0/doc/source/admin/index.rst0000666000175100017510000000025413245514472020012 0ustar zuulzuul00000000000000====================== Operator Documentation ====================== .. toctree:: :maxdepth: 2 configuration-guide advanced-configuration-guide upgrade-guide sahara-8.0.0/doc/source/_theme_rtd/0000775000175100017510000000000013245515026017164 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/_theme_rtd/theme.conf0000666000175100017510000000010713245514472021141 0ustar zuulzuul00000000000000[theme] inherit = nature stylesheet = nature.css pygments_style = tangosahara-8.0.0/doc/source/_theme_rtd/layout.html0000666000175100017510000000020513245514472021372 0ustar zuulzuul00000000000000{% extends "basic/layout.html" %} {% set css_files = css_files + ['_static/tweaks.css'] %} {% block relbar1 %}{% endblock relbar1 %}sahara-8.0.0/doc/source/intro/0000775000175100017510000000000013245515026016205 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/intro/overview.rst0000666000175100017510000002036013245514472020614 0ustar zuulzuul00000000000000Rationale ========= Introduction ------------ Apache Hadoop is an industry standard and widely adopted MapReduce implementation, it is one among a growing number of data processing frameworks. The aim of this project is to enable users to easily provision and manage clusters with Hadoop and other data processing frameworks on OpenStack. It is worth mentioning that Amazon has provided Hadoop for several years as Amazon Elastic MapReduce (EMR) service. Sahara aims to provide users with a simple means to provision Hadoop, Spark, and Storm clusters by specifying several parameters such as the framework version, cluster topology, hardware node details and more. After a user fills in all the parameters, sahara deploys the cluster in a few minutes. Also sahara provides means to scale an already provisioned cluster by adding or removing worker nodes on demand. The solution will address the following use cases: * fast provisioning of data processing clusters on OpenStack for development and quality assurance(QA). * utilization of unused compute power from a general purpose OpenStack IaaS cloud. * "Analytics as a Service" for ad-hoc or bursty analytic workloads (similar to AWS EMR). Key features are: * designed as an OpenStack component. * managed through a REST API with a user interface(UI) available as part of OpenStack Dashboard. * support for a variety of data processing frameworks: * multiple Hadoop vendor distributions. * Apache Spark and Storm. * pluggable system of Hadoop installation engines. * integration with vendor specific management tools, such as Apache Ambari and Cloudera Management Console. * predefined configuration templates with the ability to modify parameters. Details ------- The sahara product communicates with the following OpenStack services: * Dashboard (horizon) - provides a GUI with ability to use all of sahara's features. * Identity (keystone) - authenticates users and provides security tokens that are used to work with OpenStack, limiting a user's abilities in sahara to their OpenStack privileges. * Compute (nova) - used to provision VMs for data processing clusters. * Bare metal (ironic) - used to provision Baremetal nodes for data processing clusters. * Orchestration (heat) - used to provision and orchestrate the deployment of data processing clusters. * Image (glance) - stores VM images, each image containing an operating system and a pre-installed data processing distribution or framework. * Object Storage (swift) - can be used as storage for job binaries and data that will be processed or created by framework jobs. * Block Storage (cinder) - can be used to provision block storage for VM instances. * Networking (neutron) - provides networking services to data processing clusters. * DNS service (designate) - provides ability to communicate with cluster instances and Hadoop services by their hostnames. * Telemetry (ceilometer) - used to collect measures of cluster usage for metering and monitoring purposes. * Shared file systems (manila) - can be used for storage of framework job binaries and data that will be processed or created by jobs. * Key manager (barbican & castellan) - persists the authentication data like passwords and private keys in a secure storage. .. image:: ../images/openstack-interop.png :width: 800 px :scale: 99 % :align: left General Workflow ---------------- Sahara will provide two levels of abstraction for the API and UI based on the addressed use cases: cluster provisioning and analytics as a service. For fast cluster provisioning a generic workflow will be as following: * select a Hadoop (or framework) version. * select a base image with or without pre-installed data processing framework: * for base images without a pre-installed framework, sahara will support pluggable deployment engines that integrate with vendor tooling. * define cluster configuration, including cluster size, topology, and framework parameters (for example, heap size): * to ease the configuration of such parameters, configurable templates are provided. * provision the cluster; sahara will provision nodes (VMs or baremetal), install and configure the data processing framework. * perform operations on the cluster; add or remove nodes. * terminate the cluster when it is no longer needed. For analytics as a service, a generic workflow will be as following: * select one of the predefined data processing framework versions. * configure a job: * choose the type of job: pig, hive, jar-file, etc. * provide the job script source or jar location. * select input and output data location. * set the limit for the cluster size. * execute the job: * all cluster provisioning and job execution will happen transparently to the user. * if using a transient cluster, it will be removed automatically after job completion. * get the results of computations (for example, from swift). User's Perspective ------------------ While provisioning clusters through sahara, the user operates on three types of entities: Node Group Templates, Cluster Templates and Clusters. A Node Group Template describes a group of nodes within cluster. It contains a list of processes that will be launched on each instance in a group. Also a Node Group Template may provide node scoped configurations for those processes. This kind of template encapsulates hardware parameters (flavor) for the node instance and configuration for data processing framework processes running on the node. A Cluster Template is designed to bring Node Group Templates together to form a Cluster. A Cluster Template defines what Node Groups will be included and how many instances will be created for each. Some data processing framework configurations can not be applied to a single node, but to a whole Cluster. A user can specify these kinds of configurations in a Cluster Template. Sahara enables users to specify which processes should be added to an anti-affinity group within a Cluster Template. If a process is included into an anti-affinity group, it means that instances where this process is going to be launched should be scheduled to different hardware hosts. The Cluster entity represents a collection of instances that all have the same data processing framework installed. It is mainly characterized by an image with a pre-installed framework which will be used for cluster deployment. Users may choose one of the pre-configured Cluster Templates to start a Cluster. To get access to instances after a Cluster has started, the user should specify a keypair. Sahara provides several constraints on cluster framework topology. You can see all constraints in the documentation for the appropriate plugin. Each Cluster belongs to an Identity service project determined by the user. Users have access only to objects located in projects they have access to. Users can edit and delete only objects they have created or exist in their projects. Naturally, admin users have full access to every object. In this manner, sahara complies with general OpenStack access policy. Integration with Object Storage ------------------------------- The swift project provides the standard Object Storage service for OpenStack environments; it is an analog of the Amazon S3 service. As a rule it is deployed on bare metal machines. It is natural to expect data processing on OpenStack to access data stored there. Sahara provides this option with a file system implementation for swift `HADOOP-8545 `_ and `Change I6b1ba25b `_ which implements the ability to list endpoints for an object, account or container. This makes it possible to integrate swift with software that relies on data locality information to avoid network overhead. To get more information on how to enable swift support see :doc:`../user/hadoop-swift`. Pluggable Deployment and Monitoring ----------------------------------- In addition to the monitoring capabilities provided by vendor-specific Hadoop management tooling, sahara provides pluggable integration with external monitoring systems such as Nagios or Zabbix. Both deployment and monitoring tools can be installed on standalone VMs, thus allowing a single instance to manage and monitor several clusters at once. sahara-8.0.0/doc/source/intro/index.rst0000666000175100017510000000030413245514472020051 0ustar zuulzuul00000000000000=============== Sahara Overview =============== General overview of Sahara. .. toctree:: :maxdepth: 2 overview architecture Roadmap sahara-8.0.0/doc/source/intro/architecture.rst0000666000175100017510000000247213245514472021434 0ustar zuulzuul00000000000000Architecture ============ .. image:: ../images/sahara-architecture.svg :width: 800 px :scale: 100 % :align: left The Sahara architecture consists of several components: * Auth component - responsible for client authentication & authorization, communicates with the OpenStack Identity service (keystone). * DAL - Data Access Layer, persists internal models in DB. * Secure Storage Access Layer - persists the authentication data like passwords and private keys in a secure storage. * Provisioning Engine - component responsible for communication with the OpenStack Compute (nova), Orchestration (heat), Block Storage (cinder), Image (glance), and DNS (designate) services. * Vendor Plugins - pluggable mechanism responsible for configuring and launching data processing frameworks on provisioned VMs. Existing management solutions like Apache Ambari and Cloudera Management Console could be utilized for that purpose as well. * EDP - :doc:`../user/edp` responsible for scheduling and managing data processing jobs on clusters provisioned by sahara. * REST API - exposes sahara functionality via REST HTTP interface. * Python Sahara Client - like other OpenStack components, sahara has its own python client. * Sahara pages - a GUI for the sahara is located in the OpenStack Dashboard (horizon). sahara-8.0.0/doc/source/_templates/0000775000175100017510000000000013245515026017207 5ustar zuulzuul00000000000000sahara-8.0.0/doc/source/_templates/sidebarlinks.html0000666000175100017510000000046113245514472022556 0ustar zuulzuul00000000000000

Useful Links

{% if READTHEDOCS %} {% endif %} sahara-8.0.0/test-requirements-bandit.txt0000666000175100017510000000001713245514472020511 0ustar zuulzuul00000000000000bandit==0.17.3 sahara-8.0.0/devstack/0000775000175100017510000000000013245515026014611 5ustar zuulzuul00000000000000sahara-8.0.0/devstack/README.rst0000666000175100017510000000101713245514472016305 0ustar zuulzuul00000000000000====================== Enabling in Devstack ====================== 1. Download DevStack 2. Add this repo as an external repository in ``local.conf`` .. sourcecode:: bash [[local|localrc]] enable_plugin sahara git://git.openstack.org/openstack/sahara enable_plugin heat git://git.openstack.org/openstack/heat Optionally, a git refspec may be provided as follows: .. sourcecode:: bash [[local|localrc]] enable_plugin sahara git://git.openstack.org/openstack/sahara 3. run ``stack.sh`` sahara-8.0.0/devstack/settings0000666000175100017510000000270113245514472016402 0ustar zuulzuul00000000000000#!/bin/bash # Settings needed for the Sahara plugin # ------------------------------------- # Set up default directories SAHARACLIENT_DIR=$DEST/python-saharaclient SAHARA_DIR=$DEST/sahara SAHARACLIENT_REPO=${SAHARACLIENT_REPO:-\ ${GIT_BASE}/openstack/python-saharaclient.git} SAHARACLIENT_BRANCH=${SAHARACLIENT_BRANCH:-master} SAHARA_CONF_DIR=${SAHARA_CONF_DIR:-/etc/sahara} SAHARA_CONF_FILE=${SAHARA_CONF_DIR}/sahara.conf # TODO(slukjanov): Should we append sahara to SSL_ENABLED_SERVICES? if is_ssl_enabled_service "sahara" || is_service_enabled tls-proxy; then SAHARA_SERVICE_PROTOCOL="https" fi SAHARA_SERVICE_HOST=${SAHARA_SERVICE_HOST:-$SERVICE_HOST} SAHARA_SERVICE_PORT=${SAHARA_SERVICE_PORT:-8386} SAHARA_SERVICE_PORT_INT=${SAHARA_SERVICE_PORT_INT:-18386} SAHARA_SERVICE_PROTOCOL=${SAHARA_SERVICE_PROTOCOL:-$SERVICE_PROTOCOL} SAHARA_ENDPOINT_TYPE=${SAHARA_ENDPOINT_TYPE:-publicURL} SAHARA_AUTH_CACHE_DIR=${SAHARA_AUTH_CACHE_DIR:-/var/cache/sahara} SAHARA_ENABLED_PLUGINS=${SAHARA_ENABLED_PLUGINS:-\ vanilla,cdh,mapr,spark,storm,fake} SAHARA_BIN_DIR=$(get_python_exec_prefix) SAHARA_ENABLE_DISTRIBUTED_PERIODICS=${SAHARA_ENABLE_DISTRIBUTED_PERIODICS:-\ True} SAHARA_PERIODIC_COORDINATOR_URL=${SAHARA_PERIODIC_COORDINATOR_URL:-\ memcached://127.0.0.1:11211} #Toggle for deploying Sahara API with Apache + mod_wsgi SAHARA_USE_MOD_WSGI=${SAHARA_USE_MOD_WSGI:-False} enable_service sahara-api sahara-eng enable_service heat h-eng h-api h-api-cfn h-api-cw sahara-8.0.0/devstack/exercise.sh0000666000175100017510000000270113245514472016762 0ustar zuulzuul00000000000000#!/usr/bin/env bash # Sanity check that Sahara started if enabled echo "*********************************************************************" echo "Begin DevStack Exercise: $0" echo "*********************************************************************" # This script exits on an error so that errors don't compound and you see # only the first error that occurred. set -o errexit # Print the commands being run so that we can see the command that triggers # an error. It is also useful for following allowing as the install occurs. set -o xtrace # Settings # ======== # Keep track of the current directory EXERCISE_DIR=$(cd $(dirname "$0") && pwd) TOP_DIR=$(cd $EXERCISE_DIR/..; pwd) # Import common functions . $TOP_DIR/functions # Import configuration . $TOP_DIR/openrc # Import exercise configuration . $TOP_DIR/exerciserc is_service_enabled sahara || exit 55 if is_ssl_enabled_service "sahara" ||\ is_ssl_enabled_service "sahara-api" ||\ is_service_enabled tls-proxy; then SAHARA_SERVICE_PROTOCOL="https" fi SAHARA_SERVICE_PROTOCOL=${SAHARA_SERVICE_PROTOCOL:-$SERVICE_PROTOCOL} $CURL_GET $SAHARA_SERVICE_PROTOCOL://$SERVICE_HOST:8386/ 2>/dev/null \ | grep -q 'Auth' || die $LINENO "Sahara API isn't functioning!" set +o xtrace echo "*********************************************************************" echo "SUCCESS: End DevStack Exercise: $0" echo "*********************************************************************" sahara-8.0.0/devstack/plugin.sh0000777000175100017510000002371213245514472016461 0ustar zuulzuul00000000000000#!/bin/bash # # lib/sahara # Dependencies: # ``functions`` file # ``DEST``, ``DATA_DIR``, ``STACK_USER`` must be defined # ``stack.sh`` calls the entry points in this order: # # install_sahara # install_python_saharaclient # configure_sahara # start_sahara # stop_sahara # cleanup_sahara # Save trace setting XTRACE=$(set +o | grep xtrace) set -o xtrace # Functions # --------- # create_sahara_accounts() - Set up common required sahara accounts # # Tenant User Roles # ------------------------------ # service sahara admin function create_sahara_accounts { create_service_user "sahara" get_or_create_service "sahara" "data-processing" "Sahara Data Processing" get_or_create_endpoint "data-processing" \ "$REGION_NAME" \ "$SAHARA_SERVICE_PROTOCOL://$SAHARA_SERVICE_HOST:$SAHARA_SERVICE_PORT/v1.1/\$(tenant_id)s" \ "$SAHARA_SERVICE_PROTOCOL://$SAHARA_SERVICE_HOST:$SAHARA_SERVICE_PORT/v1.1/\$(tenant_id)s" \ "$SAHARA_SERVICE_PROTOCOL://$SAHARA_SERVICE_HOST:$SAHARA_SERVICE_PORT/v1.1/\$(tenant_id)s" } # cleanup_sahara() - Remove residual data files, anything left over from # previous runs that would need to clean up. function cleanup_sahara { # Cleanup auth cache dir sudo rm -rf $SAHARA_AUTH_CACHE_DIR if [ "$SAHARA_USE_MOD_WSGI" == "True" ]; then sudo rm -f $(apache_site_config_for sahara-api) fi } function configure_sahara_apache_wsgi { local sahara_apache_conf=$(apache_site_config_for sahara-api) local sahara_ssl="" local sahara_certfile="" local sahara_keyfile="" local venv_path="" if is_ssl_enabled_service sahara; then sahara_ssl="SSLEngine On" sahara_certfile="SSLCertificateFile $SAHARA_SSL_CERT" sahara_keyfile="SSLCertificateKeyFile $SAHARA_SSL_KEY" fi sudo cp $SAHARA_DIR/devstack/files/apache-sahara-api.template $sahara_apache_conf sudo sed -e " s|%PUBLICPORT%|$SAHARA_SERVICE_PORT|g; s|%APACHE_NAME%|$APACHE_NAME|g; s|%SAHARA_BIN_DIR%|$SAHARA_BIN_DIR|g; s|%SSLENGINE%|$sahara_ssl|g; s|%SSLCERTFILE%|$sahara_certfile|g; s|%SSLKEYFILE%|$sahara_keyfile|g; s|%USER%|$STACK_USER|g; s|%VIRTUALENV%|$venv_path|g " -i $sahara_apache_conf } # configure_sahara() - Set config files, create data dirs, etc function configure_sahara { sudo install -d -o $STACK_USER $SAHARA_CONF_DIR cp -p $SAHARA_DIR/etc/sahara/api-paste.ini $SAHARA_CONF_DIR # Create auth cache dir sudo install -d -o $STACK_USER -m 700 $SAHARA_AUTH_CACHE_DIR rm -rf $SAHARA_AUTH_CACHE_DIR/* configure_auth_token_middleware \ $SAHARA_CONF_FILE sahara $SAHARA_AUTH_CACHE_DIR # Set admin user parameters needed for trusts creation iniset $SAHARA_CONF_FILE \ trustee project_name $SERVICE_TENANT_NAME iniset $SAHARA_CONF_FILE trustee username sahara iniset $SAHARA_CONF_FILE \ trustee password $SERVICE_PASSWORD iniset $SAHARA_CONF_FILE \ trustee user_domain_name "$SERVICE_DOMAIN_NAME" iniset $SAHARA_CONF_FILE \ trustee project_domain_name "$SERVICE_DOMAIN_NAME" iniset $SAHARA_CONF_FILE \ trustee auth_url "$KEYSTONE_SERVICE_URI/v3" iniset_rpc_backend sahara $SAHARA_CONF_FILE DEFAULT # Set configuration to send notifications if is_service_enabled ceilometer; then iniset $SAHARA_CONF_FILE oslo_messaging_notifications driver "messaging" fi iniset $SAHARA_CONF_FILE DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL iniset $SAHARA_CONF_FILE DEFAULT plugins $SAHARA_ENABLED_PLUGINS iniset $SAHARA_CONF_FILE \ database connection `database_connection_url sahara` if is_service_enabled neutron; then iniset $SAHARA_CONF_FILE neutron endpoint_type $SAHARA_ENDPOINT_TYPE if is_ssl_enabled_service "neutron" \ || is_service_enabled tls-proxy; then iniset $SAHARA_CONF_FILE neutron ca_file $SSL_BUNDLE_FILE fi fi if is_ssl_enabled_service "heat" || is_service_enabled tls-proxy; then iniset $SAHARA_CONF_FILE heat ca_file $SSL_BUNDLE_FILE fi iniset $SAHARA_CONF_FILE heat endpoint_type $SAHARA_ENDPOINT_TYPE if is_ssl_enabled_service "cinder" || is_service_enabled tls-proxy; then iniset $SAHARA_CONF_FILE cinder ca_file $SSL_BUNDLE_FILE fi iniset $SAHARA_CONF_FILE cinder endpoint_type $SAHARA_ENDPOINT_TYPE if is_ssl_enabled_service "nova" || is_service_enabled tls-proxy; then iniset $SAHARA_CONF_FILE nova ca_file $SSL_BUNDLE_FILE fi iniset $SAHARA_CONF_FILE nova endpoint_type $SAHARA_ENDPOINT_TYPE if is_ssl_enabled_service "swift" || is_service_enabled tls-proxy; then iniset $SAHARA_CONF_FILE swift ca_file $SSL_BUNDLE_FILE fi iniset $SAHARA_CONF_FILE swift endpoint_type $SAHARA_ENDPOINT_TYPE if is_ssl_enabled_service "key" || is_service_enabled tls-proxy; then iniset $SAHARA_CONF_FILE keystone ca_file $SSL_BUNDLE_FILE fi iniset $SAHARA_CONF_FILE keystone endpoint_type $SAHARA_ENDPOINT_TYPE if is_ssl_enabled_service "glance" || is_service_enabled tls-proxy; then iniset $SAHARA_CONF_FILE glance ca_file $SSL_BUNDLE_FILE fi iniset $SAHARA_CONF_FILE glance endpoint_type $SAHARA_ENDPOINT_TYPE # Register SSL certificates if provided if is_ssl_enabled_service sahara; then ensure_certificates SAHARA iniset $SAHARA_CONF_FILE ssl cert_file "$SAHARA_SSL_CERT" iniset $SAHARA_CONF_FILE ssl key_file "$SAHARA_SSL_KEY" fi iniset $SAHARA_CONF_FILE DEFAULT use_syslog $SYSLOG # Format logging if [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ]; then if [ "$SAHARA_USE_MOD_WSGI" == "False" ]; then setup_colorized_logging $SAHARA_CONF_FILE DEFAULT fi fi if is_service_enabled tls-proxy; then # Set the service port for a proxy to take the original iniset $SAHARA_CONF_FILE DEFAULT port $SAHARA_SERVICE_PORT_INT fi if [ "$SAHARA_ENABLE_DISTRIBUTED_PERIODICS" == "True" ]; then # Enable distributed periodic tasks iniset $SAHARA_CONF_FILE DEFAULT periodic_coordinator_backend_url\ $SAHARA_PERIODIC_COORDINATOR_URL pip_install tooz[memcached] restart_service memcached fi recreate_database sahara $SAHARA_BIN_DIR/sahara-db-manage \ --config-file $SAHARA_CONF_FILE upgrade head if [ "$SAHARA_USE_MOD_WSGI" == "True" ]; then configure_sahara_apache_wsgi fi } # install_sahara() - Collect source and prepare function install_sahara { setup_develop $SAHARA_DIR if [ "$SAHARA_USE_MOD_WSGI" == "True" ]; then install_apache_wsgi fi } # install_python_saharaclient() - Collect source and prepare function install_python_saharaclient { if use_library_from_git "python-saharaclient"; then git_clone $SAHARACLIENT_REPO $SAHARACLIENT_DIR $SAHARACLIENT_BRANCH setup_develop $SAHARACLIENT_DIR fi } # start_sahara() - Start running processes, including screen function start_sahara { local service_port=$SAHARA_SERVICE_PORT local service_protocol=$SAHARA_SERVICE_PROTOCOL if is_service_enabled tls-proxy; then service_port=$SAHARA_SERVICE_PORT_INT service_protocol="http" fi if [ "$SAHARA_USE_MOD_WSGI" == "True" ] ; then enable_apache_site sahara-api restart_apache_server tail_log sahara-api /var/log/$APACHE_NAME/sahara-api.log else run_process sahara-all "$SAHARA_BIN_DIR/sahara-all \ --config-file $SAHARA_CONF_FILE" run_process sahara-api "$SAHARA_BIN_DIR/sahara-api \ --config-file $SAHARA_CONF_FILE" fi run_process sahara-eng "$SAHARA_BIN_DIR/sahara-engine \ --config-file $SAHARA_CONF_FILE" echo "Waiting for Sahara to start..." if ! wait_for_service $SERVICE_TIMEOUT \ $service_protocol://$SAHARA_SERVICE_HOST:$service_port; then die $LINENO "Sahara did not start" fi # Start proxies if enabled if is_service_enabled tls-proxy; then start_tls_proxy '*' $SAHARA_SERVICE_PORT \ $SAHARA_SERVICE_HOST \ $SAHARA_SERVICE_PORT_INT & fi } # configure_tempest_for_sahara() - Tune Tempest configuration for Sahara function configure_tempest_for_sahara { if is_service_enabled tempest; then iniset $TEMPEST_CONFIG service_available sahara True iniset $TEMPEST_CONFIG data-processing-feature-enabled plugins $SAHARA_ENABLED_PLUGINS fi } # stop_sahara() - Stop running processes function stop_sahara { # Kill the Sahara screen windows if [ "$SAHARA_USE_MOD_WSGI" == "True" ]; then disable_apache_site sahara-api restart_apache_server else stop_process sahara-all stop_process sahara-api stop_process sahara-eng fi } # is_sahara_enabled. This allows is_service_enabled sahara work # correctly throughout devstack. function is_sahara_enabled { if is_service_enabled sahara-api || \ is_service_enabled sahara-eng || \ is_service_enabled sahara-all; then return 0 else return 1 fi } # Dispatcher for Sahara plugin if is_service_enabled sahara; then if [[ "$1" == "stack" && "$2" == "install" ]]; then echo_summary "Installing sahara" install_sahara install_python_saharaclient cleanup_sahara elif [[ "$1" == "stack" && "$2" == "post-config" ]]; then echo_summary "Configuring sahara" configure_sahara create_sahara_accounts elif [[ "$1" == "stack" && "$2" == "extra" ]]; then echo_summary "Initializing sahara" start_sahara elif [[ "$1" == "stack" && "$2" == "test-config" ]]; then echo_summary "Configuring tempest" configure_tempest_for_sahara fi if [[ "$1" == "unstack" ]]; then stop_sahara fi if [[ "$1" == "clean" ]]; then cleanup_sahara fi fi # Restore xtrace $XTRACE # Local variables: # mode: shell-script # End: sahara-8.0.0/devstack/files/0000775000175100017510000000000013245515026015713 5ustar zuulzuul00000000000000sahara-8.0.0/devstack/files/apache-sahara-api.template0000666000175100017510000000136413245514472022707 0ustar zuulzuul00000000000000Listen %PUBLICPORT% WSGIDaemonProcess sahara-api processes=2 threads=1 user=%USER% display-name=%{GROUP} %VIRTUALENV% WSGIProcessGroup sahara-api WSGIScriptAlias / %SAHARA_BIN_DIR%/sahara-wsgi-api WSGIApplicationGroup %{GLOBAL} WSGIPassAuthorization On AllowEncodedSlashes On = 2.4> ErrorLogFormat "%{cu}t %M" ErrorLog /var/log/%APACHE_NAME%/sahara-api.log %SSLENGINE% %SSLCERTFILE% %SSLKEYFILE% = 2.4> Require all granted Order allow,deny Allow from all sahara-8.0.0/devstack/upgrade/0000775000175100017510000000000013245515026016240 5ustar zuulzuul00000000000000sahara-8.0.0/devstack/upgrade/resources.sh0000777000175100017510000001435513245514472020627 0ustar zuulzuul00000000000000#!/bin/bash set -o errexit . $GRENADE_DIR/grenaderc . $GRENADE_DIR/functions . $TOP_DIR/openrc admin admin set -o xtrace SAHARA_USER=sahara_grenade SAHARA_PROJECT=sahara_grenade SAHARA_PASS=pass SAHARA_KEY=sahara_key SAHARA_KEY_FILE=$SAVE_DIR/sahara_key.pem PUBLIC_NETWORK_NAME=${PUBLIC_NETWORK_NAME:-public} # cirros image is not appropriate for cluster creation SAHARA_IMAGE_NAME=${SAHARA_IMAGE_NAME:-fedora-heat-test-image} SAHARA_IMAGE_USER=${SAHARA_IMAGE_USER:-fedora} # custom flavor parameters SAHARA_FLAVOR_NAME=${SAHARA_FLAVOR_NAME:-sahara_flavor} SAHARA_FLAVOR_RAM=${SAHARA_FLAVOR_RAM:-1024} SAHARA_FLAVOR_DISK=${SAHARA_FLAVOR_DISK:-10} NG_TEMPLATE_NAME=ng-template-grenade CLUSTER_TEMPLATE_NAME=cluster-template-grenade CLUSTER_NAME=cluster-grenade function sahara_set_user { # set ourselves to the created sahara user OS_TENANT_NAME=$SAHARA_PROJECT OS_PROJECT_NAME=$SAHARA_PROJECT OS_USERNAME=$SAHARA_USER OS_PASSWORD=$SAHARA_PASS } function create_tenant { # create a tenant for the server eval $(openstack project create -f shell -c id $SAHARA_PROJECT) if [[ -z "$id" ]]; then die $LINENO "Didn't create $SAHARA_PROJECT project" fi resource_save sahara project_id $id } function create_user { local project_id=$id eval $(openstack user create $SAHARA_USER \ --project $project_id \ --password $SAHARA_PASS \ -f shell -c id) if [[ -z "$id" ]]; then die $LINENO "Didn't create $SAHARA_USER user" fi resource_save sahara user_id $id # Workaround for bug: https://bugs.launchpad.net/keystone/+bug/1662911 openstack role add Member --user $id --project $project_id } function create_keypair { # create key pair for access openstack keypair create $SAHARA_KEY > $SAHARA_KEY_FILE chmod 600 $SAHARA_KEY_FILE } function create_flavor { eval $(openstack flavor create -f shell -c id \ --ram $SAHARA_FLAVOR_RAM \ --disk $SAHARA_FLAVOR_DISK \ $SAHARA_FLAVOR_NAME) resource_save sahara flavor_id $id } function register_image { eval $(openstack image show \ -f shell -c id $SAHARA_IMAGE_NAME) resource_save sahara image_id $id openstack dataprocessing image register $id --username $SAHARA_IMAGE_USER openstack dataprocessing image tags set $id --tags fake 0.1 } function create_node_group_template { eval $(openstack network show -f shell -c id $PUBLIC_NETWORK_NAME) local public_net_id=$id local flavor_id=$(resource_get sahara flavor_id) openstack dataprocessing node group template create \ --name $NG_TEMPLATE_NAME \ --flavor $flavor_id \ --plugin fake \ --plugin-version 0.1 \ --processes jobtracker namenode tasktracker datanode \ --floating-ip-pool $public_net_id \ --auto-security-group } function create_cluster_template { openstack dataprocessing cluster template create \ --name $CLUSTER_TEMPLATE_NAME \ --node-groups $NG_TEMPLATE_NAME:1 } function create_cluster { local net_id=$(resource_get network net_id) local image_id=$(resource_get sahara image_id) if [[ -n "$net_id" ]]; then eval $(openstack dataprocessing cluster create \ --name $CLUSTER_NAME \ --cluster-template $CLUSTER_TEMPLATE_NAME \ --image $image_id \ --user-keypair $SAHARA_KEY \ --neutron-network $net_id \ -f shell -c id) else eval $(openstack dataprocessing cluster create \ --name $CLUSTER_NAME \ --cluster-template $CLUSTER_TEMPLATE_NAME \ --image $image_id \ --user-keypair $SAHARA_KEY \ -f shell -c id) fi resource_save sahara cluster_id $id } function wait_active_state { # wait until cluster moves to active state local timeleft=1000 while [[ $timeleft -gt 0 ]]; do eval $(openstack dataprocessing cluster show -f shell \ -c Status $CLUSTER_NAME) if [[ "$status" != "Active" ]]; then if [[ "$status" == "Error" ]]; then die $LINENO "Cluster is in Error state" fi echo "Cluster is still not in Active state" sleep 10 timeleft=$((timeleft - 10)) if [[ $timeleft == 0 ]]; then die $LINENO "Cluster hasn't moved to Active state \ during 1000 seconds" fi else break fi done } function check_active { # check that cluster is in Active state eval $(openstack dataprocessing cluster show -f shell \ -c Status $CLUSTER_NAME) if [[ "$status" != "Active" ]]; then die $LINENO "Cluster is not in Active state anymore" fi echo "Sahara verification: SUCCESS" } function create { create_tenant create_user create_flavor register_image sahara_set_user create_keypair create_node_group_template create_cluster_template create_cluster wait_active_state } function verify { : } function verify_noapi { : } function destroy { sahara_set_user set +o errexit # delete cluster check_active openstack dataprocessing cluster delete $CLUSTER_NAME --wait set -o errexit # delete cluster template openstack dataprocessing cluster template delete $CLUSTER_TEMPLATE_NAME # delete node group template openstack dataprocessing node group template delete $NG_TEMPLATE_NAME source_quiet $TOP_DIR/openrc admin admin # unregister image local image_id=$(resource_get sahara image_id) openstack dataprocessing image unregister $image_id # delete flavor openstack flavor delete $SAHARA_FLAVOR_NAME # delete user and project local user_id=$(resource_get sahara user_id) local project_id=$(resource_get sahara project_id) openstack user delete $user_id openstack project delete $project_id } # Dispatcher case $1 in "create") create ;; "verify_noapi") verify_noapi ;; "verify") verify ;; "destroy") destroy ;; "force_destroy") set +o errexit destroy ;; esac sahara-8.0.0/devstack/upgrade/from-mitaka/0000775000175100017510000000000013245515026020447 5ustar zuulzuul00000000000000sahara-8.0.0/devstack/upgrade/from-mitaka/upgrade-sahara0000777000175100017510000000054413245514472023272 0ustar zuulzuul00000000000000#!/usr/bin/env bash # ``upgrade-sahara`` function configure_sahara_upgrade { XTRACE=$(set +o | grep xtrace) set -o xtrace local old_plugins old_plugins=$(cat $SAHARA_CONF_DIR/sahara.conf | grep ^plugins) sed -i.bak "s/$old_plugins/plugins=fake,vanilla,cdh/g" $SAHARA_CONF_DIR/sahara.conf # reset to previous state $XTRACE } sahara-8.0.0/devstack/upgrade/from-liberty/0000775000175100017510000000000013245515026020653 5ustar zuulzuul00000000000000sahara-8.0.0/devstack/upgrade/from-liberty/upgrade-sahara0000666000175100017510000000044613245514472023474 0ustar zuulzuul00000000000000#!/usr/bin/env bash # ``upgrade-sahara`` function configure_sahara_upgrade { XTRACE=$(set +o | grep xtrace) set -o xtrace # Copy api-paste.ini to configuration directory cp -p $SAHARA_DIR/etc/sahara/api-paste.ini $SAHARA_CONF_DIR # reset to previous state $XTRACE } sahara-8.0.0/devstack/upgrade/upgrade.sh0000777000175100017510000000402713245514472020237 0ustar zuulzuul00000000000000#!/usr/bin/env bash # ``upgrade-sahara`` echo "*********************************************************************" echo "Begin $0" echo "*********************************************************************" # Clean up any resources that may be in use cleanup() { set +o errexit echo "********************************************************************" echo "ERROR: Abort $0" echo "********************************************************************" # Kill ourselves to signal any calling process trap 2; kill -2 $$ } trap cleanup SIGHUP SIGINT SIGTERM # Keep track of the grenade directory RUN_DIR=$(cd $(dirname "$0") && pwd) # Source params . $GRENADE_DIR/grenaderc # Import common functions . $GRENADE_DIR/functions # This script exits on an error so that errors don't compound and you see # only the first error that occurred. set -o errexit # Upgrade Sahara # ============ # Get functions from current DevStack . $TARGET_DEVSTACK_DIR/stackrc . $TARGET_DEVSTACK_DIR/lib/tls . $(dirname $(dirname $BASH_SOURCE))/plugin.sh . $(dirname $(dirname $BASH_SOURCE))/settings # Print the commands being run so that we can see the command that triggers # an error. It is also useful for following allowing as the install occurs. set -o xtrace # Save current config files for posterity [[ -d $SAVE_DIR/etc.sahara ]] || cp -pr $SAHARA_CONF_DIR $SAVE_DIR/etc.sahara # install_sahara() stack_install_service sahara install_python_saharaclient # calls upgrade-sahara for specific release upgrade_project sahara $RUN_DIR $BASE_DEVSTACK_BRANCH $TARGET_DEVSTACK_BRANCH # Migrate the database $SAHARA_BIN_DIR/sahara-db-manage --config-file $SAHARA_CONF_FILE \ upgrade head || die $LINENO "DB sync error" # Start Sahara start_sahara # Don't succeed unless the service come up ensure_services_started sahara set +o xtrace echo "*********************************************************************" echo "SUCCESS: End $0" echo "*********************************************************************" sahara-8.0.0/devstack/upgrade/settings0000666000175100017510000000231313245514472020030 0ustar zuulzuul00000000000000#!/bin/bash register_project_for_upgrade sahara register_db_to_save sahara devstack_localrc base IMAGE_URLS=\ "http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz,\ http://tarballs.openstack.org/heat-test-image/fedora-heat-test-image.qcow2" devstack_localrc base enable_plugin sahara \ git://git.openstack.org/openstack/sahara \ stable/ocata devstack_localrc base enable_plugin heat \ git://git.openstack.org/openstack/heat \ stable/ocata devstack_localrc base DEFAULT_IMAGE_NAME="cirros-0.3.2-x86_64-uec" devstack_localrc target IMAGE_URLS=\ "http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz,\ http://tarballs.openstack.org/heat-test-image/fedora-heat-test-image.qcow2" devstack_localrc target enable_plugin sahara \ git://git.openstack.org/openstack/sahara devstack_localrc target enable_plugin heat \ git://git.openstack.org/openstack/heat devstack_localrc target LIBS_FROM_GIT=python-saharaclient devstack_localrc target DEFAULT_IMAGE_NAME="cirros-0.3.2-x86_64-uec" sahara-8.0.0/devstack/upgrade/shutdown.sh0000777000175100017510000000073213245514472020462 0ustar zuulzuul00000000000000#!/bin/bash # ``shutdown-sahara`` set -o errexit . $GRENADE_DIR/grenaderc . $GRENADE_DIR/functions # We need base DevStack functions for this . $BASE_DEVSTACK_DIR/functions . $BASE_DEVSTACK_DIR/stackrc # needed for status directory . $BASE_DEVSTACK_DIR/lib/tls . ${GITDIR[sahara]}/devstack/plugin.sh set -o xtrace export ENABLED_SERVICES+=,sahara-api,sahara-eng, stop_sahara # sanity check that service is actually down ensure_services_stopped sahara-api sahara-eng sahara-8.0.0/LICENSE0000666000175100017510000002363613245514472014032 0ustar zuulzuul00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. sahara-8.0.0/HACKING.rst0000666000175100017510000000241713245514472014615 0ustar zuulzuul00000000000000Sahara Style Commandments ========================= - Step 1: Read the OpenStack Style Commandments https://docs.openstack.org/hacking/latest/ - Step 2: Read on Sahara Specific Commandments ---------------------------- Commit Messages --------------- Using a common format for commit messages will help keep our git history readable. Follow these guidelines: - [S365] First, provide a brief summary of 50 characters or less. Summaries of greater than 72 characters will be rejected by the gate. - [S364] The first line of the commit message should provide an accurate description of the change, not just a reference to a bug or blueprint. Imports ------- - [S366, S367] Organize your imports according to the ``Import order`` Dictionaries/Lists ------------------ - [S360] Ensure default arguments are not mutable. - [S368] Must use a dict comprehension instead of a dict constructor with a sequence of key-value pairs. For more information, please refer to http://legacy.python.org/dev/peps/pep-0274/ Logs ---- - [S373] Don't translate logs - [S374] You used a deprecated log level Importing json -------------- - [S375] It's more preferable to use ``jsonutils`` from ``oslo_serialization`` instead of ``json`` for operating with ``json`` objects. sahara-8.0.0/tox.ini0000666000175100017510000000773013245514472014335 0ustar zuulzuul00000000000000[tox] envlist = py35,py27,pep8,genpolicy minversion = 1.6 skipsdist = True [testenv] usedevelop = True install_command = pip install -U -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages} setenv = VIRTUAL_ENV={envdir} DISCOVER_DIRECTORY=sahara/tests/unit deps = -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt commands = ostestr {posargs} passenv = http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY [testenv:cover] commands = {toxinidir}/tools/cover.sh {posargs} [testenv:debug-py27] basepython = python2.7 commands = oslo_debug_helper -t sahara/tests/unit {posargs} [testenv:debug-py35] basepython = python3.5 commands = oslo_debug_helper -t sahara/tests/unit {posargs} [testenv:pep8] commands = flake8 {posargs} doc8 doc/source # Run bashate checks bash -c "find sahara -iname '*.sh' -print0 | xargs -0 bashate -v" bash -c "find devstack -not -name \*.template -and -not -name README.rst -and -not -name \*.json -type f -print0 | xargs -0 bashate -v" # Run security linter bandit -c bandit.yaml -r sahara -n5 -p sahara_default -x tests [testenv:genpolicy] commands = oslopolicy-sample-generator --config-file tools/config/sahara-policy-generator.conf [testenv:venv] commands = {posargs} [testenv:images] sitepackages = True commands = {posargs} [testenv:docs] commands = rm -rf doc/html doc/build rm -rf api-ref/build api-ref/html rm -rf doc/source/apidoc doc/source/api python setup.py build_sphinx sphinx-build -W -b html -d api-ref/build/doctrees api-ref/source api-ref/build/html whitelist_externals = rm [testenv:api-ref] install_command = pip install -U --force-reinstall {opts} {packages} commands = rm -rf api-ref/build api-ref/html sphinx-build -W -b html -d api-ref/build/doctrees api-ref/source api-ref/build/html whitelist_externals = rm [testenv:pylint] setenv = VIRTUAL_ENV={envdir} commands = bash tools/lintstack.sh [testenv:genconfig] commands = oslo-config-generator --config-file tools/config/config-generator.sahara.conf \ --output-file etc/sahara/sahara.conf.sample [testenv:releasenotes] commands = sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html [testenv:pip-missing-reqs] # do not install test-requirements as that will pollute the virtualenv for # determining missing packages # this also means that pip-missing-reqs must be installed separately, outside # of the requirements.txt files deps = pip_missing_reqs -r{toxinidir}/requirements.txt commands = pip-missing-reqs -d --ignore-file=sahara/tests/* sahara [testenv:debug] # It runs tests from the specified dir (default is sahara/tests) # in interactive mode, so, you could use pbr for tests debug. # Example usage: tox -e debug -- -t sahara/tests/unit some.test.path # https://docs.openstack.org/oslotest/latest/features.html#debugging-with-oslo-debug-helper commands = oslo_debug_helper -t sahara/tests/unit {posargs} [testenv:bandit] deps = -r{toxinidir}/test-requirements-bandit.txt commands = bandit -c bandit.yaml -r sahara -n5 -p sahara_default -x tests [flake8] show-source = true builtins = _ exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,tools # [H904] Delay string interpolations at logging calls # [H106] Don't put vim configuration in source files # [H203] Use assertIs(Not)None to check for None. # [H204] Use assert(Not)Equal to check for equality # [H205] Use assert(Greater|Less)(Equal) for comparison enable-extensions=H904,H106,H203,H204,H205 [hacking] import_exceptions = sahara.i18n local-check-factory = sahara.utils.hacking.checks.factory [testenv:bindep] # Do not install any requirements. We want this to be fast and work even if # system dependencies are missing, since it's used to tell you what system # dependencies are missing! This also means that bindep must be installed # separately, outside of the requirements files. deps = bindep commands = bindep test sahara-8.0.0/sahara/0000775000175100017510000000000013245515026014244 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/conductor/0000775000175100017510000000000013245515026016244 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/conductor/objects.py0000666000175100017510000002105713245514472020262 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """This module contains description of objects returned by the conductor. The actual objects returned are located in resource.py, which aim is to hide some necessary magic. Current module describes objects fields via docstrings and contains implementation of helper methods. """ import random from oslo_config import cfg from sahara.utils import configs from sahara.utils import remote CONF = cfg.CONF CONF.import_opt('node_domain', 'sahara.config') class Cluster(object): """An object representing Cluster. id name description tenant_id trust_id is_transient plugin_name hadoop_version cluster_configs - configs dict converted to object, see the docs for details default_image_id anti_affinity anti_affinity_ratio management_private_key management_public_key user_keypair_id status status_description info extra rollback_info - internal information required for rollback sahara_info - internal information about sahara settings provision_progress - list of ProvisionStep objects node_groups - list of NodeGroup objects cluster_template_id cluster_template - ClusterTemplate object use_autoconfig is_public is_protected domain_name """ def has_proxy_gateway(self): for ng in self.node_groups: if ng.is_proxy_gateway: return True def get_proxy_gateway_node(self): proxies = [] for ng in self.node_groups: if ng.is_proxy_gateway and ng.instances: proxies += ng.instances if proxies: return random.choice(proxies) return None @property def stack_name(self): extra = self.extra or {} return extra.get('heat_stack_name', self.name) def use_designate_feature(self): return CONF.use_designate and self.domain_name class NodeGroup(object): """An object representing Node Group. id name flavor_id image_id image_username node_processes - list of node processes node_configs - configs dict converted to object, see the docs for details volumes_per_node volumes_size volumes_availability_zone - name of Cinder availability zone where to spawn volumes volume_mount_prefix volume_type floating_ip_pool - Floating IP Pool name used to assign Floating IPs to instances in this Node Group security_groups - List of security groups for instances in this Node Group auto_security_group - indicates if Sahara should create additional security group for the Node Group availability_zone - name of Nova availability zone where to spawn instances open_ports - List of ports that will be opened if auto_security_group is True is_proxy_gateway - indicates if nodes from this node group should be used as proxy to access other cluster nodes volume_local_to_instance - indicates if volumes and instances should be created on the same physical host count instances - list of Instance objects node_group_template_id node_group_template - NodeGroupTemplate object If node group belongs to cluster: cluster_id - parent Cluster ID cluster - parent Cluster object If node group belongs to cluster template: cluster_template_id - parent ClusterTemplate ID cluster_template - parent ClusterTemplate object """ def configuration(self): return configs.merge_configs(self.cluster.cluster_configs, self.node_configs) def get_image_id(self): return self.image_id or self.cluster.default_image_id class Instance(object): """An object representing Instance. id node_group_id - parent NodeGroup ID node_group - parent NodeGroup object instance_id - Nova instance ID instance_name internal_ip management_ip volumes storage_devices_number dns_hostname """ def hostname(self): return self.instance_name def fqdn(self): if self._use_designate_feature(): return self.dns_hostname else: return self.instance_name + '.' + CONF.node_domain def get_ip_or_dns_name(self): if self._use_designate_feature(): return self.dns_hostname else: return self.management_ip def remote(self): return remote.get_remote(self) def storage_paths(self): mp = [] for idx in range(1, self.storage_devices_number + 1): mp.append(self.node_group.volume_mount_prefix + str(idx)) if not mp: mp = ['/mnt'] return mp def _use_designate_feature(self): return CONF.use_designate and self.dns_hostname class ClusterTemplate(object): """An object representing Cluster Template. id name description cluster_configs - configs dict converted to object, see the docs for details default_image_id anti_affinity tenant_id plugin_name hadoop_version node_groups - list of NodeGroup objects is_public is_protected domain_name """ class NodeGroupTemplate(object): """An object representing Node Group Template. id name description tenant_id flavor_id image_id plugin_name hadoop_version node_processes - list of node processes node_configs - configs dict converted to object, see the docs for details volumes_per_node volumes_size volumes_availability_zone volume_mount_prefix volume_type floating_ip_pool security_groups auto_security_group availability_zone is_proxy_gateway volume_local_to_instance is_public is_protected """ class Image(object): """An object representing Image. id tags username description """ # EDP Objects class DataSource(object): """An object representing Data Source. id tenant_id name description type url credentials is_public is_protected """ class JobExecution(object): """An object representing JobExecution id tenant_id job_id input_id output_id start_time end_time cluster_id info engine_job_id return_code job_configs interface extra data_source_urls is_public is_protected """ class Job(object): """An object representing Job id tenant_id name description type mains libs interface is_public is_protected """ class JobBinary(object): """An object representing JobBinary id tenant_id name description url - URLs may be the following: internal-db://URL, swift:// extra - extra may contain not only user-password but e.g. auth-token is_public is_protected """ class JobBinaryInternal(object): """An object representing JobBinaryInternal Note that the 'data' field is not returned. It uses deferred loading and must be requested explicitly with the job_binary_get_raw_data() conductor method. id tenant_id name datasize is_public is_protected """ # Events ops class ClusterProvisionStep(object): """An object representing cluster ProvisionStep id cluster_id tenant_id step_name step_type total successful events - list of Events objects assigned to the cluster """ class ClusterEvent(object): """An object representing events about cluster provision id node_group_id instance_id instance_name event_info successful step_id """ class ClusterVerification(object): """An object representing cluster verification id cluster_id status checks """ class ClusterHealthCheck(object): """An object representing health check id verification_id status description name """ sahara-8.0.0/sahara/conductor/manager.py0000666000175100017510000010037213245514472020241 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Handles database requests from other Sahara services.""" import copy from oslo_config import cfg from sahara.conductor import resource as r from sahara.db import base as db_base from sahara.service.castellan import utils as key_manager from sahara.service.edp.utils import shares from sahara.utils import configs from sahara.utils import crypto CONF = cfg.CONF CLUSTER_DEFAULTS = { "cluster_configs": {}, "status": "undefined", "anti_affinity": [], "anti_affinity_ratio": 1, "status_description": "", "info": {}, "rollback_info": {}, "sahara_info": {}, "is_public": False, "is_protected": False } NODE_GROUP_DEFAULTS = { "node_processes": [], "node_configs": {}, "volumes_per_node": 0, "volumes_size": 0, "volumes_availability_zone": None, "volume_mount_prefix": "/volumes/disk", "volume_type": None, "floating_ip_pool": None, "security_groups": None, "auto_security_group": False, "availability_zone": None, "is_proxy_gateway": False, "volume_local_to_instance": False, } NODE_GROUP_TEMPLATE_DEFAULTS = copy.deepcopy(NODE_GROUP_DEFAULTS) NODE_GROUP_TEMPLATE_DEFAULTS.update({"is_public": False, "is_protected": False}) INSTANCE_DEFAULTS = { "volumes": [], "storage_devices_number": 0 } DATA_SOURCE_DEFAULTS = { "credentials": {}, "is_public": False, "is_protected": False } JOB_DEFAULTS = { "is_public": False, "is_protected": False } JOB_BINARY_DEFAULTS = { "is_public": False, "is_protected": False } JOB_BINARY_INTERNAL_DEFAULTS = { "is_public": False, "is_protected": False } JOB_EXECUTION_DEFAULTS = { "is_public": False, "is_protected": False } def _apply_defaults(values, defaults): new_values = copy.deepcopy(defaults) new_values.update(values) return new_values class ConductorManager(db_base.Base): """This class aimed to conduct things. The methods in the base API for sahara-conductor are various proxy operations that allow other services to get specific work done without locally accessing the database. Additionally it performs some template-to-object copying magic. """ def __init__(self): super(ConductorManager, self).__init__() # Common helpers def _populate_node_groups(self, context, cluster): node_groups = cluster.get('node_groups') if not node_groups: return [] populated_node_groups = [] for node_group in node_groups: populated_node_group = self._populate_node_group(context, node_group) self._cleanup_node_group(populated_node_group) populated_node_group["tenant_id"] = context.tenant_id populated_node_groups.append( populated_node_group) return populated_node_groups def _cleanup_node_group(self, node_group): node_group.pop('id', None) node_group.pop('created_at', None) node_group.pop('updated_at', None) def _populate_node_group(self, context, node_group): node_group_merged = copy.deepcopy(NODE_GROUP_DEFAULTS) ng_tmpl_id = node_group.get('node_group_template_id') ng_tmpl = None if ng_tmpl_id: ng_tmpl = self.node_group_template_get(context, ng_tmpl_id) self._cleanup_node_group(ng_tmpl) node_group_merged.update(ng_tmpl) node_group_merged.update(node_group) if ng_tmpl: node_group_merged['node_configs'] = configs.merge_configs( ng_tmpl.get('node_configs'), node_group.get('node_configs')) return node_group_merged # Cluster ops def cluster_get(self, context, cluster, show_progress=False): """Return the cluster or None if it does not exist.""" return self.db.cluster_get(context, cluster, show_progress) def cluster_get_all(self, context, regex_search=False, **kwargs): """Get all clusters filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self.db.cluster_get_all(context, regex_search, **kwargs) def cluster_create(self, context, values): """Create a cluster from the values dictionary.""" # loading defaults merged_values = copy.deepcopy(CLUSTER_DEFAULTS) merged_values['tenant_id'] = context.tenant_id private_key, public_key = crypto.generate_key_pair() merged_values['management_private_key'] = private_key merged_values['management_public_key'] = public_key cluster_template_id = values.get('cluster_template_id') c_tmpl = None if cluster_template_id: c_tmpl = self.cluster_template_get(context, cluster_template_id) del c_tmpl['created_at'] del c_tmpl['updated_at'] del c_tmpl['id'] del c_tmpl['is_public'] del c_tmpl['is_protected'] del c_tmpl['tenant_id'] # updating with cluster_template values merged_values.update(c_tmpl) # updating with values provided in request merged_values.update(values) if c_tmpl: merged_values['cluster_configs'] = configs.merge_configs( c_tmpl.get('cluster_configs'), values.get('cluster_configs')) merged_values['node_groups'] = self._populate_node_groups( context, merged_values) return self.db.cluster_create(context, merged_values) def cluster_update(self, context, cluster, values): """Set the given properties on cluster and update it.""" values = copy.deepcopy(values) update_shares = values.get('shares') if update_shares: original_shares = ( self.db.cluster_get(context, cluster).get('shares', [])) updated_cluster = self.db.cluster_update(context, cluster, values) if update_shares: for share in update_shares: # Only call mount_shares if we have new shares to mount. # We only need one positive case to bother calling mount_shares if share not in original_shares: shares.mount_shares(r.ClusterResource(updated_cluster)) break # Any shares that were on the original, but not on the updated # list will be unmounted unmount_list = [share for share in original_shares if share not in update_shares] if len(unmount_list) > 0: shares.unmount_shares(r.ClusterResource(updated_cluster), unmount_list) return updated_cluster def cluster_destroy(self, context, cluster): """Destroy the cluster or raise if it does not exist.""" self.db.cluster_destroy(context, cluster) # Node Group ops def node_group_add(self, context, cluster, values): """Create a Node Group from the values dictionary.""" values = copy.deepcopy(values) values = self._populate_node_group(context, values) values['tenant_id'] = context.tenant_id return self.db.node_group_add(context, cluster, values) def node_group_update(self, context, node_group, values): """Set the given properties on node_group and update it.""" values = copy.deepcopy(values) self.db.node_group_update(context, node_group, values) def node_group_remove(self, context, node_group): """Destroy the node_group or raise if it does not exist.""" self.db.node_group_remove(context, node_group) # Instance ops def instance_add(self, context, node_group, values): """Create an Instance from the values dictionary.""" values = copy.deepcopy(values) values = _apply_defaults(values, INSTANCE_DEFAULTS) values['tenant_id'] = context.tenant_id return self.db.instance_add(context, node_group, values) def instance_update(self, context, instance, values): """Set the given properties on Instance and update it.""" values = copy.deepcopy(values) self.db.instance_update(context, instance, values) def instance_remove(self, context, instance): """Destroy the Instance or raise if it does not exist.""" self.db.instance_remove(context, instance) # Volumes ops def append_volume(self, context, instance, volume_id): """Append volume_id to instance.""" self.db.append_volume(context, instance, volume_id) def remove_volume(self, context, instance, volume_id): """Remove volume_id in instance.""" self.db.remove_volume(context, instance, volume_id) # Cluster Template ops def cluster_template_get(self, context, cluster_template): """Return the cluster_template or None if it does not exist.""" return self.db.cluster_template_get(context, cluster_template) def cluster_template_get_all(self, context, regex_search=False, **kwargs): """Get all cluster templates filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self.db.cluster_template_get_all(context, regex_search, **kwargs) def cluster_template_create(self, context, values): """Create a cluster_template from the values dictionary.""" values = copy.deepcopy(values) values = _apply_defaults(values, CLUSTER_DEFAULTS) values['tenant_id'] = context.tenant_id values['node_groups'] = self._populate_node_groups(context, values) return self.db.cluster_template_create(context, values) def cluster_template_destroy(self, context, cluster_template, ignore_prot_on_def=False): """Destroy the cluster_template or raise if it does not exist.""" self.db.cluster_template_destroy(context, cluster_template, ignore_prot_on_def) def cluster_template_update(self, context, id, values, ignore_prot_on_def=False): """Update a cluster_template from the values dictionary.""" values = copy.deepcopy(values) values['tenant_id'] = context.tenant_id values['id'] = id if 'node_groups' in values: values['node_groups'] = self._populate_node_groups(context, values) return self.db.cluster_template_update(context, values, ignore_prot_on_def) # Node Group Template ops def node_group_template_get(self, context, node_group_template): """Return the Node Group Template or None if it does not exist.""" return self.db.node_group_template_get(context, node_group_template) def node_group_template_get_all(self, context, regex_search=False, **kwargs): """Get all NodeGroupTemplates filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self.db.node_group_template_get_all(context, regex_search, **kwargs) def node_group_template_create(self, context, values): """Create a Node Group Template from the values dictionary.""" values = copy.deepcopy(values) values = _apply_defaults(values, NODE_GROUP_TEMPLATE_DEFAULTS) values['tenant_id'] = context.tenant_id return self.db.node_group_template_create(context, values) def node_group_template_destroy(self, context, node_group_template, ignore_prot_on_def=False): """Destroy the Node Group Template or raise if it does not exist.""" self.db.node_group_template_destroy(context, node_group_template, ignore_prot_on_def) def node_group_template_update(self, context, id, values, ignore_prot_on_def=False): """Update a Node Group Template from the values dictionary.""" values = copy.deepcopy(values) values['tenant_id'] = context.tenant_id values['id'] = id return self.db.node_group_template_update(context, values, ignore_prot_on_def) # Data Source ops def data_source_get(self, context, data_source): """Return the Data Source or None if it does not exist.""" return self.db.data_source_get(context, data_source) def data_source_get_all(self, context, regex_search=False, **kwargs): """Get all Data Sources filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self.db.data_source_get_all(context, regex_search, **kwargs) def data_source_count(self, context, **kwargs): """Count Data Sources filtered by **kwargs. Uses sqlalchemy "in_" clause for any tuple values Uses sqlalchemy "like" clause for any string values containing % """ return self.db.data_source_count(context, **kwargs) def data_source_create(self, context, values): """Create a Data Source from the values dictionary.""" values = copy.deepcopy(values) values = _apply_defaults(values, DATA_SOURCE_DEFAULTS) values['tenant_id'] = context.tenant_id # if credentials are being passed in, we use the key_manager # to store the password. if (values.get('credentials') and values['credentials'].get('password')): values['credentials']['password'] = key_manager.store_secret( values['credentials']['password'], context) return self.db.data_source_create(context, values) def data_source_destroy(self, context, data_source): """Destroy the Data Source or raise if it does not exist.""" # in cases where the credentials to access the data source are # stored with the record and the external key manager is being # used, we need to delete the key from the external manager. if (CONF.use_barbican_key_manager and not CONF.use_domain_for_proxy_users): ds_record = self.data_source_get(context, data_source) if (ds_record.get('credentials') and ds_record['credentials'].get('password')): key_manager.delete_secret( ds_record['credentials']['password'], context) return self.db.data_source_destroy(context, data_source) def data_source_update(self, context, id, values): """Update the Data Source or raise if it does not exist.""" values = copy.deepcopy(values) values["id"] = id # in cases where the credentials to access the data source are # stored with the record and the external key manager is being # used, we need to delete the old key from the manager and # create a new one. the other option here would be to retrieve # the previous key and check to see if it has changed, but it # seems less expensive to just delete the old and create a new # one. # it should be noted that the jsonschema validation ensures that # if the proxy domain is not in use then credentials must be # sent with this record. if (CONF.use_barbican_key_manager and not CONF.use_domain_for_proxy_users): # first we retrieve the original record to get the old key # uuid, and delete it. ds_record = self.data_source_get(context, id) if (ds_record.get('credentials') and ds_record['credentials'].get('password')): key_manager.delete_secret( ds_record['credentials']['password'], context) # next we create the new key. if (values.get('credentials') and values['credentials'].get('password')): values['credentials']['password'] = key_manager.store_secret( values['credentials']['password'], context) return self.db.data_source_update(context, values) # JobExecution ops def job_execution_get(self, context, job_execution): """Return the JobExecution or None if it does not exist.""" return self.db.job_execution_get(context, job_execution) def job_execution_get_all(self, context, regex_search=False, **kwargs): """Get all JobExecutions filtered by **kwargs. kwargs key values may be the names of fields in a JobExecution plus the following special values with the indicated meaning: 'cluster.name' -- name of the Cluster referenced by the JobExecution 'job.name' -- name of the Job referenced by the JobExecution 'status' -- JobExecution['info']['status'] :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self.db.job_execution_get_all(context, regex_search, **kwargs) def job_execution_count(self, context, **kwargs): """Count number of JobExecutions filtered by **kwargs. e.g. job_execution_count(cluster_id=12, input_id=123) """ return self.db.job_execution_count(context, **kwargs) def job_execution_create(self, context, values): """Create a JobExecution from the values dictionary.""" values = copy.deepcopy(values) values = _apply_defaults(values, JOB_EXECUTION_DEFAULTS) values['tenant_id'] = context.tenant_id return self.db.job_execution_create(context, values) def job_execution_update(self, context, job_execution, values): """Updates a JobExecution from the values dictionary.""" values = copy.deepcopy(values) return self.db.job_execution_update(context, job_execution, values) def job_execution_destroy(self, context, job_execution): """Destroy the JobExecution or raise if it does not exist.""" return self.db.job_execution_destroy(context, job_execution) # Job ops def job_get(self, context, job): """Return the Job or None if it does not exist.""" return self.db.job_get(context, job) def job_get_all(self, context, regex_search=False, **kwargs): """Get all Jobs filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self.db.job_get_all(context, regex_search, **kwargs) def job_create(self, context, values): """Create a Job from the values dictionary.""" values = copy.deepcopy(values) values = _apply_defaults(values, JOB_DEFAULTS) values['tenant_id'] = context.tenant_id return self.db.job_create(context, values) def job_update(self, context, job, values): """Updates a Job from the values dictionary.""" return self.db.job_update(context, job, values) def job_destroy(self, context, job): """Destroy the Job or raise if it does not exist.""" self.db.job_destroy(context, job) # JobBinary ops def job_binary_get_all(self, context, regex_search=False, **kwargs): """Get all JobBinarys filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search e.g. job_binary_get_all(name='wordcount.jar') """ return self.db.job_binary_get_all(context, regex_search, **kwargs) def job_binary_get(self, context, job_binary_id): """Return the JobBinary or None if it does not exist.""" return self.db.job_binary_get(context, job_binary_id) def job_binary_create(self, context, values): """Create a JobBinary from the values dictionary.""" values = copy.deepcopy(values) values = _apply_defaults(values, JOB_BINARY_DEFAULTS) values['tenant_id'] = context.tenant_id # if credentials are being passed in, we use the key_manager # to store the password. if values.get('extra') and values['extra'].get('password'): values['extra']['password'] = key_manager.store_secret( values['extra']['password'], context) if values.get('extra') and values['extra'].get('secretkey'): values['extra']['secretkey'] = key_manager.store_secret( values['extra']['secretkey'], context) return self.db.job_binary_create(context, values) def job_binary_destroy(self, context, job_binary): """Destroy the JobBinary or raise if it does not exist.""" # in cases where the credentials to access the job binary are # stored with the record and the external key manager is being # used, we need to delete the key from the external manager. if CONF.use_barbican_key_manager: jb_record = self.job_binary_get(context, job_binary) if not CONF.use_domain_for_proxy_users: if (jb_record.get('extra') and jb_record['extra'].get('password')): key_manager.delete_secret(jb_record['extra']['password'], context) if (jb_record.get('extra') and jb_record['extra'].get('secretkey')): key_manager.delete_secret(jb_record['extra']['secretkey'], context) self.db.job_binary_destroy(context, job_binary) def job_binary_update(self, context, id, values): """Update a JobBinary from the values dictionary.""" values = copy.deepcopy(values) values['id'] = id # in cases where the credentials to access the job binary are # stored with the record and the external key manager is being # used, we need to delete the old key from the manager and # create a new one. the other option here would be to retrieve # the previous key and check to see if it has changed, but it # seems less expensive to just delete the old and create a new # one. if CONF.use_barbican_key_manager: # first we retrieve the original record to get the old key # uuid, and delete it. # next we create the new key. jb_record = self.job_binary_get(context, id) if not CONF.use_domain_for_proxy_users: if (jb_record.get('extra') and jb_record['extra'].get('password')): key_manager.delete_secret(jb_record['extra']['password'], context) if values.get('extra') and values['extra'].get('password'): values['extra']['password'] = key_manager.store_secret( values['extra']['password'], context) if jb_record.get('extra') and jb_record['extra'].get('secretkey'): key_manager.delete_secret(jb_record['extra']['secretkey'], context) if values.get('extra') and values['extra'].get('secretkey'): values['extra']['secretkey'] = key_manager.store_secret( values['extra']['secretkey'], context) return self.db.job_binary_update(context, values) # JobBinaryInternal ops def job_binary_internal_get_all(self, context, regex_search=False, **kwargs): """Get all JobBinaryInternals filtered by **kwargs. The JobBinaryInternals returned do not contain a data field. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self.db.job_binary_internal_get_all(context, regex_search, **kwargs) def job_binary_internal_get(self, context, job_binary_internal_id): """Return the JobBinaryInternal or None if it does not exist The JobBinaryInternal returned does not contain a data field. """ return self.db.job_binary_internal_get(context, job_binary_internal_id) def job_binary_internal_create(self, context, values): """Create a JobBinaryInternal from the values dictionary.""" # Since values["data"] is (should be) encoded as a string # here the deepcopy of values only incs a reference count on data. # This is nice, since data could be big... values = copy.deepcopy(values) values = _apply_defaults(values, JOB_BINARY_INTERNAL_DEFAULTS) values['tenant_id'] = context.tenant_id return self.db.job_binary_internal_create(context, values) def job_binary_internal_destroy(self, context, job_binary_internal): """Destroy the JobBinaryInternal or raise if it does not exist.""" self.db.job_binary_internal_destroy(context, job_binary_internal) def job_binary_internal_get_raw_data(self, context, job_binary_internal_id): """Return the binary data field from a JobBinaryInternal.""" return self.db.job_binary_internal_get_raw_data( context, job_binary_internal_id) def job_binary_internal_update(self, context, id, values): """Updates a JobBinaryInternal from the values dictionary.""" return self.db.job_binary_internal_update(context, id, values) # Events ops def cluster_provision_step_add(self, context, cluster_id, values): """Create a provisioning step assigned to cluster from values dict.""" return self.db.cluster_provision_step_add(context, cluster_id, values) def cluster_provision_step_update(self, context, provision_step): """Update the cluster provisioning step.""" return self.db.cluster_provision_step_update(context, provision_step) def cluster_provision_progress_update(self, context, cluster_id): """Return cluster with provision progress updated field.""" return self.db.cluster_provision_progress_update(context, cluster_id) def cluster_event_add(self, context, provision_step, values): """Assign new event to the specified provision step.""" return self.db.cluster_event_add(context, provision_step, values) # Cluster verifications / health checks ops def cluster_verification_add(self, context, cluster_id, values): """Return created verification for the specified cluster.""" return self.db.cluster_verification_add(context, cluster_id, values) def cluster_verification_get(self, context, verification_id): """Return verification with the specified verification_id.""" return self.db.cluster_verification_get(context, verification_id) def cluster_verification_update(self, context, verification_id, values): """Return updated verification with the specified verification_id.""" return self.db.cluster_verification_update( context, verification_id, values) def cluster_verification_delete(self, context, verification_id): """"Delete verification with the specified id.""" return self.db.cluster_verification_delete(context, verification_id) def cluster_health_check_add(self, context, verification_id, values): """Return created health check in the specified verification.""" return self.db.cluster_health_check_add( context, verification_id, values) def cluster_health_check_get(self, context, health_check_id): """Return health check with the specified health_check_id.""" return self.db.cluster_health_check_get(context, health_check_id) def cluster_health_check_update(self, context, health_check_id, values): """Return updated health check with the specified health_check_id.""" return self.db.cluster_health_check_update( context, health_check_id, values) def plugin_create(self, context, values): """Return created DB entry for plugin.""" return self.db.plugin_create(context, values) def plugin_get(self, context, name): """Return DB entry for plugin.""" return self.db.plugin_get(context, name) def plugin_get_all(self, context): """Return DB entries for all plugins.""" return self.db.plugin_get_all(context) def plugin_update(self, context, name, values): """Return updated DB entry for plugin.""" return self.db.plugin_update(context, name, values) def plugin_remove(self, context, name): """Remove DB entry for plugin.""" return self.db.plugin_remove(context, name) sahara-8.0.0/sahara/conductor/api.py0000666000175100017510000006037513245514472017410 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Handles all requests to the conductor service.""" from oslo_config import cfg from sahara.conductor import manager from sahara.conductor import resource as r conductor_opts = [ cfg.BoolOpt('use_local', default=True, help='Perform sahara-conductor operations locally.'), ] conductor_group = cfg.OptGroup(name='conductor', title='Conductor Options') CONF = cfg.CONF CONF.register_group(conductor_group) CONF.register_opts(conductor_opts, conductor_group) def _get_id(obj): """Return object id. Allows usage of both an object or an object's ID as a parameter when dealing with relationships. """ try: return obj.id except AttributeError: return obj class LocalApi(object): """A local version of the conductor API. It does database updates locally instead of via RPC. """ def __init__(self): self._manager = manager.ConductorManager() # Cluster ops @r.wrap(r.ClusterResource) def cluster_get(self, context, cluster, show_progress=False): """Return the cluster or None if it does not exist.""" return self._manager.cluster_get( context, _get_id(cluster), show_progress) @r.wrap(r.ClusterResource) def cluster_get_all(self, context, regex_search=False, **kwargs): """Get all clusters filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self._manager.cluster_get_all(context, regex_search, **kwargs) @r.wrap(r.ClusterResource) def cluster_create(self, context, values): """Create a cluster from the values dictionary. :returns: the created cluster. """ return self._manager.cluster_create(context, values) @r.wrap(r.ClusterResource) def cluster_update(self, context, cluster, values): """Update the cluster with the given values dictionary. :returns: the updated cluster. """ return self._manager.cluster_update(context, _get_id(cluster), values) def cluster_destroy(self, context, cluster): """Destroy the cluster or raise if it does not exist. :returns: None. """ self._manager.cluster_destroy(context, _get_id(cluster)) # Node Group ops def node_group_add(self, context, cluster, values): """Create a node group from the values dictionary. :returns: ID of the created node group. """ return self._manager.node_group_add(context, _get_id(cluster), values) def node_group_update(self, context, node_group, values): """Update the node group with the given values dictionary. :returns: None. """ self._manager.node_group_update(context, _get_id(node_group), values) def node_group_remove(self, context, node_group): """Destroy the node group or raise if it does not exist. :returns: None. """ self._manager.node_group_remove(context, _get_id(node_group)) # Instance ops def instance_add(self, context, node_group, values): """Create an instance from the values dictionary. :returns: ID of the created instance. """ return self._manager.instance_add(context, _get_id(node_group), values) def instance_update(self, context, instance, values): """Update the instance with the given values dictionary. :returns: None. """ self._manager.instance_update(context, _get_id(instance), values) def instance_remove(self, context, instance): """Destroy the instance or raise if it does not exist. :returns: None. """ self._manager.instance_remove(context, _get_id(instance)) # Volumes ops def append_volume(self, context, instance, volume_id): """Append volume_id to instance.""" self._manager.append_volume(context, _get_id(instance), volume_id) def remove_volume(self, context, instance, volume_id): """Remove volume_id in instance.""" self._manager.remove_volume(context, _get_id(instance), volume_id) # Cluster Template ops @r.wrap(r.ClusterTemplateResource) def cluster_template_get(self, context, cluster_template): """Return the cluster template or None if it does not exist.""" return self._manager.cluster_template_get(context, _get_id(cluster_template)) @r.wrap(r.ClusterTemplateResource) def cluster_template_get_all(self, context, regex_search=False, **kwargs): """Get all cluster templates filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self._manager.cluster_template_get_all(context, regex_search, **kwargs) @r.wrap(r.ClusterTemplateResource) def cluster_template_create(self, context, values): """Create a cluster template from the values dictionary. :returns: the created cluster template """ return self._manager.cluster_template_create(context, values) def cluster_template_destroy(self, context, cluster_template, ignore_prot_on_def=False): """Destroy the cluster template or raise if it does not exist. :returns: None """ self._manager.cluster_template_destroy(context, _get_id(cluster_template), ignore_prot_on_def) @r.wrap(r.ClusterTemplateResource) def cluster_template_update(self, context, id, cluster_template, ignore_prot_on_def=False): """Update the cluster template or raise if it does not exist. :returns: the updated cluster template """ return self._manager.cluster_template_update(context, id, cluster_template, ignore_prot_on_def) # Node Group Template ops @r.wrap(r.NodeGroupTemplateResource) def node_group_template_get(self, context, node_group_template): """Return the node group template or None if it does not exist.""" return self._manager.node_group_template_get( context, _get_id(node_group_template)) @r.wrap(r.NodeGroupTemplateResource) def node_group_template_get_all(self, context, regex_search=False, **kwargs): """Get all node group templates filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self._manager.node_group_template_get_all( context, regex_search, **kwargs) @r.wrap(r.NodeGroupTemplateResource) def node_group_template_create(self, context, values): """Create a node group template from the values dictionary. :returns: the created node group template """ return self._manager.node_group_template_create(context, values) def node_group_template_destroy(self, context, node_group_template, ignore_prot_on_def=False): """Destroy the node group template or raise if it does not exist. :returns: None """ self._manager.node_group_template_destroy(context, _get_id(node_group_template), ignore_prot_on_def) @r.wrap(r.NodeGroupTemplateResource) def node_group_template_update(self, context, id, values, ignore_prot_on_def=False): """Update a node group template from the values dictionary. :returns: the updated node group template """ return self._manager.node_group_template_update(context, id, values, ignore_prot_on_def) # Data Source ops @r.wrap(r.DataSource) def data_source_get(self, context, data_source): """Return the Data Source or None if it does not exist.""" return self._manager.data_source_get(context, _get_id(data_source)) @r.wrap(r.DataSource) def data_source_get_all(self, context, regex_search=False, **kwargs): """Get all Data Sources filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self._manager.data_source_get_all(context, regex_search, **kwargs) def data_source_count(self, context, **kwargs): """Count Data Sources filtered by **kwargs. Uses sqlalchemy "in_" clause for any tuple values Uses sqlalchemy "like" clause for any string values containing % """ return self._manager.data_source_count(context, **kwargs) @r.wrap(r.DataSource) def data_source_create(self, context, values): """Create a Data Source from the values dictionary.""" return self._manager.data_source_create(context, values) def data_source_destroy(self, context, data_source): """Destroy the Data Source or raise if it does not exist.""" self._manager.data_source_destroy(context, _get_id(data_source)) @r.wrap(r.DataSource) def data_source_update(self, context, id, values): """Update an existing Data Source""" return self._manager.data_source_update(context, id, values) # JobExecution ops @r.wrap(r.JobExecution) def job_execution_get(self, context, job_execution): """Return the JobExecution or None if it does not exist.""" return self._manager.job_execution_get(context, _get_id(job_execution)) @r.wrap(r.JobExecution) def job_execution_get_all(self, context, regex_search=False, **kwargs): """Get all JobExecutions filtered by **kwargs. kwargs key values may be the names of fields in a JobExecution plus the following special values with the indicated meaning: 'cluster.name' -- name of the Cluster referenced by the JobExecution 'job.name' -- name of the Job referenced by the JobExecution 'status' -- JobExecution['info']['status'] :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self._manager.job_execution_get_all(context, regex_search, **kwargs) def job_execution_count(self, context, **kwargs): """Count number of JobExecutions filtered by **kwargs. e.g. job_execution_count(cluster_id=12, input_id=123) """ return self._manager.job_execution_count(context, **kwargs) @r.wrap(r.JobExecution) def job_execution_create(self, context, values): """Create a JobExecution from the values dictionary.""" return self._manager.job_execution_create(context, values) @r.wrap(r.JobExecution) def job_execution_update(self, context, job_execution, values): """Update the JobExecution or raise if it does not exist.""" return self._manager.job_execution_update(context, _get_id(job_execution), values) def job_execution_destroy(self, context, job_execution): """Destroy the JobExecution or raise if it does not exist.""" self._manager.job_execution_destroy(context, _get_id(job_execution)) # Job ops @r.wrap(r.Job) def job_get(self, context, job): """Return the Job or None if it does not exist.""" return self._manager.job_get(context, _get_id(job)) @r.wrap(r.Job) def job_get_all(self, context, regex_search=False, **kwargs): """Get all Jobs filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self._manager.job_get_all(context, regex_search, **kwargs) @r.wrap(r.Job) def job_create(self, context, values): """Create a Job from the values dictionary.""" return self._manager.job_create(context, values) @r.wrap(r.Job) def job_update(self, context, job, values): """Update the Job or raise if it does not exist.""" return self._manager.job_update(context, _get_id(job), values) def job_destroy(self, context, job): """Destroy the Job or raise if it does not exist.""" self._manager.job_destroy(context, _get_id(job)) def job_main_name(self, context, job): """Return the name of the first main JobBinary or None. At present the 'mains' element is expected to contain a single element. In the future if 'mains' contains more than one element we will need a scheme or convention for retrieving a name from the list of binaries. :param job: This is expected to be a Job object """ if job.mains: binary = self.job_binary_get(context, job.mains[0]) if binary is not None: return binary["name"] return None def job_lib_names(self, context, job): """Return the name of all job lib binaries or an empty list. :param job: This is expected to be a Job object """ lib_ids = job.libs or [] binaries = (self.job_binary_get(context, lib_id) for lib_id in lib_ids) return [binary["name"] for binary in binaries if binary is not None] # JobBinary ops @r.wrap(r.JobBinary) def job_binary_get_all(self, context, regex_search=False, **kwargs): """Get all JobBinarys filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self._manager.job_binary_get_all(context, regex_search, **kwargs) @r.wrap(r.JobBinary) def job_binary_get(self, context, job_binary): """Return the JobBinary or None if it does not exist.""" return self._manager.job_binary_get(context, _get_id(job_binary)) @r.wrap(r.JobBinary) def job_binary_create(self, context, values): """Create a JobBinary from the values dictionary.""" return self._manager.job_binary_create(context, values) def job_binary_destroy(self, context, job_binary): """Destroy the JobBinary or raise if it does not exist.""" self._manager.job_binary_destroy(context, _get_id(job_binary)) @r.wrap(r.JobBinary) def job_binary_update(self, context, id, values): """Update a JobBinary from the values dictionary.""" return self._manager.job_binary_update(context, id, values) # JobBinaryInternal ops @r.wrap(r.JobBinaryInternal) def job_binary_internal_get_all(self, context, regex_search=False, **kwargs): """Get all JobBinaryInternals filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return self._manager.job_binary_internal_get_all( context, regex_search, **kwargs) @r.wrap(r.JobBinaryInternal) def job_binary_internal_get(self, context, job_binary_internal): """Return the JobBinaryInternal or None if it does not exist.""" return self._manager.job_binary_internal_get( context, _get_id(job_binary_internal)) @r.wrap(r.JobBinaryInternal) def job_binary_internal_create(self, context, values): """Create a JobBinaryInternal from the values dictionary.""" return self._manager.job_binary_internal_create(context, values) def job_binary_internal_destroy(self, context, job_binary_internal_id): """Destroy the JobBinaryInternal or raise if it does not exist.""" self._manager.job_binary_internal_destroy( context, _get_id(job_binary_internal_id)) def job_binary_internal_get_raw_data(self, context, job_binary_internal_id): """Return the binary data field from a JobBinaryInternal.""" return self._manager.job_binary_internal_get_raw_data( context, job_binary_internal_id) @r.wrap(r.JobBinaryInternal) def job_binary_internal_update(self, context, job_binary_internal, values): """Update a JobBinaryInternal from the values dictionary.""" return self._manager.job_binary_internal_update( context, _get_id(job_binary_internal), values) # Events ops def cluster_provision_step_add(self, context, cluster_id, values): """Create a provisioning step assigned to cluster from values dict.""" return self._manager.cluster_provision_step_add( context, cluster_id, values) def cluster_provision_step_update(self, context, provision_step): """Update the cluster provisioning step.""" return self._manager.cluster_provision_step_update( context, provision_step) def cluster_provision_progress_update(self, context, cluster_id): """Return cluster with provision progress updated field.""" return self._manager.cluster_provision_progress_update( context, cluster_id) def cluster_event_add(self, context, provision_step, values): """Assign new event to the specified provision step.""" return self._manager.cluster_event_add( context, provision_step, values) @r.wrap(r.ClusterVerificationResource) def cluster_verification_add(self, context, cluster_id, values): """Return created verification for the specified cluster.""" return self._manager.cluster_verification_add( context, _get_id(cluster_id), values) @r.wrap(r.ClusterVerificationResource) def cluster_verification_get(self, context, verification_id): """Return verification with the specified verification_id.""" return self._manager.cluster_verification_get( context, _get_id(verification_id)) @r.wrap(r.ClusterVerificationResource) def cluster_verification_update(self, context, verification_id, values): """Return updated verification with the specified verification_id.""" return self._manager.cluster_verification_update( context, _get_id(verification_id), values) def cluster_verification_delete(self, context, verification_id): """"Delete verification with the specified id.""" return self._manager.cluster_verification_delete( context, _get_id(verification_id)) @r.wrap(r.ClusterHealthCheckResource) def cluster_health_check_add(self, context, verification_id, values): """Return created health check in the specified verification.""" return self._manager.cluster_health_check_add( context, _get_id(verification_id), values) @r.wrap(r.ClusterHealthCheckResource) def cluster_health_check_get(self, context, health_check_id): """Return health check with the specified health_check_id.""" return self._manager.cluster_health_check_get( context, _get_id(health_check_id)) @r.wrap(r.ClusterHealthCheckResource) def cluster_health_check_update(self, context, health_check_id, values): """Return updated health check with the specified health_check_id.""" return self._manager.cluster_health_check_update( context, _get_id(health_check_id), values) def plugin_create(self, context, values): """Return created DB entry for plugin.""" return self._manager.plugin_create(context, values) def plugin_get(self, context, name): """Return DB entry for plugin.""" return self._manager.plugin_get(context, name) def plugin_get_all(self, context): """Return DB entries for all plugins.""" return self._manager.plugin_get_all(context) def plugin_update(self, context, name, values): """Return updated DB entry for plugin.""" return self._manager.plugin_update(context, name, values) def plugin_remove(self, context, name): """Remove DB entry for plugin.""" return self._manager.plugin_remove(context, name) class RemoteApi(LocalApi): """Conductor API that does updates via RPC to the ConductorManager.""" # TODO(slukjanov): it should override _manager and only necessary functions sahara-8.0.0/sahara/conductor/resource.py0000666000175100017510000002307613245514472020463 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Provides means to wrap dicts coming from DB layer in objects. The Conductor can fetch only values represented by JSON. That limitation comes from Oslo RPC implementation. This module provides means to wrap a fetched value, always dictionary, into an immutable Resource object. A descendant of Resource class might provide back references to parent objects and helper methods. """ import datetime import six from sahara.conductor import objects from sahara import exceptions as ex from sahara.i18n import _ from sahara.swift import swift_helper from sahara.utils import types def wrap(resource_class): """A decorator wraps dict returned by a given function into a Resource.""" def decorator(func): def handle(*args, **kwargs): ret = func(*args, **kwargs) if isinstance(ret, types.Page): return types.Page([resource_class(el) for el in ret], ret.prev, ret.next) elif isinstance(ret, list): return [resource_class(el) for el in ret] elif ret: return resource_class(ret) else: return None return handle return decorator class Resource(types.FrozenDict): """Represents dictionary as an immutable object. Enhancing it with back references and helper methods. For instance, the following dictionary: {'first': {'a': 1, 'b': 2}, 'second': [1,2,3]} after wrapping with Resource will look like an object, let it be 'res' with the following fields: res.first res.second 'res.first' will in turn be wrapped into Resource with two fields: res.first.a == 1 res.first.b == 2 'res.second', which is a list, will be transformed into a tuple for immutability: res.second == (1,2,3) Additional helper methods could be specified in descendant classes. '_children' specifies children of that specific Resource in the following format: {refname: (child_class, backref_name)} Back reference is a reference to parent object which is injected into a Resource during wrapping. """ _resource_name = 'resource' _children = {} _filter_fields = [] _sanitize_fields = {} def __init__(self, dct): super(Resource, self).__setattr__('_initial_dict', dct) newdct = dict() for refname, entity in six.iteritems(dct): newdct[refname] = self._wrap_entity(refname, entity) super(Resource, self).__init__(newdct) def to_dict(self): """Return dictionary representing the Resource for REST API. On the way filter out fields which shouldn't be exposed. """ return self._to_dict(None) def to_wrapped_dict(self): return {self._resource_name: self.to_dict()} # Construction def _wrap_entity(self, refname, entity): if isinstance(entity, Resource): # that is a back reference return entity elif isinstance(entity, list): return self._wrap_list(refname, entity) elif isinstance(entity, dict): return self._wrap_dict(refname, entity) elif self._is_passthrough_type(entity): return entity else: raise TypeError(_("Unsupported type: %s") % type(entity).__name__) def _wrap_list(self, refname, lst): newlst = [self._wrap_entity(refname, entity) for entity in lst] return types.FrozenList(newlst) def _wrap_dict(self, refname, dct): if refname in self._children: dct = dict(dct) child_class = self._children[refname][0] backref_name = self._children[refname][1] if backref_name: dct[backref_name] = self return child_class(dct) else: return Resource(dct) def _is_passthrough_type(self, entity): return (entity is None or isinstance(entity, (six.integer_types, float, datetime.datetime, six.string_types))) # Conversion to dict def _to_dict(self, backref): dct = dict() for refname, entity in six.iteritems(self): if refname != backref and refname not in self._filter_fields: childs_backref = None if refname in self._children: childs_backref = self._children[refname][1] dct[refname] = self._entity_to_dict(entity, childs_backref) sanitize = self._sanitize_fields.get(refname) if sanitize is not None: dct[refname] = sanitize(self, dct[refname]) return dct def _entity_to_dict(self, entity, childs_backref): if isinstance(entity, Resource): return entity._to_dict(childs_backref) elif isinstance(entity, list): return self._list_to_dict(entity, childs_backref) elif entity is not None: return entity def _list_to_dict(self, lst, childs_backref): return [self._entity_to_dict(entity, childs_backref) for entity in lst] def __getattr__(self, item): return self[item] def __setattr__(self, *args): raise ex.FrozenClassError(self) class NodeGroupTemplateResource(Resource, objects.NodeGroupTemplate): _resource_name = 'node_group_template' class InstanceResource(Resource, objects.Instance): _filter_fields = ['tenant_id', 'node_group_id', "volumes"] @property def cluster_id(self): return self.node_group.cluster_id @property def cluster(self): return self.node_group.cluster class NodeGroupResource(Resource, objects.NodeGroup): _children = { 'instances': (InstanceResource, 'node_group'), 'node_group_template': (NodeGroupTemplateResource, None) } _filter_fields = ['tenant_id', 'cluster_id', 'cluster_template_id', 'image_username', 'open_ports'] class ClusterTemplateResource(Resource, objects.ClusterTemplate): _resource_name = 'cluster_template' _children = { 'node_groups': (NodeGroupResource, 'cluster_template') } class ClusterHealthCheckResource(Resource, objects.ClusterHealthCheck): _resource_name = 'cluster_health_check' class ClusterVerificationResource(Resource, objects.ClusterVerification): _resource_name = 'cluster_verification' _children = { 'checks': (ClusterHealthCheckResource, 'verification') } class ClusterResource(Resource, objects.Cluster): def sanitize_cluster_configs(self, cluster_configs): if 'proxy_configs' in cluster_configs: del cluster_configs['proxy_configs'] return cluster_configs _resource_name = 'cluster' _children = { 'node_groups': (NodeGroupResource, 'cluster'), 'cluster_template': (ClusterTemplateResource, None), 'verification': (ClusterVerificationResource, 'cluster') } _filter_fields = ['management_private_key', 'extra', 'rollback_info', 'sahara_info'] _sanitize_fields = {'cluster_configs': sanitize_cluster_configs} class ImageResource(Resource, objects.Image): _resource_name = 'image' @property def dict(self): return self.to_dict() @property def wrapped_dict(self): return {'image': self.dict} def _sanitize_image_properties(self, image_props): if 'links' in image_props: del image_props['links'] return image_props _sanitize_fields = {'links': _sanitize_image_properties} # EDP Resources class DataSource(Resource, objects.DataSource): _resource_name = "data_source" _filter_fields = ['credentials'] class JobExecution(Resource, objects.JobExecution): def sanitize_job_configs(self, job_configs): if 'configs' in job_configs: configs = job_configs['configs'] if swift_helper.HADOOP_SWIFT_USERNAME in configs: configs[swift_helper.HADOOP_SWIFT_USERNAME] = "" if swift_helper.HADOOP_SWIFT_PASSWORD in configs: configs[swift_helper.HADOOP_SWIFT_PASSWORD] = "" if 'trusts' in job_configs: del job_configs['trusts'] if 'proxy_configs' in job_configs: del job_configs['proxy_configs'] return job_configs def sanitize_info(self, info): if 'actions' in info: for d in info['actions']: if 'conf' in d: del d['conf'] return info _resource_name = "job_execution" _filter_fields = ['extra'] _sanitize_fields = {'job_configs': sanitize_job_configs, 'info': sanitize_info} # TODO(egafford): Sanitize interface ("secret" bool field on job args?) class JobBinary(Resource, objects.JobBinary): _resource_name = "job_binary" _filter_fields = ['extra'] class JobBinaryInternal(Resource, objects.JobBinaryInternal): _resource_name = "job_binary_internal" class Job(Resource, objects.Job): _resource_name = "job" _children = { 'mains': (JobBinary, None), 'libs': (JobBinary, None) } sahara-8.0.0/sahara/conductor/__init__.py0000666000175100017510000000223513245514472020365 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from sahara.conductor import api as conductor_api from sahara.i18n import _ def Api(use_local=True, **kwargs): """Creates local or remote conductor Api. Creation of local or remote conductor Api depends on passed arg 'use_local' and config option 'use_local' in 'conductor' group. """ if cfg.CONF.conductor.use_local or use_local: api = conductor_api.LocalApi else: raise NotImplementedError( _("Remote conductor isn't implemented yet.")) # api = conductor.RemoteApi return api(**kwargs) API = Api() sahara-8.0.0/sahara/api/0000775000175100017510000000000013245515026015015 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/api/v2/0000775000175100017510000000000013245515026015344 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/api/v2/cluster_templates.py0000666000175100017510000000746513245514472021477 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api.v2 import cluster_templates as api from sahara.service import validation as v from sahara.service.validations import cluster_template_schema as ct_schema from sahara.service.validations import cluster_templates as v_ct import sahara.utils.api as u rest = u.RestV2('cluster-templates', __name__) @rest.get('/cluster-templates') @acl.enforce("data-processing:cluster-templates:get_all") @v.check_exists(api.get_cluster_template, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_cluster_templates) def cluster_templates_list(): result = api.get_cluster_templates(**u.get_request_args().to_dict()) return u.render(res=result, name='cluster_templates') @rest.post('/cluster-templates') @acl.enforce("data-processing:cluster-templates:create") @v.validate(ct_schema.CLUSTER_TEMPLATE_SCHEMA_V2, v_ct.check_cluster_template_create) def cluster_templates_create(data): # renaming hadoop_version -> plugin_version # this can be removed once APIv1 is deprecated data['hadoop_version'] = data['plugin_version'] del data['plugin_version'] return u.render(api.create_cluster_template(data).to_wrapped_dict()) @rest.get('/cluster-templates/') @acl.enforce("data-processing:cluster-templates:get") @v.check_exists(api.get_cluster_template, 'cluster_template_id') def cluster_templates_get(cluster_template_id): return u.to_wrapped_dict(api.get_cluster_template, cluster_template_id) @rest.patch('/cluster-templates/') @acl.enforce("data-processing:cluster-templates:modify") @v.check_exists(api.get_cluster_template, 'cluster_template_id') @v.validate(ct_schema.CLUSTER_TEMPLATE_UPDATE_SCHEMA_V2, v_ct.check_cluster_template_update) def cluster_templates_update(cluster_template_id, data): if data.get('plugin_version', None): data['hadoop_version'] = data['plugin_version'] del data['plugin_version'] return u.to_wrapped_dict( api.update_cluster_template, cluster_template_id, data) @rest.delete('/cluster-templates/') @acl.enforce("data-processing:cluster-templates:delete") @v.check_exists(api.get_cluster_template, 'cluster_template_id') @v.validate(None, v_ct.check_cluster_template_usage) def cluster_templates_delete(cluster_template_id): api.terminate_cluster_template(cluster_template_id) return u.render() def _cluster_template_export_helper(template): template.pop('id') template.pop('updated_at') template.pop('created_at') template.pop('tenant_id') template.pop('is_default') template['default_image_id'] = '{default_image_id}' template['node_groups'] = '{node_groups}' @rest.get('/cluster-templates//export') @acl.enforce("data-processing:cluster-templates:get") @v.check_exists(api.get_cluster_template, 'cluster_template_id') def cluster_template_export(cluster_template_id): content = u.to_wrapped_dict_no_render( api.export_cluster_template, cluster_template_id) _cluster_template_export_helper(content['cluster_template']) res = u.render(content) res.headers.add('Content-Disposition', 'attachment', filename='cluster_template.json') return res sahara-8.0.0/sahara/api/v2/data_sources.py0000666000175100017510000000453213245514472020404 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api.v2 import data_sources as api from sahara.service import validation as v from sahara.service.validations.edp import data_source as v_d_s from sahara.service.validations.edp import data_source_schema as v_d_s_schema import sahara.utils.api as u rest = u.RestV2('data-sources', __name__) @rest.get('/data-sources') @acl.enforce("data-processing:data-sources:get_all") @v.check_exists(api.get_data_source, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_data_sources) def data_sources_list(): result = api.get_data_sources(**u.get_request_args().to_dict()) return u.render(res=result, name='data_sources') @rest.post('/data-sources') @acl.enforce("data-processing:data-sources:register") @v.validate(v_d_s_schema.DATA_SOURCE_SCHEMA, v_d_s.check_data_source_create) def data_source_register(data): return u.render(api.register_data_source(data).to_wrapped_dict()) @rest.get('/data-sources/') @acl.enforce("data-processing:data-sources:get") @v.check_exists(api.get_data_source, 'data_source_id') def data_source_get(data_source_id): return u.to_wrapped_dict(api.get_data_source, data_source_id) @rest.delete('/data-sources/') @acl.enforce("data-processing:data-sources:delete") @v.check_exists(api.get_data_source, 'data_source_id') def data_source_delete(data_source_id): api.delete_data_source(data_source_id) return u.render() @rest.patch('/data-sources/') @acl.enforce("data-processing:data-sources:modify") @v.check_exists(api.get_data_source, 'data_source_id') @v.validate(v_d_s_schema.DATA_SOURCE_UPDATE_SCHEMA) def data_source_update(data_source_id, data): return u.to_wrapped_dict(api.data_source_update, data_source_id, data) sahara-8.0.0/sahara/api/v2/job_binaries.py0000666000175100017510000000517713245514472020364 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api.v2 import job_binaries as api from sahara.service import validation as v from sahara.service.validations.edp import job_binary as v_j_b from sahara.service.validations.edp import job_binary_schema as v_j_b_schema import sahara.utils.api as u rest = u.RestV2('job-binaries', __name__) @rest.post('/job-binaries') @acl.enforce("data-processing:job-binaries:create") @v.validate(v_j_b_schema.JOB_BINARY_SCHEMA, v_j_b.check_job_binary) def job_binary_create(data): return u.render(api.create_job_binary(data).to_wrapped_dict()) @rest.get('/job-binaries') @acl.enforce("data-processing:job-binaries:get_all") @v.check_exists(api.get_job_binary, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_job_binaries) def job_binary_list(): result = api.get_job_binaries(**u.get_request_args().to_dict()) return u.render(res=result, name='binaries') @rest.get('/job-binaries/') @acl.enforce("data-processing:job-binaries:get") @v.check_exists(api.get_job_binary, 'job_binary_id') def job_binary_get(job_binary_id): return u.to_wrapped_dict(api.get_job_binary, job_binary_id) @rest.delete('/job-binaries/') @acl.enforce("data-processing:job-binaries:delete") @v.check_exists(api.get_job_binary, id='job_binary_id') def job_binary_delete(job_binary_id): api.delete_job_binary(job_binary_id) return u.render() @rest.get('/job-binaries//data') @acl.enforce("data-processing:job-binaries:get_data") @v.check_exists(api.get_job_binary, 'job_binary_id') def job_binary_data(job_binary_id): data = api.get_job_binary_data(job_binary_id) if type(data) == dict: data = u.render(data) return data @rest.patch('/job-binaries/') @acl.enforce("data-processing:job-binaries:modify") @v.validate(v_j_b_schema.JOB_BINARY_UPDATE_SCHEMA, v_j_b.check_job_binary) def job_binary_update(job_binary_id, data): return u.render(api.update_job_binary(job_binary_id, data).to_wrapped_dict()) sahara-8.0.0/sahara/api/v2/plugins.py0000666000175100017510000000346513245514472017415 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api.v2 import plugins as api from sahara.service import validation as v from sahara.service.validations import plugins as v_p import sahara.utils.api as u rest = u.RestV2('plugins', __name__) @rest.get('/plugins') @acl.enforce("data-processing:plugins:get_all") def plugins_list(): return u.render(plugins=[p.dict for p in api.get_plugins()]) @rest.get('/plugins/') @acl.enforce("data-processing:plugins:get") @v.check_exists(api.get_plugin, plugin_name='plugin_name') def plugins_get(plugin_name): return u.render(api.get_plugin(plugin_name).wrapped_dict) @rest.get('/plugins//') @acl.enforce("data-processing:plugins:get_version") @v.check_exists(api.get_plugin, plugin_name='plugin_name', version='version') def plugins_get_version(plugin_name, version): return u.render(api.get_plugin(plugin_name, version).wrapped_dict) @rest.patch('/plugins/') @acl.enforce("data-processing:plugins:patch") @v.check_exists(api.get_plugin, plugin_name='plugin_name') @v.validate(v_p.plugin_update_validation_jsonschema(), v_p.check_plugin_update) def plugins_update(plugin_name, data): return u.render(api.update_plugin(plugin_name, data).wrapped_dict) sahara-8.0.0/sahara/api/v2/images.py0000666000175100017510000000504413245514472017174 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api.v2 import images as api from sahara.service import validation as v from sahara.service.validations import images as v_images import sahara.utils.api as u rest = u.RestV2('images', __name__) @rest.get('/images') @acl.enforce("data-processing:images:get_all") def images_list(): tags = u.get_request_args().getlist('tags') name = u.get_request_args().get('name', None) return u.render(images=[i.dict for i in api.get_images(name, tags)]) @rest.get('/images/') @acl.enforce("data-processing:images:get") @v.check_exists(api.get_image, id='image_id') def images_get(image_id): return u.render(api.get_registered_image(id=image_id).wrapped_dict) @rest.post('/images/') @acl.enforce("data-processing:images:register") @v.check_exists(api.get_image, id='image_id') @v.validate(v_images.image_register_schema, v_images.check_image_register) def images_set(image_id, data): return u.render(api.register_image(image_id, **data).wrapped_dict) @rest.delete('/images/') @acl.enforce("data-processing:images:unregister") @v.check_exists(api.get_image, id='image_id') def images_unset(image_id): api.unregister_image(image_id) return u.render() @rest.get('/images//tags') @acl.enforce("data-processing:images:get_tags") @v.check_exists(api.get_image, id='image_id') def image_tags_get(image_id): return u.render(api.get_image_tags(image_id)) @rest.put('/images//tags', status_code=200) @acl.enforce("data-processing:images:set_tags") @v.check_exists(api.get_image, id='image_id') @v.validate(v_images.image_tags_schema, v_images.check_tags) def image_tags_update(image_id, data): return u.render(api.set_image_tags(image_id, **data).wrapped_dict) @rest.delete('/images//tags') @acl.enforce("data-processing:images:remove_tags") @v.check_exists(api.get_image, id='image_id') def image_tags_delete(image_id): api.remove_image_tags(image_id) return u.render() sahara-8.0.0/sahara/api/v2/clusters.py0000666000175100017510000000651413245514472017576 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six from sahara.api import acl from sahara.service.api.v2 import clusters as api from sahara.service import validation as v from sahara.service.validations import clusters as v_c from sahara.service.validations import clusters_scaling as v_c_s from sahara.service.validations import clusters_schema as v_c_schema import sahara.utils.api as u rest = u.RestV2('clusters', __name__) @rest.get('/clusters') @acl.enforce("data-processing:clusters:get_all") @v.check_exists(api.get_cluster, 'marker') @v.validate(None, v.validate_pagination_limit) def clusters_list(): result = api.get_clusters(**u.get_request_args().to_dict()) return u.render(res=result, name='clusters') @rest.post('/clusters') @acl.enforce("data-processing:clusters:create") @v.validate(v_c_schema.CLUSTER_SCHEMA_V2, v_c.check_cluster_create) def clusters_create(data): # renaming hadoop_version -> plugin_version # this can be removed once APIv1 is deprecated data['hadoop_version'] = data['plugin_version'] del data['plugin_version'] return u.render(api.create_cluster(data).to_wrapped_dict()) @rest.post('/clusters/multiple') @acl.enforce("data-processing:clusters:create") @v.validate( v_c_schema.MULTIPLE_CLUSTER_SCHEMA_V2, v_c.check_multiple_clusters_create) def clusters_create_multiple(data): data['hadoop_version'] = data['plugin_version'] del data['plugin_version'] return u.render(api.create_multiple_clusters(data)) @rest.put('/clusters/') @acl.enforce("data-processing:clusters:scale") @v.check_exists(api.get_cluster, 'cluster_id') @v.validate(v_c_schema.CLUSTER_SCALING_SCHEMA_V2, v_c_s.check_cluster_scaling) def clusters_scale(cluster_id, data): return u.to_wrapped_dict(api.scale_cluster, cluster_id, data) @rest.get('/clusters/') @acl.enforce("data-processing:clusters:get") @v.check_exists(api.get_cluster, 'cluster_id') def clusters_get(cluster_id): data = u.get_request_args() show_events = six.text_type( data.get('show_progress', 'false')).lower() == 'true' return u.to_wrapped_dict(api.get_cluster, cluster_id, show_events) @rest.patch('/clusters/') @acl.enforce("data-processing:clusters:modify") @v.check_exists(api.get_cluster, 'cluster_id') @v.validate(v_c_schema.CLUSTER_UPDATE_SCHEMA, v_c.check_cluster_update) def clusters_update(cluster_id, data): return u.to_wrapped_dict(api.update_cluster, cluster_id, data) @rest.delete('/clusters/') @acl.enforce("data-processing:clusters:delete") @v.check_exists(api.get_cluster, 'cluster_id') @v.validate(v_c_schema.CLUSTER_DELETE_SCHEMA_V2, v_c.check_cluster_delete) def clusters_delete(cluster_id): data = u.request_data() force = data.get('force', False) api.terminate_cluster(cluster_id, force=force) return u.render() sahara-8.0.0/sahara/api/v2/__init__.py0000666000175100017510000000527113245514472017470 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """ v2 API interface package This package contains the endpoint definitions for the version 2 API. The modules in this package are named in accordance with the top-level resource primitives they represent. This module provides a convenience function to register all the endpoint blueprints to the ``/v2`` root. When creating new endpoint modules, the following steps should be taken to ensure they are properly registered with the Flask application: * create the module file with a name that indicates its endpoint * add a sahara.utils.api.RestV2 blueprint object * add an import to this module (__init__.py) * add a registration line for the new endpoint to the register_blueprint function """ from sahara.api.v2 import cluster_templates from sahara.api.v2 import clusters from sahara.api.v2 import data_sources from sahara.api.v2 import images from sahara.api.v2 import job_binaries from sahara.api.v2 import job_templates from sahara.api.v2 import job_types from sahara.api.v2 import jobs from sahara.api.v2 import node_group_templates from sahara.api.v2 import plugins def register_blueprints(app, url_prefix): """Register the v2 endpoints with a Flask application This function will take a Flask application object and register all the v2 endpoints. Register blueprints here when adding new endpoint modules. :param app: A Flask application object to register blueprints on :param url_prefix: The url prefix for the blueprints """ app.register_blueprint(cluster_templates.rest, url_prefix=url_prefix) app.register_blueprint(clusters.rest, url_prefix=url_prefix) app.register_blueprint(data_sources.rest, url_prefix=url_prefix) app.register_blueprint(images.rest, url_prefix=url_prefix) app.register_blueprint(job_binaries.rest, url_prefix=url_prefix) app.register_blueprint(jobs.rest, url_prefix=url_prefix) app.register_blueprint(job_types.rest, url_prefix=url_prefix) app.register_blueprint(job_templates.rest, url_prefix=url_prefix) app.register_blueprint(node_group_templates.rest, url_prefix=url_prefix) app.register_blueprint(plugins.rest, url_prefix=url_prefix) sahara-8.0.0/sahara/api/v2/jobs.py0000666000175100017510000000573213245514472016670 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six from sahara.api import acl from sahara.service.api.v2 import jobs as api from sahara.service import validation as v from sahara.service.validations.edp import job_execution as v_j_e from sahara.service.validations.edp import job_execution_schema as v_j_e_schema import sahara.utils.api as u rest = u.RestV2('jobs', __name__) @rest.get('/jobs') @acl.enforce("data-processing:job-executions:get_all") @v.check_exists(api.get_job_execution, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_job_executions) def jobs_list(): result = api.job_execution_list(**u.get_request_args().to_dict()) # APIv2: renaming oozie_job_id -> engine_job_id # once APIv1 is deprecated this can be # removed for je in result: je.pop('oozie_job_id', force=True) return u.render(res=result, name='jobs') @rest.post('/jobs') @acl.enforce("data-processing:jobs:execute") @v.validate(v_j_e_schema.JOB_EXEC_SCHEMA_V2, v_j_e.check_job_execution) def jobs_execute(data): return u.render(api.execute_job(data).to_wrapped_dict()) @rest.get('/jobs/') @acl.enforce("data-processing:job-executions:get") @v.check_exists(api.get_job_execution, id='job_id') def jobs_get(job_id): data = u.get_request_args() refresh_status = six.text_type( data.get('refresh_status', 'false')).lower() == 'true' result = u.to_wrapped_dict_no_render( api.get_job_execution, job_id, refresh_status) result['job_execution']['engine_job_id'] = ( result['job_execution']['oozie_job_id'] ) del result['job_execution']['oozie_job_id'] return u.render(result) @rest.patch('/jobs/') @acl.enforce("data-processing:job-executions:modify") @v.check_exists(api.get_job_execution, id='job_id') @v.validate( v_j_e_schema.JOB_EXEC_UPDATE_SCHEMA, v_j_e.check_job_execution_update) def jobs_update(job_id, data): result = u.to_wrapped_dict_no_render( api.update_job_execution, job_id, data) result['job_execution']['engine_job_id'] = ( result['job_execution']['oozie_job_id'] ) del result['job_execution']['oozie_job_id'] return u.render(result) @rest.delete('/jobs/') @acl.enforce("data-processing:job-executions:delete") @v.check_exists(api.get_job_execution, id='job_id') @v.validate(None, v_j_e.check_job_execution_delete) def jobs_delete(job_id): api.delete_job_execution(job_id) return u.render() sahara-8.0.0/sahara/api/v2/node_group_templates.py0000666000175100017510000001012413245514472022141 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api.v2 import node_group_templates as api from sahara.service import validation as v from sahara.service.validations import node_group_template_schema as ngt_schema from sahara.service.validations import node_group_templates as v_ngt import sahara.utils.api as u rest = u.RestV2('node-group-templates', __name__) @rest.get('/node-group-templates') @acl.enforce("data-processing:node-group-templates:get_all") @v.check_exists(api.get_node_group_template, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_node_group_templates) def node_group_templates_list(): result = api.get_node_group_templates(**u.get_request_args().to_dict()) return u.render(res=result, name="node_group_templates") @rest.post('/node-group-templates') @acl.enforce("data-processing:node-group-templates:create") @v.validate(ngt_schema.NODE_GROUP_TEMPLATE_SCHEMA_V2, v_ngt.check_node_group_template_create) def node_group_templates_create(data): # renaming hadoop_version -> plugin_version # this can be removed once APIv1 is deprecated data['hadoop_version'] = data['plugin_version'] del data['plugin_version'] return u.render(api.create_node_group_template(data).to_wrapped_dict()) @rest.get('/node-group-templates/') @acl.enforce("data-processing:node-group-templates:get") @v.check_exists(api.get_node_group_template, 'node_group_template_id') def node_group_templates_get(node_group_template_id): return u.to_wrapped_dict( api.get_node_group_template, node_group_template_id) @rest.patch('/node-group-templates/') @acl.enforce("data-processing:node-group-templates:modify") @v.check_exists(api.get_node_group_template, 'node_group_template_id') @v.validate(ngt_schema.NODE_GROUP_TEMPLATE_UPDATE_SCHEMA_V2, v_ngt.check_node_group_template_update) def node_group_templates_update(node_group_template_id, data): if data.get('plugin_version', None): data['hadoop_version'] = data['plugin_version'] del data['plugin_version'] return u.to_wrapped_dict( api.update_node_group_template, node_group_template_id, data) @rest.delete('/node-group-templates/') @acl.enforce("data-processing:node-group-templates:delete") @v.check_exists(api.get_node_group_template, 'node_group_template_id') @v.validate(None, v_ngt.check_node_group_template_usage) def node_group_templates_delete(node_group_template_id): api.terminate_node_group_template(node_group_template_id) return u.render() def _node_group_template_export_helper(template): template.pop('id') template.pop('updated_at') template.pop('created_at') template.pop('tenant_id') template.pop('is_default') template['flavor_id'] = '{flavor_id}' template['security_groups'] = '{security_groups}' template['image_id'] = '{image_id}' template['floating_ip_pool'] = '{floating_ip_pool}' @rest.get('/node-group-templates//export') @acl.enforce("data-processing:node-group-templates:get") @v.check_exists(api.get_node_group_template, 'node_group_template_id') def node_group_template_export(node_group_template_id): content = u.to_wrapped_dict_no_render( api.export_node_group_template, node_group_template_id) _node_group_template_export_helper(content['node_group_template']) res = u.render(content) res.headers.add('Content-Disposition', 'attachment', filename='node_group_template.json') return res sahara-8.0.0/sahara/api/v2/job_templates.py0000666000175100017510000000515013245514472020555 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api.v2 import job_templates as api from sahara.service import validation as v from sahara.service.validations.edp import job as v_j from sahara.service.validations.edp import job_schema as v_j_schema import sahara.utils.api as u rest = u.RestV2('job-templates', __name__) @rest.get('/job-templates') @acl.enforce("data-processing:job-templates:get_all") @v.check_exists(api.get_job_templates, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_jobs) def job_templates_list(): result = api.get_job_templates(**u.get_request_args().to_dict()) return u.render(res=result, name='job_templates') @rest.post('/job-templates') @acl.enforce("data-processing:job-templates:create") @v.validate(v_j_schema.JOB_SCHEMA, v_j.check_mains_libs, v_j.check_interface) def job_templates_create(data): return u.render(api.create_job_template(data).to_wrapped_dict()) @rest.get('/job-templates/') @acl.enforce("data-processing:job-templates:get") @v.check_exists(api.get_job_templates, id='job_templates_id') def job_templates_get(job_templates_id): return u.to_wrapped_dict(api.get_job_template, job_templates_id) @rest.patch('/job-templates/') @acl.enforce("data-processing:jobs:modify") @v.check_exists(api.get_job_templates, id='job_templates_id') @v.validate(v_j_schema.JOB_UPDATE_SCHEMA) def job_templates_update(job_templates_id, data): return u.to_wrapped_dict(api.update_job_template, job_templates_id, data) @rest.delete('/job-templates/') @acl.enforce("data-processing:jobs:delete") @v.check_exists(api.get_job_templates, id='job_templates_id') def job_templates_delete(job_templates_id): api.delete_job_template(job_templates_id) return u.render() @rest.get('/job-templates/config-hints/') @acl.enforce("data-processing:jobs:get_config_hints") @v.check_exists(api.get_job_config_hints, job_type='job_type') def job_config_hints_get(job_type): return u.render(api.get_job_config_hints(job_type)) sahara-8.0.0/sahara/api/v2/job_types.py0000666000175100017510000000211313245514472017717 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api.v2 import job_types as api import sahara.utils.api as u rest = u.RestV2('job-types', __name__) @rest.get('/job-types') @acl.enforce("data-processing:job-types:get_all") def job_types_get(): # We want to use flat=False with to_dict() so that # the value of each arg is given as a list. This supports # filters of the form ?type=Pig&type=Java, etc. return u.render(job_types=api.get_job_types( **u.get_request_args().to_dict(flat=False))) sahara-8.0.0/sahara/api/base.py0000666000175100017510000000140113245514472016303 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ import sahara.utils.api as u def not_implemented(): return u.internal_error( 501, NotImplementedError(_("This API operation isn't implemented"))) sahara-8.0.0/sahara/api/__init__.py0000666000175100017510000000000013245514472017122 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/api/middleware/0000775000175100017510000000000013245515026017132 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/api/middleware/sahara_middleware.py0000666000175100017510000000532013245514472023146 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import flask from oslo_config import cfg import six from werkzeug import exceptions as werkzeug_exceptions from sahara.api import v10 as api_v10 from sahara.api import v11 as api_v11 from sahara.api import v2 as api_v2 from sahara import context from sahara.utils import api as api_utils CONF = cfg.CONF def build_app(): """App builder (wsgi). Entry point for Sahara REST API server """ app = flask.Flask('sahara.api') @app.teardown_request def teardown_request(_ex=None): context.set_ctx(None) app.register_blueprint(api_v10.rest, url_prefix='/v1.0') app.register_blueprint(api_v10.rest, url_prefix='/v1.1') app.register_blueprint(api_v11.rest, url_prefix='/v1.1') class _JSONErrorHandler(dict): def __getitem__(self, ex): status_code = (ex.code if isinstance(ex, werkzeug_exceptions.HTTPException) else 500) description = (ex.description if isinstance(ex, werkzeug_exceptions.HTTPException) else str(ex)) return api_utils.render({'error': status_code, 'error_message': description}, status=status_code) make_json_error = _JSONErrorHandler() for code in six.iterkeys(werkzeug_exceptions.default_exceptions): app.error_handler_spec[None][code] = make_json_error return app def build_v2_app(): """App builder (wsgi). Entry point for Experimental V2 Sahara REST API server """ app = build_app() api_v2.register_blueprints(app, url_prefix='/v2') return app class Router(object): def __call__(self, environ, response): return self.app(environ, response) @classmethod def factory(cls, global_config, **local_config): cls.app = build_app() return cls(**local_config) class RouterV2(object): def __call__(self, environ, response): return self.app(environ, response) @classmethod def factory(cls, global_config, **local_config): cls.app = build_v2_app() return cls(**local_config) sahara-8.0.0/sahara/api/middleware/version_discovery.py0000666000175100017510000000451413245514472023272 0ustar zuulzuul00000000000000# Copyright (c) 2018 Massachusetts Open Cloud # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re from oslo_middleware import base from oslo_serialization import jsonutils import webob import webob.dec class VersionResponseMiddlewareV1(base.Middleware): @webob.dec.wsgify def __call__(self, req): """Respond to a request for all Sahara API versions.""" path = req.environ['PATH_INFO'] if re.match(r"^/*$", path): response = webob.Response(request=req, status=300, content_type="application/json") response.body = jsonutils.dumps(self._get_versions(req)) return response else: return self.application def _get_versions(self, req): """Populate the version response with APIv1 stuff.""" version_response = { "versions": [ {"id": "v1.0", "status": "SUPPORTED", "links": self._get_links("1.0", req) }, {"id": "v1.1", "status": "CURRENT", "links": self._get_links("1.1", req) } ] } return version_response @staticmethod def _get_links(version, req): href = "%s/v%s/" % (req.host_url, version) return [{"rel": "self", "href": href}] class VersionResponseMiddlewareV2(VersionResponseMiddlewareV1): def _get_versions(self, req): """Populate the version response with APIv1 and APIv2 stuff.""" version_response = ( super(VersionResponseMiddlewareV2, self)._get_versions(req) ) version_response["versions"].append( {"id": "v2", "status": "EXPERIMENTAL", "links": self._get_links("2", req) } ) return version_response sahara-8.0.0/sahara/api/middleware/auth_valid.py0000666000175100017510000000426613245514472021642 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_log import log as logging from oslo_middleware import base from oslo_utils import strutils from oslo_utils import uuidutils import webob import webob.exc as ex from sahara.i18n import _ LOG = logging.getLogger(__name__) class AuthValidator(base.Middleware): """Handles token auth results and tenants.""" @webob.dec.wsgify def __call__(self, req): """Ensures that tenants in url and token are equal. Handle incoming request by checking tenant info prom the headers and url ({tenant_id} url attribute). Pass request downstream on success. Reject request if tenant_id from headers not equals to tenant_id from url. """ token_tenant = req.environ.get("HTTP_X_TENANT_ID") if not token_tenant: LOG.warning("Can't get tenant_id from env") raise ex.HTTPServiceUnavailable() path = req.environ['PATH_INFO'] if path != '/': try: version, possibly_url_tenant, rest = ( strutils.split_path(path, 2, 3, True) ) except ValueError: LOG.warning("Incorrect path: {path}".format(path=path)) raise ex.HTTPNotFound(_("Incorrect path")) if uuidutils.is_uuid_like(possibly_url_tenant): url_tenant = possibly_url_tenant if token_tenant != url_tenant: LOG.debug("Unauthorized: token tenant != requested tenant") raise ex.HTTPUnauthorized( _('Token tenant != requested tenant')) return self.application sahara-8.0.0/sahara/api/middleware/__init__.py0000666000175100017510000000000013245514472021237 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/api/v11.py0000666000175100017510000002545213245514472016014 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.api import acl from sahara.service.api import v11 as api from sahara.service import validation as v from sahara.service.validations.edp import data_source as v_d_s from sahara.service.validations.edp import data_source_schema as v_d_s_schema from sahara.service.validations.edp import job as v_j from sahara.service.validations.edp import job_binary as v_j_b from sahara.service.validations.edp import job_binary_internal as v_j_b_i from sahara.service.validations.edp import job_binary_internal_schema as vjbi_s from sahara.service.validations.edp import job_binary_schema as v_j_b_schema from sahara.service.validations.edp import job_execution as v_j_e from sahara.service.validations.edp import job_execution_schema as v_j_e_schema from sahara.service.validations.edp import job_schema as v_j_schema import sahara.utils.api as u rest = u.Rest('v11', __name__) # Job execution ops @rest.get('/job-executions') @acl.enforce("data-processing:job-executions:get_all") @v.check_exists(api.get_job_execution, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_job_executions) def job_executions_list(): result = api.job_execution_list( **u.get_request_args().to_dict()) return u.render(res=result, name='job_executions') @rest.get('/job-executions/') @acl.enforce("data-processing:job-executions:get") @v.check_exists(api.get_job_execution, id='job_id') def job_executions(job_id): return u.to_wrapped_dict(api.get_job_execution, job_id) @rest.get('/job-executions//refresh-status') @acl.enforce("data-processing:job-executions:refresh_status") @v.check_exists(api.get_job_execution, id='job_id') def job_executions_status(job_id): return u.to_wrapped_dict(api.get_job_execution_status, job_id) @rest.get('/job-executions//cancel') @acl.enforce("data-processing:job-executions:cancel") @v.check_exists(api.get_job_execution, id='job_id') @v.validate(None, v_j_e.check_job_execution_cancel) def job_executions_cancel(job_id): return u.to_wrapped_dict(api.cancel_job_execution, job_id) @rest.patch('/job-executions/') @acl.enforce("data-processing:job-executions:modify") @v.check_exists(api.get_job_execution, id='job_id') @v.validate(v_j_e_schema.JOB_EXEC_UPDATE_SCHEMA, v_j_e.check_job_execution_update, v_j_e.check_job_status_update) def job_executions_update(job_id, data): return u.to_wrapped_dict(api.update_job_execution, job_id, data) @rest.delete('/job-executions/') @acl.enforce("data-processing:job-executions:delete") @v.check_exists(api.get_job_execution, id='job_id') @v.validate(None, v_j_e.check_job_execution_delete) def job_executions_delete(job_id): api.delete_job_execution(job_id) return u.render() # Data source ops @rest.get('/data-sources') @acl.enforce("data-processing:data-sources:get_all") @v.check_exists(api.get_data_source, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_data_sources) def data_sources_list(): result = api.get_data_sources(**u.get_request_args().to_dict()) return u.render(res=result, name='data_sources') @rest.post('/data-sources') @acl.enforce("data-processing:data-sources:register") @v.validate(v_d_s_schema.DATA_SOURCE_SCHEMA, v_d_s.check_data_source_create) def data_source_register(data): return u.render(api.register_data_source(data).to_wrapped_dict()) @rest.get('/data-sources/') @acl.enforce("data-processing:data-sources:get") @v.check_exists(api.get_data_source, 'data_source_id') def data_source_get(data_source_id): return u.to_wrapped_dict(api.get_data_source, data_source_id) @rest.delete('/data-sources/') @acl.enforce("data-processing:data-sources:delete") @v.check_exists(api.get_data_source, 'data_source_id') def data_source_delete(data_source_id): api.delete_data_source(data_source_id) return u.render() @rest.put('/data-sources/') @acl.enforce("data-processing:data-sources:modify") @v.check_exists(api.get_data_source, 'data_source_id') @v.validate( v_d_s_schema.DATA_SOURCE_UPDATE_SCHEMA, v_d_s.check_data_source_update) def data_source_update(data_source_id, data): return u.to_wrapped_dict(api.data_source_update, data_source_id, data) # Job ops @rest.get('/jobs') @acl.enforce("data-processing:jobs:get_all") @v.check_exists(api.get_job, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_jobs) def job_list(): result = api.get_jobs(**u.get_request_args().to_dict()) return u.render(res=result, name='jobs') @rest.post('/jobs') @acl.enforce("data-processing:jobs:create") @v.validate(v_j_schema.JOB_SCHEMA, v_j.check_mains_libs, v_j.check_interface) def job_create(data): return u.render(api.create_job(data).to_wrapped_dict()) @rest.get('/jobs/') @acl.enforce("data-processing:jobs:get") @v.check_exists(api.get_job, id='job_templates_id') def job_get(job_templates_id): return u.to_wrapped_dict(api.get_job, job_templates_id) @rest.patch('/jobs/') @acl.enforce("data-processing:jobs:modify") @v.check_exists(api.get_job, id='job_templates_id') @v.validate(v_j_schema.JOB_UPDATE_SCHEMA) def job_update(job_templates_id, data): return u.to_wrapped_dict(api.update_job, job_templates_id, data) @rest.delete('/jobs/') @acl.enforce("data-processing:jobs:delete") @v.check_exists(api.get_job, id='job_templates_id') def job_delete(job_templates_id): api.delete_job(job_templates_id) return u.render() @rest.post('/jobs//execute') @acl.enforce("data-processing:jobs:execute") @v.check_exists(api.get_job, id='job_templates_id') @v.validate(v_j_e_schema.JOB_EXEC_SCHEMA, v_j_e.check_job_execution) def job_execute(job_templates_id, data): return u.render(job_execution=api.execute_job( job_templates_id, data).to_dict()) @rest.get('/jobs/config-hints/') @acl.enforce("data-processing:jobs:get_config_hints") @v.check_exists(api.get_job_config_hints, job_type='job_type') def job_config_hints_get(job_type): return u.render(api.get_job_config_hints(job_type)) @rest.get('/job-types') @acl.enforce("data-processing:job-types:get_all") def job_types_get(): # We want to use flat=False with to_dict() so that # the value of each arg is given as a list. This supports # filters of the form ?type=Pig&type=Java, etc. return u.render(job_types=api.get_job_types( **u.get_request_args().to_dict(flat=False))) # Job binary ops @rest.post('/job-binaries') @acl.enforce("data-processing:job-binaries:create") @v.validate(v_j_b_schema.JOB_BINARY_SCHEMA, v_j_b.check_job_binary) def job_binary_create(data): return u.render(api.create_job_binary(data).to_wrapped_dict()) @rest.get('/job-binaries') @acl.enforce("data-processing:job-binaries:get_all") @v.check_exists(api.get_job_binaries, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_job_binaries) def job_binary_list(): result = api.get_job_binaries(**u.get_request_args().to_dict()) return u.render(res=result, name='binaries') @rest.get('/job-binaries/') @acl.enforce("data-processing:job-binaries:get") @v.check_exists(api.get_job_binary, 'job_binary_id') def job_binary_get(job_binary_id): return u.to_wrapped_dict(api.get_job_binary, job_binary_id) @rest.delete('/job-binaries/') @acl.enforce("data-processing:job-binaries:delete") @v.check_exists(api.get_job_binary, id='job_binary_id') def job_binary_delete(job_binary_id): api.delete_job_binary(job_binary_id) return u.render() @rest.get('/job-binaries//data') @acl.enforce("data-processing:job-binaries:get_data") @v.check_exists(api.get_job_binary, 'job_binary_id') def job_binary_data(job_binary_id): data = api.get_job_binary_data(job_binary_id) if type(data) == dict: data = u.render(data) return data @rest.put('/job-binaries/') @acl.enforce("data-processing:job-binaries:modify") @v.validate(v_j_b_schema.JOB_BINARY_UPDATE_SCHEMA, v_j_b.check_job_binary) def job_binary_update(job_binary_id, data): return u.render(api.update_job_binary(job_binary_id, data).to_wrapped_dict()) # Job binary internals ops @rest.put_file('/job-binary-internals/') @acl.enforce("data-processing:job-binary-internals:create") @v.validate(None, v_j_b_i.check_job_binary_internal) def job_binary_internal_create(**values): return u.render(api.create_job_binary_internal(values).to_wrapped_dict()) @rest.get('/job-binary-internals') @acl.enforce("data-processing:job-binary-internals:get_all") @v.check_exists(api.get_job_binary_internal, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_job_binary_internals) def job_binary_internal_list(): result = api.get_job_binary_internals(**u.get_request_args().to_dict()) return u.render(res=result, name='binaries') @rest.get('/job-binary-internals/') @acl.enforce("data-processing:job-binary-internals:get") @v.check_exists(api.get_job_binary_internal, 'job_binary_internal_id') def job_binary_internal_get(job_binary_internal_id): return u.to_wrapped_dict( api.get_job_binary_internal, job_binary_internal_id) @rest.delete('/job-binary-internals/') @acl.enforce("data-processing:job-binary-internals:delete") @v.check_exists(api.get_job_binary_internal, 'job_binary_internal_id') def job_binary_internal_delete(job_binary_internal_id): api.delete_job_binary_internal(job_binary_internal_id) return u.render() @rest.get('/job-binary-internals//data') @acl.enforce("data-processing:job-binary-internals:get_data") @v.check_exists(api.get_job_binary_internal, 'job_binary_internal_id') def job_binary_internal_data(job_binary_internal_id): return api.get_job_binary_internal_data(job_binary_internal_id) @rest.patch('/job-binary-internals/') @acl.enforce("data-processing:job-binaries:modify") @v.check_exists(api.get_job_binary_internal, 'job_binary_internal_id') @v.validate(vjbi_s.JOB_BINARY_UPDATE_SCHEMA) def job_binary_internal_update(job_binary_internal_id, data): return u.to_wrapped_dict( api.update_job_binary_internal, job_binary_internal_id, data) sahara-8.0.0/sahara/api/v10.py0000666000175100017510000002767513245514472016024 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six from sahara.api import acl from sahara.service.api import v10 as api from sahara.service import validation as v from sahara.service.validations import cluster_template_schema as ct_schema from sahara.service.validations import cluster_templates as v_ct from sahara.service.validations import clusters as v_c from sahara.service.validations import clusters_scaling as v_c_s from sahara.service.validations import clusters_schema as v_c_schema from sahara.service.validations import images as v_images from sahara.service.validations import node_group_template_schema as ngt_schema from sahara.service.validations import node_group_templates as v_ngt from sahara.service.validations import plugins as v_p import sahara.utils.api as u rest = u.Rest('v10', __name__) # Cluster ops @rest.get('/clusters') @acl.enforce("data-processing:clusters:get_all") @v.check_exists(api.get_cluster, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_clusters) def clusters_list(): result = api.get_clusters(**u.get_request_args().to_dict()) return u.render(res=result, name='clusters') @rest.post('/clusters') @acl.enforce("data-processing:clusters:create") @v.validate(v_c_schema.CLUSTER_SCHEMA, v_c.check_cluster_create) def clusters_create(data): return u.render(api.create_cluster(data).to_wrapped_dict()) @rest.post('/clusters/multiple') @acl.enforce("data-processing:clusters:create") @v.validate( v_c_schema.MULTIPLE_CLUSTER_SCHEMA, v_c.check_multiple_clusters_create) def clusters_create_multiple(data): return u.render(api.create_multiple_clusters(data)) @rest.put('/clusters/') @acl.enforce("data-processing:clusters:scale") @v.check_exists(api.get_cluster, 'cluster_id') @v.validate(v_c_schema.CLUSTER_SCALING_SCHEMA, v_c_s.check_cluster_scaling) def clusters_scale(cluster_id, data): return u.to_wrapped_dict(api.scale_cluster, cluster_id, data) @rest.get('/clusters/') @acl.enforce("data-processing:clusters:get") @v.check_exists(api.get_cluster, 'cluster_id') def clusters_get(cluster_id): data = u.get_request_args() show_events = six.text_type( data.get('show_progress', 'false')).lower() == 'true' return u.to_wrapped_dict(api.get_cluster, cluster_id, show_events) @rest.patch('/clusters/') @acl.enforce("data-processing:clusters:modify") @v.check_exists(api.get_cluster, 'cluster_id') @v.validate(v_c_schema.CLUSTER_UPDATE_SCHEMA, v_c.check_cluster_update) def clusters_update(cluster_id, data): return u.to_wrapped_dict(api.update_cluster, cluster_id, data) @rest.delete('/clusters/') @acl.enforce("data-processing:clusters:delete") @v.check_exists(api.get_cluster, 'cluster_id') @v.validate(None, v_c.check_cluster_delete) def clusters_delete(cluster_id): api.terminate_cluster(cluster_id) return u.render() # ClusterTemplate ops @rest.get('/cluster-templates') @acl.enforce("data-processing:cluster-templates:get_all") @v.check_exists(api.get_cluster_template, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_cluster_templates) def cluster_templates_list(): result = api.get_cluster_templates( **u.get_request_args().to_dict()) return u.render(res=result, name='cluster_templates') @rest.post('/cluster-templates') @acl.enforce("data-processing:cluster-templates:create") @v.validate(ct_schema.CLUSTER_TEMPLATE_SCHEMA, v_ct.check_cluster_template_create) def cluster_templates_create(data): return u.render(api.create_cluster_template(data).to_wrapped_dict()) @rest.get('/cluster-templates/') @acl.enforce("data-processing:cluster-templates:get") @v.check_exists(api.get_cluster_template, 'cluster_template_id') def cluster_templates_get(cluster_template_id): return u.to_wrapped_dict(api.get_cluster_template, cluster_template_id) @rest.put('/cluster-templates/') @acl.enforce("data-processing:cluster-templates:modify") @v.check_exists(api.get_cluster_template, 'cluster_template_id') @v.validate(ct_schema.CLUSTER_TEMPLATE_UPDATE_SCHEMA, v_ct.check_cluster_template_update) def cluster_templates_update(cluster_template_id, data): return u.to_wrapped_dict( api.update_cluster_template, cluster_template_id, data) @rest.delete('/cluster-templates/') @acl.enforce("data-processing:cluster-templates:delete") @v.check_exists(api.get_cluster_template, 'cluster_template_id') @v.validate(None, v_ct.check_cluster_template_usage) def cluster_templates_delete(cluster_template_id): api.terminate_cluster_template(cluster_template_id) return u.render() def _cluster_template_export_helper(template): template.pop('id') template.pop('updated_at') template.pop('created_at') template.pop('tenant_id') template.pop('is_default') template['default_image_id'] = '{default_image_id}' template['node_groups'] = '{node_groups}' @rest.get('/cluster-templates//export') @acl.enforce("data-processing:cluster-templates:get") @v.check_exists(api.get_cluster_template, 'cluster_template_id') def cluster_template_export(cluster_template_id): content = u.to_wrapped_dict_no_render( api.get_cluster_template, cluster_template_id) _cluster_template_export_helper(content['cluster_template']) res = u.render(content) res.headers.add('Content-Disposition', 'attachment', filename='cluster_template.json') return res # NodeGroupTemplate ops @rest.get('/node-group-templates') @acl.enforce("data-processing:node-group-templates:get_all") @v.check_exists(api.get_node_group_template, 'marker') @v.validate(None, v.validate_pagination_limit, v.validate_sorting_node_group_templates) def node_group_templates_list(): result = api.get_node_group_templates( **u.get_request_args().to_dict()) return u.render(res=result, name='node_group_templates') @rest.post('/node-group-templates') @acl.enforce("data-processing:node-group-templates:create") @v.validate(ngt_schema.NODE_GROUP_TEMPLATE_SCHEMA, v_ngt.check_node_group_template_create) def node_group_templates_create(data): return u.render(api.create_node_group_template(data).to_wrapped_dict()) @rest.get('/node-group-templates/') @acl.enforce("data-processing:node-group-templates:get") @v.check_exists(api.get_node_group_template, 'node_group_template_id') def node_group_templates_get(node_group_template_id): return u.to_wrapped_dict( api.get_node_group_template, node_group_template_id) @rest.put('/node-group-templates/') @acl.enforce("data-processing:node-group-templates:modify") @v.check_exists(api.get_node_group_template, 'node_group_template_id') @v.validate(ngt_schema.NODE_GROUP_TEMPLATE_UPDATE_SCHEMA, v_ngt.check_node_group_template_update) def node_group_templates_update(node_group_template_id, data): return u.to_wrapped_dict( api.update_node_group_template, node_group_template_id, data) @rest.delete('/node-group-templates/') @acl.enforce("data-processing:node-group-templates:delete") @v.check_exists(api.get_node_group_template, 'node_group_template_id') @v.validate(None, v_ngt.check_node_group_template_usage) def node_group_templates_delete(node_group_template_id): api.terminate_node_group_template(node_group_template_id) return u.render() def _node_group_template_export_helper(template): template.pop('id') template.pop('updated_at') template.pop('created_at') template.pop('tenant_id') template.pop('is_default') template['flavor_id'] = '{flavor_id}' template['security_groups'] = '{security_groups}' template['image_id'] = '{image_id}' template['floating_ip_pool'] = '{floating_ip_pool}' @rest.get('/node-group-templates//export') @acl.enforce("data-processing:node-group-templates:get") @v.check_exists(api.get_node_group_template, 'node_group_template_id') def node_group_template_export(node_group_template_id): content = u.to_wrapped_dict_no_render( api.export_node_group_template, node_group_template_id) _node_group_template_export_helper(content['node_group_template']) res = u.render(content) res.headers.add('Content-Disposition', 'attachment', filename='node_group_template.json') return res # Plugins ops @rest.get('/plugins') @acl.enforce("data-processing:plugins:get_all") def plugins_list(): return u.render(plugins=[p.dict for p in api.get_plugins()]) @rest.get('/plugins/') @acl.enforce("data-processing:plugins:get") @v.check_exists(api.get_plugin, plugin_name='plugin_name') def plugins_get(plugin_name): return u.render(api.get_plugin(plugin_name).wrapped_dict) @rest.get('/plugins//') @acl.enforce("data-processing:plugins:get_version") @v.check_exists(api.get_plugin, plugin_name='plugin_name', version='version') def plugins_get_version(plugin_name, version): return u.render(api.get_plugin(plugin_name, version).wrapped_dict) @rest.patch('/plugins/') @acl.enforce("data-processing:plugins:patch") @v.check_exists(api.get_plugin, plugin_name='plugin_name') @v.validate(v_p.plugin_update_validation_jsonschema(), v_p.check_plugin_update) def plugins_update(plugin_name, data): return u.render(api.update_plugin(plugin_name, data).wrapped_dict) @rest.post_file('/plugins///convert-config/') @acl.enforce("data-processing:plugins:convert_config") @v.check_exists(api.get_plugin, plugin_name='plugin_name', version='version') @v.validate(None, v_p.check_convert_to_template) def plugins_convert_to_cluster_template(plugin_name, version, name, data): # There is no plugins that supports converting to cluster template # The last plugin with support of that is no longer supported pass # Image Registry ops @rest.get('/images') @acl.enforce("data-processing:images:get_all") def images_list(): tags = u.get_request_args().getlist('tags') name = u.get_request_args().get('name', None) return u.render(images=[i.dict for i in api.get_images(name, tags)]) @rest.get('/images/') @acl.enforce("data-processing:images:get") @v.check_exists(api.get_image, id='image_id') def images_get(image_id): return u.render(api.get_registered_image(image_id=image_id).wrapped_dict) @rest.post('/images/') @acl.enforce("data-processing:images:register") @v.check_exists(api.get_image, id='image_id') @v.validate(v_images.image_register_schema, v_images.check_image_register) def images_set(image_id, data): return u.render(api.register_image(image_id, **data).wrapped_dict) @rest.delete('/images/') @acl.enforce("data-processing:images:unregister") @v.check_exists(api.get_image, id='image_id') def images_unset(image_id): api.unregister_image(image_id) return u.render() @rest.post('/images//tag') @acl.enforce("data-processing:images:add_tags") @v.check_exists(api.get_image, id='image_id') @v.validate(v_images.image_tags_schema, v_images.check_tags) def image_tags_add(image_id, data): return u.render(api.add_image_tags(image_id, **data).wrapped_dict) @rest.post('/images//untag') @acl.enforce("data-processing:images:remove_tags") @v.check_exists(api.get_image, id='image_id') @v.validate(v_images.image_tags_schema) def image_tags_delete(image_id, data): return u.render(api.remove_image_tags(image_id, **data).wrapped_dict) sahara-8.0.0/sahara/api/acl.py0000666000175100017510000000245713245514472016144 0ustar zuulzuul00000000000000# # Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. """Policy Engine For Sahara""" import functools from oslo_config import cfg from oslo_policy import policy from sahara.common import policies from sahara import context from sahara import exceptions ENFORCER = None def setup_policy(): global ENFORCER ENFORCER = policy.Enforcer(cfg.CONF) ENFORCER.register_defaults(policies.list_rules()) def enforce(rule): def decorator(func): @functools.wraps(func) def handler(*args, **kwargs): ctx = context.ctx() ENFORCER.enforce(rule, {}, ctx.to_dict(), do_raise=True, exc=exceptions.Forbidden) return func(*args, **kwargs) return handler return decorator sahara-8.0.0/sahara/db/0000775000175100017510000000000013245515026014631 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/db/templates/0000775000175100017510000000000013245515026016627 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/db/templates/cli.py0000666000175100017510000002020413245514472017754 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import print_function import sys from oslo_config import cfg from oslo_log import log import pkg_resources as pkg from sahara.db.templates import api from sahara import version LOG = log.getLogger(__name__) CONF = cfg.CONF def extra_option_checks(): if not CONF.database.connection: print("No database connection string was specified in configuration", file=sys.stderr) sys.exit(1) if CONF.command.name in ['update', 'delete']: if CONF.command.plugin_version and not CONF.command.plugin_name: print("The --plugin-version option is not valid " "without --plugin-name", file=sys.stderr) sys.exit(-1) if CONF.command.name == "update": # Options handling probably needs some refactoring in the future. # For now, though, we touch the conductor which ultimately touches # the plugins.base. Use the "plugins" option there as a default # list of plugins to process, since those are the plugins that # will be loaded by Sahara if not CONF.command.plugin_name: if "plugins" in CONF and CONF.plugins: LOG.info("Using plugin list {plugins} from " "config".format(plugins=CONF.plugins)) else: print("No plugins specified with --plugin-name " "or config", file=sys.stderr) sys.exit(-1) def add_command_parsers(subparsers): # Note, there is no 'list' command here because the client # or REST can be used for list operations. Default templates # will display, and templates will show the 'is_default' field. def add_id(parser): parser.add_argument('--id', required=True, help='The id of the default ' 'template to delete') def add_tenant_id(parser): parser.add_argument('-t', '--tenant_id', required=True, help='Tenant ID for database operations.') def add_name_and_tenant_id(parser): parser.add_argument('--name', dest='template_name', required=True, help='Name of the default template') add_tenant_id(parser) def add_plugin_name_and_version(parser, require_plugin_name=False): plugin_name_help = ('Only process templates containing ' 'a "plugin_name" field matching ' 'one of these values.') if not require_plugin_name: extra = (' The default list of plugin names ' 'is taken from the "plugins" parameter in ' 'the [DEFAULT] config section.') plugin_name_help += extra parser.add_argument('-p', '--plugin-name', nargs="*", required=require_plugin_name, help=plugin_name_help) parser.add_argument('-pv', '--plugin-version', nargs="*", help='Only process templates containing a ' '"hadoop_version" field matching one of ' 'these values. This option is ' 'only valid if --plugin-name is specified ' 'as well. A version specified ' 'here may optionally be prefixed with a ' 'plugin name and a dot, for example ' '"vanilla.1.2.1". Dotted versions only ' 'apply to the plugin named in the ' 'prefix. Versions without a prefix apply to ' 'all plugins.') fname = pkg.resource_filename(version.version_info.package, "plugins/default_templates") # update command parser = subparsers.add_parser('update', help='Update the default template set') parser.add_argument('-d', '--directory', default=fname, help='Template directory. Default is %s' % fname) parser.add_argument('-n', '--norecurse', action='store_true', help='Do not descend into subdirectories') add_plugin_name_and_version(parser) add_tenant_id(parser) parser.set_defaults(func=api.do_update) # delete command parser = subparsers.add_parser('delete', help='Delete default templates ' 'by plugin and version') add_plugin_name_and_version(parser, require_plugin_name=True) add_tenant_id(parser) parser.set_defaults(func=api.do_delete) # node-group-template-delete command parser = subparsers.add_parser('node-group-template-delete', help='Delete a default ' 'node group template by name') add_name_and_tenant_id(parser) parser.set_defaults(func=api.do_node_group_template_delete) # cluster-template-delete command parser = subparsers.add_parser('cluster-template-delete', help='Delete a default ' 'cluster template by name') add_name_and_tenant_id(parser) parser.set_defaults(func=api.do_cluster_template_delete) # node-group-template-delete-id command parser = subparsers.add_parser('node-group-template-delete-id', help='Delete a default ' 'node group template by id') add_id(parser) parser.set_defaults(func=api.do_node_group_template_delete_by_id) # cluster-template-delete-id command parser = subparsers.add_parser('cluster-template-delete-id', help='Delete a default ' 'cluster template by id') add_id(parser) parser.set_defaults(func=api.do_cluster_template_delete_by_id) command_opt = cfg.SubCommandOpt('command', title='Command', help='Available commands', handler=add_command_parsers) CONF.register_cli_opt(command_opt) def unregister_extra_cli_opt(name): try: for cli in CONF._cli_opts: if cli['opt'].name == name: CONF.unregister_opt(cli['opt']) except Exception: pass # Remove a few extra CLI opts that we picked up via imports # Do this early so that they do not appear in the help for extra_opt in ["log-exchange", "host", "port"]: unregister_extra_cli_opt(extra_opt) def main(): # TODO(tmckay): Work on restricting the options # pulled in by imports which show up in the help. # If we find a nice way to do this the calls to # unregister_extra_cli_opt() can be removed CONF(project='sahara') # For some reason, this is necessary to clear cached values # and re-read configs. For instance, if this is not done # here the 'plugins' value will not reflect the value from # the config file on the command line CONF.reload_config_files() log.setup(CONF, "sahara") # If we have to enforce extra option checks, like one option # requires another, do it here extra_option_checks() # Since this may be scripted, record the command in the log # so a user can know exactly what was done LOG.info("Command: {command}".format(command=' '.join(sys.argv))) api.set_logger(LOG) api.set_conf(CONF) CONF.command.func() LOG.info("Finished {command}".format(command=CONF.command.name)) sahara-8.0.0/sahara/db/templates/README.rst0000666000175100017510000002300213245514472020321 0ustar zuulzuul00000000000000Sahara Default Template CLI =========================== The *sahara-templates* application is a simple CLI for managing default templates in Sahara. This document gives an overview of default templates and explains how to use the CLI. Default Templates Overview -------------------------- The goal of the default template facility in Sahara is to make cluster launching quick and easy by providing users with a stable set of pre-generated node group and cluster templates for each of the Sahara provisioning plugins. Template sets are defined in .json files grouped into directories. The CLI reads these template sets and writes directly to the Sahara database. Default templates may only be created, modified, or deleted via the CLI -- operations through the python-saharaclient or REST API are restricted. JSON Files ---------- Cluster and node group templates are defined in .json files. A very simple cluster template JSON file might look like this: .. code:: python { "plugin_name": "vanilla", "hadoop_version": "2.7.1", "node_groups": [ { "name": "master", "count": 1, "node_group_template_id": "{master}" }, { "name": "worker", "count": 3, "node_group_template_id": "{worker}" } ], "name": "cluster-template" } The values of the *node_group_template_id* fields are the names of node group templates in set braces. In this example, *master* and *worker* are the names of node group templates defined in .json files in the same directory. When the CLI processes the directory, it will create the node group templates first and then substitute the appropriate id values for the name references when it creates the cluster template. Configuration Files and Value Substitutions ------------------------------------------- The CLI supports value substitution for a limited set of fields. For cluster templates, the following fields may use substitution: * default_image_id * neutron_management_network For node group templates, the following fields may use substitution: * image_id * flavor_id * floating_ip_pool Substitution is indicated for one of these fields in a .json file when the value is the name of the field in set braces. Here is an example of a node group template file that uses substitution for *flavor_id*: .. code:: python { "plugin_name": "vanilla", "hadoop_version": "2.7.1", "node_processes": [ "namenode", "resourcemanager", "oozie", "historyserver" ], "name": "master", "flavor_id": "{flavor_id}", "floating_ip_pool": "{floating_ip_pool}" } The values for *flavor_id* and *floating_ip_pool* in this template will come from a configuration file. If a configuration value is found for the substitution, the value will be replaced. If a configuration value is not found, the field will be omitted from the template. (In this example, *flavor_id* is a required field of node group templates and the template will fail validation if there is no substitution value specified. However, *floating_ip_pool* is not required and so the template will still pass validation if it is omitted). The CLI will look for configuration sections with names based on the *plugin_name*, *hadoop_version*, and *name* fields in the template. It will look for sections in the following order: * **[]** May contain fields only for the type of the named template If templates are named in an **unambiguous** way, the template name alone can be a used as the name of the config section. This produces shorter names and aids readability when there is a one-to-one mapping between template names and config sections. * **[__]** May contain fields only for the type of the named template This form unambiguously applies to a specific template for a specific plugin. * **[_]** May contain node group or cluster template fields * **[]** May contain node group or cluster template fields * **[DEFAULT]** May contain node group or cluster template fields If we have the following configuration file in our example the CLI will find the value of *flavor_id* for the *master* template in the first configuration section and the value for *floating_ip_pool* in the third section: .. code:: python [vanilla_2.7.1_master] # This is named for the plugin, version, and template. # It may contain only node group template fields. flavor_id = 5 image_id = b7883f8a-9a7f-42cc-89a2-d3c8b1cc7b28 [vanilla_2.7.1] # This is named for the plugin and version. # It may contain fields for both node group and cluster templates. flavor_id = 4 neutron_management_network = 9973da0b-68eb-497d-bd48-d85aca37f088 [vanilla] # This is named for the plugin. # It may contain fields for both node group and cluster templates. flavor_id = 3 default_image_id = 89de8d21-9743-4d20-873e-7677973416dd floating_ip_pool = my_pool [DEFAULT] # This is the normal default section. # It may contain fields for both node group and cluster templates. flavor_id = 2 Sample Configuration File ------------------------- A sample configuration file is provided in *sahara/plugins/default_templates/template.conf*. This file sets the *flavor_id* for most of the node group templates supplied with Sahara to 2 which indicates the *m1.small* flavor in a default OpenStack deployment. The master node templates for the CDH plugin have the *flavor_id* set to 4 which indicates the *m1.large* flavor, since these nodes require more resources. This configuration file may be used with the CLI as is, or it may be copied and modified. Note that multiple configuration files may be passed to the CLI by repeating the *--config-file* option. Other Special Configuration Parameters -------------------------------------- The only configuration parameter that is strictly required is the *connection* parameter in the *database* section. Without this value the CLI will not be able to connect to the Sahara database. By default, the CLI will use the value of the *plugins* parameter in the [DEFAULT] section on *update* to filter the templates that will be created or updated. This parameter in Sahara defaults to the set of fully supported plugins. To restrict the set of plugins for the *update* operation set this parameter or use the *--plugin-name* option. Directory Structure ------------------- The structure of the directory holding .json files for the CLI is very flexible. The CLI will begin processing at the designated starting directory and recurse through subdirectories. At each directory level, the CLI will look for .json files to define a set of default templates. Cluster templates may reference node group templates in the same set by name. Templates at different levels in the directory structure are not in the same set. Plugin name and version are determined from the values in the .json files, not by the file names or the directory structure. Recursion may be turned off with the "-n" option (see below). The default starting directory is *sahara/plugins/default_templates* Example CLI Commands -------------------- For ``update``, ``delete``, ``node-group-template-delete``, and ``cluster-template-delete`` operations, the tenant must always be specified. For ``node-group-template-delete-id`` and ``cluster-template-delete-id`` tenant is not required. All useful information about activity by the CLI is logged Create/update all of the default templates bundled with Sahara. Use the standard Sahara configuration file in */etc/sahara/sahara.conf* to specify the plugin list and the database connection string and another configuration file to supply the *flavor_id* values:: $ sahara-templates --config-file /etc/sahara/sahara.conf --config-file myconfig update -t $TENANT_ID Create/update default templates from the directory *mypath*:: $ sahara-templates --config-file myconfig update -t $TENANT_ID -d mypath Create/update default templates from the directory *mypath* but do not descend into subdirectories:: $ sahara-templates --config-file myconfig update -t $TENANT_ID -d mypath -n Create/update default templates bundled with Sahara for just the vanilla plugin:: $ sahara-templates --config-file myconfig update -t $TENANT_ID -p vanilla Create/update default templates bundled with Sahara for just version 2.7.1 of the vanilla plugin:: $ sahara-templates --config-file myconfig update -t $TENANT_ID -p vanilla -pv 2.7.1 Create/update default templates bundled with Sahara for just version 2.7.1 of the vanilla plugin and version 2.0.6 of the hdp plugin:: $ sahara-templates --config-file myconfig update -t $TENANT_ID -p vanilla -pv vanilla.2.7.1 -p hdp -pv hdp.2.0.6 Delete default templates for the vanilla plugin:: $ sahara-templates --config-file myconfig delete -t $TENANT_ID -p vanilla Delete default templates for version 2.7.1 of the vanilla plugin:: $ sahara-templates --config-file myconfig delete -t $TENANT_ID -p vanilla -pv 2.7.1 Delete a specific node group template by ID:: $ sahara-templates --config-file myconfig node-group-template-delete-id --id ID Delete a specific cluster template by ID:: $ sahara-templates --config-file myconfig cluster-template-delete-id --id ID Delete a specific node group template by name:: $ sahara-templates --config-file myconfig node-group-template-delete --name NAME -t $TENANT_ID Delete a specific cluster template by name:: $ sahara-templates --config-file myconfig cluster-template-delete --name NAME -t $TENANT_ID sahara-8.0.0/sahara/db/templates/api.py0000666000175100017510000007552113245514472017772 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import os import jsonschema from oslo_config import cfg from oslo_serialization import jsonutils as json from oslo_utils import uuidutils import six from sahara import conductor from sahara.db.templates import utils as u from sahara.service.validations import cluster_template_schema as clt from sahara.service.validations import node_group_template_schema as ngt from sahara.utils import api_validator LOG = None CONF = None # This is broken out to support testability def set_logger(log): global LOG LOG = log # This is broken out to support testability def set_conf(conf): global CONF CONF = conf ng_validator = api_validator.ApiValidator(ngt.NODE_GROUP_TEMPLATE_SCHEMA) ct_validator = api_validator.ApiValidator(clt.CLUSTER_TEMPLATE_SCHEMA) # Options that we allow to be replaced in a node group template node_group_template_opts = [ cfg.StrOpt('image_id', help='Image id field for a node group template.'), cfg.StrOpt('flavor_id', help='Flavor id field for a node group template.'), cfg.StrOpt('floating_ip_pool', help='Floating ip pool field for a node group template.'), cfg.BoolOpt('auto_security_group', default=False, help='Auto security group field for node group template.'), cfg.ListOpt('security_groups', default=[], help='Security group field for node group template.') ] # Options that we allow to be replaced in a cluster template cluster_template_opts = [ cfg.StrOpt('default_image_id', help='Default image id field for a cluster template.'), cfg.StrOpt('neutron_management_network', help='Neutron management network ' 'field for a cluster template.')] all_template_opts = node_group_template_opts + cluster_template_opts node_group_template_opt_names = [o.name for o in node_group_template_opts] cluster_template_opt_names = [o.name for o in cluster_template_opts] # This is a local exception class that is used to exit routines # in cases where error information has already been logged. # It is caught and suppressed everywhere it is used. class Handled(Exception): pass class Context(object): '''Create a pseudo Context object Since this tool does not use the REST interface, we do not have a request from which to build a Context. ''' def __init__(self, is_admin=False, tenant_id=None): self.is_admin = is_admin self.tenant_id = tenant_id def check_usage_of_existing(ctx, ng_templates, cl_templates): '''Determine if any of the specified templates are in use This method searches for the specified templates by name and determines whether or not any existing templates are in use by a cluster or cluster template. Returns True if any of the templates are in use. :param ng_templates: A list of dictionaries. Each dictionary has a "template" entry that represents a node group template. :param cl_templates: A list of dictionaries. Each dictionary has a "template" entry that represents a cluster template :returns: True if any of the templates are in use, False otherwise ''' error = False clusters = conductor.API.cluster_get_all(ctx) for ng_info in ng_templates: ng = u.find_node_group_template_by_name(ctx, ng_info["template"]["name"]) if ng: cluster_users, template_users = u.check_node_group_template_usage( ng["id"], clusters) if cluster_users: LOG.warning("Node group template {name} " "in use by clusters {clusters}".format( name=ng["name"], clusters=cluster_users)) if template_users: LOG.warning("Node group template {name} " "in use by cluster templates {templates}".format( name=ng["name"], templates=template_users)) if cluster_users or template_users: LOG.warning("Update of node group template " "{name} is not allowed".format(name=ng["name"])) error = True for cl_info in cl_templates: cl = u.find_cluster_template_by_name(ctx, cl_info["template"]["name"]) if cl: cluster_users = u.check_cluster_template_usage(cl["id"], clusters) if cluster_users: LOG.warning("Cluster template {name} " "in use by clusters {clusters}".format( name=cl["name"], clusters=cluster_users)) LOG.warning("Update of cluster template " "{name} is not allowed".format(name=cl["name"])) error = True return error def log_skipping_dir(path, reason=""): if reason: reason = ", " + reason LOG.warning("Skipping processing for {dir}{reason}".format( dir=path, reason=reason)) def check_cluster_templates_valid(ng_templates, cl_templates): # Check that if name references to node group templates # are replaced with a uuid value that the cluster template # passes JSON validation. We don't have the real uuid yet, # but this will allow the validation test. if ng_templates: dummy_uuid = uuidutils.generate_uuid() ng_ids = {ng["template"]["name"]: dummy_uuid for ng in ng_templates} else: ng_ids = {} for cl in cl_templates: template = copy.deepcopy(cl["template"]) u.substitute_ng_ids(template, ng_ids) try: ct_validator.validate(template) except jsonschema.ValidationError as e: LOG.warning("Validation for {path} failed, {reason}".format( path=cl["path"], reason=e)) return True return False def add_config_section(section_name, options): if section_name and hasattr(CONF, section_name): # It's already been added return if section_name: group = cfg.OptGroup(name=section_name) CONF.register_group(group) CONF.register_opts(options, group) else: # Add options to the default section CONF.register_opts(options) def add_config_section_for_template(template): '''Register a config section based on the template values Check to see if the configuration files contain a section that corresponds to the template. If an appropriate section can be found, register options for the template so that the config values can be read and applied to the template via substitution (oslo supports registering groups and options at any time, before or after the config files are parsed). Corresponding section names may be of the following forms: , example "hdp-2.0.6-master" This is useful when a template naming convention is being used, so that the template name is already unambiguous __, example "hdp_2.0.6_master" This can be used if there is a name collision between templates _, example "hdp_2.0.6" , example "hdp" DEFAULT Sections are tried in the order given above. Since the first two section naming forms refer to a specific template by name, options are added based on template type. However, the other section naming forms may map to node group templates or cluster templates, so options for both are added. ''' sections = list(CONF.list_all_sections()) unique_name = "{name}".format(**template) fullname = "{plugin_name}_{hadoop_version}_{name}".format(**template) plugin_version = "{plugin_name}_{hadoop_version}".format(**template) plugin = "{plugin_name}".format(**template) section_name = None if unique_name in sections: section_name = unique_name elif fullname in sections: section_name = fullname if section_name: if u.is_node_group(template): opts = node_group_template_opts else: opts = cluster_template_opts else: if plugin_version in sections: section_name = plugin_version elif plugin in sections: section_name = plugin opts = all_template_opts add_config_section(section_name, opts) return section_name def substitute_config_values(configs, template, path): if u.is_node_group(template): opt_names = node_group_template_opt_names else: opt_names = cluster_template_opt_names for opt, value in six.iteritems(configs): if opt in opt_names and opt in template: template[opt] = value def get_configs(section): if section is None: return dict(CONF) return dict(CONF[section]) def get_plugin_name(): if CONF.command.name == "update" and ( not CONF.command.plugin_name and ( hasattr(CONF, "plugins") and CONF.plugins)): return CONF.plugins return CONF.command.plugin_name def process_files(dirname, files): node_groups = [] clusters = [] plugin_name = get_plugin_name() try: for fname in files: if os.path.splitext(fname)[1] == ".json": fpath = os.path.join(dirname, fname) with open(fpath, 'r') as fp: try: data = fp.read() template = json.loads(data) except ValueError as e: LOG.warning("Error processing {path}, " "{reason}".format(path=fpath, reason=e)) raise Handled("error processing files") # If this file doesn't contain basic fields, skip it. # If we are filtering on plugin and version make # sure the file is one that we want if not u.check_basic_fields(template) or ( not u.check_plugin_name_and_version( template, plugin_name, CONF.command.plugin_version)): continue # Look through the sections in CONF and register # options for this template if we find a section # related to the template (ie, plugin, version, name) section = add_config_section_for_template(template) LOG.debug("Using config section {section} " "for {path}".format(section=section, path=fpath)) # Attempt to resolve substitutions using the config section substitute_config_values(get_configs(section), template, fpath) file_entry = {'template': template, 'path': fpath} if u.is_node_group(template): # JSON validator try: ng_validator.validate(template) except jsonschema.ValidationError as e: LOG.warning("Validation for {path} failed, " "{reason}".format(path=fpath, reason=e)) raise Handled( "node group template validation failed") node_groups.append(file_entry) LOG.debug("Added {path} to node group " "template files".format(path=fpath)) else: clusters.append(file_entry) LOG.debug("Added {path} to cluster template " "files".format(path=fpath)) except Handled as e: log_skipping_dir(dirname, str(e)) node_groups = [] clusters = [] except Exception as e: log_skipping_dir(dirname, "unhandled exception, {reason}".format(reason=e)) node_groups = [] clusters = [] return node_groups, clusters def delete_node_group_template(ctx, template, rollback=False): rollback_msg = " on rollback" if rollback else "" # If we are not deleting something that we just created, # do usage checks to ensure that the template is not in # use by a cluster or a cluster template if not rollback: clusters = conductor.API.cluster_get_all(ctx) cluster_templates = conductor.API.cluster_template_get_all(ctx) cluster_users, template_users = u.check_node_group_template_usage( template["id"], clusters, cluster_templates) if cluster_users: LOG.warning("Node group template {info} " "in use by clusters {clusters}".format( info=u.name_and_id(template), clusters=cluster_users)) if template_users: LOG.warning("Node group template {info} " "in use by cluster templates {templates}".format( info=u.name_and_id(template), templates=template_users)) if cluster_users or template_users: LOG.warning("Deletion of node group template " "{info} failed".format(info=u.name_and_id(template))) return try: conductor.API.node_group_template_destroy(ctx, template["id"], ignore_prot_on_def=True) except Exception as e: LOG.warning("Deletion of node group template {info} " "failed{rollback}, {reason}".format( info=u.name_and_id(template), reason=e, rollback=rollback_msg)) else: LOG.info("Deleted node group template {info}{rollback}".format( info=u.name_and_id(template), rollback=rollback_msg)) def reverse_node_group_template_creates(ctx, templates): for template in templates: delete_node_group_template(ctx, template, rollback=True) def reverse_node_group_template_updates(ctx, update_info): for template, values in update_info: # values are the original values that we overwrote in the update try: conductor.API.node_group_template_update(ctx, template["id"], values, ignore_prot_on_def=True) except Exception as e: LOG.warning("Rollback of update for node group " "template {info} failed, {reason}".format( info=u.name_and_id(template), reason=e)) else: LOG.info("Rolled back update for node group " "template {info}".format(info=u.name_and_id(template))) def add_node_group_templates(ctx, node_groups): error = False ng_info = {"ids": {}, "created": [], "updated": []} def do_reversals(ng_info): reverse_node_group_template_updates(ctx, ng_info["updated"]) reverse_node_group_template_creates(ctx, ng_info["created"]) return {}, True try: for ng in node_groups: template = ng['template'] current = u.find_node_group_template_by_name(ctx, template['name']) if current: # Track what we see in the current template that is different # from our update values. Save it for possible rollback. # Note, this is not perfect because it does not recurse through # nested structures to get an exact diff, but it ensures that # we track only fields that are valid in the JSON schema updated_fields = u.value_diff(current.to_dict(), template) # Always attempt to update. Since the template value is a # combination of JSON and config values, there is no useful # timestamp we can use to skip an update. # If sqlalchemy determines no change in fields, it will not # mark it as updated. try: template = conductor.API.node_group_template_update( ctx, current['id'], template, ignore_prot_on_def=True) except Exception as e: LOG.warning("Update of node group template {info} " "failed, {reason}".format( info=u.name_and_id(current), reason=e)) raise Handled() if template['updated_at'] != current['updated_at']: ng_info["updated"].append((template, updated_fields)) LOG.info("Updated node group template {info} " "from {path}".format(info=u.name_and_id(template), path=ng["path"])) else: LOG.debug("No change to node group template {info} " "from {path}".format( info=u.name_and_id(current), path=ng['path'])) else: template['is_default'] = True try: template = conductor.API.node_group_template_create( ctx, template) except Exception as e: LOG.warning("Creation of node group template " "from {path} failed, {reason}".format( path=ng['path'], reason=e)) raise Handled() ng_info["created"].append(template) LOG.info("Created node group template {info} " "from {path}".format(info=u.name_and_id(template), path=ng["path"])) # For the purposes of substitution we need a dict of id by name ng_info["ids"][template['name']] = template['id'] except Handled: ng_info, error = do_reversals(ng_info) except Exception as e: LOG.warning("Unhandled exception while processing " "node group templates, {reason}".format(reason=e)) ng_info, error = do_reversals(ng_info) return ng_info, error def delete_cluster_template(ctx, template, rollback=False): rollback_msg = " on rollback" if rollback else "" # If we are not deleting something that we just created, # do usage checks to ensure that the template is not in # use by a cluster if not rollback: clusters = conductor.API.cluster_get_all(ctx) cluster_users = u.check_cluster_template_usage(template["id"], clusters) if cluster_users: LOG.warning("Cluster template {info} " "in use by clusters {clusters}".format( info=u.name_and_id(template), clusters=cluster_users)) LOG.warning("Deletion of cluster template " "{info} failed".format(info=u.name_and_id(template))) return try: conductor.API.cluster_template_destroy(ctx, template["id"], ignore_prot_on_def=True) except Exception as e: LOG.warning("Deletion of cluster template {info} failed{rollback}" ", {reason}".format(info=u.name_and_id(template), reason=e, rollback=rollback_msg)) else: LOG.info("Deleted cluster template {info}{rollback}".format( info=u.name_and_id(template), rollback=rollback_msg)) def reverse_cluster_template_creates(ctx, templates): for template in templates: delete_cluster_template(ctx, template, rollback=True) def reverse_cluster_template_updates(ctx, update_info): for template, values in update_info: # values are the original values that we overwrote in the update try: conductor.API.cluster_template_update(ctx, template["id"], values, ignore_prot_on_def=True) except Exception as e: LOG.warning("Rollback of update for cluster " "template {info} failed, {reason}".format( info=u.name_and_id(template), reason=e)) else: LOG.info("Rolled back update for cluster " "template {info}".format(info=u.name_and_id(template))) def add_cluster_templates(ctx, clusters, ng_dict): '''Add cluster templates to the database. The value of any node_group_template_id fields in cluster templates which reference a node group template in ng_dict by name will be changed to the id of the node group template. If there is an error in creating or updating a template, any templates that have already been created will be delete and any updates will be reversed. :param clusters: a list of dictionaries. Each dictionary has a "template" entry holding the cluster template and a "path" entry holding the path of the file from which the template was read. :param ng_dict: a dictionary of node group template ids keyed by node group template names ''' error = False created = [] updated = [] def do_reversals(created, updated): reverse_cluster_template_updates(ctx, updated) reverse_cluster_template_creates(ctx, created) return True try: for cl in clusters: template = cl['template'] # Fix up node_group_template_id fields u.substitute_ng_ids(template, ng_dict) # Name + tenant_id is unique, so search by name current = u.find_cluster_template_by_name(ctx, template['name']) if current: # Track what we see in the current template that is different # from our update values. Save it for possible rollback. # Note, this is not perfect because it does not recurse through # nested structures to get an exact diff, but it ensures that # we track only fields that are valid in the JSON schema updated_fields = u.value_diff(current.to_dict(), template) # Always attempt to update. Since the template value is a # combination of JSON and config values, there is no useful # timestamp we can use to skip an update. # If sqlalchemy determines no change in fields, it will not # mark it as updated. # TODO(tmckay): why when I change the count in an # entry in node_groups does it not count as an update? # Probably a bug try: template = conductor.API.cluster_template_update( ctx, current['id'], template, ignore_prot_on_def=True) except Exception as e: LOG.warning("Update of cluster template {info} " "failed, {reason}".format( info=u.name_and_id(current), reason=e)) raise Handled() if template['updated_at'] != current['updated_at']: updated.append((template, updated_fields)) LOG.info("Updated cluster template {info} " "from {path}".format(info=u.name_and_id(template), path=cl['path'])) else: LOG.debug("No change to cluster template {info} " "from {path}".format(info=u.name_and_id(current), path=cl["path"])) else: template["is_default"] = True try: template = conductor.API.cluster_template_create(ctx, template) except Exception as e: LOG.warning("Creation of cluster template " "from {path} failed, {reason}".format( path=cl['path'], reason=e)) raise Handled() created.append(template) LOG.info("Created cluster template {info} " "from {path}".format(info=u.name_and_id(template), path=cl['path'])) except Handled: error = do_reversals(created, updated) except Exception as e: LOG.warning("Unhandled exception while processing " "cluster templates, {reason}".format(reason=e)) error = do_reversals(created, updated) return error def do_update(): '''Create or update default templates for the specified tenant. Looks for '.json' files beginning at the specified starting directory (--directory CLI option) and descending through subdirectories by default. The .json files represent cluster templates or node group templates. All '.json' files at the same directory level are treated as a set. Cluster templates may reference node group templates in the same set. If an error occurs in processing a set, skip it and continue. If creation of cluster templates fails, any node group templates in the set that were already created will be deleted. ''' ctx = Context(tenant_id=CONF.command.tenant_id) start_dir = os.path.abspath(CONF.command.directory) for root, dirs, files in os.walk(start_dir): # Find all the template files and identify them as node_group # or cluster templates. If there is an exception in # processing the set, return empty lists. node_groups, clusters = process_files(root, files) # Now that we know what the valid node group templates are, # we can validate the cluster templates as well. if check_cluster_templates_valid(node_groups, clusters): log_skipping_dir(root, "error processing cluster templates") # If there are existing default templates that match the names # in the template files, do usage checks here to detect update # failures early (we can't update a template in use) elif check_usage_of_existing(ctx, node_groups, clusters): log_skipping_dir(root, "templates in use") else: ng_info, error = add_node_group_templates(ctx, node_groups) if error: log_skipping_dir(root, "error processing node group templates") elif add_cluster_templates(ctx, clusters, ng_info["ids"]): log_skipping_dir(root, "error processing cluster templates") # Cluster templates failed so remove the node group templates reverse_node_group_template_updates(ctx, ng_info["updated"]) reverse_node_group_template_creates(ctx, ng_info["created"]) if CONF.command.norecurse: break def do_delete(): '''Delete default templates in the specified tenant Deletion uses the --plugin-name and --plugin-version options as filters. Only templates with 'is_default=True' will be deleted. ''' ctx = Context(tenant_id=CONF.command.tenant_id) for plugin in get_plugin_name(): kwargs = {'is_default': True} kwargs['plugin_name'] = plugin # Delete cluster templates first for the sake of usage checks lst = conductor.API.cluster_template_get_all(ctx, **kwargs) for l in lst: if not u.check_plugin_version(l, CONF.command.plugin_version): continue delete_cluster_template(ctx, l) lst = conductor.API.node_group_template_get_all(ctx, **kwargs) for l in lst: if not u.check_plugin_version(l, CONF.command.plugin_version): continue delete_node_group_template(ctx, l) def do_node_group_template_delete(): ctx = Context(tenant_id=CONF.command.tenant_id) template_name = CONF.command.template_name t = u.find_node_group_template_by_name(ctx, template_name) if t: delete_node_group_template(ctx, t) else: LOG.warning("Deletion of node group template {name} failed, " "no such template".format(name=template_name)) def do_node_group_template_delete_by_id(): ctx = Context(is_admin=True) # Make sure it's a default t = conductor.API.node_group_template_get(ctx, CONF.command.id) if t: if t["is_default"]: delete_node_group_template(ctx, t) else: LOG.warning("Deletion of node group template {info} skipped, " "not a default template".format( info=u.name_and_id(t))) else: LOG.warning("Deletion of node group template {id} failed, " "no such template".format(id=CONF.command.id)) def do_cluster_template_delete(): ctx = Context(tenant_id=CONF.command.tenant_id) template_name = CONF.command.template_name t = u.find_cluster_template_by_name(ctx, template_name) if t: delete_cluster_template(ctx, t) else: LOG.warning("Deletion of cluster template {name} failed, " "no such template".format(name=template_name)) def do_cluster_template_delete_by_id(): ctx = Context(is_admin=True) # Make sure it's a default t = conductor.API.cluster_template_get(ctx, CONF.command.id) if t: if t["is_default"]: delete_cluster_template(ctx, t) else: LOG.warning("Deletion of cluster template {info} skipped, " "not a default template".format( info=u.name_and_id(t))) else: LOG.warning("Deletion of cluster template {id} failed, " "no such template".format(id=CONF.command.id)) sahara-8.0.0/sahara/db/templates/__init__.py0000666000175100017510000000000013245514472020734 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/db/templates/utils.py0000666000175100017510000001631013245514472020350 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import six from sahara import conductor def name_and_id(template): return "{name} ({id})".format(name=template["name"], id=template["id"]) def is_node_group(template): # Node group templates and cluster templates have # different required fields in validation and neither # allows additional fields. So, the presence of # node_processes or flavor_id should suffice to # identify a node group template. Check for both # to be nice, in case someone made a typo. return 'node_processes' in template or 'flavor_id' in template def substitute_ng_ids(cl, ng_dict): '''Substitute node group template ids for node group template names If the cluster template contains node group elements with node_group_template_id fields that reference node group templates by name, substitute the node group template id for the name. The name reference is expected to be a string containing a format specifier of the form "{name}", for example "{master}" :param cl: a cluster template :param ng_dict: a dictionary of node group template ids keyed by node group template names ''' for ng in cl["node_groups"]: if "node_group_template_id" in ng: val = ng["node_group_template_id"].format(**ng_dict) ng["node_group_template_id"] = val def check_basic_fields(template): return "plugin_name" in template and ( "hadoop_version" in template and ( "name" in template)) def check_plugin_version(template, plugin_versions): '''Check that the template matches the plugin versions list Tests whether or not the plugin version indicated by the template matches one of the versions specified in plugin_versions :param template: A node group or cluster template :param plugin_versions: A list of plugin version strings. These values may be regular version strings or may be the name of the plugin followed by a "." followed by a version string. :returns: True if the plugin version specified in the template matches a version in plugin_versions or plugin_versions is an empty list. Otherwise False ''' def dotted_name(template): return template['plugin_name'] + "." + template['hadoop_version'] version_matches = plugin_versions is None or ( template['hadoop_version'] in plugin_versions) or ( dotted_name(template) in plugin_versions) return version_matches def check_plugin_name_and_version(template, plugin_names, plugin_versions): '''Check that the template is for one of the specified plugins Tests whether or not the plugin name and version indicated by the template matches one of the names and one of the versions specified in plugin_names and plugin_versions :param template: A node group or cluster template :param plugin_names: A list of plugin names :param plugin_versions: A list of plugin version strings. These values may be regular version strings or may be the name of the plugin followed by a "." followed by a version string. :returns: True if the plugin name specified in the template matches a name in plugin_names or plugin_names is an empty list, and if the plugin version specified in the template matches a version in plugin_versions or plugin_versions is an empty list. Otherwise False ''' name_and_version_matches = (plugin_names is None or ( template['plugin_name'] in plugin_names)) and ( check_plugin_version(template, plugin_versions)) return name_and_version_matches # TODO(tmckay): refactor the service validation code so # that the node group template usage checks there can be reused # without incurring unnecessary dependencies def check_node_group_template_usage(node_group_template_id, cluster_list, cluster_template_list=None): cluster_template_list = cluster_template_list or [] cluster_users = [] template_users = [] for cluster in cluster_list: if (node_group_template_id in [node_group.node_group_template_id for node_group in cluster.node_groups]): cluster_users += [cluster.name] for cluster_template in cluster_template_list: if (node_group_template_id in [node_group.node_group_template_id for node_group in cluster_template.node_groups]): template_users += [cluster_template.name] return cluster_users, template_users # TODO(tmckay): refactor the service validation code so # that the cluster template usage checks there can be reused # without incurring unnecessary dependencies def check_cluster_template_usage(cluster_template_id, cluster_list): cluster_users = [] for cluster in cluster_list: if cluster_template_id == cluster.cluster_template_id: cluster_users.append(cluster.name) return cluster_users def find_node_group_template_by_name(ctx, name): t = conductor.API.node_group_template_get_all(ctx, name=name, is_default=True) if t: return t[0] return None def find_cluster_template_by_name(ctx, name): t = conductor.API.cluster_template_get_all(ctx, name=name, is_default=True) if t: return t[0] return None def value_diff(current, new_values): '''Return the entries in current that would be overwritten by new_values Returns the set of entries in current that would be overwritten if current.update(new_values) was called. :param current: A dictionary whose key values are a superset of the key values in new_values :param new_values: A dictionary ''' # Current is an existing template from the db and # template is a set of values that has been validated # against the JSON schema for the template. # Copy items from current if they are present in template. # In the case of "node_groups" the conductor does magic # to set up template relations and insures that appropriate # fields are cleaned (like "updated_at" and "id") so we # trust the conductor in that case. diff_values = {} for k, v in six.iteritems(new_values): if k in current and current[k] != v: diff_values[k] = copy.deepcopy(current[k]) return diff_values sahara-8.0.0/sahara/db/base.py0000666000175100017510000000156513245514472016132 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Base class for classes that need modular database access.""" from oslo_config import cfg from oslo_utils import importutils CONF = cfg.CONF class Base(object): """DB driver is injected in the init method.""" def __init__(self): self.db = importutils.try_import(CONF.db_driver) sahara-8.0.0/sahara/db/sqlalchemy/0000775000175100017510000000000013245515026016773 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/db/sqlalchemy/model_base.py0000666000175100017510000000333213245514472021446 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_db.sqlalchemy import models as oslo_models from sqlalchemy.ext import declarative from sqlalchemy.orm import attributes class _SaharaBase(oslo_models.ModelBase, oslo_models.TimestampMixin): """Base class for all SQLAlchemy DB Models.""" def to_dict(self): """sqlalchemy based automatic to_dict method.""" d = {} # if a column is unloaded at this point, it is # probably deferred. We do not want to access it # here and thereby cause it to load... unloaded = attributes.instance_state(self).unloaded for col in self.__table__.columns: if col.name not in unloaded: d[col.name] = getattr(self, col.name) datetime_to_str(d, 'created_at') datetime_to_str(d, 'updated_at') return d def datetime_to_str(dct, attr_name): if dct.get(attr_name) is not None: value = dct[attr_name].isoformat('T') ms_delimiter = value.find(".") if ms_delimiter != -1: # Removing ms from time value = value[:ms_delimiter] dct[attr_name] = value SaharaBase = declarative.declarative_base(cls=_SaharaBase) sahara-8.0.0/sahara/db/sqlalchemy/models.py0000666000175100017510000004727013245514472020650 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils import sqlalchemy as sa from sqlalchemy.orm import relationship from sahara.db.sqlalchemy import model_base as mb from sahara.db.sqlalchemy import types as st # Helpers def _generate_unicode_uuid(): return uuidutils.generate_uuid() def _id_column(): return sa.Column(sa.String(36), primary_key=True, default=_generate_unicode_uuid) # Main objects: Cluster, NodeGroup, Instance class Cluster(mb.SaharaBase): """Contains all info about cluster.""" __tablename__ = 'clusters' __table_args__ = ( sa.UniqueConstraint('name', 'tenant_id'), ) id = _id_column() name = sa.Column(sa.String(80), nullable=False) description = sa.Column(sa.Text) tenant_id = sa.Column(sa.String(36)) trust_id = sa.Column(sa.String(36)) is_transient = sa.Column(sa.Boolean, default=False) plugin_name = sa.Column(sa.String(80), nullable=False) hadoop_version = sa.Column(sa.String(80), nullable=False) cluster_configs = sa.Column(st.JsonDictType()) default_image_id = sa.Column(sa.String(36)) neutron_management_network = sa.Column(sa.String(36)) anti_affinity = sa.Column(st.JsonListType()) anti_affinity_ratio = sa.Column(sa.Integer, default=1) management_private_key = sa.Column(sa.Text, nullable=False) management_public_key = sa.Column(sa.Text, nullable=False) user_keypair_id = sa.Column(sa.String(80)) status = sa.Column(sa.String(80)) status_description = sa.Column(st.LongText()) info = sa.Column(st.JsonDictType()) extra = sa.Column(st.JsonDictType()) rollback_info = sa.Column(st.JsonDictType()) sahara_info = sa.Column(st.JsonDictType()) use_autoconfig = sa.Column(sa.Boolean(), default=True) provision_progress = relationship('ClusterProvisionStep', cascade="all,delete", backref='cluster', lazy='subquery') verification = relationship('ClusterVerification', cascade="all,delete", backref="cluster", lazy='joined') node_groups = relationship('NodeGroup', cascade="all,delete", backref='cluster', lazy='subquery') cluster_template_id = sa.Column(sa.String(36), sa.ForeignKey('cluster_templates.id')) cluster_template = relationship('ClusterTemplate', backref="clusters") shares = sa.Column(st.JsonListType()) is_public = sa.Column(sa.Boolean()) is_protected = sa.Column(sa.Boolean()) domain_name = sa.Column(sa.String(255)) def to_dict(self, show_progress=False): d = super(Cluster, self).to_dict() d['node_groups'] = [ng.to_dict() for ng in self.node_groups] d['provision_progress'] = [pp.to_dict(show_progress) for pp in self.provision_progress] if self.verification: d['verification'] = self.verification[0].to_dict() return d class NodeGroup(mb.SaharaBase): """Specifies group of nodes within a cluster.""" __tablename__ = 'node_groups' __table_args__ = ( sa.UniqueConstraint('name', 'cluster_id'), ) id = _id_column() name = sa.Column(sa.String(80), nullable=False) tenant_id = sa.Column(sa.String(36)) flavor_id = sa.Column(sa.String(36), nullable=False) image_id = sa.Column(sa.String(36)) image_username = sa.Column(sa.String(36)) node_processes = sa.Column(st.JsonListType()) node_configs = sa.Column(st.JsonDictType()) volumes_per_node = sa.Column(sa.Integer) volumes_size = sa.Column(sa.Integer) volumes_availability_zone = sa.Column(sa.String(255)) volume_mount_prefix = sa.Column(sa.String(80)) volume_type = sa.Column(sa.String(255)) count = sa.Column(sa.Integer, nullable=False) use_autoconfig = sa.Column(sa.Boolean(), default=True) instances = relationship('Instance', cascade="all,delete", backref='node_group', order_by="Instance.instance_name", lazy='subquery') cluster_id = sa.Column(sa.String(36), sa.ForeignKey('clusters.id')) node_group_template_id = sa.Column(sa.String(36), sa.ForeignKey( 'node_group_templates.id')) node_group_template = relationship('NodeGroupTemplate', backref="node_groups") floating_ip_pool = sa.Column(sa.String(36)) security_groups = sa.Column(st.JsonListType()) auto_security_group = sa.Column(sa.Boolean()) availability_zone = sa.Column(sa.String(255)) open_ports = sa.Column(st.JsonListType()) is_proxy_gateway = sa.Column(sa.Boolean()) volume_local_to_instance = sa.Column(sa.Boolean()) shares = sa.Column(st.JsonListType()) def to_dict(self): d = super(NodeGroup, self).to_dict() d['instances'] = [i.to_dict() for i in self.instances] return d class Instance(mb.SaharaBase): """An OpenStack instance created for the cluster.""" __tablename__ = 'instances' __table_args__ = ( sa.UniqueConstraint('instance_id', 'node_group_id'), ) id = _id_column() tenant_id = sa.Column(sa.String(36)) node_group_id = sa.Column(sa.String(36), sa.ForeignKey('node_groups.id')) instance_id = sa.Column(sa.String(36)) instance_name = sa.Column(sa.String(80), nullable=False) internal_ip = sa.Column(sa.String(45)) management_ip = sa.Column(sa.String(45)) volumes = sa.Column(st.JsonListType()) storage_devices_number = sa.Column(sa.Integer) dns_hostname = sa.Column(sa.String(255)) # Template objects: ClusterTemplate, NodeGroupTemplate, TemplatesRelation class ClusterTemplate(mb.SaharaBase): """Template for Cluster.""" __tablename__ = 'cluster_templates' __table_args__ = ( sa.UniqueConstraint('name', 'tenant_id'), ) id = _id_column() name = sa.Column(sa.String(80), nullable=False) description = sa.Column(sa.Text) cluster_configs = sa.Column(st.JsonDictType()) default_image_id = sa.Column(sa.String(36)) anti_affinity = sa.Column(st.JsonListType()) tenant_id = sa.Column(sa.String(36)) neutron_management_network = sa.Column(sa.String(36)) plugin_name = sa.Column(sa.String(80), nullable=False) hadoop_version = sa.Column(sa.String(80), nullable=False) node_groups = relationship('TemplatesRelation', cascade="all,delete", backref='cluster_template', lazy='subquery') is_default = sa.Column(sa.Boolean(), default=False) use_autoconfig = sa.Column(sa.Boolean(), default=True) shares = sa.Column(st.JsonListType()) is_public = sa.Column(sa.Boolean()) is_protected = sa.Column(sa.Boolean()) domain_name = sa.Column(sa.String(255)) def to_dict(self): d = super(ClusterTemplate, self).to_dict() d['node_groups'] = [tr.to_dict() for tr in self.node_groups] return d class NodeGroupTemplate(mb.SaharaBase): """Template for NodeGroup.""" __tablename__ = 'node_group_templates' __table_args__ = ( sa.UniqueConstraint('name', 'tenant_id'), ) id = _id_column() name = sa.Column(sa.String(80), nullable=False) description = sa.Column(sa.Text) tenant_id = sa.Column(sa.String(36)) flavor_id = sa.Column(sa.String(36), nullable=False) image_id = sa.Column(sa.String(36)) plugin_name = sa.Column(sa.String(80), nullable=False) hadoop_version = sa.Column(sa.String(80), nullable=False) node_processes = sa.Column(st.JsonListType()) node_configs = sa.Column(st.JsonDictType()) volumes_per_node = sa.Column(sa.Integer, nullable=False) volumes_size = sa.Column(sa.Integer) volumes_availability_zone = sa.Column(sa.String(255)) volume_mount_prefix = sa.Column(sa.String(80)) volume_type = sa.Column(sa.String(255)) floating_ip_pool = sa.Column(sa.String(36)) security_groups = sa.Column(st.JsonListType()) auto_security_group = sa.Column(sa.Boolean()) availability_zone = sa.Column(sa.String(255)) is_proxy_gateway = sa.Column(sa.Boolean()) volume_local_to_instance = sa.Column(sa.Boolean()) is_default = sa.Column(sa.Boolean(), default=False) use_autoconfig = sa.Column(sa.Boolean(), default=True) shares = sa.Column(st.JsonListType()) is_public = sa.Column(sa.Boolean()) is_protected = sa.Column(sa.Boolean()) class TemplatesRelation(mb.SaharaBase): """NodeGroupTemplate - ClusterTemplate relationship. In fact, it's a template of NodeGroup in Cluster. """ __tablename__ = 'templates_relations' id = _id_column() tenant_id = sa.Column(sa.String(36)) name = sa.Column(sa.String(80), nullable=False) flavor_id = sa.Column(sa.String(36), nullable=False) image_id = sa.Column(sa.String(36)) node_processes = sa.Column(st.JsonListType()) node_configs = sa.Column(st.JsonDictType()) volumes_per_node = sa.Column(sa.Integer) volumes_size = sa.Column(sa.Integer) volumes_availability_zone = sa.Column(sa.String(255)) volume_mount_prefix = sa.Column(sa.String(80)) volume_type = sa.Column(sa.String(255)) count = sa.Column(sa.Integer, nullable=False) use_autoconfig = sa.Column(sa.Boolean(), default=True) cluster_template_id = sa.Column(sa.String(36), sa.ForeignKey('cluster_templates.id')) node_group_template_id = sa.Column(sa.String(36), sa.ForeignKey( 'node_group_templates.id')) node_group_template = relationship('NodeGroupTemplate', backref="templates_relations") floating_ip_pool = sa.Column(sa.String(36)) security_groups = sa.Column(st.JsonListType()) auto_security_group = sa.Column(sa.Boolean()) availability_zone = sa.Column(sa.String(255)) is_proxy_gateway = sa.Column(sa.Boolean()) volume_local_to_instance = sa.Column(sa.Boolean()) shares = sa.Column(st.JsonListType()) # EDP objects: DataSource, Job, Job Execution, JobBinary class DataSource(mb.SaharaBase): """DataSource - represent a diffident types of data sources. e.g. Swift, Cassandra etc. """ __tablename__ = 'data_sources' __table_args__ = ( sa.UniqueConstraint('name', 'tenant_id'), ) id = _id_column() tenant_id = sa.Column(sa.String(36)) name = sa.Column(sa.String(80), nullable=False) description = sa.Column(sa.Text()) type = sa.Column(sa.String(80), nullable=False) url = sa.Column(sa.String(256), nullable=False) credentials = sa.Column(st.JsonDictType()) is_public = sa.Column(sa.Boolean()) is_protected = sa.Column(sa.Boolean()) class JobExecution(mb.SaharaBase): """JobExecution - represent a job execution of specific cluster.""" __tablename__ = 'job_executions' id = _id_column() tenant_id = sa.Column(sa.String(36)) job_id = sa.Column(sa.String(36), sa.ForeignKey('jobs.id')) input_id = sa.Column(sa.String(36), sa.ForeignKey('data_sources.id')) output_id = sa.Column(sa.String(36), sa.ForeignKey('data_sources.id')) start_time = sa.Column(sa.DateTime()) end_time = sa.Column(sa.DateTime()) cluster_id = sa.Column(sa.String(36), sa.ForeignKey('clusters.id')) info = sa.Column(st.JsonDictType()) engine_job_id = sa.Column(sa.String(100)) return_code = sa.Column(sa.String(80)) job_configs = sa.Column(st.JsonDictType()) extra = sa.Column(st.JsonDictType()) data_source_urls = sa.Column(st.JsonDictType()) is_public = sa.Column(sa.Boolean()) is_protected = sa.Column(sa.Boolean()) def to_dict(self): d = super(JobExecution, self).to_dict() # The oozie_job_id filed is renamed to engine_job_id # to make this field more universal. But, we need to # carry both engine_job_id and oozie_job_id until we # can deprecate "oozie_job_id". d['oozie_job_id'] = self.engine_job_id return d mains_association = sa.Table("mains_association", mb.SaharaBase.metadata, sa.Column("Job_id", sa.String(36), sa.ForeignKey("jobs.id")), sa.Column("JobBinary_id", sa.String(36), sa.ForeignKey("job_binaries.id")) ) libs_association = sa.Table("libs_association", mb.SaharaBase.metadata, sa.Column("Job_id", sa.String(36), sa.ForeignKey("jobs.id")), sa.Column("JobBinary_id", sa.String(36), sa.ForeignKey("job_binaries.id")) ) class Job(mb.SaharaBase): """Job - description and location of a job binary.""" __tablename__ = 'jobs' __table_args__ = ( sa.UniqueConstraint('name', 'tenant_id'), ) id = _id_column() tenant_id = sa.Column(sa.String(36)) name = sa.Column(sa.String(80), nullable=False) description = sa.Column(sa.Text()) type = sa.Column(sa.String(80), nullable=False) is_public = sa.Column(sa.Boolean()) is_protected = sa.Column(sa.Boolean()) mains = relationship("JobBinary", secondary=mains_association, lazy="subquery") libs = relationship("JobBinary", secondary=libs_association, lazy="subquery") interface = relationship('JobInterfaceArgument', cascade="all,delete", order_by="JobInterfaceArgument.order", backref='job', lazy='subquery') def to_dict(self): d = super(Job, self).to_dict() d['mains'] = [jb.to_dict() for jb in self.mains] d['libs'] = [jb.to_dict() for jb in self.libs] d['interface'] = [arg.to_dict() for arg in self.interface] return d class JobInterfaceArgument(mb.SaharaBase): """JobInterfaceArgument - Configuration setting for a specific job.""" __tablename__ = 'job_interface_arguments' __table_args__ = ( sa.UniqueConstraint('job_id', 'name'), sa.UniqueConstraint('job_id', 'order') ) id = _id_column() job_id = sa.Column(sa.String(36), sa.ForeignKey('jobs.id'), nullable=False) tenant_id = sa.Column(sa.String(36)) name = sa.Column(sa.String(80), nullable=False) description = sa.Column(sa.Text()) mapping_type = sa.Column(sa.String(80), nullable=False) location = sa.Column(sa.Text(), nullable=False) value_type = sa.Column(sa.String(80), nullable=False) required = sa.Column(sa.Boolean(), nullable=False) order = sa.Column(sa.SmallInteger(), nullable=False) default = sa.Column(sa.Text()) class JobBinaryInternal(mb.SaharaBase): """JobBinaryInternal - raw binary storage for executable jobs.""" __tablename__ = 'job_binary_internal' __table_args__ = ( sa.UniqueConstraint('name', 'tenant_id'), ) id = _id_column() tenant_id = sa.Column(sa.String(36)) name = sa.Column(sa.String(80), nullable=False) data = sa.orm.deferred(sa.Column(st.LargeBinary())) datasize = sa.Column(sa.BIGINT) is_public = sa.Column(sa.Boolean()) is_protected = sa.Column(sa.Boolean()) class JobBinary(mb.SaharaBase): """JobBinary - raw binary storage for executable jobs.""" __tablename__ = 'job_binaries' __table_args__ = ( sa.UniqueConstraint('name', 'tenant_id'), ) id = _id_column() tenant_id = sa.Column(sa.String(36)) name = sa.Column(sa.String(80), nullable=False) description = sa.Column(sa.Text()) url = sa.Column(sa.String(256), nullable=False) extra = sa.Column(st.JsonDictType()) is_public = sa.Column(sa.Boolean()) is_protected = sa.Column(sa.Boolean()) class ClusterEvent(mb.SaharaBase): """"Event - represent a info about current provision step.""" __tablename__ = 'cluster_events' __table_args__ = ( sa.UniqueConstraint('id', 'step_id'), ) id = _id_column() node_group_id = sa.Column(sa.String(36)) instance_id = sa.Column(sa.String(36)) instance_name = sa.Column(sa.String(80)) event_info = sa.Column(sa.Text) successful = sa.Column(sa.Boolean, nullable=False) step_id = sa.Column(sa.String(36), sa.ForeignKey( 'cluster_provision_steps.id')) class ClusterProvisionStep(mb.SaharaBase): """ProvisionStep - represent a current provision step of cluster.""" __tablename__ = 'cluster_provision_steps' __table_args__ = ( sa.UniqueConstraint('id', 'cluster_id'), ) id = _id_column() cluster_id = sa.Column(sa.String(36), sa.ForeignKey('clusters.id')) tenant_id = sa.Column(sa.String(36)) step_name = sa.Column(sa.String(80)) step_type = sa.Column(sa.String(36)) total = sa.Column(sa.Integer) successful = sa.Column(sa.Boolean, nullable=True) events = relationship('ClusterEvent', cascade="all,delete", backref='ClusterProvisionStep', lazy='subquery') def to_dict(self, show_progress): d = super(ClusterProvisionStep, self).to_dict() if show_progress: d['events'] = [event.to_dict() for event in self.events] return d class ClusterVerification(mb.SaharaBase): """ClusterVerification represent results of cluster health checks.""" __tablename__ = 'cluster_verifications' __table_args__ = (sa.UniqueConstraint('id', 'cluster_id'),) id = _id_column() cluster_id = sa.Column( sa.String(36), sa.ForeignKey('clusters.id')) status = sa.Column(sa.String(15)) checks = relationship( 'ClusterHealthCheck', cascade="all,delete", backref='ClusterVerification', lazy='subquery') def to_dict(self): base = super(ClusterVerification, self).to_dict() base['checks'] = [check.to_dict() for check in self.checks] return base class ClusterHealthCheck(mb.SaharaBase): """ClusterHealthCheck respresent cluster health check.""" __tablename__ = 'cluster_health_checks' __table_args__ = (sa.UniqueConstraint('id', 'verification_id'),) id = _id_column() verification_id = sa.Column( sa.String(36), sa.ForeignKey('cluster_verifications.id')) status = sa.Column(sa.String(15)) description = sa.Column(sa.Text) name = sa.Column(sa.String(80)) class PluginData(mb.SaharaBase): """Plugin Data represents Provisioning Plugin.""" __tablename__ = 'plugin_data' __table_args__ = ( sa.UniqueConstraint('name', 'tenant_id'), ) id = _id_column() tenant_id = sa.Column(sa.String(36), nullable=False) name = sa.Column(sa.String(15), nullable=False) plugin_labels = sa.Column(st.JsonDictType()) version_labels = sa.Column(st.JsonDictType()) sahara-8.0.0/sahara/db/sqlalchemy/api.py0000666000175100017510000015710113245514472020131 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Implementation of SQLAlchemy backend.""" import copy import sys import threading from oslo_config import cfg from oslo_db import exception as db_exc from oslo_db.sqlalchemy import session as db_session from oslo_db.sqlalchemy import utils from oslo_log import log as logging import six import sqlalchemy as sa from sahara.db.sqlalchemy import models as m from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.validations import acl as validate from sahara.utils import types LOG = logging.getLogger(__name__) CONF = cfg.CONF _FACADE = None _LOCK = threading.Lock() def _create_facade_lazily(): global _LOCK, _FACADE if _FACADE is None: with _LOCK: if _FACADE is None: _FACADE = db_session.EngineFacade.from_config(CONF, sqlite_fk=True) return _FACADE def get_engine(): facade = _create_facade_lazily() return facade.get_engine() def get_session(**kwargs): facade = _create_facade_lazily() return facade.get_session(**kwargs) def _parse_sorting_args(sort_by): if sort_by is None: sort_by = "id" if sort_by[0] == "-": return sort_by[1:], "desc" return sort_by, "asc" def _get_prev_and_next_objects(objects, limit, marker, order=None): if order == 'desc': objects.reverse() position = None if limit is None: return None, None if marker: for pos, obj in enumerate(objects): if obj.id == marker.id: position = pos break else: return None, None if position - limit >= 0: prev_marker = objects[position - limit].id else: prev_marker = None if position + limit < len(objects): next_marker = objects[position + limit].id else: next_marker = None else: if limit < len(objects): next_marker = objects[limit - 1].id else: next_marker = None prev_marker = None return prev_marker, next_marker def cleanup(): global _FACADE _FACADE = None def get_backend(): """The backend is this module itself.""" return sys.modules[__name__] def model_query(model, context, session=None, project_only=True): """Query helper. :param model: base model to query :param context: context to query under :param project_only: if present and context is user-type, then restrict query to match the context's tenant_id. """ session = session or get_session() query = session.query(model) if project_only and not context.is_admin: query = query.filter( (model.tenant_id == context.tenant_id) | getattr(model, 'is_public', False)) return query def count_query(model, context, session=None, project_only=None): """Count query helper. :param model: base model to query :param context: context to query under :param project_only: if present and context is user-type, then restrict query to match the context's project_id. """ return model_query(sa.func.count(model.id), context, session, project_only) def in_filter(query, cls, search_opts): """Add 'in' filters for specified columns. Add a sqlalchemy 'in' filter to the query for any entry in the 'search_opts' dict where the key is the name of a column in 'cls' and the value is a tuple. This allows the value of a column to be matched against multiple possible values (OR). Return the modified query and any entries in search_opts whose keys do not match columns or whose values are not tuples. :param query: a non-null query object :param cls: the database model class that filters will apply to :param search_opts: a dictionary whose key/value entries are interpreted as column names and search values :returns: a tuple containing the modified query and a dictionary of unused search_opts """ if not search_opts: return query, search_opts remaining = {} for k, v in six.iteritems(search_opts): if type(v) == tuple and k in cls.__table__.columns: col = cls.__table__.columns[k] query = query.filter(col.in_(v)) else: remaining[k] = v return query, remaining def like_filter(query, cls, search_opts): """Add 'like' filters for specified columns. Add a sqlalchemy 'like' filter to the query for any entry in the 'search_opts' dict where the key is the name of a column in 'cls' and the value is a string containing '%'. This allows the value of a column to be matched against simple sql string patterns using LIKE and the '%' wildcard. Return the modified query and any entries in search_opts whose keys do not match columns or whose values are not strings containing '%'. :param query: a non-null query object :param cls: the database model class the filters will apply to :param search_opts: a dictionary whose key/value entries are interpreted as column names and search patterns :returns: a tuple containing the modified query and a dictionary of unused search_opts """ if not search_opts: return query, search_opts remaining = {} for k, v in six.iteritems(search_opts): if isinstance(v, six.string_types) and ( '%' in v and k in cls.__table__.columns): col = cls.__table__.columns[k] query = query.filter(col.like(v)) else: remaining[k] = v return query, remaining def _get_regex_op(connection): db = connection.split(':')[0].split('+')[0] regexp_op_map = { 'postgresql': '~', 'mysql': 'REGEXP' } return regexp_op_map.get(db, None) def regex_filter(query, cls, regex_cols, search_opts): """Add regex filters for specified columns. Add a regex filter to the query for any entry in the 'search_opts' dict where the key is the name of a column in 'cls' and listed in 'regex_cols' and the value is a string. Return the modified query and any entries in search_opts whose keys do not match columns or whose values are not strings. This is only supported for mysql and postgres. For other databases, the query is not altered. :param query: a non-null query object :param cls: the database model class the filters will apply to :param regex_cols: a list of columns for which regex is supported :param search_opts: a dictionary whose key/value entries are interpreted as column names and search patterns :returns: a tuple containing the modified query and a dictionary of unused search_opts """ regex_op = _get_regex_op(CONF.database.connection) if not regex_op: return query, copy.copy(search_opts) remaining = {} for k, v in six.iteritems(search_opts): if isinstance(v, six.string_types) and ( k in cls.__table__.columns and k in regex_cols): col = cls.__table__.columns[k] query = query.filter(col.op(regex_op)(v)) else: remaining[k] = v return query, remaining def setup_db(): try: engine = get_engine() m.Cluster.metadata.create_all(engine) except sa.exc.OperationalError as e: LOG.warning("Database registration exception: {exc}".format(exc=e)) return False return True def drop_db(): try: engine = get_engine() m.Cluster.metadata.drop_all(engine) except Exception as e: LOG.warning("Database shutdown exception: {exc}".format(exc=e)) return False return True # Cluster ops def _cluster_get(context, session, cluster_id): query = model_query(m.Cluster, context, session) return query.filter_by(id=cluster_id).first() def cluster_get(context, cluster_id): return _cluster_get(context, get_session(), cluster_id) def cluster_get_all(context, regex_search=False, limit=None, marker=None, sort_by=None, **kwargs): sort_by, order = _parse_sorting_args(sort_by) regex_cols = ['name', 'description', 'plugin_name', 'tenant_id'] query = model_query(m.Cluster, context) if regex_search: query, kwargs = regex_filter(query, m.Cluster, regex_cols, kwargs) limit = int(limit) if limit else None marker = cluster_get(context, marker) prev_marker, next_marker = _get_prev_and_next_objects( query.filter_by(**kwargs).order_by(sort_by).all(), limit, marker, order=order) result = utils.paginate_query(query.filter_by(**kwargs), m.Cluster, limit, [sort_by], marker, order) return types.Page(result, prev_marker, next_marker) def cluster_create(context, values): values = values.copy() cluster = m.Cluster() node_groups = values.pop("node_groups", []) cluster.update(values) session = get_session() try: with session.begin(): session.add(cluster) session.flush(objects=[cluster]) for ng in node_groups: node_group = m.NodeGroup() node_group.update(ng) node_group.update({"cluster_id": cluster.id}) session.add(node_group) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for object %(object)s. Failed on columns: " "%(columns)s") % {"object": e.value, "columns": e.columns}) return cluster_get(context, cluster.id) def cluster_update(context, cluster_id, values): session = get_session() try: with session.begin(): cluster = _cluster_get(context, session, cluster_id) if cluster is None: raise ex.NotFoundException(cluster_id, _("Cluster id '%s' not found!")) cluster.update(values) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for Cluster: %s") % e.columns) return cluster def cluster_destroy(context, cluster_id): session = get_session() with session.begin(): cluster = _cluster_get(context, session, cluster_id) if not cluster: raise ex.NotFoundException(cluster_id, _("Cluster id '%s' not found!")) session.delete(cluster) # Node Group ops def _node_group_get(context, session, node_group_id): query = model_query(m.NodeGroup, context, session) return query.filter_by(id=node_group_id).first() def node_group_add(context, cluster_id, values): session = get_session() with session.begin(): cluster = _cluster_get(context, session, cluster_id) if not cluster: raise ex.NotFoundException(cluster_id, _("Cluster id '%s' not found!")) node_group = m.NodeGroup() node_group.update({"cluster_id": cluster_id}) node_group.update(values) session.add(node_group) return node_group.id def node_group_update(context, node_group_id, values): session = get_session() with session.begin(): node_group = _node_group_get(context, session, node_group_id) if not node_group: raise ex.NotFoundException(node_group_id, _("Node Group id '%s' not found!")) node_group.update(values) def node_group_remove(context, node_group_id): session = get_session() with session.begin(): node_group = _node_group_get(context, session, node_group_id) if not node_group: raise ex.NotFoundException(node_group_id, _("Node Group id '%s' not found!")) session.delete(node_group) # Instance ops def _instance_get(context, session, instance_id): query = model_query(m.Instance, context, session) return query.filter_by(id=instance_id).first() def instance_add(context, node_group_id, values): session = get_session() with session.begin(): node_group = _node_group_get(context, session, node_group_id) if not node_group: raise ex.NotFoundException(node_group_id, _("Node Group id '%s' not found!")) instance = m.Instance() instance.update({"node_group_id": node_group_id}) instance.update(values) session.add(instance) node_group = _node_group_get(context, session, node_group_id) node_group.count += 1 return instance.id def instance_update(context, instance_id, values): session = get_session() with session.begin(): instance = _instance_get(context, session, instance_id) if not instance: raise ex.NotFoundException(instance_id, _("Instance id '%s' not found!")) instance.update(values) def instance_remove(context, instance_id): session = get_session() with session.begin(): instance = _instance_get(context, session, instance_id) if not instance: raise ex.NotFoundException(instance_id, _("Instance id '%s' not found!")) session.delete(instance) node_group_id = instance.node_group_id node_group = _node_group_get(context, session, node_group_id) node_group.count -= 1 # Volumes ops def append_volume(context, instance_id, volume_id): session = get_session() with session.begin(): instance = _instance_get(context, session, instance_id) if not instance: raise ex.NotFoundException(instance_id, _("Instance id '%s' not found!")) instance.volumes.append(volume_id) def remove_volume(context, instance_id, volume_id): session = get_session() with session.begin(): instance = _instance_get(context, session, instance_id) if not instance: raise ex.NotFoundException(instance_id, _("Instance id '%s' not found!")) instance.volumes.remove(volume_id) # Cluster Template ops def _cluster_template_get(context, session, cluster_template_id): query = model_query(m.ClusterTemplate, context, session) return query.filter_by(id=cluster_template_id).first() def cluster_template_get(context, cluster_template_id): return _cluster_template_get(context, get_session(), cluster_template_id) def cluster_template_get_all(context, regex_search=False, marker=None, limit=None, sort_by=None, **kwargs): regex_cols = ['name', 'description', 'plugin_name', 'tenant_id'] sort_by, order = _parse_sorting_args(sort_by) query = model_query(m.ClusterTemplate, context) if regex_search: query, kwargs = regex_filter(query, m.ClusterTemplate, regex_cols, kwargs) limit = int(limit) if limit else None marker = cluster_template_get(context, marker) prev_marker, next_marker = _get_prev_and_next_objects( query.filter_by(**kwargs).order_by(sort_by).all(), limit, marker, order=order) result = utils.paginate_query(query.filter_by(**kwargs), m.ClusterTemplate, limit, [sort_by], marker, order) return types.Page(result, prev_marker, next_marker) def cluster_template_create(context, values): values = values.copy() cluster_template = m.ClusterTemplate() node_groups = values.pop("node_groups") or [] cluster_template.update(values) session = get_session() try: with session.begin(): session.add(cluster_template) session.flush(objects=[cluster_template]) for ng in node_groups: node_group = m.TemplatesRelation() node_group.update({"cluster_template_id": cluster_template.id}) node_group.update(ng) session.add(node_group) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for object %(object)s. Failed on columns: " "%(columns)s") % {"object": e.value, "columns": e.columns}) return cluster_template_get(context, cluster_template.id) def cluster_template_destroy(context, cluster_template_id, ignore_prot_on_def=False): session = get_session() with session.begin(): cluster_template = _cluster_template_get(context, session, cluster_template_id) if not cluster_template: raise ex.NotFoundException( cluster_template_id, _("Cluster Template id '%s' not found!")) validate.check_tenant_for_delete(context, cluster_template) if not (cluster_template.is_default and ignore_prot_on_def): validate.check_protected_from_delete(cluster_template) session.delete(cluster_template) def cluster_template_update(context, values, ignore_prot_on_def=False): explicit_node_groups = "node_groups" in values if explicit_node_groups: node_groups = values.pop("node_groups") if node_groups is None: node_groups = [] session = get_session() cluster_template_id = values['id'] try: with session.begin(): cluster_template = (_cluster_template_get( context, session, cluster_template_id)) if not cluster_template: raise ex.NotFoundException( cluster_template_id, _("Cluster Template id '%s' not found!")) validate.check_tenant_for_update(context, cluster_template) if not (cluster_template.is_default and ignore_prot_on_def): validate.check_protected_from_update(cluster_template, values) if len(cluster_template.clusters) > 0: raise ex.UpdateFailedException( cluster_template_id, _("Cluster Template id '%s' can not be updated. " "It is referenced by at least one cluster.") ) cluster_template.update(values) # The flush here will cause a duplicate entry exception if # unique constraints are violated, before we go ahead and delete # the node group templates session.flush(objects=[cluster_template]) # If node_groups has not been specified, then we are # keeping the old ones so don't delete! if explicit_node_groups: model_query(m.TemplatesRelation, context, session=session).filter_by( cluster_template_id=cluster_template_id).delete() for ng in node_groups: node_group = m.TemplatesRelation() node_group.update(ng) node_group.update({"cluster_template_id": cluster_template_id}) session.add(node_group) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for ClusterTemplate: %s") % e.columns) return cluster_template_get(context, cluster_template_id) # Node Group Template ops def _node_group_template_get(context, session, node_group_template_id): query = model_query(m.NodeGroupTemplate, context, session) return query.filter_by(id=node_group_template_id).first() def node_group_template_get(context, node_group_template_id): return _node_group_template_get(context, get_session(), node_group_template_id) def node_group_template_get_all(context, regex_search=False, marker=None, limit=None, sort_by=None, **kwargs): sort_by, order = _parse_sorting_args(sort_by) regex_cols = ['name', 'description', 'plugin_name', 'tenant_id'] limit = int(limit) if limit else None query = model_query(m.NodeGroupTemplate, context) if regex_search: query, kwargs = regex_filter(query, m.NodeGroupTemplate, regex_cols, kwargs) marker = node_group_template_get(context, marker) prev_marker, next_marker = _get_prev_and_next_objects( query.filter_by(**kwargs).order_by(sort_by).all(), limit, marker, order=order) result = utils.paginate_query( query.filter_by(**kwargs), m.NodeGroupTemplate, limit, [sort_by], marker, order) return types.Page(result, prev_marker, next_marker) def node_group_template_create(context, values): node_group_template = m.NodeGroupTemplate() node_group_template.update(values) session = get_session() try: with session.begin(): session.add(node_group_template) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for NodeGroupTemplate: %s") % e.columns) return node_group_template def node_group_template_destroy(context, node_group_template_id, ignore_prot_on_def=False): session = get_session() with session.begin(): node_group_template = _node_group_template_get(context, session, node_group_template_id) if not node_group_template: raise ex.NotFoundException( node_group_template_id, _("Node Group Template id '%s' not found!")) validate.check_tenant_for_delete(context, node_group_template) if not (node_group_template.is_default and ignore_prot_on_def): validate.check_protected_from_delete(node_group_template) session.delete(node_group_template) def node_group_template_update(context, values, ignore_prot_on_def=False): session = get_session() try: with session.begin(): ngt_id = values['id'] ngt = _node_group_template_get(context, session, ngt_id) if not ngt: raise ex.NotFoundException( ngt_id, _("NodeGroupTemplate id '%s' not found")) validate.check_tenant_for_update(context, ngt) if not (ngt.is_default and ignore_prot_on_def): validate.check_protected_from_update(ngt, values) # Check to see that the node group template to be updated is not in # use by an existing cluster. for template_relationship in ngt.templates_relations: if len(template_relationship.cluster_template.clusters) > 0: raise ex.UpdateFailedException( ngt_id, _("NodeGroupTemplate id '%s' can not be updated. " "It is referenced by an existing cluster.") ) ngt.update(values) # Here we update any cluster templates that reference the # updated node group template for template_relationship in ngt.templates_relations: ct_id = template_relationship.cluster_template_id ct = cluster_template_get( context, template_relationship.cluster_template_id) node_groups = ct.node_groups ct_node_groups = [] for ng in node_groups: # Need to fill in all node groups, not just # the modified group ng_to_add = ng if ng.node_group_template_id == ngt_id: # use the updated node group template ng_to_add = ngt ng_to_add = ng_to_add.to_dict() ng_to_add.update( {"count": ng["count"], "node_group_template_id": ng.node_group_template_id}) ng_to_add.pop("updated_at", None) ng_to_add.pop("created_at", None) ng_to_add.pop("id", None) ct_node_groups.append(ng_to_add) ct_update = {"id": ct_id, "node_groups": ct_node_groups} cluster_template_update(context, ct_update, ignore_prot_on_def) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for NodeGroupTemplate: %s") % e.columns) return ngt # Data Source ops def _data_source_get(context, session, data_source_id): query = model_query(m.DataSource, context, session) return query.filter_by(id=data_source_id).first() def data_source_get(context, data_source_id): return _data_source_get(context, get_session(), data_source_id) def data_source_count(context, **kwargs): """Count DataSource objects filtered by search criteria in kwargs. Entries in kwargs indicate column names and search values. 'in' filters will be used to search for any entries in kwargs that name DataSource columns and have values of type tuple. This allows column values to match multiple values (OR) 'like' filters will be used for any entries in kwargs that name DataSource columns and have string values containing '%'. This allows column values to match simple wildcards. Any other entries in kwargs will be searched for using filter_by() """ query = model_query(m.DataSource, context) query, kwargs = in_filter(query, m.DataSource, kwargs) query, kwargs = like_filter(query, m.DataSource, kwargs) # Use normal filter_by for remaining keys return query.filter_by(**kwargs).count() def data_source_get_all(context, regex_search=False, limit=None, marker=None, sort_by=None, **kwargs): regex_cols = ['name', 'description', 'url'] sort_by, order = _parse_sorting_args(sort_by) query = model_query(m.DataSource, context) if regex_search: query, kwargs = regex_filter(query, m.DataSource, regex_cols, kwargs) limit = int(limit) if limit else None marker = data_source_get(context, marker) prev_marker, next_marker = _get_prev_and_next_objects( query.filter_by(**kwargs).order_by(sort_by).all(), limit, marker, order=order) result = utils.paginate_query(query.filter_by(**kwargs), m.DataSource, limit, [sort_by], marker, order) return types.Page(result, prev_marker, next_marker) def data_source_create(context, values): data_source = m.DataSource() data_source.update(values) session = get_session() try: with session.begin(): session.add(data_source) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for DataSource: %s") % e.columns) return data_source def data_source_destroy(context, data_source_id): session = get_session() try: with session.begin(): data_source = _data_source_get(context, session, data_source_id) if not data_source: raise ex.NotFoundException( data_source_id, _("Data Source id '%s' not found!")) validate.check_tenant_for_delete(context, data_source) validate.check_protected_from_delete(data_source) session.delete(data_source) except db_exc.DBError as e: msg = ("foreign key constraint" in six.text_type(e) and _(" on foreign key constraint") or "") raise ex.DeletionFailed(_("Data Source deletion failed%s") % msg) def data_source_update(context, values): session = get_session() try: with session.begin(): ds_id = values['id'] data_source = _data_source_get(context, session, ds_id) if not data_source: raise ex.NotFoundException( ds_id, _("DataSource id '%s' not found")) validate.check_tenant_for_update(context, data_source) validate.check_protected_from_update(data_source, values) data_source.update(values) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for DataSource: %s") % e.columns) return data_source # JobExecution ops def _job_execution_get(context, session, job_execution_id): query = model_query(m.JobExecution, context, session) return query.filter_by(id=job_execution_id).first() def job_execution_get(context, job_execution_id): return _job_execution_get(context, get_session(), job_execution_id) def job_execution_get_all(context, regex_search=False, limit=None, marker=None, sort_by=None, **kwargs): """Get all JobExecutions filtered by **kwargs. kwargs key values may be the names of fields in a JobExecution plus the following special values with the indicated meaning: 'cluster.name' -- name of the Cluster referenced by the JobExecution 'job.name' -- name of the Job referenced by the JobExecution 'status' -- JobExecution['info']['status'] e.g. job_execution_get_all(cluster_id=12, input_id=123) job_execution_get_all(**{'cluster.name': 'test', 'job.name': 'wordcount'}) """ sort_by, order = _parse_sorting_args(sort_by) regex_cols = ['job.name', 'cluster.name'] # Remove the external fields if present, they'll # be handled with a join and filter externals = {k: kwargs.pop(k) for k in ['cluster.name', 'job.name', 'status'] if k in kwargs} # At this time, none of the fields in m.JobExecution itself # are candidates for regex search, however this code fragment # should remain in case that changes. This is the correct place # to insert regex filters on the m.JobExecution class query = model_query(m.JobExecution, context) if regex_search: query, kwargs = regex_filter(query, m.JobExecution, regex_cols, kwargs) # Filter JobExecution by the remaining kwargs. This has to be done # before application of the joins and filters because those # change the class that query.filter_by will apply to query = query.filter_by(**kwargs) # Now add the joins and filters for the externals if 'cluster.name' in externals: search_opts = {'name': externals['cluster.name']} query = query.join(m.Cluster) if regex_filter and 'cluster.name' in regex_cols: query, search_opts = regex_filter(query, m.Cluster, ['name'], search_opts) query = query.filter_by(**search_opts) if 'job.name' in externals: search_opts = {'name': externals['job.name']} query = query.join(m.Job) if regex_filter and 'job.name' in regex_cols: query, search_opts = regex_filter(query, m.Job, ['name'], search_opts) query = query.filter_by(**search_opts) res = query.order_by(sort_by).all() if order == 'desc': res.reverse() # 'info' is a JsonDictType which is stored as a string. # It would be possible to search for the substring containing # the value of 'status' in 'info', but 'info' also contains # data returned from a client and not managed by Sahara. # In the case of Oozie jobs, for example, other fields (actions) # also contain 'status'. Therefore we can't filter on it reliably # by a substring search in the query. if 'status' in externals: status = externals['status'].lower() res = [je for je in res if ( je['info'] and je['info'].get('status', '').lower() == status)] res_page = res if marker: n = None for i, je in enumerate(res): if je['id'] == marker: n = i if n: res_page = res[n:] if limit: limit = int(limit) res_page = res_page[:limit] if limit < len(res_page) else res_page marker = job_execution_get(context, marker) prev_marker, next_marker = _get_prev_and_next_objects( res, limit, marker) return types.Page(res_page, prev_marker, next_marker) def job_execution_count(context, **kwargs): query = count_query(m.JobExecution, context) return query.filter_by(**kwargs).first()[0] def _get_config_section(configs, mapping_type): if mapping_type not in configs: configs[mapping_type] = [] if mapping_type == "args" else {} return configs[mapping_type] def _merge_execution_interface(job_ex, job, execution_interface): """Merges the interface for a job execution with that of its job.""" configs = job_ex.job_configs or {} nonexistent = object() positional_args = {} for arg in job.interface: value = nonexistent typed_configs = _get_config_section(configs, arg.mapping_type) # Interface args are our first choice for the value. if arg.name in execution_interface: value = execution_interface[arg.name] else: # If a default exists, we can use that, but... if arg.default is not None: value = arg.default # We should prefer an argument passed through the # job_configs that maps to the same location. if arg.mapping_type != "args": value = typed_configs.get(arg.location, value) if value is not nonexistent: if arg.mapping_type != "args": typed_configs[arg.location] = value else: positional_args[int(arg.location)] = value if positional_args: positional_args = [positional_args[i] for i in range(len(positional_args))] configs["args"] = positional_args + configs["args"] if configs and not job_ex.job_configs: job_ex.job_configs = configs def job_execution_create(context, values): session = get_session() execution_interface = values.pop('interface', {}) job_ex = m.JobExecution() job_ex.update(values) try: with session.begin(): job_ex.interface = [] job = _job_get(context, session, job_ex.job_id) if job.interface: _merge_execution_interface(job_ex, job, execution_interface) session.add(job_ex) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for JobExecution: %s") % e.columns) return job_ex def job_execution_update(context, job_execution_id, values): session = get_session() with session.begin(): job_ex = _job_execution_get(context, session, job_execution_id) if not job_ex: raise ex.NotFoundException(job_execution_id, _("JobExecution id '%s' not found!")) job_ex.update(values) session.add(job_ex) return job_ex def job_execution_destroy(context, job_execution_id): session = get_session() with session.begin(): job_ex = _job_execution_get(context, session, job_execution_id) if not job_ex: raise ex.NotFoundException(job_execution_id, _("JobExecution id '%s' not found!")) session.delete(job_ex) # Job ops def _job_get(context, session, job_id): query = model_query(m.Job, context, session) return query.filter_by(id=job_id).first() def job_get(context, job_id): return _job_get(context, get_session(), job_id) def job_get_all(context, regex_search=False, limit=None, marker=None, sort_by=None, **kwargs): regex_cols = ['name', 'description'] sort_by, order = _parse_sorting_args(sort_by) query = model_query(m.Job, context) if regex_search: query, kwargs = regex_filter(query, m.Job, regex_cols, kwargs) limit = int(limit) if limit else None marker = job_get(context, marker) prev_marker, next_marker = _get_prev_and_next_objects( query.filter_by(**kwargs).order_by(sort_by).all(), limit, marker, order=order) result = utils.paginate_query(query.filter_by(**kwargs), m.Job, limit, [sort_by], marker, order) return types.Page(result, prev_marker, next_marker) def _append_job_binaries(context, session, from_list, to_list): for job_binary_id in from_list: job_binary = model_query( m.JobBinary, context, session).filter_by(id=job_binary_id).first() if job_binary is not None: to_list.append(job_binary) def _append_interface(context, from_list, to_list): for order, argument_values in enumerate(from_list): argument_values['tenant_id'] = context.tenant_id argument_values['order'] = order argument = m.JobInterfaceArgument() argument.update(argument_values) to_list.append(argument) def job_create(context, values): mains = values.pop("mains", []) libs = values.pop("libs", []) interface = values.pop("interface", []) session = get_session() try: with session.begin(): job = m.Job() job.update(values) # These are 'lazy' objects. The initialization below # is needed here because it provides libs, mains, and # interface to be initialized within a session even if # the lists are empty job.mains = [] job.libs = [] job.interface = [] _append_job_binaries(context, session, mains, job.mains) _append_job_binaries(context, session, libs, job.libs) _append_interface(context, interface, job.interface) session.add(job) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for Job: %s") % e.columns) return job def job_update(context, job_id, values): session = get_session() try: with session.begin(): job = _job_get(context, session, job_id) if not job: raise ex.NotFoundException(job_id, _("Job id '%s' not found!")) validate.check_tenant_for_update(context, job) validate.check_protected_from_update(job, values) job.update(values) session.add(job) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for Job: %s") % e.columns) return job def job_destroy(context, job_id): session = get_session() try: with session.begin(): job = _job_get(context, session, job_id) if not job: raise ex.NotFoundException(job_id, _("Job id '%s' not found!")) validate.check_tenant_for_delete(context, job) validate.check_protected_from_delete(job) session.delete(job) except db_exc.DBError as e: msg = ("foreign key constraint" in six.text_type(e) and _(" on foreign key constraint") or "") raise ex.DeletionFailed(_("Job deletion failed%s") % msg) # JobBinary ops def _job_binary_get(context, session, job_binary_id): query = model_query(m.JobBinary, context, session) return query.filter_by(id=job_binary_id).first() def job_binary_get_all(context, regex_search=False, limit=None, marker=None, sort_by=None, **kwargs): sort_by, order = _parse_sorting_args(sort_by) regex_cols = ['name', 'description', 'url'] query = model_query(m.JobBinary, context) if regex_search: query, kwargs = regex_filter(query, m.JobBinary, regex_cols, kwargs) limit = int(limit) if limit else None marker = job_binary_get(context, marker) prev_marker, next_marker = _get_prev_and_next_objects( query.filter_by(**kwargs).order_by(sort_by).all(), limit, marker, order=order) result = utils.paginate_query(query.filter_by(**kwargs), m.JobBinary, limit, [sort_by], marker, order) return types.Page(result, prev_marker, next_marker) def job_binary_get(context, job_binary_id): """Returns a JobBinary object that does not contain a data field The data column uses deferred loading. """ return _job_binary_get(context, get_session(), job_binary_id) def job_binary_create(context, values): """Returns a JobBinary that does not contain a data field The data column uses deferred loading. """ job_binary = m.JobBinary() job_binary.update(values) session = get_session() try: with session.begin(): session.add(job_binary) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for JobBinary: %s") % e.columns) return job_binary def job_binary_update(context, values): """Returns a JobBinary updated with the provided values.""" jb_id = values["id"] session = get_session() try: with session.begin(): jb = _job_binary_get(context, session, jb_id) if not jb: raise ex.NotFoundException( jb_id, _("JobBinary id '%s' not found")) validate.check_tenant_for_update(context, jb) validate.check_protected_from_update(jb, values) # We do not want to update the url for internal binaries new_url = values.get("url", None) if new_url and "internal-db://" in jb["url"]: if jb["url"] != new_url: raise ex.UpdateFailedException( jb_id, _("The url for JobBinary Id '%s' can not " "be updated because it is an internal-db url.")) jobs = job_execution_get_all(context) pending_jobs = [job for job in jobs if job.info["status"] == "PENDING"] if len(pending_jobs) > 0: for job in pending_jobs: if _check_job_binary_referenced( context, session, jb_id, job.job_id): raise ex.UpdateFailedException( jb_id, _("JobBinary Id '%s' is used in a PENDING job " "and can not be updated.")) jb.update(values) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for JobBinary: %s") % e.columns) return jb def _check_job_binary_referenced(ctx, session, job_binary_id, job_id=None): args = {"JobBinary_id": job_binary_id} if job_id: args["Job_id"] = job_id mains = model_query(m.mains_association, ctx, session, project_only=False).filter_by(**args) libs = model_query(m.libs_association, ctx, session, project_only=False).filter_by(**args) return mains.first() is not None or libs.first() is not None def job_binary_destroy(context, job_binary_id): session = get_session() with session.begin(): job_binary = _job_binary_get(context, session, job_binary_id) if not job_binary: raise ex.NotFoundException(job_binary_id, _("JobBinary id '%s' not found!")) validate.check_tenant_for_delete(context, job_binary) validate.check_protected_from_delete(job_binary) if _check_job_binary_referenced(context, session, job_binary_id): raise ex.DeletionFailed( _("JobBinary is referenced and cannot be deleted")) session.delete(job_binary) # JobBinaryInternal ops def _job_binary_internal_get(context, session, job_binary_internal_id): query = model_query(m.JobBinaryInternal, context, session) return query.filter_by(id=job_binary_internal_id).first() def job_binary_internal_get_all(context, regex_search=False, limit=None, marker=None, sort_by=None, **kwargs): """Returns JobBinaryInternal objects that do not contain a data field The data column uses deferred loading. """ sort_by, order = _parse_sorting_args(sort_by) regex_cols = ['name'] query = model_query(m.JobBinaryInternal, context) if regex_search: query, kwargs = regex_filter(query, m.JobBinaryInternal, regex_cols, kwargs) limit = int(limit) if limit else None marker = job_binary_internal_get(context, marker) prev_marker, next_marker = _get_prev_and_next_objects( query.filter_by(**kwargs).order_by(sort_by).all(), limit, marker, order=order) result = utils.paginate_query(query.filter_by(**kwargs), m.JobBinaryInternal, limit, [sort_by], marker, order) return types.Page(result, prev_marker, next_marker) def job_binary_internal_get(context, job_binary_internal_id): """Returns a JobBinaryInternal object that does not contain a data field The data column uses deferred loading. """ return _job_binary_internal_get(context, get_session(), job_binary_internal_id) def job_binary_internal_get_raw_data(context, job_binary_internal_id): """Returns only the data field for the specified JobBinaryInternal.""" query = model_query(m.JobBinaryInternal, context) res = query.filter_by(id=job_binary_internal_id).first() if res is not None: datasize_KB = res.datasize / 1024.0 if datasize_KB > CONF.job_binary_max_KB: raise ex.DataTooBigException( round(datasize_KB, 1), CONF.job_binary_max_KB, _("Size of internal binary (%(size)sKB) is greater than the " "maximum (%(maximum)sKB)")) # This assignment is sufficient to load the deferred column res = res.data return res def job_binary_internal_create(context, values): """Returns a JobBinaryInternal that does not contain a data field The data column uses deferred loading. """ values["datasize"] = len(values["data"]) datasize_KB = values["datasize"] / 1024.0 if datasize_KB > CONF.job_binary_max_KB: raise ex.DataTooBigException( round(datasize_KB, 1), CONF.job_binary_max_KB, _("Size of internal binary (%(size)sKB) is greater " "than the maximum (%(maximum)sKB)")) job_binary_int = m.JobBinaryInternal() job_binary_int.update(values) session = get_session() try: with session.begin(): session.add(job_binary_int) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for JobBinaryInternal: %s") % e.columns) return job_binary_internal_get(context, job_binary_int.id) def job_binary_internal_destroy(context, job_binary_internal_id): session = get_session() with session.begin(): job_binary_internal = _job_binary_internal_get(context, session, job_binary_internal_id) if not job_binary_internal: raise ex.NotFoundException( job_binary_internal_id, _("JobBinaryInternal id '%s' not found!")) validate.check_tenant_for_delete(context, job_binary_internal) validate.check_protected_from_delete(job_binary_internal) session.delete(job_binary_internal) def job_binary_internal_update(context, job_binary_internal_id, values): """Returns a JobBinary updated with the provided values.""" session = get_session() try: with session.begin(): j_b_i = _job_binary_internal_get( context, session, job_binary_internal_id) if not j_b_i: raise ex.NotFoundException( job_binary_internal_id, _("JobBinaryInternal id '%s' not found!")) validate.check_tenant_for_update(context, j_b_i) validate.check_protected_from_update(j_b_i, values) j_b_i.update(values) except db_exc.DBDuplicateEntry as e: raise ex.DBDuplicateEntry( _("Duplicate entry for JobBinaryInternal: %s") % e.columns) return j_b_i # Events ops def _cluster_provision_step_get(context, session, provision_step_id): query = model_query(m.ClusterProvisionStep, context, session) return query.filter_by(id=provision_step_id).first() def _cluster_provision_step_update(context, session, step_id): step = _cluster_provision_step_get(context, session, step_id) if step is None: raise ex.NotFoundException( step_id, _("Cluster Provision Step id '%s' not found!")) if step.successful is not None: return if len(step.events) == step.total: for event in step.events: session.delete(event) step.update({'successful': True}) def cluster_provision_step_add(context, cluster_id, values): session = get_session() with session.begin(): cluster = _cluster_get(context, session, cluster_id) if not cluster: raise ex.NotFoundException(cluster_id, _("Cluster id '%s' not found!")) provision_step = m.ClusterProvisionStep() values['cluster_id'] = cluster_id values['tenant_id'] = context.tenant_id provision_step.update(values) session.add(provision_step) return provision_step.id def cluster_provision_step_update(context, step_id): if CONF.disable_event_log: return session = get_session() with session.begin(): _cluster_provision_step_update(context, session, step_id) def cluster_provision_progress_update(context, cluster_id): if CONF.disable_event_log: return _cluster_get(context, get_session(), cluster_id) session = get_session() with session.begin(): cluster = _cluster_get(context, session, cluster_id) if cluster is None: raise ex.NotFoundException(cluster_id, _("Cluster id '%s' not found!")) for step in cluster.provision_progress: if step.successful is None: _cluster_provision_step_update(context, session, step.id) result_cluster = _cluster_get(context, session, cluster_id) return result_cluster def cluster_event_add(context, step_id, values): session = get_session() with session.begin(): provision_step = _cluster_provision_step_get( context, session, step_id) if not provision_step: raise ex.NotFoundException( step_id, _("Cluster Provision Step id '%s' not found!")) event = m.ClusterEvent() values['step_id'] = step_id if not values['successful']: provision_step.update({'successful': False}) event.update(values) session.add(event) return event.id # Cluster verifications / health check ops def _cluster_verification_get(context, session, verification_id): # tenant id is not presented query = model_query(m.ClusterVerification, context, session, project_only=False) return query.filter_by(id=verification_id).first() def cluster_verification_get(context, verification_id): return _cluster_verification_get(context, get_session(), verification_id) def cluster_verification_add(context, cluster_id, values): session = get_session() with session.begin(): cluster = _cluster_get(context, session, cluster_id) if not cluster: raise ex.NotFoundException( cluster_id, _("Cluster id '%s' not found!")) verification = m.ClusterVerification() values['cluster_id'] = cluster_id verification.update(values) session.add(verification) return verification def cluster_verification_update(context, verification_id, values): session = get_session() with session.begin(): verification = _cluster_verification_get( context, session, verification_id) if not verification: raise ex.NotFoundException( verification_id, _("Verification id '%s' not found!")) verification.update(values) return verification def cluster_verification_delete(context, verification_id): session = get_session() with session.begin(): verification = _cluster_verification_get( context, session, verification_id) if not verification: raise ex.NotFoundException( verification_id, _("Verification id '%s' not found!")) for check in verification.checks: session.delete(check) session.delete(verification) def _cluster_health_check_get(context, session, health_check_id): # tenant id is not presented query = model_query(m.ClusterHealthCheck, context, session, project_only=False) return query.filter_by(id=health_check_id).first() def cluster_health_check_get(context, health_check_id): return _cluster_health_check_get(context, get_session(), health_check_id) def cluster_health_check_add(context, verification_id, values): session = get_session() with session.begin(): verification = _cluster_verification_get( context, session, verification_id) if not verification: raise ex.NotFoundException( verification_id, _("Verification id '%s' not found!")) health_check = m.ClusterHealthCheck() values['verification_id'] = verification_id values['tenant_id'] = context.tenant_id health_check.update(values) session.add(health_check) return health_check def cluster_health_check_update(context, health_check_id, values): session = get_session() with session.begin(): health_check = _cluster_health_check_get( context, session, health_check_id) if not health_check: raise ex.NotFoundException( health_check_id, _("Health check id '%s' not found!")) health_check.update(values) return health_check def _plugin_get(context, session, name): query = model_query(m.PluginData, context, session) return query.filter_by(name=name).first() def plugin_get(context, name): session = get_session() with session.begin(): data = _plugin_get(context, session, name) return data def plugin_create(context, values): session = get_session() with session.begin(): plugin = m.PluginData() values['tenant_id'] = context.tenant_id plugin.update(values) session.add(plugin) return plugin def plugin_get_all(context): query = model_query(m.PluginData, context) return query.all() def plugin_update(context, name, values): session = get_session() with session.begin(): plugin = _plugin_get(context, session, name) if not plugin: raise ex.NotFoundException(name, _("Plugin name '%s' not found!")) plugin.update(values) return plugin def plugin_remove(context, name): session = get_session() with session.begin(): plugin = _plugin_get(context, session, name) if not plugin: raise ex.NotFoundException(name, _("Plugin name '%s' not found!")) session.delete(plugin) sahara-8.0.0/sahara/db/sqlalchemy/__init__.py0000666000175100017510000000000013245514472021100 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/db/sqlalchemy/types.py0000666000175100017510000000672313245514472020527 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils import sqlalchemy as sa from sqlalchemy.dialects import mysql from sqlalchemy.ext import mutable class JsonEncoded(sa.TypeDecorator): """Represents an immutable structure as a json-encoded string.""" impl = sa.Text def process_bind_param(self, value, dialect): if value is not None: value = jsonutils.dumps(value) return value def process_result_value(self, value, dialect): if value is not None: value = jsonutils.loads(value) return value class MutableDict(mutable.Mutable, dict): @classmethod def coerce(cls, key, value): """Convert plain dictionaries to MutableDict.""" if not isinstance(value, MutableDict): if isinstance(value, dict): return MutableDict(value) # this call will raise ValueError return mutable.Mutable.coerce(key, value) else: return value def update(self, e=None, **f): """Detect dictionary update events and emit change events.""" dict.update(self, e, **f) self.changed() def __setitem__(self, key, value): """Detect dictionary set events and emit change events.""" dict.__setitem__(self, key, value) self.changed() def __delitem__(self, key): """Detect dictionary del events and emit change events.""" dict.__delitem__(self, key) self.changed() class MutableList(mutable.Mutable, list): @classmethod def coerce(cls, key, value): """Convert plain lists to MutableList.""" if not isinstance(value, MutableList): if isinstance(value, list): return MutableList(value) # this call will raise ValueError return mutable.Mutable.coerce(key, value) else: return value def append(self, value): """Detect list add events and emit change events.""" list.append(self, value) self.changed() def remove(self, value): """Removes an item by value and emit change events.""" list.remove(self, value) self.changed() def __setitem__(self, key, value): """Detect list set events and emit change events.""" list.__setitem__(self, key, value) self.changed() def __delitem__(self, i): """Detect list del events and emit change events.""" list.__delitem__(self, i) self.changed() def JsonDictType(): """Returns an SQLAlchemy Column Type suitable to store a Json dict.""" return MutableDict.as_mutable(JsonEncoded) def JsonListType(): """Returns an SQLAlchemy Column Type suitable to store a Json array.""" return MutableList.as_mutable(JsonEncoded) def LargeBinary(): return sa.LargeBinary().with_variant(mysql.LONGBLOB(), 'mysql') def LongText(): return sa.Text().with_variant(mysql.LONGTEXT(), 'mysql') sahara-8.0.0/sahara/db/api.py0000666000175100017510000004726513245514472016000 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Defines interface for DB access. Functions in this module are imported into the sahara.db namespace. Call these functions from sahara.db namespace, not the sahara.db.api namespace. All functions in this module return objects that implement a dictionary-like interface. **Related Flags** :db_backend: string to lookup in the list of LazyPluggable backends. `sqlalchemy` is the only supported backend right now. :sql_connection: string specifying the sqlalchemy connection to use, like: `mysql://user:password@localhost/sahara`. """ from oslo_config import cfg from oslo_db import api as db_api from oslo_db import options from sahara.utils import types CONF = cfg.CONF options.set_defaults(CONF) _BACKEND_MAPPING = { 'sqlalchemy': 'sahara.db.sqlalchemy.api', } IMPL = db_api.DBAPI.from_config(CONF, backend_mapping=_BACKEND_MAPPING) def setup_db(): """Set up database, create tables, etc. Return True on success, False otherwise """ return IMPL.setup_db() def drop_db(): """Drop database. Return True on success, False otherwise """ return IMPL.drop_db() # Helpers for building constraints / equality checks def constraint(**conditions): """Return a constraint object suitable for use with some updates.""" return IMPL.constraint(**conditions) def equal_any(*values): """Return an equality condition object suitable for use in a constraint. Equal_any conditions require that a model object's attribute equal any one of the given values. """ return IMPL.equal_any(*values) def not_equal(*values): """Return an inequality condition object suitable for use in a constraint. Not_equal conditions require that a model object's attribute differs from all of the given values. """ return IMPL.not_equal(*values) def to_dict(func): def decorator(*args, **kwargs): res = func(*args, **kwargs) if isinstance(res, types.Page): return types.Page([item.to_dict() for item in res], res.prev, res.next) if isinstance(res, list): return [item.to_dict() for item in res] if res: return res.to_dict() else: return None return decorator # Cluster ops def cluster_get(context, cluster, show_progress=False): """Return the cluster or None if it does not exist.""" if show_progress: cluster = IMPL.cluster_provision_progress_update(context, cluster) else: cluster = IMPL.cluster_get(context, cluster) if cluster: return cluster.to_dict(show_progress) return None @to_dict def cluster_get_all(context, regex_search=False, **kwargs): """Get all clusters filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return IMPL.cluster_get_all(context, regex_search, **kwargs) @to_dict def cluster_create(context, values): """Create a cluster from the values dictionary.""" return IMPL.cluster_create(context, values) @to_dict def cluster_update(context, cluster, values): """Set the given properties on cluster and update it.""" return IMPL.cluster_update(context, cluster, values) def cluster_destroy(context, cluster): """Destroy the cluster or raise if it does not exist.""" IMPL.cluster_destroy(context, cluster) # Node Group ops def node_group_add(context, cluster, values): """Create a Node Group from the values dictionary.""" return IMPL.node_group_add(context, cluster, values) def node_group_update(context, node_group, values): """Set the given properties on node_group and update it.""" IMPL.node_group_update(context, node_group, values) def node_group_remove(context, node_group): """Destroy the node_group or raise if it does not exist.""" IMPL.node_group_remove(context, node_group) # Instance ops def instance_add(context, node_group, values): """Create an Instance from the values dictionary.""" return IMPL.instance_add(context, node_group, values) def instance_update(context, instance, values): """Set the given properties on Instance and update it.""" IMPL.instance_update(context, instance, values) def instance_remove(context, instance): """Destroy the Instance or raise if it does not exist.""" IMPL.instance_remove(context, instance) # Volumes ops def append_volume(context, instance, volume_id): """Append volume_id to instance.""" IMPL.append_volume(context, instance, volume_id) def remove_volume(context, instance, volume_id): """Remove volume_id in instance.""" IMPL.remove_volume(context, instance, volume_id) # Cluster Template ops @to_dict def cluster_template_get(context, cluster_template): """Return the cluster_template or None if it does not exist.""" return IMPL.cluster_template_get(context, cluster_template) @to_dict def cluster_template_get_all(context, regex_search=False, **kwargs): """Get all cluster templates filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return IMPL.cluster_template_get_all(context, regex_search, **kwargs) @to_dict def cluster_template_create(context, values): """Create a cluster_template from the values dictionary.""" return IMPL.cluster_template_create(context, values) def cluster_template_destroy(context, cluster_template, ignore_prot_on_def=False): """Destroy the cluster_template or raise if it does not exist.""" IMPL.cluster_template_destroy(context, cluster_template, ignore_prot_on_def) @to_dict def cluster_template_update(context, values, ignore_prot_on_def=False): """Update a cluster_template from the values dictionary.""" return IMPL.cluster_template_update(context, values, ignore_prot_on_def) # Node Group Template ops @to_dict def node_group_template_get(context, node_group_template): """Return the Node Group Template or None if it does not exist.""" return IMPL.node_group_template_get(context, node_group_template) @to_dict def node_group_template_get_all(context, regex_search=False, **kwargs): """Get all Node Group Templates filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return IMPL.node_group_template_get_all(context, regex_search, **kwargs) @to_dict def node_group_template_create(context, values): """Create a Node Group Template from the values dictionary.""" return IMPL.node_group_template_create(context, values) def node_group_template_destroy(context, node_group_template, ignore_prot_on_def=False): """Destroy the Node Group Template or raise if it does not exist.""" IMPL.node_group_template_destroy(context, node_group_template, ignore_prot_on_def) @to_dict def node_group_template_update(context, node_group_template, ignore_prot_on_def=False): """Update a Node Group Template from the values in a dictionary.""" return IMPL.node_group_template_update(context, node_group_template, ignore_prot_on_def) # Data Source ops @to_dict def data_source_get(context, data_source): """Return the Data Source or None if it does not exist.""" return IMPL.data_source_get(context, data_source) @to_dict def data_source_get_all(context, regex_search=False, **kwargs): """Get all Data Sources filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return IMPL.data_source_get_all(context, regex_search, **kwargs) def data_source_count(context, **kwargs): """Count Data Sources filtered by **kwargs. Uses sqlalchemy "in_" clause for any tuple values Uses sqlalchemy "like" clause for any string values containing % """ return IMPL.data_source_count(context, **kwargs) @to_dict def data_source_create(context, values): """Create a Data Source from the values dictionary.""" return IMPL.data_source_create(context, values) def data_source_destroy(context, data_source): """Destroy the Data Source or raise if it does not exist.""" IMPL.data_source_destroy(context, data_source) @to_dict def data_source_update(context, data_source): """Create a Data Source from the values dictionary.""" return IMPL.data_source_update(context, data_source) # JobExecutions ops @to_dict def job_execution_get(context, job_execution): """Return the JobExecution or None if it does not exist.""" return IMPL.job_execution_get(context, job_execution) @to_dict def job_execution_get_all(context, regex_search=False, **kwargs): """Get all JobExecutions filtered by **kwargs. kwargs key values may be the names of fields in a JobExecution plus the following special values with the indicated meaning: 'cluster.name' -- name of the Cluster referenced by the JobExecution 'job.name' -- name of the Job referenced by the JobExecution 'status' -- JobExecution['info']['status'] :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return IMPL.job_execution_get_all(context, regex_search, **kwargs) def job_execution_count(context, **kwargs): """Count number of JobExecutions filtered by **kwargs. e.g. job_execution_count(cluster_id=12, input_id=123) """ return IMPL.job_execution_count(context, **kwargs) @to_dict def job_execution_create(context, values): """Create a JobExecution from the values dictionary.""" return IMPL.job_execution_create(context, values) @to_dict def job_execution_update(context, job_execution, values): """Create a JobExecution from the values dictionary.""" return IMPL.job_execution_update(context, job_execution, values) def job_execution_destroy(context, job_execution): """Destroy the JobExecution or raise if it does not exist.""" IMPL.job_execution_destroy(context, job_execution) # Job ops @to_dict def job_get(context, job): """Return the Job or None if it does not exist.""" return IMPL.job_get(context, job) @to_dict def job_get_all(context, regex_search=False, **kwargs): """Get all Jobs filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return IMPL.job_get_all(context, regex_search, **kwargs) @to_dict def job_create(context, values): """Create a Job from the values dictionary.""" return IMPL.job_create(context, values) @to_dict def job_update(context, job, values): """Update a Job from the values dictionary.""" return IMPL.job_update(context, job, values) def job_destroy(context, job): """Destroy the Job or raise if it does not exist.""" IMPL.job_destroy(context, job) @to_dict def job_binary_get_all(context, regex_search=False, **kwargs): """Get all JobBinarys filtered by **kwargs. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return IMPL.job_binary_get_all(context, regex_search, **kwargs) @to_dict def job_binary_get(context, job_binary): """Return the JobBinary or None if it does not exist.""" return IMPL.job_binary_get(context, job_binary) @to_dict def job_binary_create(context, values): """Create a JobBinary from the values dictionary.""" return IMPL.job_binary_create(context, values) def job_binary_destroy(context, job_binary): """Destroy the JobBinary or raise if it does not exist.""" IMPL.job_binary_destroy(context, job_binary) @to_dict def job_binary_update(context, values): """Update the JobBinary with the provided values""" return IMPL.job_binary_update(context, values) @to_dict def job_binary_internal_get_all(context, regex_search=False, **kwargs): """Get all JobBinaryInternals filtered by **kwargs. The JobBinaryInternals returned do not contain a data field. :param context: The context, and associated authentication, to use with this operation :param regex_search: If True, enable regex matching for filter values. See the user guide for more information on how regex matching is handled. If False, no regex matching is done. :param kwargs: Specifies values for named fields by which to constrain the search """ return IMPL.job_binary_internal_get_all(context, regex_search, **kwargs) @to_dict def job_binary_internal_get(context, job_binary_internal): """Return the JobBinaryInternal or None if it does not exist.""" return IMPL.job_binary_internal_get(context, job_binary_internal) @to_dict def job_binary_internal_create(context, values): """Create a JobBinaryInternal from the values dictionary.""" return IMPL.job_binary_internal_create(context, values) def job_binary_internal_destroy(context, job_binary_internal): """Destroy the JobBinaryInternal or raise if it does not exist.""" IMPL.job_binary_internal_destroy(context, job_binary_internal) def job_binary_internal_get_raw_data(context, job_binary_internal_id): """Return the binary data field from the specified JobBinaryInternal.""" return IMPL.job_binary_internal_get_raw_data(context, job_binary_internal_id) @to_dict def job_binary_internal_update(context, job_binary_internal, values): """Update the JobBinaryInternal with the provided values""" return IMPL.job_binary_internal_update( context, job_binary_internal, values) # Events ops def cluster_provision_step_add(context, cluster_id, values): """Create a cluster assigned ProvisionStep from the values dictionary.""" return IMPL.cluster_provision_step_add(context, cluster_id, values) def cluster_provision_step_update(context, step_id): """Updates provision step.""" return IMPL.cluster_provision_step_update(context, step_id) def cluster_provision_progress_update(context, cluster_id): """Return cluster with provision progress updated field.""" return IMPL.cluster_provision_progress_update(context, cluster_id) def cluster_event_add(context, provision_step, values): """Assign new event to the specified provision step.""" return IMPL.cluster_event_add(context, provision_step, values) # Health verifications / checks ops @to_dict def cluster_verification_add(context, cluster_id, values): """Return created verification for the specified cluster.""" return IMPL.cluster_verification_add(context, cluster_id, values) @to_dict def cluster_verification_get(context, verification_id): """Return verification with the specified verification_id.""" return IMPL.cluster_verification_get(context, verification_id) @to_dict def cluster_verification_update(context, verification_id, values): """Return updated verification with the specified verification_id.""" return IMPL.cluster_verification_update(context, verification_id, values) def cluster_verification_delete(context, verification_id): """"Delete verification with the specified id.""" return IMPL.cluster_verification_delete(context, verification_id) @to_dict def cluster_health_check_add(context, verification_id, values): """Return created health check in the specified verification.""" return IMPL.cluster_health_check_add(context, verification_id, values) @to_dict def cluster_health_check_get(context, health_check_id): """Return health check with the specified health_check_id.""" return IMPL.cluster_health_check_get(context, health_check_id) @to_dict def cluster_health_check_update(context, health_check_id, values): """Return updated health check with the specified health_check_id.""" return IMPL.cluster_health_check_update(context, health_check_id, values) @to_dict def plugin_create(context, values): """Return created DB entry for plugin.""" return IMPL.plugin_create(context, values) @to_dict def plugin_get(context, name): """Return DB entry for plugin.""" return IMPL.plugin_get(context, name) @to_dict def plugin_get_all(context): """Return DB entries of all plugins.""" return IMPL.plugin_get_all(context) @to_dict def plugin_update(context, name, values): """Return updated DB entry for plugin.""" return IMPL.plugin_update(context, name, values) def plugin_remove(context, name): """Remove DB entry for plugin.""" return IMPL.plugin_remove(context, name) sahara-8.0.0/sahara/db/__init__.py0000666000175100017510000000121613245514472016750 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """ DB abstraction for Sahara """ from sahara.db.api import * # noqa sahara-8.0.0/sahara/db/migration/0000775000175100017510000000000013245515026016622 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/db/migration/alembic_migrations/0000775000175100017510000000000013245515026022452 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/0000775000175100017510000000000013245515026024322 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/024_manila_shares.py0000666000175100017510000000254713245514472030105 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """manila_shares Revision ID: 024 Revises: 023 Create Date: 2015-07-20 14:51:20.275823 """ # revision identifiers, used by Alembic. revision = '024' down_revision = '023' from alembic import op import sqlalchemy as sa from sahara.db.sqlalchemy import types as st MYSQL_ENGINE = 'InnoDB' MYSQL_CHARSET = 'utf8' def upgrade(): op.add_column('node_group_templates', sa.Column('shares', st.JsonEncoded())) op.add_column('node_groups', sa.Column('shares', st.JsonEncoded())) op.add_column('templates_relations', sa.Column('shares', st.JsonEncoded())) op.add_column('clusters', sa.Column('shares', st.JsonEncoded())) op.add_column('cluster_templates', sa.Column('shares', st.JsonEncoded())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/015_add_events_objects.py0000666000175100017510000000746613245514472031131 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add_events_objects Revision ID: 015 Revises: 014 Create Date: 2014-11-07 15:20:21.806128 """ # revision identifiers, used by Alembic. revision = '015' down_revision = '014' from alembic import op import sqlalchemy as sa MYSQL_ENGINE = 'InnoDB' MYSQL_CHARSET = 'utf8' def upgrade(): op.create_table('cluster_provision_steps', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('cluster_id', sa.String(length=36), nullable=True), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('step_name', sa.String(length=80), nullable=True), sa.Column('step_type', sa.String(length=36), nullable=True), sa.Column('completed', sa.Integer(), nullable=True), sa.Column('total', sa.Integer(), nullable=True), sa.Column('successful', sa.Boolean(), nullable=True), sa.Column('started_at', sa.DateTime(), nullable=True), sa.Column('completed_at', sa.DateTime(), nullable=True), sa.ForeignKeyConstraint(['cluster_id'], ['clusters.id'], ), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('id', 'cluster_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('cluster_events', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('node_group_id', sa.String(length=36), nullable=True), sa.Column('instance_id', sa.String(length=36), nullable=True), sa.Column('instance_name', sa.String(length=80), nullable=True), sa.Column('event_info', sa.Text(), nullable=True), sa.Column('successful', sa.Boolean(), nullable=False), sa.Column('step_id', sa.String(length=36), nullable=True), sa.ForeignKeyConstraint( ['step_id'], ['cluster_provision_steps.id'], ), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('id', 'step_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/030-health-check.py0000666000175100017510000000431513245514472027525 0ustar zuulzuul00000000000000# Copyright 2016 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """health-check Revision ID: 029 Revises: 028 Create Date: 2016-01-26 16:11:46.008367 """ # revision identifiers, used by Alembic. revision = '030' down_revision = '029' from alembic import op import sqlalchemy as sa MYSQL_ENGINE = 'InnoDB' MYSQL_CHARSET = 'utf8' def upgrade(): op.create_table( 'cluster_verifications', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('cluster_id', sa.String(length=36), nullable=True), sa.Column('status', sa.String(length=15), nullable=True), sa.ForeignKeyConstraint(['cluster_id'], ['clusters.id'], ), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('id', 'cluster_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table( 'cluster_health_checks', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('verification_id', sa.String(length=36), nullable=True), sa.Column('status', sa.String(length=15), nullable=True), sa.Column('description', sa.Text(), nullable=True), sa.Column('name', sa.String(length=80), nullable=True), sa.ForeignKeyConstraint( ['verification_id'], ['cluster_verifications.id'], ), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('id', 'verification_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/001_icehouse.py0000666000175100017510000004426413245514472027100 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Icehouse release Revision ID: 001 Revises: None Create Date: 2014-04-01 20:46:25.783444 """ # revision identifiers, used by Alembic. revision = '001' down_revision = None from alembic import op import sqlalchemy as sa from sahara.db.sqlalchemy import types as st MYSQL_ENGINE = 'InnoDB' MYSQL_CHARSET = 'utf8' def upgrade(): op.create_table('jobs', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('description', sa.Text(), nullable=True), sa.Column('type', sa.String(length=80), nullable=False), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'tenant_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('node_group_templates', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('description', sa.Text(), nullable=True), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('flavor_id', sa.String(length=36), nullable=False), sa.Column('image_id', sa.String(length=36), nullable=True), sa.Column('plugin_name', sa.String(length=80), nullable=False), sa.Column('hadoop_version', sa.String(length=80), nullable=False), sa.Column('node_processes', st.JsonEncoded(), nullable=True), sa.Column('node_configs', st.JsonEncoded(), nullable=True), sa.Column('volumes_per_node', sa.Integer(), nullable=False), sa.Column('volumes_size', sa.Integer(), nullable=True), sa.Column('volume_mount_prefix', sa.String(length=80), nullable=True), sa.Column('floating_ip_pool', sa.String(length=36), nullable=True), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'tenant_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('cluster_templates', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('description', sa.Text(), nullable=True), sa.Column('cluster_configs', st.JsonEncoded(), nullable=True), sa.Column('default_image_id', sa.String(length=36), nullable=True), sa.Column('anti_affinity', st.JsonEncoded(), nullable=True), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('neutron_management_network', sa.String(length=36), nullable=True), sa.Column('plugin_name', sa.String(length=80), nullable=False), sa.Column('hadoop_version', sa.String(length=80), nullable=False), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'tenant_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('job_binary_internal', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('data', st.LargeBinary(), nullable=True), sa.Column('datasize', sa.BIGINT(), nullable=True), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'tenant_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('job_binaries', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('description', sa.Text(), nullable=True), sa.Column('url', sa.String(length=256), nullable=False), sa.Column('extra', st.JsonEncoded(), nullable=True), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'tenant_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('data_sources', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('description', sa.Text(), nullable=True), sa.Column('type', sa.String(length=80), nullable=False), sa.Column('url', sa.String(length=256), nullable=False), sa.Column('credentials', st.JsonEncoded(), nullable=True), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'tenant_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('libs_association', sa.Column('Job_id', sa.String(length=36), nullable=True), sa.Column('JobBinary_id', sa.String(length=36), nullable=True), sa.ForeignKeyConstraint(['JobBinary_id'], ['job_binaries.id'], ), sa.ForeignKeyConstraint(['Job_id'], ['jobs.id'], ), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('clusters', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('description', sa.Text(), nullable=True), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('trust_id', sa.String(length=36), nullable=True), sa.Column('is_transient', sa.Boolean(), nullable=True), sa.Column('plugin_name', sa.String(length=80), nullable=False), sa.Column('hadoop_version', sa.String(length=80), nullable=False), sa.Column('cluster_configs', st.JsonEncoded(), nullable=True), sa.Column('default_image_id', sa.String(length=36), nullable=True), sa.Column('neutron_management_network', sa.String(length=36), nullable=True), sa.Column('anti_affinity', st.JsonEncoded(), nullable=True), sa.Column('management_private_key', sa.Text(), nullable=False), sa.Column('management_public_key', sa.Text(), nullable=False), sa.Column('user_keypair_id', sa.String(length=80), nullable=True), sa.Column('status', sa.String(length=80), nullable=True), sa.Column('status_description', sa.String(length=200), nullable=True), sa.Column('info', st.JsonEncoded(), nullable=True), sa.Column('extra', st.JsonEncoded(), nullable=True), sa.Column('cluster_template_id', sa.String(length=36), nullable=True), sa.ForeignKeyConstraint(['cluster_template_id'], ['cluster_templates.id'], ), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'tenant_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('templates_relations', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('flavor_id', sa.String(length=36), nullable=False), sa.Column('image_id', sa.String(length=36), nullable=True), sa.Column('node_processes', st.JsonEncoded(), nullable=True), sa.Column('node_configs', st.JsonEncoded(), nullable=True), sa.Column('volumes_per_node', sa.Integer(), nullable=True), sa.Column('volumes_size', sa.Integer(), nullable=True), sa.Column('volume_mount_prefix', sa.String(length=80), nullable=True), sa.Column('count', sa.Integer(), nullable=False), sa.Column('cluster_template_id', sa.String(length=36), nullable=True), sa.Column('node_group_template_id', sa.String(length=36), nullable=True), sa.Column('floating_ip_pool', sa.String(length=36), nullable=True), sa.ForeignKeyConstraint(['cluster_template_id'], ['cluster_templates.id'], ), sa.ForeignKeyConstraint(['node_group_template_id'], ['node_group_templates.id'], ), sa.PrimaryKeyConstraint('id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('mains_association', sa.Column('Job_id', sa.String(length=36), nullable=True), sa.Column('JobBinary_id', sa.String(length=36), nullable=True), sa.ForeignKeyConstraint(['JobBinary_id'], ['job_binaries.id'], ), sa.ForeignKeyConstraint(['Job_id'], ['jobs.id'], ), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('job_executions', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('job_id', sa.String(length=36), nullable=True), sa.Column('input_id', sa.String(length=36), nullable=True), sa.Column('output_id', sa.String(length=36), nullable=True), sa.Column('start_time', sa.DateTime(), nullable=True), sa.Column('end_time', sa.DateTime(), nullable=True), sa.Column('cluster_id', sa.String(length=36), nullable=True), sa.Column('info', st.JsonEncoded(), nullable=True), sa.Column('progress', sa.Float(), nullable=True), sa.Column('oozie_job_id', sa.String(length=100), nullable=True), sa.Column('return_code', sa.String(length=80), nullable=True), sa.Column('job_configs', st.JsonEncoded(), nullable=True), sa.Column('extra', st.JsonEncoded(), nullable=True), sa.ForeignKeyConstraint(['cluster_id'], ['clusters.id'], ), sa.ForeignKeyConstraint(['input_id'], ['data_sources.id'], ), sa.ForeignKeyConstraint(['job_id'], ['jobs.id'], ), sa.ForeignKeyConstraint(['output_id'], ['data_sources.id'], ), sa.PrimaryKeyConstraint('id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('node_groups', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('name', sa.String(length=80), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('flavor_id', sa.String(length=36), nullable=False), sa.Column('image_id', sa.String(length=36), nullable=True), sa.Column('image_username', sa.String(length=36), nullable=True), sa.Column('node_processes', st.JsonEncoded(), nullable=True), sa.Column('node_configs', st.JsonEncoded(), nullable=True), sa.Column('volumes_per_node', sa.Integer(), nullable=True), sa.Column('volumes_size', sa.Integer(), nullable=True), sa.Column('volume_mount_prefix', sa.String(length=80), nullable=True), sa.Column('count', sa.Integer(), nullable=False), sa.Column('cluster_id', sa.String(length=36), nullable=True), sa.Column('node_group_template_id', sa.String(length=36), nullable=True), sa.Column('floating_ip_pool', sa.String(length=36), nullable=True), sa.ForeignKeyConstraint(['cluster_id'], ['clusters.id'], ), sa.ForeignKeyConstraint(['node_group_template_id'], ['node_group_templates.id'], ), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'cluster_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) op.create_table('instances', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('node_group_id', sa.String(length=36), nullable=True), sa.Column('instance_id', sa.String(length=36), nullable=True), sa.Column('instance_name', sa.String(length=80), nullable=False), sa.Column('internal_ip', sa.String(length=15), nullable=True), sa.Column('management_ip', sa.String(length=15), nullable=True), sa.Column('volumes', st.JsonEncoded(), nullable=True), sa.ForeignKeyConstraint(['node_group_id'], ['node_groups.id'], ), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('instance_id', 'node_group_id'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/017_drop_progress.py0000666000175100017510000000154313245514472030164 0ustar zuulzuul00000000000000# Copyright 2015 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """drop progress in JobExecution Revision ID: 017 Revises: 016 Create Date: 2015-02-25 09:23:04.390388 """ # revision identifiers, used by Alembic. revision = '017' down_revision = '016' from alembic import op def upgrade(): op.drop_column('job_executions', 'progress') sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/031_added_plugins_table.py0000666000175100017510000000270713245514472031244 0ustar zuulzuul00000000000000# Copyright 2016 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """added_plugins_table Revision ID: 031 Revises: 030 Create Date: 2016-06-21 13:32:40.151321 """ from alembic import op import sqlalchemy as sa from sahara.db.sqlalchemy import types as st # revision identifiers, used by Alembic. revision = '031' down_revision = '030' def upgrade(): op.create_table( 'plugin_data', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=False), sa.Column('name', sa.String(length=15), nullable=False), sa.Column('plugin_labels', st.JsonEncoded(), nullable=True), sa.Column('version_labels', st.JsonEncoded(), nullable=True), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('name', 'tenant_id') ) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/025_increase_ip_column_size.py0000666000175100017510000000207413245514472032163 0ustar zuulzuul00000000000000# Copyright 2015 Telles Nobrega # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Increase internal_ip and management_ip column size to work with IPv6 Revision ID: 025 Revises: 024 Create Date: 2015-07-17 09:58:22.128263 """ # revision identifiers, used by Alembic. revision = '025' down_revision = '024' from alembic import op import sqlalchemy as sa def upgrade(): op.alter_column('instances', 'internal_ip', type_=sa.String(45), nullable=True) op.alter_column('instances', 'management_ip', type_=sa.String(45), nullable=True) ././@LongLink0000000000000000000000000000015500000000000011216 Lustar 00000000000000sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/033_add_anti_affinity_ratio_field_to_cluster.pysahara-8.0.0/sahara/db/migration/alembic_migrations/versions/033_add_anti_affinity_ratio_field_to_cl0000666000175100017510000000167113245514472034022 0ustar zuulzuul00000000000000# Copyright 2016 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """033_add anti_affinity_ratio field to cluster Revision ID: 033 Revises: 032 Create Date: 2016-01-05 09:40:25.941365 """ # revision identifiers, used by Alembic. revision = '033' down_revision = '032' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('clusters', sa.Column('anti_affinity_ratio', sa.Integer())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/002_placeholder.py0000666000175100017510000000142113245514472027543 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """placeholder Revision ID: 002 Revises: 001 Create Date: 2014-04-01 21:04:47.941098 """ # revision identifiers, used by Alembic. revision = '002' down_revision = '001' def upgrade(): pass sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/014_add_volume_type.py0000666000175100017510000000237213245514472030452 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add_volume_type Revision ID: 014 Revises: 013 Create Date: 2014-10-09 12:47:17.871520 """ # revision identifiers, used by Alembic. revision = '014' down_revision = '013' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('node_group_templates', sa.Column('volume_type', sa.String(length=255), nullable=True)) op.add_column('node_groups', sa.Column('volume_type', sa.String(length=255), nullable=True)) op.add_column('templates_relations', sa.Column('volume_type', sa.String(length=255), nullable=True)) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/006_placeholder.py0000666000175100017510000000142113245514472027547 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """placeholder Revision ID: 006 Revises: 005 Create Date: 2014-04-01 21:04:52.194332 """ # revision identifiers, used by Alembic. revision = '006' down_revision = '005' def upgrade(): pass sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/023_add_use_autoconfig.py0000666000175100017510000000245113245514472031112 0ustar zuulzuul00000000000000# Copyright 2015 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add_use_autoconfig Revision ID: 023 Revises: 022 Create Date: 2015-04-24 14:51:39.582085 """ # revision identifiers, used by Alembic. revision = '023' down_revision = '022' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('clusters', sa.Column('use_autoconfig', sa.Boolean())) op.add_column('cluster_templates', sa.Column('use_autoconfig', sa.Boolean())) op.add_column('node_group_templates', sa.Column('use_autoconfig', sa.Boolean())) op.add_column('node_groups', sa.Column('use_autoconfig', sa.Boolean())) op.add_column('templates_relations', sa.Column('use_autoconfig', sa.Boolean())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/009_rollback_info.py0000666000175100017510000000172513245514472030103 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add rollback info to cluster Revision ID: 009 Revises: 008 Create Date: 2014-06-25 22:36:00.783444 """ # revision identifiers, used by Alembic. revision = '009' down_revision = '008' from alembic import op import sqlalchemy as sa from sahara.db.sqlalchemy import types as st def upgrade(): op.add_column('clusters', sa.Column('rollback_info', st.JsonEncoded())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/021_datasource_placeholders.py0000666000175100017510000000200013245514472032133 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Add data_source_urls to job_executions to support placeholders Revision ID: 021 Revises: 020 Create Date: 2015-02-24 12:47:17.871520 """ # revision identifiers, used by Alembic. revision = '021' down_revision = '020' from alembic import op import sqlalchemy as sa from sahara.db.sqlalchemy import types as st def upgrade(): op.add_column('job_executions', sa.Column('data_source_urls', st.JsonEncoded())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/016_is_proxy_gateway.py0000666000175100017510000000216213245514472030666 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Add is_proxy_gateway Revision ID: 016 Revises: 015 Create Date: 2014-11-10 12:47:17.871520 """ # revision identifiers, used by Alembic. revision = '016' down_revision = '015' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('node_group_templates', sa.Column('is_proxy_gateway', sa.Boolean())) op.add_column('node_groups', sa.Column('is_proxy_gateway', sa.Boolean())) op.add_column('templates_relations', sa.Column('is_proxy_gateway', sa.Boolean())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/010_auto_security_groups.py0000666000175100017510000000243613245514472031565 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add auto_security_groups flag to node group Revision ID: 010 Revises: 009 Create Date: 2014-07-21 14:31:49.685689 """ # revision identifiers, used by Alembic. revision = '010' down_revision = '009' from alembic import op import sqlalchemy as sa from sahara.db.sqlalchemy import types as st def upgrade(): op.add_column('node_group_templates', sa.Column('auto_security_group', sa.Boolean())) op.add_column('node_groups', sa.Column('auto_security_group', sa.Boolean())) op.add_column('templates_relations', sa.Column('auto_security_group', sa.Boolean())) op.add_column('node_groups', sa.Column('open_ports', st.JsonEncoded())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/029_set_is_protected_on_is_default.py0000666000175100017510000000254613245514472033535 0ustar zuulzuul00000000000000# Copyright 2015 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """set is_protected on is_default Revision ID: 029 Revises: 028 Create Date: 2015-11-4 12:41:52.571258 """ # revision identifiers, used by Alembic. revision = '029' down_revision = '028' from alembic import op import sqlalchemy as sa from sqlalchemy.sql import table, column def upgrade(): ng = table('node_group_templates', column('is_protected', sa.Boolean), column('is_default', sa.Boolean)) op.execute( ng.update().where( ng.c.is_default).values({'is_protected': True}) ) clt = table('cluster_templates', column('is_protected', sa.Boolean), column('is_default', sa.Boolean)) op.execute( clt.update().where( clt.c.is_default).values({'is_protected': True}) ) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/026_add_is_public_is_protected.py0000666000175100017510000000443713245514472032626 0ustar zuulzuul00000000000000# Copyright 2015 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add is_public and is_protected flags Revision ID: 026 Revises: 025 Create Date: 2015-06-24 12:41:52.571258 """ # revision identifiers, used by Alembic. revision = '026' down_revision = '025' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('clusters', sa.Column('is_public', sa.Boolean()),) op.add_column('cluster_templates', sa.Column('is_public', sa.Boolean())) op.add_column('node_group_templates', sa.Column('is_public', sa.Boolean())) op.add_column('data_sources', sa.Column('is_public', sa.Boolean())) op.add_column('job_executions', sa.Column('is_public', sa.Boolean())) op.add_column('jobs', sa.Column('is_public', sa.Boolean())) op.add_column('job_binary_internal', sa.Column('is_public', sa.Boolean())) op.add_column('job_binaries', sa.Column('is_public', sa.Boolean())) op.add_column('clusters', sa.Column('is_protected', sa.Boolean())) op.add_column('cluster_templates', sa.Column('is_protected', sa.Boolean())) op.add_column('node_group_templates', sa.Column('is_protected', sa.Boolean())) op.add_column('data_sources', sa.Column('is_protected', sa.Boolean())) op.add_column('job_executions', sa.Column('is_protected', sa.Boolean())) op.add_column('jobs', sa.Column('is_protected', sa.Boolean())) op.add_column('job_binary_internal', sa.Column('is_protected', sa.Boolean())) op.add_column('job_binaries', sa.Column('is_protected', sa.Boolean())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/019_is_default_for_templates.py0000666000175100017510000000210713245514472032336 0ustar zuulzuul00000000000000# Copyright 2015 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Add is_default field for cluster and node_group templates Revision ID: 019 Revises: 018 Create Date: 2015-03-02 14:32:04.415021 """ # revision identifiers, used by Alembic. revision = '019' down_revision = '018' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('cluster_templates', sa.Column('is_default', sa.Boolean(), nullable=True)) op.add_column('node_group_templates', sa.Column('is_default', sa.Boolean(), nullable=True)) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/012_availability_zone.py0000666000175100017510000000232213245514472030770 0ustar zuulzuul00000000000000# -*- coding: utf-8 -*- # Copyright 2014, Adrien Vergé # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add availability_zone field to node groups Revision ID: 012 Revises: 011 Create Date: 2014-09-08 15:37:00.000000 """ # revision identifiers, used by Alembic. revision = '012' down_revision = '011' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('node_group_templates', sa.Column('availability_zone', sa.String(length=255))) op.add_column('node_groups', sa.Column('availability_zone', sa.String(length=255))) op.add_column('templates_relations', sa.Column('availability_zone', sa.String(length=255))) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/008_security_groups.py0000666000175100017510000000227513245514472030545 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add security_groups field to node groups Revision ID: 008 Revises: 007 Create Date: 2014-07-15 14:31:49.685689 """ # revision identifiers, used by Alembic. revision = '008' down_revision = '007' from alembic import op import sqlalchemy as sa from sahara.db.sqlalchemy import types as st def upgrade(): op.add_column('node_group_templates', sa.Column('security_groups', st.JsonEncoded())) op.add_column('node_groups', sa.Column('security_groups', st.JsonEncoded())) op.add_column('templates_relations', sa.Column('security_groups', st.JsonEncoded())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/013_volumes_availability_zone.py0000666000175100017510000000251013245514472032542 0ustar zuulzuul00000000000000# -*- coding: utf-8 -*- # Copyright 2014, Adrien Vergé # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add volumes_availability_zone field to node groups Revision ID: 013 Revises: 012 Create Date: 2014-09-08 15:37:00.000000 """ # revision identifiers, used by Alembic. revision = '013' down_revision = '012' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('node_group_templates', sa.Column('volumes_availability_zone', sa.String(length=255))) op.add_column('node_groups', sa.Column('volumes_availability_zone', sa.String(length=255))) op.add_column('templates_relations', sa.Column('volumes_availability_zone', sa.String(length=255))) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/022_add_job_interface.py0000666000175100017510000000500413245514472030666 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add_job_interface Revision ID: 022 Revises: 021 Create Date: 2015-01-27 15:53:22.128263 """ # revision identifiers, used by Alembic. revision = '022' down_revision = '021' from alembic import op import sqlalchemy as sa MYSQL_ENGINE = 'InnoDB' MYSQL_CHARSET = 'utf8' def upgrade(): op.create_table('job_interface_arguments', sa.Column('created_at', sa.DateTime(), nullable=True), sa.Column('updated_at', sa.DateTime(), nullable=True), sa.Column('id', sa.String(length=36), nullable=False), sa.Column('job_id', sa.String(length=36), nullable=False), sa.Column('tenant_id', sa.String(length=36), nullable=True), sa.Column('name', sa.String(80), nullable=False), sa.Column('description', sa.Text()), sa.Column('mapping_type', sa.String(80), nullable=False), sa.Column('location', sa.Text(), nullable=False), sa.Column('value_type', sa.String(80), nullable=False), sa.Column('required', sa.Boolean(), nullable=False), sa.Column('order', sa.SmallInteger(), nullable=False), sa.Column('default', sa.Text()), sa.ForeignKeyConstraint(['job_id'], ['jobs.id']), sa.PrimaryKeyConstraint('id'), sa.UniqueConstraint('job_id', 'order'), sa.UniqueConstraint('job_id', 'name'), mysql_engine=MYSQL_ENGINE, mysql_charset=MYSQL_CHARSET) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/032_add_domain_name.py0000666000175100017510000000220013245514472030337 0ustar zuulzuul00000000000000# Copyright 2016 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """032_add_domain_name Revision ID: 032 Revises: 031 Create Date: 2016-07-21 13:33:33.674853 """ # revision identifiers, used by Alembic. revision = '032' down_revision = '031' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('cluster_templates', sa.Column( 'domain_name', sa.String(length=255), nullable=True)) op.add_column('clusters', sa.Column( 'domain_name', sa.String(length=255), nullable=True)) op.add_column('instances', sa.Column( 'dns_hostname', sa.String(length=255), nullable=True)) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/004_placeholder.py0000666000175100017510000000142113245514472027545 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """placeholder Revision ID: 004 Revises: 003 Create Date: 2014-04-01 21:04:57.627883 """ # revision identifiers, used by Alembic. revision = '004' down_revision = '003' def upgrade(): pass sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/007_increase_status_description_size.py0000666000175100017510000000203313245514472034117 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """convert clusters.status_description to LongText Revision ID: 007 Revises: 006 Create Date: 2014-06-20 22:36:00.783444 """ # revision identifiers, used by Alembic. revision = '007' down_revision = '006' from alembic import op from sahara.db.sqlalchemy import types as st def upgrade(): op.alter_column('clusters', 'status_description', type_=st.LongText(), existing_nullable=True, existing_server_default=None) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/018_volume_local_to_instance.py0000666000175100017510000000222713245514472032344 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add volume_local_to_instance flag Revision ID: 018 Revises: 017 Create Date: 2015-03-03 14:35:43.625429 """ # revision identifiers, used by Alembic. revision = '018' down_revision = '017' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('node_group_templates', sa.Column('volume_local_to_instance', sa.Boolean())) op.add_column('node_groups', sa.Column('volume_local_to_instance', sa.Boolean())) op.add_column('templates_relations', sa.Column('volume_local_to_instance', sa.Boolean())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/003_placeholder.py0000666000175100017510000000142113245514472027544 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """placeholder Revision ID: 003 Revises: 002 Create Date: 2014-04-01 21:05:00.270366 """ # revision identifiers, used by Alembic. revision = '003' down_revision = '002' def upgrade(): pass sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/005_placeholder.py0000666000175100017510000000142113245514472027546 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """placeholder Revision ID: 005 Revises: 004 Create Date: 2014-04-01 21:04:54.928605 """ # revision identifiers, used by Alembic. revision = '005' down_revision = '004' def upgrade(): pass sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/011_sahara_info.py0000666000175100017510000000173213245514472027540 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add Sahara settings info to cluster Revision ID: 011 Revises: 010 Create Date: 2014-08-26 22:36:00.783444 """ # revision identifiers, used by Alembic. revision = '011' down_revision = '010' from alembic import op import sqlalchemy as sa from sahara.db.sqlalchemy import types as st def upgrade(): op.add_column('clusters', sa.Column('sahara_info', st.JsonEncoded())) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/020_remove_redandunt_progress_ops.py0000666000175100017510000000174713245514472033442 0ustar zuulzuul00000000000000# Copyright 2015 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """remove redandunt progress ops Revision ID: 020 Revises: 019 Create Date: 2015-02-26 15:01:41.015076 """ # revision identifiers, used by Alembic. revision = '020' down_revision = '019' from alembic import op def upgrade(): op.drop_column('cluster_provision_steps', 'completed_at') op.drop_column('cluster_provision_steps', 'completed') op.drop_column('cluster_provision_steps', 'started_at') sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/027_rename_oozie_job_id.py0000666000175100017510000000173413245514472031261 0ustar zuulzuul00000000000000# Copyright 2015 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Rename oozie_job_id Revision ID: 027 Revises: 026 Create Date: 2015-07-27 14:31:02.413053 """ # revision identifiers, used by Alembic. revision = '027' down_revision = '026' from alembic import op import sqlalchemy as sa def upgrade(): op.alter_column('job_executions', 'oozie_job_id', new_column_name="engine_job_id", type_=sa.String(length=100)) sahara-8.0.0/sahara/db/migration/alembic_migrations/versions/028_storage_devices_number.py0000666000175100017510000000172613245514472032017 0ustar zuulzuul00000000000000# Copyright 2015 OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """add_storage_devices_number Revision ID: 028 Revises: 027 Create Date: 2015-07-20 16:56:23.562710 """ # revision identifiers, used by Alembic. revision = '028' down_revision = '027' from alembic import op import sqlalchemy as sa def upgrade(): op.add_column('instances', sa.Column('storage_devices_number', sa.Integer(), nullable=True)) sahara-8.0.0/sahara/db/migration/alembic_migrations/env.py0000666000175100017510000000522313245514472023624 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # Based on Neutron's migration/cli.py from __future__ import with_statement from logging import config as c from alembic import context from oslo_utils import importutils from sqlalchemy import create_engine from sqlalchemy import pool from sahara.db.sqlalchemy import model_base importutils.try_import('sahara.db.sqlalchemy.models') # this is the Alembic Config object, which provides # access to the values within the .ini file in use. config = context.config sahara_config = config.sahara_config # Interpret the config file for Python logging. # This line sets up loggers basically. c.fileConfig(config.config_file_name) # add your model's MetaData object here # for 'autogenerate' support # from myapp import mymodel # target_metadata = mymodel.Base.metadata target_metadata = model_base.SaharaBase.metadata # other values from the config, defined by the needs of env.py, # can be acquired: # my_important_option = config.get_main_option("my_important_option") # ... etc. def run_migrations_offline(): """Run migrations in 'offline' mode. This configures the context with just a URL and not an Engine, though an Engine is acceptable here as well. By skipping the Engine creation we don't even need a DBAPI to be available. Calls to context.execute() here emit the given string to the script output. """ context.configure(url=sahara_config.database.connection) with context.begin_transaction(): context.run_migrations() def run_migrations_online(): """Run migrations in 'online' mode. In this scenario we need to create an Engine and associate a connection with the context. """ engine = create_engine( sahara_config.database.connection, poolclass=pool.NullPool) connection = engine.connect() context.configure( connection=connection, target_metadata=target_metadata) try: with context.begin_transaction(): context.run_migrations() finally: connection.close() if context.is_offline_mode(): run_migrations_offline() else: run_migrations_online() sahara-8.0.0/sahara/db/migration/alembic_migrations/README.md0000666000175100017510000000465513245514472023751 0ustar zuulzuul00000000000000 The migrations in `alembic_migrations/versions` contain the changes needed to migrate between Sahara database revisions. A migration occurs by executing a script that details the changes needed to upgrade the database. The migration scripts are ordered so that multiple scripts can run sequentially. The scripts are executed by Sahara's migration wrapper which uses the Alembic library to manage the migration. Sahara supports migration from Icehouse or later. You can upgrade to the latest database version via: ``` $ sahara-db-manage --config-file /path/to/sahara.conf upgrade head ``` To check the current database version: ``` $ sahara-db-manage --config-file /path/to/sahara.conf current ``` To create a script to run the migration offline: ``` $ sahara-db-manage --config-file /path/to/sahara.conf upgrade head --sql ``` To run the offline migration between specific migration versions: ``` $ sahara-db-manage --config-file /path/to/sahara.conf upgrade : --sql ``` Upgrade the database incrementally: ``` $ sahara-db-manage --config-file /path/to/sahara.conf upgrade --delta <# of revs> ``` Create new revision: ``` $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" --autogenerate ``` Create a blank file: ``` $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" ``` This command does not perform any migrations, it only sets the revision. Revision may be any existing revision. Use this command carefully. ``` $ sahara-db-manage --config-file /path/to/sahara.conf stamp ``` To verify that the timeline does branch, you can run this command: ``` $ sahara-db-manage --config-file /path/to/sahara.conf check_migration ``` If the migration path does branch, you can find the branch point via: ``` $ sahara-db-manage --config-file /path/to/sahara.conf history ``` sahara-8.0.0/sahara/db/migration/alembic_migrations/script.py.mako0000666000175100017510000000167013245514472025270 0ustar zuulzuul00000000000000# Copyright ${create_date.year} OpenStack Foundation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """${message} Revision ID: ${up_revision} Revises: ${down_revision} Create Date: ${create_date} """ # revision identifiers, used by Alembic. revision = ${repr(up_revision)} down_revision = ${repr(down_revision)} from alembic import op import sqlalchemy as sa ${imports if imports else ""} def upgrade(): ${upgrades if upgrades else "pass"} sahara-8.0.0/sahara/db/migration/cli.py0000666000175100017510000000662513245514472017762 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from alembic import command as alembic_cmd from alembic import config as alembic_cfg from alembic import util as alembic_u from oslo_config import cfg from sahara.i18n import _ CONF = cfg.CONF def do_alembic_command(config, cmd, *args, **kwargs): try: getattr(alembic_cmd, cmd)(config, *args, **kwargs) except alembic_u.CommandError as e: alembic_u.err(str(e)) def do_check_migration(config, _cmd): do_alembic_command(config, 'branches') def do_upgrade_downgrade(config, cmd): if not CONF.command.revision and not CONF.command.delta: raise SystemExit(_('You must provide a revision or relative delta')) revision = CONF.command.revision if CONF.command.delta: sign = '+' if CONF.command.name == 'upgrade' else '-' revision = sign + str(CONF.command.delta) do_alembic_command(config, cmd, revision, sql=CONF.command.sql) def do_stamp(config, cmd): do_alembic_command(config, cmd, CONF.command.revision, sql=CONF.command.sql) def do_revision(config, cmd): do_alembic_command(config, cmd, message=CONF.command.message, autogenerate=CONF.command.autogenerate, sql=CONF.command.sql) def add_command_parsers(subparsers): for name in ['current', 'history', 'branches']: parser = subparsers.add_parser(name) parser.set_defaults(func=do_alembic_command) parser = subparsers.add_parser('check_migration') parser.set_defaults(func=do_check_migration) parser = subparsers.add_parser('upgrade') parser.add_argument('--delta', type=int) parser.add_argument('--sql', action='store_true') parser.add_argument('revision', nargs='?') parser.set_defaults(func=do_upgrade_downgrade) parser = subparsers.add_parser('stamp') parser.add_argument('--sql', action='store_true') parser.add_argument('revision') parser.set_defaults(func=do_stamp) parser = subparsers.add_parser('revision') parser.add_argument('-m', '--message') parser.add_argument('--autogenerate', action='store_true') parser.add_argument('--sql', action='store_true') parser.set_defaults(func=do_revision) command_opt = cfg.SubCommandOpt('command', title='Command', help='Available commands', handler=add_command_parsers) CONF.register_cli_opt(command_opt) def main(): config = alembic_cfg.Config( os.path.join(os.path.dirname(__file__), 'alembic.ini') ) config.set_main_option('script_location', 'sahara.db.migration:alembic_migrations') # attach the Sahara conf to the Alembic conf config.sahara_config = CONF CONF(project='sahara') CONF.command.func(config, CONF.command.name) sahara-8.0.0/sahara/db/migration/alembic.ini0000666000175100017510000000170013245514472020723 0ustar zuulzuul00000000000000# A generic, single database configuration. [alembic] # path to migration scripts script_location = sahara/db/migration/alembic_migrations # template used to generate migration files # file_template = %%(rev)s_%%(slug)s # max length of characters to apply to the # "slug" field #truncate_slug_length = 40 # set to 'true' to run the environment during # the 'revision' command, regardless of autogenerate # revision_environment = false sqlalchemy.url = # Logging configuration [loggers] keys = root,sqlalchemy,alembic [handlers] keys = console [formatters] keys = generic [logger_root] level = WARN handlers = console qualname = [logger_sqlalchemy] level = WARN handlers = qualname = sqlalchemy.engine [logger_alembic] level = INFO handlers = qualname = alembic [handler_console] class = StreamHandler args = (sys.stderr,) level = NOTSET formatter = generic [formatter_generic] format = %(levelname)-5.5s [%(name)s] %(message)s datefmt = %H:%M:%S sahara-8.0.0/sahara/db/migration/__init__.py0000666000175100017510000000000013245514472020727 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/config.py0000666000175100017510000002011313245514472016066 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import itertools # loading keystonemiddleware opts because sahara uses these options in code from keystonemiddleware import opts # noqa from oslo_config import cfg from oslo_log import log from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import opts as plugins_base from sahara.service.castellan import config as castellan from sahara.service.edp.data_sources import opts as data_source from sahara.service.edp.job_binaries import opts as job_binary from sahara.topology import topology_helper from sahara.utils.notification import sender from sahara.utils.openstack import cinder from sahara.utils.openstack import keystone from sahara.utils import remote from sahara import version cli_opts = [ cfg.HostAddressOpt('host', default='0.0.0.0', help='Hostname or IP address that will be used ' 'to listen on.'), cfg.PortOpt('port', default=8386, help='Port that will be used to listen on.'), cfg.BoolOpt('log-exchange', default=False, help='Log request/response exchange details: environ, ' 'headers and bodies.') ] edp_opts = [ cfg.IntOpt('job_binary_max_KB', default=5120, help='Maximum length of job binary data in kilobytes that ' 'may be stored or retrieved in a single operation.'), cfg.IntOpt('job_canceling_timeout', default=300, help='Timeout for canceling job execution (in seconds). ' 'Sahara will try to cancel job execution during ' 'this time.'), cfg.BoolOpt('edp_internal_db_enabled', default=True, help='Use Sahara internal db to store job binaries.') ] db_opts = [ cfg.StrOpt('db_driver', default='sahara.db', help='Driver to use for database access.') ] networking_opts = [ cfg.BoolOpt('use_floating_ips', default=True, help='If set to True, Sahara will use floating IPs to ' 'communicate with instances. To make sure that all ' 'instances have floating IPs assigned, make sure ' 'that all Node Groups have "floating_ip_pool" ' 'parameter defined.'), cfg.StrOpt('node_domain', default='novalocal', help="The suffix of the node's FQDN."), cfg.BoolOpt('use_namespaces', default=False, help="Use network namespaces for communication."), cfg.BoolOpt('use_rootwrap', default=False, help="Use rootwrap facility to allow non-root users to run " "the sahara services and access private network IPs " "(only valid to use in conjunction with " "use_namespaces=True)"), cfg.StrOpt('rootwrap_command', default='sudo sahara-rootwrap /etc/sahara/rootwrap.conf', help="Rootwrap command to leverage. Use in conjunction with " "use_rootwrap=True") ] dns_opts = [ cfg.BoolOpt('use_designate', default=False, help='Use Designate for internal and external hostnames ' 'resolution'), cfg.ListOpt('nameservers', default=[], help="IP addresses of Designate nameservers. " "This is required if 'use_designate' is True") ] CONF = cfg.CONF CONF.register_cli_opts(cli_opts) CONF.register_opts(networking_opts) CONF.register_opts(edp_opts) CONF.register_opts(db_opts) CONF.register_opts(dns_opts) log.register_options(CONF) sahara_default_log_levels = [ 'stevedore=INFO', 'eventlet.wsgi.server=WARN', 'paramiko=WARN', 'requests=WARN', 'neutronclient=INFO', ] log.set_defaults( default_log_levels=log.get_default_log_levels()+sahara_default_log_levels) def list_opts(): # NOTE (vgridnev): we make these import here to avoid problems # with importing unregistered options in sahara code. # As example, importing 'node_domain' in # sahara/conductor/objects.py from sahara.conductor import api from sahara import main as sahara_main from sahara.service import coordinator from sahara.service.edp import job_utils from sahara.service.heat import heat_engine from sahara.service.heat import templates from sahara.service import periodic from sahara.swift import swift_helper from sahara.utils import cluster_progress_ops as cpo from sahara.utils.openstack import base from sahara.utils.openstack import glance from sahara.utils.openstack import heat from sahara.utils.openstack import manila from sahara.utils.openstack import neutron from sahara.utils.openstack import nova from sahara.utils.openstack import swift from sahara.utils import poll_utils from sahara.utils import proxy from sahara.utils import ssh_remote return [ (None, itertools.chain(cli_opts, edp_opts, networking_opts, dns_opts, db_opts, plugins_base.opts, topology_helper.opts, keystone.opts, remote.ssh_opts, sahara_main.opts, job_utils.opts, periodic.periodic_opts, coordinator.coordinator_opts, proxy.opts, cpo.event_log_opts, base.opts, heat_engine.heat_engine_opts, templates.heat_engine_opts, ssh_remote.ssh_config_options, castellan.opts, data_source.opts, job_binary.opts)), (poll_utils.timeouts.name, itertools.chain(poll_utils.timeouts_opts)), (api.conductor_group.name, itertools.chain(api.conductor_opts)), (cinder.cinder_group.name, itertools.chain(cinder.opts)), (glance.glance_group.name, itertools.chain(glance.opts)), (heat.heat_group.name, itertools.chain(heat.opts)), (manila.manila_group.name, itertools.chain(manila.opts)), (neutron.neutron_group.name, itertools.chain(neutron.opts)), (nova.nova_group.name, itertools.chain(nova.opts)), (swift.swift_group.name, itertools.chain(swift.opts)), (keystone.keystone_group.name, itertools.chain(keystone.ssl_opts)), (keystone.trustee_group.name, itertools.chain(keystone.trustee_opts)), (base.retries.name, itertools.chain(base.opts)), (swift_helper.public_endpoint_cert_group.name, itertools.chain(swift_helper.opts)), (castellan.castellan_group.name, itertools.chain(castellan.castellan_opts)), (sender.notifier_opts_group, sender.notifier_opts) ] def parse_configs(conf_files=None): try: version_string = version.version_info.version_string() CONF(project='sahara', version=version_string, default_config_files=conf_files) except cfg.RequiredOptError as roe: raise ex.ConfigurationError( _("Option '%(option)s' is required for config group '%(group)s'") % {'option': roe.opt_name, 'group': roe.group.name}) sahara-8.0.0/sahara/version.py0000666000175100017510000000121513245514472016310 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from pbr import version version_info = version.VersionInfo('sahara') sahara-8.0.0/sahara/common/0000775000175100017510000000000013245515026015534 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/common/config.py0000666000175100017510000000300313245514472017355 0ustar zuulzuul00000000000000# Copyright (c) 2016 Hewlett Packard Enterprise Development Corporation, LP # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_middleware import cors def set_config_defaults(): """This method updates all configuration default values.""" set_cors_middleware_defaults() def set_cors_middleware_defaults(): """Update default configuration options for oslo.middleware.""" cors.set_defaults( allow_headers=['X-Auth-Token', 'X-Identity-Status', 'X-Roles', 'X-Service-Catalog', 'X-User-Id', 'X-Tenant-Id', 'X-OpenStack-Request-ID'], expose_headers=['X-Auth-Token', 'X-Subject-Token', 'X-Service-Token', 'X-OpenStack-Request-ID'], allow_methods=['GET', 'PUT', 'POST', 'DELETE', 'PATCH'] ) sahara-8.0.0/sahara/common/policies/0000775000175100017510000000000013245515026017343 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/common/policies/cluster_templates.py0000666000175100017510000000524613245514472023471 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base cluster_templates_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTER_TEMPLATES % 'create', check_str=base.UNPROTECTED, description='Create cluster template.', operations=[{'path': '/v1.1/{project_id}/cluster-templates', 'method': 'POST'}, {'path': '/v2/cluster-templates', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTER_TEMPLATES % 'delete', check_str=base.UNPROTECTED, description='Delete a cluster template.', operations=[ {'path': '/v1.1/{project_id}/cluster-templates/{cluster_temp_id}', 'method': 'DELETE'}, {'path': '/v2/cluster-templates/{cluster_temp_id}', 'method': 'DELETE'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTER_TEMPLATES % 'modify', check_str=base.UNPROTECTED, description='Update cluster template.', operations=[ {'path': '/v1.1/{project_id}/cluster-templates/{cluster_temp_id}', 'method': 'PUT'}, {'path': '/v2/cluster-templates/{cluster_temp_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTER_TEMPLATES % 'get', check_str=base.UNPROTECTED, description='Show cluster template details.', operations=[ {'path': '/v1.1/{project_id}/cluster-templates/{cluster_temp_id}', 'method': 'GET'}, {'path': '/v2/cluster-templates/{cluster_temp_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTER_TEMPLATES % 'get_all', check_str=base.UNPROTECTED, description='List cluster templates.', operations=[{'path': '/v1.1/{project_id}/cluster-templates', 'method': 'GET'}, {'path': '/v2/cluster-templates', 'method': 'GET'}]), ] def list_rules(): return cluster_templates_policies sahara-8.0.0/sahara/common/policies/data_sources.py0000666000175100017510000000506213245514472022402 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base data_sources_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_DATA_SOURCES % 'get_all', check_str=base.UNPROTECTED, description='List data sources.', operations=[{'path': '/v1.1/{project_id}/data-sources', 'method': 'GET'}, {'path': '/v2/data-sources', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_DATA_SOURCES % 'get', check_str=base.UNPROTECTED, description='Show data source details.', operations=[ {'path': '/v1.1/{project_id}/data-sources/{data_source_id}', 'method': 'GET'}, {'path': '/v2/data-sources/{data_source_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_DATA_SOURCES % 'register', check_str=base.UNPROTECTED, description='Create data source.', operations=[{'path': '/v1.1/{project_id}/data-sources', 'method': 'POST'}, {'path': '/v2/data-sources', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_DATA_SOURCES % 'modify', check_str=base.UNPROTECTED, description='Update data source.', operations=[ {'path': '/v1.1/{project_id}/data-sources/{data_source_id}', 'method': 'PUT'}, {'path': '/v2/data-sources/{data_source_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_DATA_SOURCES % 'delete', check_str=base.UNPROTECTED, description='Delete data source.', operations=[ {'path': '/v1.1/{project_id}/data-sources/{data_source_id}', 'method': 'DELETE'}, {'path': '/v2/data-sources/{data_source_id}', 'method': 'DELETE'}]), ] def list_rules(): return data_sources_policies sahara-8.0.0/sahara/common/policies/base.py0000666000175100017510000000311413245514472020634 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslo_policy import policy DATA_PROCESSING = 'data-processing:%s' DATA_PROCESSING_CLUSTERS = DATA_PROCESSING % 'clusters:%s' DATA_PROCESSING_CLUSTER_TEMPLATES = DATA_PROCESSING % 'cluster-templates:%s' DATA_PROCESSING_DATA_SOURCES = DATA_PROCESSING % 'data-sources:%s' DATA_PROCESSING_IMAGES = DATA_PROCESSING % 'images:%s' DATA_PROCESSING_JOB_BINARIES = DATA_PROCESSING % 'job-binaries:%s' DATA_PROCESSING_JOB_EXECUTIONS = DATA_PROCESSING % 'job-executions:%s' DATA_PROCESSING_JOB_TEMPLATES = DATA_PROCESSING % 'job-templates:%s' DATA_PROCESSING_JOB_TYPES = DATA_PROCESSING % 'job-types:%s' DATA_PROCESSING_JOBS = DATA_PROCESSING % 'jobs:%s' DATA_PROCESSING_PLUGINS = DATA_PROCESSING % 'plugins:%s' DATA_PROCESSING_NODE_GROUP_TEMPLATES = ( DATA_PROCESSING % 'node-group-templates:%s') DATA_PROCESSING_JOB_BINARY_INTERNALS = ( DATA_PROCESSING % 'job-binary-internals:%s') UNPROTECTED = '' ROLE_ADMIN = 'role:admin' rules = [ policy.RuleDefault( name='context_is_admin', check_str=ROLE_ADMIN), ] def list_rules(): return rules sahara-8.0.0/sahara/common/policies/job_binaries.py0000666000175100017510000000571613245514472022362 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base job_binaries_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARIES % 'get_all', check_str=base.UNPROTECTED, description='List job binaries.', operations=[{'path': '/v1.1/{project_id}/job-binaries', 'method': 'GET'}, {'path': '/v2/job-binaries', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARIES % 'create', check_str=base.UNPROTECTED, description='Create job binary.', operations=[{'path': '/v1.1/{project_id}/job-binaries', 'method': 'POST'}, {'path': '/v2/job-binaries', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARIES % 'get_data', check_str=base.UNPROTECTED, description='Show job binary data.', operations=[ {'path': '/v1.1/{project_id}/job-binaries/{job-binary_id}/data', 'method': 'POST'}, {'path': '/v2/job-binaries/{job-binary_id}/data', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARIES % 'modify', check_str=base.UNPROTECTED, description='Update job binary.', operations=[ {'path': '/v1.1/{project_id}/job-binaries/{job-binary_id}', 'method': 'PUT'}, {'path': '/v2/job-binaries/{job-binary_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARIES % 'get', check_str=base.UNPROTECTED, description='Show job binary details.', operations=[{'path': '/v1.1/{project_id}/job-binaries/{job_binary_id}', 'method': 'GET'}, {'path': '/v2/job-binaries/{job_binary_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARIES % 'delete', check_str=base.UNPROTECTED, description='Delete job binary.', operations=[{'path': '/v1.1/{project_id}/job-binaries/{job_binary_id}', 'method': 'DELETE'}, {'path': '/v2/job-binaries/{job_binary_id}', 'method': 'DELETE'}]), ] def list_rules(): return job_binaries_policies sahara-8.0.0/sahara/common/policies/plugins.py0000666000175100017510000000500713245514472021406 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base plugins_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_PLUGINS % 'get_all', check_str=base.UNPROTECTED, description='List plugins.', operations=[{'path': '/v1.1/{project_id}/plugins', 'method': 'GET'}, {'path': '/v2/plugins', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_PLUGINS % 'get_version', check_str=base.UNPROTECTED, description='Show plugins version details.', operations=[ {'path': '/v1.1/{project_id}/plugins/{plugin_name}/{version}', 'method': 'GET'}, {'path': '/v2/plugins/{plugin_name}/{version}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_PLUGINS % 'get', check_str=base.UNPROTECTED, description='Show plugin details.', operations=[{'path': '/v1.1/{project_id}/plugins/{plugin_name}', 'method': 'GET'}, {'path': '/v2/plugins/{plugin_name}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_PLUGINS % 'convert_config', check_str=base.UNPROTECTED, description='Convert plugins to cluster template', operations=[ {'path': ('/v1.1/{project_id}/plugins/{plugin_name}/' '{version}/convert-config/{name}'), 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_PLUGINS % 'patch', check_str=base.ROLE_ADMIN, description='Update plugin details.', operations=[{'path': '/v1.1/{project_id}/plugins/{plugin_name}', 'method': 'PATCH'}, {'path': '/v2/plugins/{plugin_name}', 'method': 'PATCH'}]), ] def list_rules(): return plugins_policies sahara-8.0.0/sahara/common/policies/images.py0000666000175100017510000000635713245514472021203 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base images_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_IMAGES % 'set_tags', check_str=base.UNPROTECTED, description='Add tags to image.', operations=[{'path': '/v2/images/{image_id}/tags', 'method': 'PUT'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_IMAGES % 'add_tags', check_str=base.UNPROTECTED, description='Add tags to image.', operations=[{'path': '/v1.1/{project_id}/images/{image_id}/tag', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_IMAGES % 'register', check_str=base.UNPROTECTED, description='Register image.', operations=[{'path': '/v1.1/{project_id}/images/{image_id}', 'method': 'POST'}, {'path': '/v2/images/{image_id}', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_IMAGES % 'get_all', check_str=base.UNPROTECTED, description='List images.', operations=[{'path': '/v1.1/{project_id}/images', 'method': 'GET'}, {'path': '/v2/images', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_IMAGES % 'unregister', check_str=base.UNPROTECTED, description='Unregister image.', operations=[{'path': '/v1.1/{project_id}/images/{image_id}', 'method': 'POST'}, {'path': '/v2/images/{image_id}', 'method': 'DELETE'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_IMAGES % 'get', check_str=base.UNPROTECTED, description='Show image details.', operations=[{'path': '/v1.1/{project_id}/images/{image_id}', 'method': 'GET'}, {'path': '/v2/images/{image_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_IMAGES % 'remove_tags', check_str=base.UNPROTECTED, description='Remove tags from image.', operations=[{'path': '/v1.1/{project_id}/images/{image_id}/untag', 'method': 'POST'}, {'path': '/v2/images/{image_id}/tags', 'method': 'DELETE'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_IMAGES % 'get_tags', check_str=base.UNPROTECTED, description='List tags on an image.', operations=[{'path': '/v2/images/{image_id}/tags', 'method': 'GET'}]), ] def list_rules(): return images_policies sahara-8.0.0/sahara/common/policies/clusters.py0000666000175100017510000000555213245514472021576 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base clusters_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTERS % 'scale', check_str=base.UNPROTECTED, description='Scale cluster.', operations=[{'path': '/v1.1/{project_id}/clusters/{cluster_id}', 'method': 'PUT'}, {'path': '/v2/clusters/{cluster_id}', 'method': 'PUT'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTERS % 'get_all', check_str=base.UNPROTECTED, description='List available clusters', operations=[{'path': '/v1.1/{project_id}/clusters', 'method': 'GET'}, {'path': '/v2/clusters', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTERS % 'create', check_str=base.UNPROTECTED, description='Create cluster.', operations=[{'path': '/v1.1/{project_id}/clusters', 'method': 'POST'}, {'path': '/v2/clusters', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTERS % 'get', check_str=base.UNPROTECTED, description='Show details of a cluster.', operations=[{'path': '/v1.1/{project_id}/clusters/{cluster_id}', 'method': 'GET'}, {'path': '/v2/clusters/{cluster_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTERS % 'modify', check_str=base.UNPROTECTED, description='Modify a cluster.', operations=[{'path': '/v1.1/{project_id}/clusters/{cluster_id}', 'method': 'PATCH'}, {'path': '/v2/clusters/{cluster_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_CLUSTERS % 'delete', check_str=base.UNPROTECTED, description='Delete a cluster.', operations=[{'path': '/v1.1/{project_id}/clusters/{cluster_id}', 'method': 'DELETE'}, {'path': '/v2/clusters/{cluster_id}', 'method': 'DELETE'}]), ] def list_rules(): return clusters_policies sahara-8.0.0/sahara/common/policies/__init__.py0000666000175100017510000000321413245514472021462 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import itertools from sahara.common.policies import base from sahara.common.policies import cluster_templates from sahara.common.policies import clusters from sahara.common.policies import data_sources from sahara.common.policies import images from sahara.common.policies import job_binaries from sahara.common.policies import job_binary_internals from sahara.common.policies import job_executions from sahara.common.policies import job_templates from sahara.common.policies import job_types from sahara.common.policies import jobs from sahara.common.policies import node_group_templates from sahara.common.policies import plugins def list_rules(): return itertools.chain( base.list_rules(), clusters.list_rules(), cluster_templates.list_rules(), data_sources.list_rules(), images.list_rules(), job_binaries.list_rules(), job_binary_internals.list_rules(), job_executions.list_rules(), job_types.list_rules(), job_templates.list_rules(), jobs.list_rules(), node_group_templates.list_rules(), plugins.list_rules() ) sahara-8.0.0/sahara/common/policies/jobs.py0000666000175100017510000000557213245514472020671 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base jobs_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOBS % 'execute', check_str=base.UNPROTECTED, description='Run job.', operations=[{'path': '/v1.1/{project_id}/jobs/{job_id}/execute', 'method': 'POST'}, {'path': '/v2/jobs', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOBS % 'get', check_str=base.UNPROTECTED, description='Show job details.', operations=[{'path': '/v1.1/{project_id}/jobs/{job_id}', 'method': 'GET'}, {'path': '/v2/jobs/{job_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOBS % 'create', check_str=base.UNPROTECTED, description='Create job.', operations=[{'path': '/v1.1/{project_id}/jobs', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOBS % 'get_all', check_str=base.UNPROTECTED, description='List jobs.', operations=[{'path': '/v1.1/{project_id}/jobs', 'method': 'GET'}, {'path': '/v2/jobs', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOBS % 'modify', check_str=base.UNPROTECTED, description='Update job object.', operations=[{'path': '/v1.1/{project_id}/jobs/{job_id}', 'method': 'PATCH'}, {'path': '/v2/jobs/{job_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOBS % 'get_config_hints', check_str=base.UNPROTECTED, description='Get job config hints.', operations=[ {'path': '/v1.1/{project_id}/jobs/get_config_hints/{job_type}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOBS % 'delete', check_str=base.UNPROTECTED, description='Remove job.', operations=[{'path': '/v1.1/{project_id}/jobs/{job_id}', 'method': 'DELETE'}, {'path': '/v2/jobs/{job_id}', 'method': 'DELETE'}]), ] def list_rules(): return jobs_policies sahara-8.0.0/sahara/common/policies/node_group_templates.py0000666000175100017510000000543713245514472024153 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base node_group_templates_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_NODE_GROUP_TEMPLATES % 'get_all', check_str=base.UNPROTECTED, description='List node group templates.', operations=[{'path': '/v1.1/{project_id}/node-group-templates', 'method': 'GET'}, {'path': '/v2/node-group-templates', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_NODE_GROUP_TEMPLATES % 'create', check_str=base.UNPROTECTED, description='Create node group template.', operations=[{'path': '/v1.1/{project_id}/node-group-templates', 'method': 'POST'}, {'path': '/v2/node-group-templates', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_NODE_GROUP_TEMPLATES % 'get', check_str=base.UNPROTECTED, description='Show node group template details.', operations=[ {'path': '/v1.1/{project_id}/node-group-templates/{node_group_temp_id}', 'method': 'GET'}, {'path': '/v2/node-group-templates/{node_group_temp_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_NODE_GROUP_TEMPLATES % 'modify', check_str=base.UNPROTECTED, description='Update node group template.', operations=[ {'path': '/v1.1/{project_id}/node-group-templates/{node_group_temp_id}', 'method': 'PUT'}, {'path': '/v2/node-group-templates/{node_group_temp_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_NODE_GROUP_TEMPLATES % 'delete', check_str=base.UNPROTECTED, description='Delete node group template.', operations=[ {'path': '/v1.1/{project_id}/node-group-templates/{node_group_temp_id}', 'method': 'DELETE'}, {'path': '/v2/node-group-templates/{node_group_temp_id}', 'method': 'DELETE'}]), ] def list_rules(): return node_group_templates_policies sahara-8.0.0/sahara/common/policies/job_templates.py0000666000175100017510000000453213245514472022557 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base job_templates_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_TEMPLATES % 'get', check_str=base.UNPROTECTED, description='Show job template details.', operations=[{'path': '/v2/job-templates/{job_temp_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_TEMPLATES % 'create', check_str=base.UNPROTECTED, description='Create job templates.', operations=[{'path': '/v2/job-templates', 'method': 'POST'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_TEMPLATES % 'get_all', check_str=base.UNPROTECTED, description='List job templates.', operations=[{'path': '/v2/job-templates', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_TEMPLATES % 'modify', check_str=base.UNPROTECTED, description='Update job template.', operations=[{'path': '/v2/job-templates/{job_temp_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_TEMPLATES % 'get_config_hints', check_str=base.UNPROTECTED, description='Get job template config hints.', operations=[ {'path': '/v2/job-templates/config-hints/{job_type}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_TEMPLATES % 'delete', check_str=base.UNPROTECTED, description='Remove job template.', operations=[{'path': '/v2/job-templates/{job_temp_id}', 'method': 'DELETE'}]), ] def list_rules(): return job_templates_policies sahara-8.0.0/sahara/common/policies/job_binary_internals.py0000666000175100017510000000512613245514472024124 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base job_binary_internals_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARY_INTERNALS % 'get', check_str=base.UNPROTECTED, description='Show job binary internal details.', operations=[{ 'path': '/v1.1/{project_id}/job-binary-internals/{job_bin_int_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARY_INTERNALS % 'get_all', check_str=base.UNPROTECTED, description='List job binary internals.', operations=[{'path': '/v1.1/{project_id}/job-binary-internals', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARY_INTERNALS % 'create', check_str=base.UNPROTECTED, description='Create job binary internals.', operations=[{'path': '/v1.1/{project_id}/job-binary-internals/{name}', 'method': 'PUT'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARY_INTERNALS % 'get_data', check_str=base.UNPROTECTED, description='Show job binary internal data.', operations=[{ 'path': '/v1.1/{project_id}/job-binary-internals/{job_bin_int_id}/data', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARY_INTERNALS % 'modify', check_str=base.UNPROTECTED, description='Update job binary internal.', operations=[{ 'path': '/v1.1/{project_id}/job-binary-internals/{job_bin_int_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_BINARY_INTERNALS % 'delete', check_str=base.UNPROTECTED, description='Delete job binary internals.', operations=[{ 'path': '/v1.1/{project_id}/job-binary-internals/{job_bin_int_id}', 'method': 'DELETE'}]), ] def list_rules(): return job_binary_internals_policies sahara-8.0.0/sahara/common/policies/job_executions.py0000666000175100017510000000477713245514472022762 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base job_executions_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_EXECUTIONS % 'get', check_str=base.UNPROTECTED, description='Show job executions details.', operations=[{'path': '/v1.1/{project_id}/job-executions/{job_exec_id}', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_EXECUTIONS % 'modify', check_str=base.UNPROTECTED, description='Update job execution.', operations=[{'path': '/v1.1/{project_id}/job-executions/{job_exec_id}', 'method': 'PATCH'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_EXECUTIONS % 'get_all', check_str=base.UNPROTECTED, description='List job executions.', operations=[{'path': '/v1.1/{project_id}/job-executions', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_EXECUTIONS % 'refresh_status', check_str=base.UNPROTECTED, description='Refresh job execution status.', operations=[ {'path': '/v1.1/{project_id}/job-executions/{job_exec_id}/refresh-status', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_EXECUTIONS % 'cancel', check_str=base.UNPROTECTED, description='Cancel job execution.', operations=[{'path': '/v1.1/{project_id}/job-executions/{job_exec_id}/cancel', 'method': 'GET'}]), policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_EXECUTIONS % 'delete', check_str=base.UNPROTECTED, description='Delete job execution.', operations=[{'path': '/v1.1/{project_id}/job-executions/{job_exec_id}', 'method': 'DELETE'}]), ] def list_rules(): return job_executions_policies sahara-8.0.0/sahara/common/policies/job_types.py0000666000175100017510000000203013245514472021714 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy from sahara.common.policies import base job_types_policies = [ policy.DocumentedRuleDefault( name=base.DATA_PROCESSING_JOB_TYPES % 'get_all', check_str=base.UNPROTECTED, description='List job types.', operations=[{'path': '/v1.1/{project_id}/job-types', 'method': 'GET'}, {'path': '/v2/job-types', 'method': 'GET'}]), ] def list_rules(): return job_types_policies sahara-8.0.0/sahara/common/__init__.py0000666000175100017510000000000013245514472017641 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/0000775000175100017510000000000013245515027015405 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/rpc.py0000666000175100017510000000717213245514472016557 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # Copyright (c) 2013 Julien Danjou # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging import oslo_messaging as messaging from oslo_messaging.rpc import dispatcher from oslo_serialization import jsonutils from sahara import context MESSAGING_TRANSPORT = None NOTIFICATION_TRANSPORT = None NOTIFIER = None CONF = cfg.CONF LOG = logging.getLogger(__name__) class ContextSerializer(messaging.Serializer): def __init__(self, base): self._base = base def serialize_entity(self, ctxt, entity): return self._base.serialize_entity(ctxt, entity) def deserialize_entity(self, ctxt, entity): return self._base.deserialize_entity(ctxt, entity) @staticmethod def serialize_context(ctxt): return ctxt.to_dict() @staticmethod def deserialize_context(ctxt): pass class JsonPayloadSerializer(messaging.NoOpSerializer): @classmethod def serialize_entity(cls, context, entity): return jsonutils.to_primitive(entity, convert_instances=True) class RPCClient(object): def __init__(self, target): global MESSAGING_TRANSPORT self.__client = messaging.RPCClient( target=target, transport=MESSAGING_TRANSPORT, ) def cast(self, name, **kwargs): ctx = context.current() self.__client.cast(ctx.to_dict(), name, **kwargs) def call(self, name, **kwargs): ctx = context.current() return self.__client.call(ctx.to_dict(), name, **kwargs) class RPCServer(object): def __init__(self, target): global MESSAGING_TRANSPORT access_policy = dispatcher.DefaultRPCAccessPolicy self.__server = messaging.get_rpc_server( target=target, transport=MESSAGING_TRANSPORT, endpoints=[self], executor='eventlet', access_policy=access_policy) def get_service(self): return self.__server def setup_service_messaging(): global MESSAGING_TRANSPORT if MESSAGING_TRANSPORT: # Already is up return MESSAGING_TRANSPORT = messaging.get_rpc_transport(cfg.CONF) def setup_notifications(): global NOTIFICATION_TRANSPORT, NOTIFIER, MESSAGING_TRANSPORT try: NOTIFICATION_TRANSPORT = \ messaging.get_notification_transport(cfg.CONF) except Exception: LOG.error("Unable to setup notification transport. Reusing " "service transport for that.") setup_service_messaging() NOTIFICATION_TRANSPORT = MESSAGING_TRANSPORT serializer = ContextSerializer(JsonPayloadSerializer()) NOTIFIER = messaging.Notifier(NOTIFICATION_TRANSPORT, serializer=serializer) def setup(service_name): """Initialise the oslo_messaging layer.""" messaging.set_transport_defaults('sahara') setup_notifications() if service_name != 'all-in-one': setup_service_messaging() def get_notifier(publisher_id): """Return a configured oslo_messaging notifier.""" return NOTIFIER.prepare(publisher_id=publisher_id) sahara-8.0.0/sahara/utils/general.py0000666000175100017510000000404313245514472017402 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re import six NATURAL_SORT_RE = re.compile('([0-9]+)') def find_dict(iterable, **rules): """Search for dict in iterable of dicts using specified key-value rules.""" for item in iterable: # assert all key-value pairs from rules dict ok = True for k, v in six.iteritems(rules): ok = ok and k in item and item[k] == v if ok: return item return None def find(lst, **kwargs): for obj in lst: match = True for attr, value in kwargs.items(): if getattr(obj, attr) != value: match = False if match: return obj return None def get_by_id(lst, id): for obj in lst: if obj.id == id: return obj return None # Taken from http://stackoverflow.com/questions/4836710/does- # python-have-a-built-in-function-for-string-natural-sort def natural_sort_key(s): return [int(text) if text.isdigit() else text.lower() for text in re.split(NATURAL_SORT_RE, s)] def generate_instance_name(cluster_name, node_group_name, index): return ("%s-%s-%03d" % (cluster_name, node_group_name, index)).lower() def generate_auto_security_group_name(node_group): return ("%s-%s-%s" % (node_group.cluster.name, node_group.name, node_group.id[:8])).lower() def generate_aa_group_name(cluster_name, server_group_index): return ("%s-aa-group-%d" % (cluster_name, server_group_index)).lower() sahara-8.0.0/sahara/utils/proxy.py0000666000175100017510000002634313245514472017155 0ustar zuulzuul00000000000000# Copyright (c) 2014 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging from oslo_utils import uuidutils import six from sahara import conductor as c from sahara import context from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.castellan import utils as key_manager from sahara.service.edp import job_utils from sahara.service import trusts as t from sahara.swift import utils as su from sahara.utils.openstack import base as b from sahara.utils.openstack import keystone as k PROXY_DOMAIN = None conductor = c.API LOG = logging.getLogger(__name__) CONF = cfg.CONF opts = [ cfg.BoolOpt('use_domain_for_proxy_users', default=False, help='Enables Sahara to use a domain for creating temporary ' 'proxy users to access Swift. If this is enabled ' 'a domain must be created for Sahara to use.'), cfg.StrOpt('proxy_user_domain_name', default=None, help='The domain Sahara will use to create new proxy users ' 'for Swift object access.'), cfg.ListOpt('proxy_user_role_names', default=['Member'], help='A list of the role names that the proxy user should ' 'assume through trust for Swift object access.') ] CONF.register_opts(opts) def create_proxy_user_for_job_execution(job_execution): '''Creates a proxy user and adds the credentials to the job execution :param job_execution: The job execution model to update ''' username = 'job_{0}'.format(job_execution.id) password = key_manager.store_secret(proxy_user_create(username)) current_user = k.auth() proxy_user = k.auth_for_proxy(username, password) trust_id = t.create_trust(trustor=current_user, trustee=proxy_user, role_names=CONF.proxy_user_role_names) update = {'job_configs': job_execution.job_configs.to_dict()} update['job_configs']['proxy_configs'] = { 'proxy_username': username, 'proxy_password': password, 'proxy_trust_id': trust_id } conductor.job_execution_update(context.ctx(), job_execution, update) def delete_proxy_user_for_job_execution(job_execution): '''Delete a proxy user based on a JobExecution :param job_execution: The job execution with proxy user information :returns: An updated job_configs dictionary or None ''' proxy_configs = job_execution.job_configs.get('proxy_configs') if proxy_configs is not None: proxy_username = proxy_configs.get('proxy_username') proxy_trust_id = proxy_configs.get('proxy_trust_id') proxy_user = k.auth_for_proxy(proxy_username, key_manager.get_secret( proxy_configs.get('proxy_password')), proxy_trust_id) t.delete_trust(proxy_user, proxy_trust_id) proxy_user_delete(proxy_username) key_manager.delete_secret(proxy_configs.get('proxy_password')) update = job_execution.job_configs.to_dict() del update['proxy_configs'] return update return None def create_proxy_user_for_cluster(cluster): '''Creates a proxy user and adds the credentials to the cluster :param cluster: The cluster model to update ''' if cluster.cluster_configs.get('proxy_configs'): return cluster username = 'cluster_{0}'.format(cluster.id) password = key_manager.store_secret(proxy_user_create(username)) current_user = k.auth() proxy_user = k.auth_for_proxy(username, password) trust_id = t.create_trust(trustor=current_user, trustee=proxy_user, role_names=CONF.proxy_user_role_names) update = {'cluster_configs': cluster.cluster_configs.to_dict()} update['cluster_configs']['proxy_configs'] = { 'proxy_username': username, 'proxy_password': password, 'proxy_trust_id': trust_id } return conductor.cluster_update(context.ctx(), cluster, update) def delete_proxy_user_for_cluster(cluster): '''Delete a proxy user based on a Cluster :param cluster: The cluster model with proxy user information ''' proxy_configs = cluster.cluster_configs.get('proxy_configs') if proxy_configs is not None: proxy_username = proxy_configs.get('proxy_username') proxy_trust_id = proxy_configs.get('proxy_trust_id') proxy_user = k.auth_for_proxy(proxy_username, key_manager.get_secret( proxy_configs.get('proxy_password')), proxy_trust_id) t.delete_trust(proxy_user, proxy_trust_id) proxy_user_delete(proxy_username) key_manager.delete_secret(proxy_configs.get('proxy_password')) update = {'cluster_configs': cluster.cluster_configs.to_dict()} del update['cluster_configs']['proxy_configs'] conductor.cluster_update(context.ctx(), cluster, update) def domain_for_proxy(): '''Return the proxy domain or None If configured to use the proxy domain, this function will return that domain. If not configured to use the proxy domain, this function will return None. If the proxy domain can't be found this will raise an exception. :returns: A Keystone Domain object or None. :raises ConfigurationError: If the domain is requested but not specified. :raises NotFoundException: If the domain name is specified but cannot be found. ''' if CONF.use_domain_for_proxy_users is False: return None if CONF.proxy_user_domain_name is None: raise ex.ConfigurationError(_('Proxy domain requested but not ' 'specified.')) admin = k.client_for_admin() global PROXY_DOMAIN if not PROXY_DOMAIN: domain_list = b.execute_with_retries( admin.domains.list, name=CONF.proxy_user_domain_name) if len(domain_list) == 0: raise ex.NotFoundException( value=CONF.proxy_user_domain_name, message_template=_('Failed to find domain %s')) # the domain name should be globally unique in Keystone if len(domain_list) > 1: raise ex.NotFoundException( value=CONF.proxy_user_domain_name, message_template=_('Unexpected results found when searching ' 'for domain %s')) PROXY_DOMAIN = domain_list[0] return PROXY_DOMAIN def job_execution_requires_proxy_user(job_execution): '''Returns True if the job execution requires a proxy user.''' def _check_values(values): return any(value.startswith( su.SWIFT_INTERNAL_PREFIX) for value in values if ( isinstance(value, six.string_types))) if CONF.use_domain_for_proxy_users is False: return False paths = [conductor.data_source_get(context.ctx(), job_execution.output_id), conductor.data_source_get(context.ctx(), job_execution.input_id)] if _check_values(ds.url for ds in paths if ds): return True if _check_values(six.itervalues( job_execution.job_configs.get('configs', {}))): return True if _check_values(six.itervalues( job_execution.job_configs.get('params', {}))): return True if _check_values(job_execution.job_configs.get('args', [])): return True job = conductor.job_get(context.ctx(), job_execution.job_id) if _check_values(main.url for main in job.mains): return True if _check_values(lib.url for lib in job.libs): return True # We did the simple checks, now if data_source referencing is # enabled and we have values that could be a name or uuid, # query for data_sources that match and contain a swift path by_name, by_uuid = job_utils.may_contain_data_source_refs( job_execution.job_configs) if by_name: names = tuple(job_utils.find_possible_data_source_refs_by_name( job_execution.job_configs)) # do a query here for name in names and path starts with swift-prefix if names and conductor.data_source_count( context.ctx(), name=names, url=su.SWIFT_INTERNAL_PREFIX+'%') > 0: return True if by_uuid: uuids = tuple(job_utils.find_possible_data_source_refs_by_uuid( job_execution.job_configs)) # do a query here for id in uuids and path starts with swift-prefix if uuids and conductor.data_source_count( context.ctx(), id=uuids, url=su.SWIFT_INTERNAL_PREFIX+'%') > 0: return True return False def proxy_domain_users_list(): '''Return a list of all users in the proxy domain.''' admin = k.client_for_admin() domain = domain_for_proxy() if domain: return b.execute_with_retries(admin.users.list, domain=domain.id) return [] def proxy_user_create(username): '''Create a new user in the proxy domain Creates the username specified with a random password. :param username: The name of the new user. :returns: The password created for the user. ''' admin = k.client_for_admin() domain = domain_for_proxy() password = uuidutils.generate_uuid() b.execute_with_retries( admin.users.create, name=username, password=password, domain=domain.id) LOG.debug('Created proxy user {username}'.format(username=username)) return password def proxy_user_delete(username=None, user_id=None): '''Delete the user from the proxy domain. :param username: The name of the user to delete. :param user_id: The id of the user to delete, if provided this overrides the username. :raises NotFoundException: If there is an error locating the user in the proxy domain. ''' admin = k.client_for_admin() if not user_id: domain = domain_for_proxy() user_list = b.execute_with_retries( admin.users.list, domain=domain.id, name=username) if len(user_list) == 0: raise ex.NotFoundException( value=username, message_template=_('Failed to find user %s')) if len(user_list) > 1: raise ex.NotFoundException( value=username, message_template=_('Unexpected results found when searching ' 'for user %s')) user_id = user_list[0].id b.execute_with_retries(admin.users.delete, user_id) LOG.debug('Deleted proxy user id {user_id}'.format(user_id=user_id)) sahara-8.0.0/sahara/utils/xmlutils.py0000666000175100017510000001357013245514472017653 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re import xml.dom.minidom as xml import pkg_resources as pkg from sahara import version # hadoop.xml related utils def load_hadoop_xml_defaults(file_name): doc = load_xml_document(file_name) configs = [] prop = doc.getElementsByTagName('property') for elements in prop: configs.append({ "name": get_text_from_node(elements, 'name'), "value": _adjust_field(get_text_from_node(elements, 'value')), "description": _adjust_field( get_text_from_node(elements, 'description')) }) return configs def parse_hadoop_xml_with_name_and_value(data): doc = xml.parseString(data) configs = [] prop = doc.getElementsByTagName('property') for elements in prop: configs.append({ 'name': get_text_from_node(elements, 'name'), 'value': get_text_from_node(elements, 'value') }) return configs def _get_node_element(element, name): element = element.getElementsByTagName(name) return element[0] if element and element[0].hasChildNodes() else None def create_hadoop_xml(configs, config_filter=None): doc = xml.Document() pi = doc.createProcessingInstruction('xml-stylesheet', 'type="text/xsl" ' 'href="configuration.xsl"') doc.insertBefore(pi, doc.firstChild) # Create the base element configuration = doc.createElement('configuration') doc.appendChild(configuration) default_configs = [] if config_filter is not None: default_configs = [cfg['name'] for cfg in config_filter] for name in sorted(configs): if name in default_configs or config_filter is None: add_property_to_configuration(doc, name, configs[name]) # Return newly created XML return doc.toprettyxml(indent=" ") def create_elements_xml(configs): doc = xml.Document() text = '' for name in sorted(configs): element = doc.createElement('property') add_text_element_to_element(doc, element, 'name', name) add_text_element_to_element(doc, element, 'value', configs[name]) text += element.toprettyxml(indent=" ") return text # basic utils def load_xml_document(file_name, strip=False): fname = pkg.resource_filename( version.version_info.package, file_name) if strip: with open(fname, "r") as f: doc = "".join(line.strip() for line in f) return xml.parseString(doc) else: return xml.parse(fname) def get_text_from_node(element, name): element = element.getElementsByTagName(name) if element else None return element[0].firstChild.nodeValue if ( element and element[0].hasChildNodes()) else '' def _adjust_field(text): return re.sub(r"\n *|\t", "", str(text)) def add_property_to_configuration(doc, name, value): prop = add_child(doc, 'configuration', 'property') add_text_element_to_element(doc, prop, 'name', name) add_text_element_to_element(doc, prop, 'value', value) def add_properties_to_configuration(doc, parent_for_conf, configs): get_and_create_if_not_exist(doc, parent_for_conf, 'configuration') for n in sorted(filter(lambda x: x, configs)): add_property_to_configuration(doc, n, configs[n]) def add_child(doc, parent, tag_to_add): actions = doc.getElementsByTagName(parent) actions[0].appendChild(doc.createElement(tag_to_add)) return actions[0].lastChild def add_element(doc, parent, element): actions = doc.getElementsByTagName(parent) actions[0].appendChild(element) return actions[0].lastChild def get_and_create_if_not_exist(doc, parent, element): prop = doc.getElementsByTagName(element) if len(prop) != 0: return prop[0] return add_child(doc, parent, element) def add_text_element_to_tag(doc, parent_tag, element, value): prop = add_child(doc, parent_tag, element) prop.appendChild(doc.createTextNode(str(value))) def add_text_element_to_element(doc, parent, element, value): parent.appendChild(doc.createElement(element)) try: parent.lastChild.appendChild(doc.createTextNode(str(value))) except UnicodeEncodeError: parent.lastChild.appendChild(doc.createTextNode( str(value.encode('utf8')))) def add_equal_separated_dict(doc, parent_tag, each_elem_tag, value): for k in sorted(filter(lambda x: x, value)): if k: add_text_element_to_tag(doc, parent_tag, each_elem_tag, "%s=%s" % (k, value[k])) def add_attributes_to_element(doc, tag, attributes): element = doc.getElementsByTagName(tag)[0] for name, value in attributes.items(): element.setAttribute(name, value) def add_tagged_list(doc, parent_tag, each_elem_tag, values): for v in values: add_text_element_to_tag(doc, parent_tag, each_elem_tag, v) def get_property_dict(elem): res = {} properties = elem.getElementsByTagName('property') for prop in properties: k = get_text_from_node(prop, 'name') v = get_text_from_node(prop, 'value') res[k] = v return res def get_param_dict(elem): res = {} params = elem.getElementsByTagName('param') for param in params: k, v = param.firstChild.nodeValue.split('=') res[k] = v return res sahara-8.0.0/sahara/utils/cluster_progress_ops.py0000666000175100017510000001352313245514472022256 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools from oslo_config import cfg from oslo_utils import excutils from oslo_utils import timeutils import six from sahara import conductor as c from sahara.conductor import resource from sahara import context from sahara.utils import cluster as cluster_utils conductor = c.API CONF = cfg.CONF event_log_opts = [ cfg.BoolOpt('disable_event_log', default=False, help="Disables event log feature.") ] CONF.register_opts(event_log_opts) def add_successful_event(instance): if CONF.disable_event_log: return cluster_id = instance.cluster_id step_id = get_current_provisioning_step(cluster_id) if step_id: conductor.cluster_event_add(context.ctx(), step_id, { 'successful': True, 'node_group_id': instance.node_group_id, 'instance_id': instance.instance_id, 'instance_name': instance.instance_name, 'event_info': None, }) def add_fail_event(instance, exception): if CONF.disable_event_log: return cluster_id = instance.cluster_id step_id = get_current_provisioning_step(cluster_id) event_info = six.text_type(exception) if step_id: conductor.cluster_event_add(context.ctx(), step_id, { 'successful': False, 'node_group_id': instance.node_group_id, 'instance_id': instance.instance_id, 'instance_name': instance.instance_name, 'event_info': event_info, }) def add_provisioning_step(cluster_id, step_name, total): if (CONF.disable_event_log or not cluster_utils.check_cluster_exists(cluster_id)): return prev_step = get_current_provisioning_step(cluster_id) if prev_step: conductor.cluster_provision_step_update(context.ctx(), prev_step) step_type = context.ctx().current_instance_info.step_type new_step = conductor.cluster_provision_step_add( context.ctx(), cluster_id, { 'step_name': step_name, 'step_type': step_type, 'total': total, 'started_at': timeutils.utcnow(), }) context.current().current_instance_info.step_id = new_step return new_step def get_current_provisioning_step(cluster_id): if (CONF.disable_event_log or not cluster_utils.check_cluster_exists(cluster_id)): return None current_instance_info = context.ctx().current_instance_info return current_instance_info.step_id def event_wrapper(mark_successful_on_exit, **spec): """"General event-log wrapper :param mark_successful_on_exit: should we send success event after execution of function :param spec: extra specification :parameter step: provisioning step name (only for provisioning steps with only one event) :parameter param: tuple (name, pos) with parameter specification, where 'name' is the name of the parameter of function, 'pos' is the position of the parameter of function. This parameter is used to extract info about Instance or Cluster. """ def decorator(func): @functools.wraps(func) def handler(*args, **kwargs): if CONF.disable_event_log: return func(*args, **kwargs) step_name = spec.get('step', None) instance = _find_in_args(spec, *args, **kwargs) cluster_id = instance.cluster_id if not cluster_utils.check_cluster_exists(cluster_id): return func(*args, **kwargs) if step_name: # It's single process, let's add provisioning step here add_provisioning_step(cluster_id, step_name, 1) try: value = func(*args, **kwargs) except Exception as e: with excutils.save_and_reraise_exception(): add_fail_event(instance, e) if mark_successful_on_exit: add_successful_event(instance) return value return handler return decorator def _get_info_from_instance(arg): if isinstance(arg, resource.InstanceResource): return arg return None def _get_info_from_cluster(arg): if isinstance(arg, resource.ClusterResource): return context.InstanceInfo(arg.id) return None def _get_event_info(arg): try: return arg.get_event_info() except AttributeError: return None def _get_info_from_obj(arg): functions = [_get_info_from_instance, _get_info_from_cluster, _get_event_info] for func in functions: value = func(arg) if value: return value return None def _find_in_args(spec, *args, **kwargs): param_values = spec.get('param', None) if param_values: p_name, p_pos = param_values obj = kwargs.get(p_name, None) if obj: return _get_info_from_obj(obj) return _get_info_from_obj(args[p_pos]) # If param is not specified, let's search instance in args for arg in args: val = _get_info_from_instance(arg) if val: return val for arg in kwargs.values(): val = _get_info_from_instance(arg) if val: return val # If instance not found in args, let's get instance info from context return context.ctx().current_instance_info sahara-8.0.0/sahara/utils/openstack/0000775000175100017510000000000013245515027017374 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/openstack/manila.py0000666000175100017510000000454213245514472021221 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import manilaclient.client as manila_client try: from manilaclient.common.apiclient import exceptions as manila_ex except ImportError: from manilaclient.openstack.common.apiclient import exceptions as manila_ex from oslo_config import cfg from sahara import context from sahara import exceptions as ex from sahara.i18n import _ from sahara.utils.openstack import base opts = [ cfg.StrOpt('api_version', default='1', help='Version of the manila API to use.'), cfg.BoolOpt('api_insecure', default=True, help='Allow to perform insecure SSL requests to manila.'), cfg.StrOpt('ca_file', help='Location of ca certificates file to use for manila ' 'client requests.') ] manila_group = cfg.OptGroup(name='manila', title='Manila client options') CONF = cfg.CONF CONF.register_group(manila_group) CONF.register_opts(opts, group=manila_group) MANILA_PREFIX = "manila://" def client(): ctx = context.ctx() args = { 'username': ctx.username, 'project_name': ctx.tenant_name, 'project_id': ctx.tenant_id, 'input_auth_token': context.get_auth_token(), 'auth_url': base.retrieve_auth_url(), 'service_catalog_url': base.url_for(ctx.service_catalog, 'share'), 'ca_cert': CONF.manila.ca_file, 'insecure': CONF.manila.api_insecure } return manila_client.Client(CONF.manila.api_version, **args) def get_share(client_instance, share_id, raise_on_error=False): try: return client_instance.shares.get(share_id) except manila_ex.NotFound: if raise_on_error: raise ex.NotFoundException( share_id, _("Share with id %s was not found.")) else: return None sahara-8.0.0/sahara/utils/openstack/cinder.py0000666000175100017510000000677113245514472021232 0ustar zuulzuul00000000000000# -*- coding: utf-8 -*- # Copyright (c) 2013 Mirantis Inc. # Copyright (c) 2014 Adrien Vergé # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from cinderclient.v2 import client as cinder_client_v2 from cinderclient.v3 import client as cinder_client_v3 from keystoneauth1 import exceptions as keystone_exceptions from oslo_config import cfg from oslo_log import log as logging from sahara import context from sahara.service import sessions from sahara.utils.openstack import base from sahara.utils.openstack import keystone LOG = logging.getLogger(__name__) opts = [ cfg.IntOpt('api_version', default=3, help='Version of the Cinder API to use.', deprecated_name='cinder_api_version'), cfg.BoolOpt('api_insecure', default=False, help='Allow to perform insecure SSL requests to cinder.'), cfg.StrOpt('ca_file', help='Location of ca certificates file to use for cinder ' 'client requests.'), cfg.StrOpt("endpoint_type", default="internalURL", help="Endpoint type for cinder client requests") ] cinder_group = cfg.OptGroup(name='cinder', title='Cinder client options') CONF = cfg.CONF CONF.register_group(cinder_group) CONF.register_opts(opts, group=cinder_group) def validate_config(): if CONF.cinder.api_version == 2: LOG.warning('The Cinder v2 API is deprecated. You should set ' 'cinder.api_version=3 in your sahara.conf file.') elif CONF.cinder.api_version != 3: LOG.warning('Unsupported Cinder API version: {bad}. Please set a ' 'correct value for cinder.api_version in your ' 'sahara.conf file (currently supported versions are: ' '{supported}). Falling back to Cinder API version 3.' .format(bad=CONF.cinder.api_version, supported=[2, 3])) CONF.set_override('api_version', 3, group='cinder') def client(): session = sessions.cache().get_session(sessions.SESSION_TYPE_CINDER) auth = keystone.auth() if CONF.cinder.api_version == 2: cinder = cinder_client_v2.Client( session=session, auth=auth, endpoint_type=CONF.cinder.endpoint_type, region_name=CONF.os_region_name) else: cinder = cinder_client_v3.Client( session=session, auth=auth, endpoint_type=CONF.cinder.endpoint_type, region_name=CONF.os_region_name) return cinder def check_cinder_exists(): if CONF.cinder.api_version == 2: service_type = 'volumev2' else: service_type = 'volumev3' try: base.url_for(context.current().service_catalog, service_type, endpoint_type=CONF.cinder.endpoint_type) return True except keystone_exceptions.EndpointNotFound: return False def get_volume(volume_id): return base.execute_with_retries(client().volumes.get, volume_id) sahara-8.0.0/sahara/utils/openstack/swift.py0000666000175100017510000000705413245514472021115 0ustar zuulzuul00000000000000# Copyright (c) 2014 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import swiftclient from sahara import context from sahara.swift import swift_helper as sh from sahara.swift import utils as su from sahara.utils.openstack import base from sahara.utils.openstack import keystone as k opts = [ cfg.BoolOpt('api_insecure', default=False, help='Allow to perform insecure SSL requests to swift.'), cfg.StrOpt('ca_file', help='Location of ca certificates file to use for swift ' 'client requests.'), cfg.StrOpt("endpoint_type", default="internalURL", help="Endpoint type for swift client requests") ] swift_group = cfg.OptGroup(name='swift', title='Swift client options') CONF = cfg.CONF CONF.register_group(swift_group) CONF.register_opts(opts, group=swift_group) def client(username, password, trust_id=None): '''return a Swift client This will return a Swift client for the specified username scoped to the current context project, unless a trust identifier is specified. If a trust identifier is present then the Swift client will be created based on a preauthorized token generated by the username scoped to the trust identifier. :param username: The username for the Swift client :param password: The password associated with the username :param trust_id: A trust identifier for scoping the username (optional) :returns: A Swift client object ''' if trust_id: proxyauth = k.auth_for_proxy(username, password, trust_id) return client_from_token(k.token_from_auth(proxyauth)) else: return swiftclient.Connection( auth_version='3', cacert=CONF.swift.ca_file, insecure=CONF.swift.api_insecure, authurl=su.retrieve_auth_url(CONF.keystone.endpoint_type), user=username, key=password, tenant_name=sh.retrieve_tenant(), retries=CONF.retries.retries_number, retry_on_ratelimit=True, starting_backoff=CONF.retries.retry_after, max_backoff=CONF.retries.retry_after) def client_from_token(token=None): if not token: token = context.get_auth_token() '''return a Swift client authenticated from a token.''' return swiftclient.Connection(auth_version='3', cacert=CONF.swift.ca_file, insecure=CONF.swift.api_insecure, preauthurl=base.url_for( service_type="object-store", endpoint_type=CONF.swift.endpoint_type), preauthtoken=token, retries=CONF.retries.retries_number, retry_on_ratelimit=True, starting_backoff=CONF.retries.retry_after, max_backoff=CONF.retries.retry_after) sahara-8.0.0/sahara/utils/openstack/keystone.py0000666000175100017510000002340713245514472021622 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re from keystoneauth1 import identity as keystone_identity from keystoneclient.v2_0 import client as keystone_client from keystoneclient.v3 import client as keystone_client_v3 from oslo_config import cfg from oslo_log import log as logging from sahara import context from sahara.service import sessions from sahara.utils.openstack import base LOG = logging.getLogger(__name__) def _get_keystoneauth_cfg(name): """get the keystone auth cfg Fetch value of keystone_authtoken group from config file when not available as part of GroupAttr. :rtype: String :param name: property name to be retrieved """ try: value_list = CONF._namespace._get_file_value([('keystone_authtoken', name)]) cfg_val = value_list[0] if name == "auth_url" and not re.findall(r'\/v[2-3].*', cfg_val): cfg_val += "/v3" return cfg_val except KeyError: if name in ["user_domain_name", "project_domain_name"]: return "Default" else: raise def validate_config(): if any(map(lambda o: getattr(CONF.trustee, o) is None, CONF.trustee)): for replace_opt in CONF.trustee: CONF.set_override(replace_opt, _get_keystoneauth_cfg(replace_opt), group="trustee") LOG.warning(""" __ __ _ \ \ / /_ _ _ __ _ __ (_)_ __ __ _ \ \ /\ / / _` | '__| '_ \| | '_ \ / _` | \ V V / (_| | | | | | | | | | | (_| | \_/\_/ \__,_|_| |_| |_|_|_| |_|\__, | |___/ Using the [keystone_authtoken] user as the Sahara trustee user directly is deprecated. Please add the trustee credentials you need to the [trustee] section of your sahara.conf file. """) opts = [ # TODO(alazarev) Move to [keystone] section cfg.BoolOpt('use_identity_api_v3', default=True, help='Enables Sahara to use Keystone API v3. ' 'If that flag is disabled, ' 'per-job clusters will not be terminated ' 'automatically.') ] ssl_opts = [ cfg.BoolOpt('api_insecure', default=False, help='Allow to perform insecure SSL requests to keystone.'), cfg.StrOpt('ca_file', help='Location of ca certificates file to use for keystone ' 'client requests.'), cfg.StrOpt("endpoint_type", default="internalURL", help="Endpoint type for keystone client requests") ] keystone_group = cfg.OptGroup(name='keystone', title='Keystone client options') trustee_opts = [ cfg.StrOpt('username', help='Username for trusts creation'), cfg.StrOpt('password', help='Password for trusts creation'), cfg.StrOpt('project_name', help='Project name for trusts creation'), cfg.StrOpt('user_domain_name', help='User domain name for trusts creation', default="Default"), cfg.StrOpt('project_domain_name', help='Project domain name for trusts creation', default="Default"), cfg.StrOpt('auth_url', help='Auth url for trusts creation'), ] trustee_group = cfg.OptGroup(name='trustee', title="Trustee options") CONF = cfg.CONF CONF.register_group(keystone_group) CONF.register_group(trustee_group) CONF.register_opts(opts) CONF.register_opts(ssl_opts, group=keystone_group) CONF.register_opts(trustee_opts, group=trustee_group) def auth(): '''Return a token auth plugin for the current context.''' ctx = context.current() return ctx.auth_plugin or token_auth(token=context.get_auth_token(), project_id=ctx.tenant_id) def auth_for_admin(project_name=None, trust_id=None): '''Return an auth plugin for the admin. :param project_name: a project to scope the auth with (optional). :param trust_id: a trust to scope the auth with (optional). :returns: an auth plugin object for the admin. ''' # TODO(elmiko) revisit the project_domain_name if we start getting # into federated authentication. it will need to match the domain that # the project_name exists in. auth = _password_auth( username=CONF.trustee.username, password=CONF.trustee.password, project_name=project_name, user_domain_name=CONF.trustee.user_domain_name, project_domain_name=CONF.trustee.project_domain_name, trust_id=trust_id) return auth def auth_for_proxy(username, password, trust_id=None): '''Return an auth plugin for the proxy user. :param username: the name of the proxy user. :param password: the proxy user's password. :param trust_id: a trust to scope the auth with (optional). :returns: an auth plugin object for the proxy user. ''' auth = _password_auth( username=username, password=password, user_domain_name=CONF.proxy_user_domain_name, trust_id=trust_id) return auth def client(): '''Return the current context client.''' return client_from_auth(auth()) def client_for_admin(): '''Return the Sahara admin user client.''' auth = auth_for_admin( project_name=CONF.trustee.project_name) return client_from_auth(auth) def client_from_auth(auth): '''Return a session based client from the auth plugin provided. A session is obtained from the global session cache. :param auth: the auth plugin object to use in client creation. :returns: a keystone client ''' session = sessions.cache().get_session(sessions.SESSION_TYPE_KEYSTONE) if CONF.use_identity_api_v3: client_class = keystone_client_v3.Client else: client_class = keystone_client.Client return client_class(session=session, auth=auth) def project_id_from_auth(auth): '''Return the project id associated with an auth plugin. :param auth: the auth plugin to inspect. :returns: the project id associated with the auth plugin. ''' return auth.get_project_id( sessions.cache().get_session(sessions.SESSION_TYPE_KEYSTONE)) def service_catalog_from_auth(auth): '''Return the service catalog associated with an auth plugin. :param auth: the auth plugin to inspect. :returns: a list containing the service catalog. ''' access_info = auth.get_access( sessions.cache().get_session(sessions.SESSION_TYPE_KEYSTONE)) if access_info.has_service_catalog(): return access_info.service_catalog.catalog else: return [] def token_auth(token, project_id=None, project_name=None, project_domain_name='Default'): '''Return a token auth plugin object. :param token: the token to use for authentication. :param project_id: the project(ex. tenant) id to scope the auth. :returns: a token auth plugin object. ''' token_kwargs = dict( auth_url=base.retrieve_auth_url(CONF.keystone.endpoint_type), token=token ) if CONF.use_identity_api_v3: token_kwargs.update(dict( project_id=project_id, project_name=project_name, project_domain_name=project_domain_name, )) auth = keystone_identity.v3.Token(**token_kwargs) else: token_kwargs.update(dict( tenant_id=project_id, tenant_name=project_name, )) auth = keystone_identity.v2.Token(**token_kwargs) return auth def token_from_auth(auth): '''Return an authentication token from an auth plugin. :param auth: the auth plugin to acquire a token from. :returns: an auth token in string format. ''' return sessions.cache().token_for_auth(auth) def user_id_from_auth(auth): '''Return a user id associated with an auth plugin. :param auth: the auth plugin to inspect. :returns: a token associated with the auth. ''' return auth.get_user_id(sessions.cache().get_session( sessions.SESSION_TYPE_KEYSTONE)) def _password_auth(username, password, project_name=None, user_domain_name=None, project_domain_name=None, trust_id=None): '''Return a password auth plugin object. :param username: the user to authenticate as. :param password: the user's password. :param project_name: the project(ex. tenant) name to scope the auth. :param user_domain_name: the domain the user belongs to. :param project_domain_name: the domain the project belongs to. :param trust_id: a trust id to scope the auth. :returns: a password auth plugin object. ''' passwd_kwargs = dict( auth_url=CONF.trustee.auth_url, username=username, password=password ) if CONF.use_identity_api_v3: passwd_kwargs.update(dict( project_name=project_name, user_domain_name=user_domain_name, project_domain_name=project_domain_name, trust_id=trust_id )) auth = keystone_identity.v3.Password(**passwd_kwargs) else: passwd_kwargs.update(dict( tenant_name=project_name, trust_id=trust_id )) auth = keystone_identity.v2.Password(**passwd_kwargs) return auth sahara-8.0.0/sahara/utils/openstack/base.py0000666000175100017510000001025613245514472020671 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re from keystoneauth1.access import service_catalog as keystone_service_catalog from keystoneauth1 import exceptions as keystone_ex from oslo_config import cfg from oslo_log import log as logging from oslo_serialization import jsonutils as json from six.moves.urllib import parse as urlparse from sahara import context from sahara import exceptions as ex LOG = logging.getLogger(__name__) # List of the errors, that can be retried ERRORS_TO_RETRY = [408, 413, 429, 500, 502, 503, 504] opts = [ cfg.IntOpt('retries_number', default=5, help='Number of times to retry the request to client before ' 'failing'), cfg.IntOpt('retry_after', default=10, help='Time between the retries to client (in seconds).') ] retries = cfg.OptGroup(name='retries', title='OpenStack clients calls retries') CONF = cfg.CONF CONF.register_group(retries) CONF.register_opts(opts, group=retries) def url_for(service_catalog=None, service_type='identity', endpoint_type="internalURL"): if not service_catalog: service_catalog = context.current().service_catalog try: return keystone_service_catalog.ServiceCatalogV2( json.loads(service_catalog)).url_for( service_type=service_type, interface=endpoint_type, region_name=CONF.os_region_name) except keystone_ex.EndpointNotFound: return keystone_service_catalog.ServiceCatalogV3( json.loads(service_catalog)).url_for( service_type=service_type, interface=endpoint_type, region_name=CONF.os_region_name) def prepare_auth_url(auth_url, version): info = urlparse.urlparse(auth_url) url_path = info.path.rstrip("/") # replacing current api version to empty string url_path = re.sub('/(v3/auth|v3|v2\.0)', '', url_path) url_path = (url_path + "/" + version).lstrip("/") return "%s://%s/%s" % (info[:2] + (url_path,)) def retrieve_auth_url(endpoint_type="internalURL", version=None): if not version: version = 'v3' if CONF.use_identity_api_v3 else 'v2.0' ctx = context.current() if ctx.service_catalog: auth_url = url_for(ctx.service_catalog, 'identity', endpoint_type) else: auth_url = CONF.trustee.auth_url return prepare_auth_url(auth_url, version) def execute_with_retries(method, *args, **kwargs): attempts = CONF.retries.retries_number + 1 while attempts > 0: try: return method(*args, **kwargs) except Exception as e: error_code = getattr(e, 'http_status', None) or getattr( e, 'status_code', None) or getattr(e, 'code', None) if error_code in ERRORS_TO_RETRY: LOG.warning('Occasional error occurred during "{method}" ' 'execution: {error_msg} ({error_code}). ' 'Operation will be retried.'.format( method=method.__name__, error_msg=e, error_code=error_code)) attempts -= 1 retry_after = getattr(e, 'retry_after', 0) context.sleep(max(retry_after, CONF.retries.retry_after)) else: LOG.debug('Permanent error occurred during "{method}" ' 'execution: {error_msg}.'.format( method=method.__name__, error_msg=e)) raise e else: attempts = CONF.retries.retries_number raise ex.MaxRetriesExceeded(attempts, method.__name__) sahara-8.0.0/sahara/utils/openstack/heat.py0000666000175100017510000001015613245514472020677 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from heatclient import client as heat_client from oslo_config import cfg from sahara import context from sahara import exceptions as ex from sahara.i18n import _ from sahara.service import sessions from sahara.utils.openstack import base from sahara.utils.openstack import keystone opts = [ cfg.BoolOpt('api_insecure', default=False, help='Allow to perform insecure SSL requests to heat.'), cfg.StrOpt('ca_file', help='Location of ca certificates file to use for heat ' 'client requests.'), cfg.StrOpt("endpoint_type", default="internalURL", help="Endpoint type for heat client requests") ] heat_group = cfg.OptGroup(name='heat', title='Heat client options') CONF = cfg.CONF CONF.register_group(heat_group) CONF.register_opts(opts, group=heat_group) def client(): ctx = context.ctx() session = sessions.cache().get_heat_session() heat_url = base.url_for(ctx.service_catalog, 'orchestration', endpoint_type=CONF.heat.endpoint_type) return heat_client.Client( '1', endpoint=heat_url, session=session, auth=keystone.auth(), region_name=CONF.os_region_name) def get_stack(stack_name, raise_on_missing=True): for stack in base.execute_with_retries( client().stacks.list, show_hidden=True, filters={'name': stack_name}): return stack if not raise_on_missing: return None raise ex.NotFoundException({'stack': stack_name}, _('Failed to find stack %(stack)s')) def delete_stack(cluster): stack_name = cluster.stack_name base.execute_with_retries(client().stacks.delete, stack_name) stack = get_stack(stack_name, raise_on_missing=False) while stack is not None: # Valid states: IN_PROGRESS, empty and COMPLETE if stack.status in ['IN_PROGRESS', '', 'COMPLETE']: context.sleep(5) else: raise ex.HeatStackException( message=_( "Cannot delete heat stack {name}, reason: " "stack status: {status}, status reason: {reason}").format( name=stack_name, status=stack.status, reason=stack.stack_status_reason)) stack = get_stack(stack_name, raise_on_missing=False) def abandon_stack(cluster): '''Put stack in deleting state if not already, then abandon it.''' heat_client = client() stack_name = cluster.stack_name base.execute_with_retries(heat_client.stacks.delete, stack_name) # TODO(jfreud): sleep between calls? base.execute_with_retries(heat_client.stacks.abandon, stack_name) def get_stack_outputs(cluster): stack = get_stack(cluster.stack_name) stack.get() return stack.outputs def _verify_completion(stack, is_update=False, last_update_time=None): # NOTE: expected empty status because status of stack # maybe is not set in heat database if stack.status in ['IN_PROGRESS', '']: return False if is_update and stack.status == 'COMPLETE': if stack.updated_time == last_update_time: return False return True def wait_stack_completion(cluster, is_update=False, last_updated_time=None): stack_name = cluster.stack_name stack = get_stack(stack_name) while not _verify_completion(stack, is_update, last_updated_time): context.sleep(1) stack = get_stack(stack_name) if stack.status != 'COMPLETE': raise ex.HeatStackException(stack.stack_status_reason) sahara-8.0.0/sahara/utils/openstack/images.py0000666000175100017510000001363313245514472021226 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools import six from sahara.conductor import resource from sahara import exceptions as exc from sahara.utils.openstack import glance PROP_DESCR = '_sahara_description' PROP_USERNAME = '_sahara_username' PROP_TAG = '_sahara_tag_' PROP_ALL_TAGS = '_all_tags' def image_manager(): return SaharaImageManager() def wrap_entity(func): @functools.wraps(func) def handle(*args, **kwargs): res = func(*args, **kwargs) if isinstance(res, list): images = [] for image in res: image = _transform_image_props(image) images.append(resource.ImageResource(image)) return images else: res = _transform_image_props(res) return resource.ImageResource(res) return handle def _get_all_tags(image_props): tags = [] for key, value in image_props.iteritems(): if key.startswith(PROP_TAG) and value: tags.append(key) return tags def _get_meta_prop(image_props, prop, default=None): if PROP_ALL_TAGS == prop: return _get_all_tags(image_props) return image_props.get(prop, default) def _parse_tags(image_props): tags = _get_meta_prop(image_props, PROP_ALL_TAGS) return [t.replace(PROP_TAG, "") for t in tags] def _serialize_metadata(image): data = {} for key, value in image.iteritems(): if key.startswith('_sahara') and value: data[key] = value return data def _get_compat_values(image): data = {} # TODO(vgridnev): Drop these values from APIv2 data["OS-EXT-IMG-SIZE:size"] = image.size data['metadata'] = _serialize_metadata(image) data["minDisk"] = getattr(image, 'min_disk', 0) data["minRam"] = getattr(image, 'min_ram', 0) data["progress"] = getattr(image, 'progress', 100) data["status"] = image.status.upper() data['created'] = image.created_at data['updated'] = image.updated_at return data def _transform_image_props(image): data = _get_compat_values(image) data['username'] = _get_meta_prop(image, PROP_USERNAME, "") data['description'] = _get_meta_prop(image, PROP_DESCR, "") data['tags'] = _parse_tags(image) data['id'] = image.id data["name"] = image.name return data def _ensure_tags(tags): if not tags: return [] return [tags] if isinstance(tags, six.string_types) else tags class SaharaImageManager(object): """SaharaImageManager This class is intermediate layer between sahara and glanceclient.v2.images. It provides additional sahara properties for image such as description, image tags and image username. """ def __init__(self): self.client = glance.client().images @wrap_entity def get(self, image_id): image = self.client.get(image_id) return image @wrap_entity def find(self, **kwargs): images = self.client.list(**kwargs) num_matches = len(images) if num_matches == 0: raise exc.NotFoundException(kwargs, "No images matching %s.") elif num_matches > 1: raise exc.NoUniqueMatchException(response=images, query=kwargs) else: return images[0] @wrap_entity def list(self): return list(self.client.list()) def set_meta(self, image_id, meta): self.client.update(image_id, remove_props=None, **meta) def delete_meta(self, image_id, meta_list): self.client.update(image_id, remove_props=meta_list) def set_image_info(self, image_id, username, description=None): """Sets human-readable information for image. For example: Ubuntu 15 x64 with Java 1.7 and Apache Hadoop 2.1, ubuntu """ meta = {PROP_USERNAME: username} if description: meta[PROP_DESCR] = description self.set_meta(image_id, meta) def unset_image_info(self, image_id): """Unsets all Sahara-related information. It removes username, description and tags from the specified image. """ image = self.get(image_id) meta = [PROP_TAG + tag for tag in image.tags] if image.description is not None: meta += [PROP_DESCR] if image.username is not None: meta += [PROP_USERNAME] self.delete_meta(image_id, meta) def tag(self, image_id, tags): """Adds tags to the specified image.""" tags = _ensure_tags(tags) self.set_meta(image_id, {PROP_TAG + tag: 'True' for tag in tags}) def untag(self, image_id, tags): """Removes tags from the specified image.""" tags = _ensure_tags(tags) self.delete_meta(image_id, [PROP_TAG + tag for tag in tags]) def list_by_tags(self, tags): """Returns images having all of the specified tags.""" tags = _ensure_tags(tags) return [i for i in self.list() if set(tags).issubset(i.tags)] def list_registered(self, name=None, tags=None): tags = _ensure_tags(tags) images_list = [i for i in self.list() if i.username and set(tags).issubset(i.tags)] if name: return [i for i in images_list if i.name == name] else: return images_list def get_registered_image(self, image_id): img = self.get(image_id) if img.username: return img else: raise exc.ImageNotRegistered(image_id) sahara-8.0.0/sahara/utils/openstack/__init__.py0000666000175100017510000000000013245514472021500 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/openstack/neutron.py0000666000175100017510000000744113245514472021453 0ustar zuulzuul00000000000000# Copyright (c) 2013 Hortonworks, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from neutronclient.common import exceptions as n_ex from neutronclient.neutron import client as neutron_cli from oslo_config import cfg from oslo_log import log as logging from sahara import exceptions as ex from sahara.i18n import _ from sahara.service import sessions from sahara.utils.openstack import base from sahara.utils.openstack import keystone opts = [ cfg.BoolOpt('api_insecure', default=False, help='Allow to perform insecure SSL requests to neutron.'), cfg.StrOpt('ca_file', help='Location of ca certificates file to use for neutron ' 'client requests.'), cfg.StrOpt("endpoint_type", default="internalURL", help="Endpoint type for neutron client requests") ] neutron_group = cfg.OptGroup(name='neutron', title='Neutron client options') CONF = cfg.CONF CONF.register_group(neutron_group) CONF.register_opts(opts, group=neutron_group) LOG = logging.getLogger(__name__) def client(auth=None): if not auth: auth = keystone.auth() session = sessions.cache().get_session(sessions.SESSION_TYPE_NEUTRON) neutron = neutron_cli.Client('2.0', session=session, auth=auth, endpoint_type=CONF.neutron.endpoint_type, region_name=CONF.os_region_name) return neutron class NeutronClient(object): neutron = None routers = {} def __init__(self, network, token, tenant_name, auth=None): if not auth: auth = keystone.token_auth(token=token, project_name=tenant_name) self.neutron = client(auth) self.network = network def get_router(self): matching_router = NeutronClient.routers.get(self.network, None) if matching_router: LOG.debug('Returning cached qrouter') return matching_router['id'] routers = self.neutron.list_routers()['routers'] for router in routers: device_id = router['id'] ports = base.execute_with_retries( self.neutron.list_ports, device_id=device_id)['ports'] port = next((port for port in ports if port['network_id'] == self.network), None) if port: matching_router = router NeutronClient.routers[self.network] = matching_router break if not matching_router: raise ex.SystemError(_('Neutron router corresponding to network ' '%s is not found') % self.network) return matching_router['id'] def get_private_network_cidrs(cluster): neutron_client = client() private_net = base.execute_with_retries(neutron_client.show_network, cluster.neutron_management_network) cidrs = [] for subnet_id in private_net['network']['subnets']: subnet = base.execute_with_retries( neutron_client.show_subnet, subnet_id) cidrs.append(subnet['subnet']['cidr']) return cidrs def get_network(id): try: return base.execute_with_retries( client().find_resource_by_id, 'network', id) except n_ex.NotFound: return None sahara-8.0.0/sahara/utils/openstack/glance.py0000666000175100017510000000313013245514472021201 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from glanceclient import client as glance_client from oslo_config import cfg from sahara.service import sessions from sahara.utils.openstack import keystone opts = [ cfg.BoolOpt('api_insecure', default=False, help='Allow to perform insecure SSL requests to glance.'), cfg.StrOpt('ca_file', help='Location of ca certificates file to use for glance ' 'client requests.'), cfg.StrOpt("endpoint_type", default="internalURL", help="Endpoint type for glance client requests"), ] glance_group = cfg.OptGroup(name='glance', title='Glance client options') CONF = cfg.CONF CONF.register_group(glance_group) CONF.register_opts(opts, group=glance_group) def client(): session = sessions.cache().get_session(sessions.SESSION_TYPE_GLANCE) glance = glance_client.Client('2', session=session, auth=keystone.auth(), interface=CONF.glance.endpoint_type) return glance sahara-8.0.0/sahara/utils/openstack/nova.py0000666000175100017510000000357713245514472020732 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from novaclient import client as nova_client from oslo_config import cfg from sahara.service import sessions import sahara.utils.openstack.base as base from sahara.utils.openstack import keystone opts = [ cfg.BoolOpt('api_insecure', default=False, help='Allow to perform insecure SSL requests to nova.'), cfg.StrOpt('ca_file', help='Location of ca certificates file to use for nova ' 'client requests.'), cfg.StrOpt("endpoint_type", default="internalURL", help="Endpoint type for nova client requests") ] nova_group = cfg.OptGroup(name='nova', title='Nova client options') CONF = cfg.CONF CONF.register_group(nova_group) CONF.register_opts(opts, group=nova_group) def client(): session = sessions.cache().get_session(sessions.SESSION_TYPE_NOVA) nova = nova_client.Client('2', session=session, auth=keystone.auth(), endpoint_type=CONF.nova.endpoint_type, region_name=CONF.os_region_name) return nova def get_flavor(**kwargs): return base.execute_with_retries(client().flavors.find, **kwargs) def get_instance_info(instance): return base.execute_with_retries( client().servers.get, instance.instance_id) sahara-8.0.0/sahara/utils/resources.py0000666000175100017510000000412513245514472020000 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import inspect import six class BaseResource(object): __resource_name__ = 'base' __filter_cols__ = [] @property def dict(self): return self.to_dict() @property def wrapped_dict(self): return {self.__resource_name__: self.dict} @property def __all_filter_cols__(self): cls = self.__class__ if not hasattr(cls, '__mro_filter_cols__'): filter_cols = [] for base_cls in inspect.getmro(cls): filter_cols += getattr(base_cls, '__filter_cols__', []) cls.__mro_filter_cols__ = set(filter_cols) return cls.__mro_filter_cols__ def _filter_field(self, k): return k == '_sa_instance_state' or k in self.__all_filter_cols__ def to_dict(self): dictionary = self.__dict__.copy() return {k: v for k, v in six.iteritems(dictionary) if not self._filter_field(k)} def as_resource(self): return Resource(self.__resource_name__, self.to_dict()) class Resource(BaseResource): def __init__(self, _name, _info): self._name = _name self.__resource_name__ = _name self._info = _info def __getattr__(self, k): if k not in self.__dict__: return self._info.get(k) return self.__dict__[k] def __repr__(self): return '<%s %s>' % (self._name, self._info) def __eq__(self, other): return self._name == other._name and self._info == other._info def to_dict(self): return self._info.copy() sahara-8.0.0/sahara/utils/hacking/0000775000175100017510000000000013245515027017011 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/hacking/commit_message.py0000666000175100017510000000615213245514472022370 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import os import re import subprocess # nosec from hacking import core class GitCheck(core.GlobalCheck): """Base-class for Git related checks.""" def _get_commit_title(self): # Check if we're inside a git checkout try: subp = subprocess.Popen( # nosec ['git', 'rev-parse', '--show-toplevel'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) gitdir = subp.communicate()[0].rstrip() except OSError: # "git" was not found return None if not os.path.exists(gitdir): return None # Get title of most recent commit subp = subprocess.Popen( # nosec ['git', 'log', '--no-merges', '--pretty=%s', '-1'], stdout=subprocess.PIPE) title = subp.communicate()[0] if subp.returncode: raise Exception("git log failed with code %s" % subp.returncode) return title.decode('utf-8') class OnceGitCheckCommitTitleBug(GitCheck): """Check git commit messages for bugs. OpenStack HACKING recommends not referencing a bug or blueprint in first line. It should provide an accurate description of the change S364 """ name = "GitCheckCommitTitleBug" # From https://github.com/openstack/openstack-ci-puppet # /blob/master/modules/gerrit/manifests/init.pp#L74 # Changeid|bug|blueprint GIT_REGEX = re.compile( r'(I[0-9a-f]{8,40})|' '([Bb]ug|[Ll][Pp])[\s\#:]*(\d+)|' '([Bb]lue[Pp]rint|[Bb][Pp])[\s\#:]*([A-Za-z0-9\\-]+)') def run_once(self): title = self._get_commit_title() # NOTE(jogo) if match regex but over 3 words, acceptable title if (title and self.GIT_REGEX.search(title) is not None and len(title.split()) <= 3): return (1, 0, "S364: git commit title ('%s') should provide an accurate " "description of the change, not just a reference to a bug " "or blueprint" % title.strip(), self.name) class OnceGitCheckCommitTitleLength(GitCheck): """Check git commit message length. HACKING recommends commit titles 50 chars or less, but enforces a 72 character limit S365 Title limited to 72 chars """ name = "GitCheckCommitTitleLength" def run_once(self): title = self._get_commit_title() if title and len(title) > 72: return ( 1, 0, "S365: git commit title ('%s') should be under 50 chars" % title.strip(), self.name) sahara-8.0.0/sahara/utils/hacking/import_checks.py0000666000175100017510000005501713245514472022232 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import imp from hacking import core # NOTE(Kezar): This checks a good enough if we have only py2.7 supported. # As soon as we'll get py3.x we need to drop it or rewrite. You can read more # about it in dev-list archive, topic: "[hacking]proposed rules drop for 1.0" def _find_module(module, path=None): mod_base = module parent_path = None while '.' in mod_base: first, _, mod_base = mod_base.partition('.') parent_path = path _, path, _ = imp.find_module(first, path) path = [path] try: _, path, _ = imp.find_module(mod_base, path) except ImportError: # NOTE(bnemec): There are two reasons we might get here: 1) A # non-module import and 2) an import of a namespace module that is # in the same namespace as the current project, which caused us to # recurse into the project namespace but fail to find the third-party # module. For 1), we won't be able to import it as a module, so we # return the parent module's path, but for 2) the import below should # succeed, so we re-raise the ImportError because the module was # legitimately not found in this path. try: __import__(module) except ImportError: # Non-module import, return the parent path if we have it if parent_path: return parent_path raise raise return path module_cache = dict() # List of all Python 2 stdlib modules - anything not in this list will be # allowed in either the stdlib or third-party groups to allow for Python 3 # stdlib additions. # The list was generated via the following script, which is a variation on # the one found here: # http://stackoverflow.com/questions/6463918/how-can-i-get-a-list-of-all-the-python-standard-library-modules """ from distutils import sysconfig import os import sys std_lib = sysconfig.get_python_lib(standard_lib=True) prefix_len = len(std_lib) + 1 modules = '' line = '[' mod_list = [] for top, dirs, files in os.walk(std_lib): for name in files: if 'site-packages' not in top: if name == '__init__.py': full_name = top[prefix_len:].replace('/', '.') mod_list.append(full_name) elif name.endswith('.py'): full_name = top.replace('/', '.') + '.' full_name += name[:-3] full_name = full_name[prefix_len:] mod_list.append(full_name) elif name.endswith('.so') and top.endswith('lib-dynload'): full_name = name[:-3] if full_name.endswith('module'): full_name = full_name[:-6] mod_list.append(full_name) for name in sys.builtin_module_names: mod_list.append(name) mod_list.sort() for mod in mod_list: if len(line + mod) + 8 > 79: modules += '\n' + line line = ' ' line += "'%s', " % mod print modules + ']' """ py2_stdlib = [ 'BaseHTTPServer', 'Bastion', 'CGIHTTPServer', 'ConfigParser', 'Cookie', 'DocXMLRPCServer', 'HTMLParser', 'MimeWriter', 'Queue', 'SimpleHTTPServer', 'SimpleXMLRPCServer', 'SocketServer', 'StringIO', 'UserDict', 'UserList', 'UserString', '_LWPCookieJar', '_MozillaCookieJar', '__builtin__', '__future__', '__main__', '__phello__.foo', '_abcoll', '_ast', '_bisect', '_bsddb', '_codecs', '_codecs_cn', '_codecs_hk', '_codecs_iso2022', '_codecs_jp', '_codecs_kr', '_codecs_tw', '_collections', '_crypt', '_csv', '_ctypes', '_curses', '_curses_panel', '_elementtree', '_functools', '_hashlib', '_heapq', '_hotshot', '_io', '_json', '_locale', '_lsprof', '_multibytecodec', '_multiprocessing', '_osx_support', '_pyio', '_random', '_socket', '_sqlite3', '_sre', '_ssl', '_strptime', '_struct', '_symtable', '_sysconfigdata', '_threading_local', '_warnings', '_weakref', '_weakrefset', 'abc', 'aifc', 'antigravity', 'anydbm', 'argparse', 'array', 'ast', 'asynchat', 'asyncore', 'atexit', 'audiodev', 'audioop', 'base64', 'bdb', 'binascii', 'binhex', 'bisect', 'bsddb', 'bsddb.db', 'bsddb.dbobj', 'bsddb.dbrecio', 'bsddb.dbshelve', 'bsddb.dbtables', 'bsddb.dbutils', 'bz2', 'cPickle', 'cProfile', 'cStringIO', 'calendar', 'cgi', 'cgitb', 'chunk', 'cmath', 'cmd', 'code', 'codecs', 'codeop', 'collections', 'colorsys', 'commands', 'compileall', 'compiler', 'compiler.ast', 'compiler.consts', 'compiler.future', 'compiler.misc', 'compiler.pyassem', 'compiler.pycodegen', 'compiler.symbols', 'compiler.syntax', 'compiler.transformer', 'compiler.visitor', 'contextlib', 'cookielib', 'copy', 'copy_reg', 'crypt', 'csv', 'ctypes', 'ctypes._endian', 'ctypes.macholib', 'ctypes.macholib.dyld', 'ctypes.macholib.dylib', 'ctypes.macholib.framework', 'ctypes.util', 'ctypes.wintypes', 'curses', 'curses.ascii', 'curses.has_key', 'curses.panel', 'curses.textpad', 'curses.wrapper', 'datetime', 'dbhash', 'dbm', 'decimal', 'difflib', 'dircache', 'dis', 'distutils', 'distutils.archive_util', 'distutils.bcppcompiler', 'distutils.ccompiler', 'distutils.cmd', 'distutils.command', 'distutils.command.bdist', 'distutils.command.bdist_dumb', 'distutils.command.bdist_msi', 'distutils.command.bdist_rpm', 'distutils.command.bdist_wininst', 'distutils.command.build', 'distutils.command.build_clib', 'distutils.command.build_ext', 'distutils.command.build_py', 'distutils.command.build_scripts', 'distutils.command.check', 'distutils.command.clean', 'distutils.command.config', 'distutils.command.install', 'distutils.command.install_data', 'distutils.command.install_egg_info', 'distutils.command.install_headers', 'distutils.command.install_lib', 'distutils.command.install_scripts', 'distutils.command.register', 'distutils.command.sdist', 'distutils.command.upload', 'distutils.config', 'distutils.core', 'distutils.cygwinccompiler', 'distutils.debug', 'distutils.dep_util', 'distutils.dir_util', 'distutils.dist', 'distutils.emxccompiler', 'distutils.errors', 'distutils.extension', 'distutils.fancy_getopt', 'distutils.file_util', 'distutils.filelist', 'distutils.log', 'distutils.msvc9compiler', 'distutils.msvccompiler', 'distutils.spawn', 'distutils.sysconfig', 'distutils.text_file', 'distutils.unixccompiler', 'distutils.util', 'distutils.version', 'distutils.versionpredicate', 'dl', 'doctest', 'dumbdbm', 'dummy_thread', 'dummy_threading', 'email', 'email._parseaddr', 'email.base64mime', 'email.charset', 'email.encoders', 'email.errors', 'email.feedparser', 'email.generator', 'email.header', 'email.iterators', 'email.message', 'email.mime', 'email.mime.application', 'email.mime.audio', 'email.mime.base', 'email.mime.image', 'email.mime.message', 'email.mime.multipart', 'email.mime.nonmultipart', 'email.mime.text', 'email.parser', 'email.quoprimime', 'email.utils', 'encodings', 'encodings.aliases', 'encodings.ascii', 'encodings.base64_codec', 'encodings.big5', 'encodings.big5hkscs', 'encodings.bz2_codec', 'encodings.charmap', 'encodings.cp037', 'encodings.cp1006', 'encodings.cp1026', 'encodings.cp1140', 'encodings.cp1250', 'encodings.cp1251', 'encodings.cp1252', 'encodings.cp1253', 'encodings.cp1254', 'encodings.cp1255', 'encodings.cp1256', 'encodings.cp1257', 'encodings.cp1258', 'encodings.cp424', 'encodings.cp437', 'encodings.cp500', 'encodings.cp720', 'encodings.cp737', 'encodings.cp775', 'encodings.cp850', 'encodings.cp852', 'encodings.cp855', 'encodings.cp856', 'encodings.cp857', 'encodings.cp858', 'encodings.cp860', 'encodings.cp861', 'encodings.cp862', 'encodings.cp863', 'encodings.cp864', 'encodings.cp865', 'encodings.cp866', 'encodings.cp869', 'encodings.cp874', 'encodings.cp875', 'encodings.cp932', 'encodings.cp949', 'encodings.cp950', 'encodings.euc_jis_2004', 'encodings.euc_jisx0213', 'encodings.euc_jp', 'encodings.euc_kr', 'encodings.gb18030', 'encodings.gb2312', 'encodings.gbk', 'encodings.hex_codec', 'encodings.hp_roman8', 'encodings.hz', 'encodings.idna', 'encodings.iso2022_jp', 'encodings.iso2022_jp_1', 'encodings.iso2022_jp_2', 'encodings.iso2022_jp_2004', 'encodings.iso2022_jp_3', 'encodings.iso2022_jp_ext', 'encodings.iso2022_kr', 'encodings.iso8859_1', 'encodings.iso8859_10', 'encodings.iso8859_11', 'encodings.iso8859_13', 'encodings.iso8859_14', 'encodings.iso8859_15', 'encodings.iso8859_16', 'encodings.iso8859_2', 'encodings.iso8859_3', 'encodings.iso8859_4', 'encodings.iso8859_5', 'encodings.iso8859_6', 'encodings.iso8859_7', 'encodings.iso8859_8', 'encodings.iso8859_9', 'encodings.johab', 'encodings.koi8_r', 'encodings.koi8_u', 'encodings.latin_1', 'encodings.mac_arabic', 'encodings.mac_centeuro', 'encodings.mac_croatian', 'encodings.mac_cyrillic', 'encodings.mac_farsi', 'encodings.mac_greek', 'encodings.mac_iceland', 'encodings.mac_latin2', 'encodings.mac_roman', 'encodings.mac_romanian', 'encodings.mac_turkish', 'encodings.mbcs', 'encodings.palmos', 'encodings.ptcp154', 'encodings.punycode', 'encodings.quopri_codec', 'encodings.raw_unicode_escape', 'encodings.rot_13', 'encodings.shift_jis', 'encodings.shift_jis_2004', 'encodings.shift_jisx0213', 'encodings.string_escape', 'encodings.tis_620', 'encodings.undefined', 'encodings.unicode_escape', 'encodings.unicode_internal', 'encodings.utf_16', 'encodings.utf_16_be', 'encodings.utf_16_le', 'encodings.utf_32', 'encodings.utf_32_be', 'encodings.utf_32_le', 'encodings.utf_7', 'encodings.utf_8', 'encodings.utf_8_sig', 'encodings.uu_codec', 'encodings.zlib_codec', 'errno', 'exceptions', 'fcntl', 'filecmp', 'fileinput', 'fnmatch', 'formatter', 'fpformat', 'fractions', 'ftplib', 'functools', 'future_builtins', 'gc', 'gdbm', 'genericpath', 'getopt', 'getpass', 'gettext', 'glob', 'grp', 'gzip', 'hashlib', 'heapq', 'hmac', 'hotshot', 'hotshot.log', 'hotshot.stats', 'hotshot.stones', 'htmlentitydefs', 'htmllib', 'httplib', 'idlelib', 'idlelib.AutoComplete', 'idlelib.AutoCompleteWindow', 'idlelib.AutoExpand', 'idlelib.Bindings', 'idlelib.CallTipWindow', 'idlelib.CallTips', 'idlelib.ClassBrowser', 'idlelib.CodeContext', 'idlelib.ColorDelegator', 'idlelib.Debugger', 'idlelib.Delegator', 'idlelib.EditorWindow', 'idlelib.FileList', 'idlelib.FormatParagraph', 'idlelib.GrepDialog', 'idlelib.HyperParser', 'idlelib.IOBinding', 'idlelib.IdleHistory', 'idlelib.MultiCall', 'idlelib.MultiStatusBar', 'idlelib.ObjectBrowser', 'idlelib.OutputWindow', 'idlelib.ParenMatch', 'idlelib.PathBrowser', 'idlelib.Percolator', 'idlelib.PyParse', 'idlelib.PyShell', 'idlelib.RemoteDebugger', 'idlelib.RemoteObjectBrowser', 'idlelib.ReplaceDialog', 'idlelib.RstripExtension', 'idlelib.ScriptBinding', 'idlelib.ScrolledList', 'idlelib.SearchDialog', 'idlelib.SearchDialogBase', 'idlelib.SearchEngine', 'idlelib.StackViewer', 'idlelib.ToolTip', 'idlelib.TreeWidget', 'idlelib.UndoDelegator', 'idlelib.WidgetRedirector', 'idlelib.WindowList', 'idlelib.ZoomHeight', 'idlelib.aboutDialog', 'idlelib.configDialog', 'idlelib.configHandler', 'idlelib.configHelpSourceEdit', 'idlelib.configSectionNameDialog', 'idlelib.dynOptionMenuWidget', 'idlelib.idle', 'idlelib.idlever', 'idlelib.keybindingDialog', 'idlelib.macosxSupport', 'idlelib.rpc', 'idlelib.run', 'idlelib.tabbedpages', 'idlelib.textView', 'ihooks', 'imageop', 'imaplib', 'imghdr', 'imp', 'importlib', 'imputil', 'inspect', 'io', 'itertools', 'json', 'json.decoder', 'json.encoder', 'json.scanner', 'json.tool', 'keyword', 'lib2to3', 'lib2to3.__main__', 'lib2to3.btm_matcher', 'lib2to3.btm_utils', 'lib2to3.fixer_base', 'lib2to3.fixer_util', 'lib2to3.fixes', 'lib2to3.fixes.fix_apply', 'lib2to3.fixes.fix_basestring', 'lib2to3.fixes.fix_buffer', 'lib2to3.fixes.fix_callable', 'lib2to3.fixes.fix_dict', 'lib2to3.fixes.fix_except', 'lib2to3.fixes.fix_exec', 'lib2to3.fixes.fix_execfile', 'lib2to3.fixes.fix_exitfunc', 'lib2to3.fixes.fix_filter', 'lib2to3.fixes.fix_funcattrs', 'lib2to3.fixes.fix_future', 'lib2to3.fixes.fix_getcwdu', 'lib2to3.fixes.fix_has_key', 'lib2to3.fixes.fix_idioms', 'lib2to3.fixes.fix_import', 'lib2to3.fixes.fix_imports', 'lib2to3.fixes.fix_imports2', 'lib2to3.fixes.fix_input', 'lib2to3.fixes.fix_intern', 'lib2to3.fixes.fix_isinstance', 'lib2to3.fixes.fix_itertools', 'lib2to3.fixes.fix_itertools_imports', 'lib2to3.fixes.fix_long', 'lib2to3.fixes.fix_map', 'lib2to3.fixes.fix_metaclass', 'lib2to3.fixes.fix_methodattrs', 'lib2to3.fixes.fix_ne', 'lib2to3.fixes.fix_next', 'lib2to3.fixes.fix_nonzero', 'lib2to3.fixes.fix_numliterals', 'lib2to3.fixes.fix_operator', 'lib2to3.fixes.fix_paren', 'lib2to3.fixes.fix_print', 'lib2to3.fixes.fix_raise', 'lib2to3.fixes.fix_raw_input', 'lib2to3.fixes.fix_reduce', 'lib2to3.fixes.fix_renames', 'lib2to3.fixes.fix_repr', 'lib2to3.fixes.fix_set_literal', 'lib2to3.fixes.fix_standarderror', 'lib2to3.fixes.fix_sys_exc', 'lib2to3.fixes.fix_throw', 'lib2to3.fixes.fix_tuple_params', 'lib2to3.fixes.fix_types', 'lib2to3.fixes.fix_unicode', 'lib2to3.fixes.fix_urllib', 'lib2to3.fixes.fix_ws_comma', 'lib2to3.fixes.fix_xrange', 'lib2to3.fixes.fix_xreadlines', 'lib2to3.fixes.fix_zip', 'lib2to3.main', 'lib2to3.patcomp', 'lib2to3.pgen2', 'lib2to3.pgen2.conv', 'lib2to3.pgen2.driver', 'lib2to3.pgen2.grammar', 'lib2to3.pgen2.literals', 'lib2to3.pgen2.parse', 'lib2to3.pgen2.pgen', 'lib2to3.pgen2.token', 'lib2to3.pgen2.tokenize', 'lib2to3.pygram', 'lib2to3.pytree', 'lib2to3.refactor', 'linecache', 'linuxaudiodev', 'locale', 'logging', 'logging.config', 'logging.handlers', 'macpath', 'macurl2path', 'mailbox', 'mailcap', 'markupbase', 'marshal', 'math', 'md5', 'mhlib', 'mimetools', 'mimetypes', 'mimify', 'mmap', 'modulefinder', 'multifile', 'multiprocessing', 'multiprocessing.connection', 'multiprocessing.dummy', 'multiprocessing.dummy.connection', 'multiprocessing.forking', 'multiprocessing.heap', 'multiprocessing.managers', 'multiprocessing.pool', 'multiprocessing.process', 'multiprocessing.queues', 'multiprocessing.reduction', 'multiprocessing.sharedctypes', 'multiprocessing.synchronize', 'multiprocessing.util', 'mutex', 'netrc', 'new', 'nis', 'nntplib', 'ntpath', 'nturl2path', 'numbers', 'opcode', 'operator', 'optparse', 'os', 'os2emxpath', 'ossaudiodev', 'parser', 'pdb', 'pickle', 'pickletools', 'pipes', 'pkgutil', 'plat-linux2.CDROM', 'plat-linux2.DLFCN', 'plat-linux2.IN', 'plat-linux2.TYPES', 'platform', 'plistlib', 'popen2', 'poplib', 'posix', 'posixfile', 'posixpath', 'pprint', 'profile', 'pstats', 'pty', 'pwd', 'py_compile', 'pyclbr', 'pydoc', 'pydoc_data', 'pydoc_data.topics', 'pyexpat', 'quopri', 'random', 're', 'readline', 'repr', 'resource', 'rexec', 'rfc822', 'rlcompleter', 'robotparser', 'runpy', 'sched', 'select', 'sets', 'sgmllib', 'sha', 'shelve', 'shlex', 'shutil', 'signal', 'site', 'smtpd', 'smtplib', 'sndhdr', 'socket', 'spwd', 'sqlite3', 'sqlite3.dbapi2', 'sqlite3.dump', 'sre', 'sre_compile', 'sre_constants', 'sre_parse', 'ssl', 'stat', 'statvfs', 'string', 'stringold', 'stringprep', 'strop', 'struct', 'subprocess', 'sunau', 'sunaudio', 'symbol', 'symtable', 'sys', 'sysconfig', 'syslog', 'tabnanny', 'tarfile', 'telnetlib', 'tempfile', 'termios', 'test', 'test.test_support', 'textwrap', 'this', 'thread', 'threading', 'time', 'timeit', 'timing', 'toaiff', 'token', 'tokenize', 'trace', 'traceback', 'tty', 'types', 'unicodedata', 'unittest', 'unittest.__main__', 'unittest.case', 'unittest.loader', 'unittest.main', 'unittest.result', 'unittest.runner', 'unittest.signals', 'unittest.suite', 'unittest.test', 'unittest.test.dummy', 'unittest.test.support', 'unittest.test.test_assertions', 'unittest.test.test_break', 'unittest.test.test_case', 'unittest.test.test_discovery', 'unittest.test.test_functiontestcase', 'unittest.test.test_loader', 'unittest.test.test_program', 'unittest.test.test_result', 'unittest.test.test_runner', 'unittest.test.test_setups', 'unittest.test.test_skipping', 'unittest.test.test_suite', 'unittest.util', 'urllib', 'urllib2', 'urlparse', 'user', 'uu', 'uuid', 'warnings', 'wave', 'weakref', 'webbrowser', 'whichdb', 'wsgiref', 'wsgiref.handlers', 'wsgiref.headers', 'wsgiref.simple_server', 'wsgiref.util', 'wsgiref.validate', 'xdrlib', 'xml', 'xml.dom', 'xml.dom.NodeFilter', 'xml.dom.domreg', 'xml.dom.expatbuilder', 'xml.dom.minicompat', 'xml.dom.minidom', 'xml.dom.pulldom', 'xml.dom.xmlbuilder', 'xml.etree', 'xml.etree.ElementInclude', 'xml.etree.ElementPath', 'xml.etree.ElementTree', 'xml.etree.cElementTree', 'xml.parsers', 'xml.parsers.expat', 'xml.sax', 'xml.sax._exceptions', 'xml.sax.expatreader', 'xml.sax.handler', 'xml.sax.saxutils', 'xml.sax.xmlreader', 'xmllib', 'xmlrpclib', 'xxsubtype', 'zipfile', ] # Dynamic modules that can't be auto-discovered by the script above manual_stdlib = ['os.path', ] py2_stdlib.extend(manual_stdlib) def _get_import_type(module): if module in module_cache: return module_cache[module] def cache_type(module_type): module_cache[module] = module_type return module_type # Check static stdlib list if module in py2_stdlib: return cache_type('stdlib') # Check if the module is local try: _find_module(module, ['.']) # If the previous line succeeded then it must be a project module return cache_type('project') except ImportError: pass # Otherwise treat it as third-party - this means we may treat some stdlib # modules as third-party, but that's okay because we are allowing # third-party libs in the stdlib section. return cache_type('third-party') @core.flake8ext def hacking_import_groups(logical_line, blank_before, previous_logical, indent_level, previous_indent_level, physical_line, noqa): r"""Check that imports are grouped correctly. OpenStack HACKING guide recommendation for imports: imports grouped such that Python standard library imports are together, third party library imports are together, and project imports are together Okay: import os\nimport sys\n\nimport six\n\nimport hacking Okay: import six\nimport znon_existent_package Okay: import os\nimport threading S366: import mock\nimport os S366: import hacking\nimport os S366: import hacking\nimport nonexistent S366: import hacking\nimport mock """ if (noqa or blank_before > 0 or indent_level != previous_indent_level): return normalized_line = core.import_normalize(logical_line.strip()).split() normalized_previous = core.import_normalize(previous_logical. strip()).split() def compatible(previous, current): if previous == current: return True if normalized_line and normalized_line[0] == 'import': current_type = _get_import_type(normalized_line[1]) if normalized_previous and normalized_previous[0] == 'import': previous_type = _get_import_type(normalized_previous[1]) if not compatible(previous_type, current_type): yield(0, 'S366: imports not grouped correctly ' '(%s: %s, %s: %s)' % (normalized_previous[1], previous_type, normalized_line[1], current_type)) class ImportGroupData(object): """A class to hold persistent state data for import group checks. To verify import grouping, it is necessary to know the current group for the current file. This can not always be known solely from the current and previous line, so this class can be used to keep track. """ # NOTE(bnemec): *args is needed because the test code tries to run this # as a flake8 check and passes an argument to it. def __init__(self, *args): self.current_group = None self.current_filename = None self.current_import = None together_data = ImportGroupData() @core.flake8ext def hacking_import_groups_together(logical_line, blank_lines, indent_level, previous_indent_level, line_number, physical_line, filename, noqa): r"""Check that like imports are grouped together. OpenStack HACKING guide recommendation for imports: Imports should be grouped together by type. Okay: import os\nimport sys Okay: try:\n import foo\nexcept ImportError:\n pass\n\nimport six Okay: import abc\nimport mock\n\nimport six Okay: import eventlet\neventlet.monkey_patch()\n\nimport copy S367: import mock\n\nimport six S367: import os\n\nimport sys S367: import mock\nimport os\n\nimport sys """ if line_number == 1 or filename != together_data.current_filename: together_data.current_group = None together_data.current_filename = filename if noqa: return def update_current_group(current): together_data.current_group = current normalized_line = core.import_normalize(logical_line.strip()).split() if normalized_line: if normalized_line[0] == 'import': current_type = _get_import_type(normalized_line[1]) previous_import = together_data.current_import together_data.current_import = normalized_line[1] matched = current_type == together_data.current_group update_current_group(current_type) if (matched and indent_level == previous_indent_level and blank_lines >= 1): yield(0, 'S367: like imports should be grouped together (%s ' 'and %s from %s are separated by whitespace)' % (previous_import, together_data.current_import, current_type)) else: # Reset on non-import code together_data.current_group = None sahara-8.0.0/sahara/utils/hacking/__init__.py0000666000175100017510000000000013245514472021115 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/hacking/checks.py0000666000175100017510000001054713245514472020637 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import pep8 import re import tokenize from sahara.utils.hacking import commit_message from sahara.utils.hacking import import_checks from sahara.utils.hacking import logging_checks RE_OSLO_IMPORTS = (re.compile(r"(((from)|(import))\s+oslo\.)"), re.compile(r"(from\s+oslo\s+import)")) RE_DICT_CONSTRUCTOR_WITH_LIST_COPY = re.compile(r".*\bdict\((\[)?(\(|\[)") RE_USE_JSONUTILS_INVALID_LINE = re.compile(r"(import\s+json)") RE_USE_JSONUTILS_VALID_LINE = re.compile(r"(import\s+jsonschema)") RE_MUTABLE_DEFAULT_ARGS = re.compile(r"^\s*def .+\((.+=\{\}|.+=\[\])") def _starts_with_any(line, *prefixes): for prefix in prefixes: if line.startswith(prefix): return True return False def _any_in(line, *sublines): for subline in sublines: if subline in line: return True return False def import_db_only_in_conductor(logical_line, filename): """Check that db calls are only in conductor module and in tests. S361 """ if _any_in(filename, "sahara/conductor", "sahara/tests", "sahara/db"): return if _starts_with_any(logical_line, "from sahara import db", "from sahara.db", "import sahara.db"): yield (0, "S361: sahara.db import only allowed in " "sahara/conductor/*") def hacking_no_author_attr(logical_line, tokens): """__author__ should not be used. S362: __author__ = slukjanov """ for token_type, text, start_index, _, _ in tokens: if token_type == tokenize.NAME and text == "__author__": yield (start_index[1], "S362: __author__ should not be used") def check_oslo_namespace_imports(logical_line): """Check to prevent old oslo namespace usage. S363 """ if re.match(RE_OSLO_IMPORTS[0], logical_line): yield(0, "S363: '%s' must be used instead of '%s'." % ( logical_line.replace('oslo.', 'oslo_'), logical_line)) if re.match(RE_OSLO_IMPORTS[1], logical_line): yield(0, "S363: '%s' must be used instead of '%s'" % ( 'import oslo_%s' % logical_line.split()[-1], logical_line)) def dict_constructor_with_list_copy(logical_line): """Check to prevent dict constructor with a sequence of key-value pairs. S368 """ if RE_DICT_CONSTRUCTOR_WITH_LIST_COPY.match(logical_line): yield (0, 'S368: Must use a dict comprehension instead of a dict ' 'constructor with a sequence of key-value pairs.') def use_jsonutils(logical_line, filename): """Check to prevent importing json in sahara code. S375 """ if pep8.noqa(logical_line): return if (RE_USE_JSONUTILS_INVALID_LINE.match(logical_line) and not RE_USE_JSONUTILS_VALID_LINE.match(logical_line)): yield(0, "S375: Use jsonutils from oslo_serialization instead" " of json") def no_mutable_default_args(logical_line): """Check to prevent mutable default argument in sahara code. S360 """ msg = "S360: Method's default argument shouldn't be mutable!" if RE_MUTABLE_DEFAULT_ARGS.match(logical_line): yield (0, msg) def factory(register): register(import_db_only_in_conductor) register(hacking_no_author_attr) register(check_oslo_namespace_imports) register(commit_message.OnceGitCheckCommitTitleBug) register(commit_message.OnceGitCheckCommitTitleLength) register(import_checks.hacking_import_groups) register(import_checks.hacking_import_groups_together) register(dict_constructor_with_list_copy) register(logging_checks.no_translate_logs) register(logging_checks.accepted_log_levels) register(use_jsonutils) register(no_mutable_default_args) sahara-8.0.0/sahara/utils/hacking/logging_checks.py0000666000175100017510000000420613245514472022340 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import re ALL_LOG_LEVELS = "info|exception|warning|critical|error|debug" RE_ACCEPTED_LOG_LEVELS = re.compile( r"(.)*LOG\.(%(levels)s)\(" % {'levels': ALL_LOG_LEVELS}) # Since _Lx() have been removed, we just need to check _() RE_TRANSLATED_LOG = re.compile( r"(.)*LOG\.(%(levels)s)\(\s*_\(" % {'levels': ALL_LOG_LEVELS}) def no_translate_logs(logical_line, filename): """Check for 'LOG.*(_(' Translators don't provide translations for log messages, and operators asked not to translate them. * This check assumes that 'LOG' is a logger. * Use filename so we can start enforcing this in specific folders instead of needing to do so all at once. S373 """ msg = "S373 Don't translate logs" if RE_TRANSLATED_LOG.match(logical_line): yield (0, msg) def accepted_log_levels(logical_line, filename): """In Sahara we use only 5 log levels. This check is needed because we don't want new contributors to use deprecated log levels. S374 """ # NOTE(Kezar): sahara/tests included because we don't require translations # in tests. sahara/db/templates provide separate cli interface so we don't # want to translate it. ignore_dirs = ["sahara/db/templates", "sahara/tests"] for directory in ignore_dirs: if directory in filename: return msg = ("S374 You used deprecated log level. Accepted log levels are " "%(levels)s" % {'levels': ALL_LOG_LEVELS}) if logical_line.startswith("LOG."): if not RE_ACCEPTED_LOG_LEVELS.search(logical_line): yield(0, msg) sahara-8.0.0/sahara/utils/procutils.py0000666000175100017510000000557213245514472020021 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import pickle # nosec import sys from eventlet.green import subprocess from eventlet import timeout as e_timeout from sahara import context from sahara import exceptions def _get_sub_executable(): return '%s/_sahara-subprocess' % os.path.dirname(sys.argv[0]) def start_subprocess(): return subprocess.Popen((sys.executable, _get_sub_executable()), close_fds=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) def run_in_subprocess(proc, func, args=None, kwargs=None, interactive=False): args = args or () kwargs = kwargs or {} try: # TODO(elmiko) these pickle usages should be reinvestigated to # determine a more secure manner to deploy remote commands. pickle.dump(func, proc.stdin) # nosec pickle.dump(args, proc.stdin) # nosec pickle.dump(kwargs, proc.stdin) # nosec proc.stdin.flush() if not interactive: result = pickle.load(proc.stdout) # nosec if 'exception' in result: raise exceptions.SubprocessException(result['exception']) return result['output'] finally: # NOTE(dmitryme): in oslo.concurrency's file processutils.py it # is suggested to sleep a little between calls to multiprocessing. # That should allow it make some necessary cleanup context.sleep(0) def _finish(cleanup_func): cleanup_func() sys.stdin.close() sys.stdout.close() sys.stderr.close() sys.exit(0) def shutdown_subprocess(proc, cleanup_func): try: with e_timeout.Timeout(5): # timeout would mean that our single-threaded subprocess # is hung on previous task which blocks _finish to complete run_in_subprocess(proc, _finish, (cleanup_func,)) except BaseException: # exception could be caused by either timeout, or # successful shutdown, ignoring anyway pass finally: kill_subprocess(proc) def kill_subprocess(proc): proc.stdin.close() proc.stdout.close() proc.stderr.close() try: proc.kill() proc.wait() except OSError: # could be caused by process already dead, so ignoring pass sahara-8.0.0/sahara/utils/wsgi.py0000666000175100017510000000502613245514472016740 0ustar zuulzuul00000000000000# Copyright 2011 OpenStack LLC. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # Only (de)serialization utils hasn't been removed to decrease requirements # number. """Utility methods for working with WSGI servers.""" import datetime from oslo_serialization import jsonutils import six from sahara import exceptions from sahara.i18n import _ class ActionDispatcher(object): """Maps method name to local methods through action name.""" def dispatch(self, *args, **kwargs): """Find and call local method.""" action = kwargs.pop('action', 'default') action_method = getattr(self, str(action), self.default) return action_method(*args, **kwargs) def default(self, data): raise NotImplementedError() class DictSerializer(ActionDispatcher): """Default request body serialization.""" def serialize(self, data, action='default'): return self.dispatch(data, action=action) def default(self, data): return "" class JSONDictSerializer(DictSerializer): """Default JSON request body serialization.""" def default(self, data): def sanitizer(obj): if isinstance(obj, datetime.datetime): _dtime = obj - datetime.timedelta(microseconds=obj.microsecond) return _dtime.isoformat() return six.text_type(obj) return jsonutils.dumps(data, default=sanitizer) class TextDeserializer(ActionDispatcher): """Default request body deserialization.""" def deserialize(self, datastring, action='default'): return self.dispatch(datastring, action=action) def default(self, datastring): return {} class JSONDeserializer(TextDeserializer): def _from_json(self, datastring): try: return jsonutils.loads(datastring) except ValueError: msg = _("cannot understand JSON") raise exceptions.MalformedRequestBody(msg) def default(self, datastring): return {'body': self._from_json(datastring)} sahara-8.0.0/sahara/utils/crypto.py0000666000175100017510000000405313245514472017306 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from oslo_concurrency import processutils import paramiko import six from sahara import exceptions as ex from sahara.i18n import _ from sahara.utils import tempfiles def to_paramiko_private_key(pkey): """Convert private key (str) to paramiko-specific RSAKey object.""" return paramiko.RSAKey(file_obj=six.StringIO(pkey)) def generate_key_pair(key_length=2048): """Create RSA key pair with specified number of bits in key. Returns tuple of private and public keys. """ with tempfiles.tempdir() as tmpdir: keyfile = os.path.join(tmpdir, 'tempkey') args = [ 'ssh-keygen', '-q', # quiet '-N', '', # w/o passphrase '-t', 'rsa', # create key of rsa type '-f', keyfile, # filename of the key file '-C', 'Generated-by-Sahara' # key comment ] if key_length is not None: args.extend(['-b', key_length]) processutils.execute(*args) if not os.path.exists(keyfile): raise ex.SystemError(_("Private key file hasn't been created")) with open(keyfile) as keyfile_fd: private_key = keyfile_fd.read() public_key_path = keyfile + '.pub' if not os.path.exists(public_key_path): raise ex.SystemError(_("Public key file hasn't been created")) with open(public_key_path) as public_key_path_fd: public_key = public_key_path_fd.read() return private_key, public_key sahara-8.0.0/sahara/utils/poll_utils.py0000666000175100017510000001253013245514472020153 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools from oslo_config import cfg from oslo_log import log as logging from oslo_utils import timeutils from sahara import context from sahara import exceptions as ex from sahara.utils import cluster as cluster_utils LOG = logging.getLogger(__name__) # set 3 hours timeout by default DEFAULT_TIMEOUT = 10800 DEFAULT_SLEEP_TIME = 5 timeouts_opts = [ # engine opts cfg.IntOpt('ips_assign_timeout', default=DEFAULT_TIMEOUT, help="Assign IPs timeout, in seconds"), cfg.IntOpt('wait_until_accessible', default=DEFAULT_TIMEOUT, help="Wait for instance accessibility, in seconds"), # direct engine opts cfg.IntOpt('delete_instances_timeout', default=DEFAULT_TIMEOUT, help="Wait for instances to be deleted, in seconds"), # volumes opts cfg.IntOpt( 'detach_volume_timeout', default=300, help='Timeout for detaching volumes from instance, in seconds'), ] timeouts = cfg.OptGroup(name='timeouts', title='Sahara timeouts') CONF = cfg.CONF CONF.register_group(timeouts) CONF.register_opts(timeouts_opts, group=timeouts) def _get_consumed(started_at): return timeutils.delta_seconds(started_at, timeutils.utcnow()) def _get_current_value(cluster, option): option_target = option.applicable_target conf = cluster.cluster_configs if option_target in conf and option.name in conf[option_target]: return conf[option_target][option.name] return option.default_value def poll(get_status, kwargs=None, args=None, operation_name=None, timeout_name=None, timeout=DEFAULT_TIMEOUT, sleep=DEFAULT_SLEEP_TIME, exception_strategy='raise'): """This util poll status of object obj during some timeout. :param get_status: function, which return current status of polling as Boolean :param kwargs: keyword arguments of function get_status :param operation_name: name of polling process :param timeout_name: name of timeout option :param timeout: value of timeout in seconds. By default, it equals to 3 hours :param sleep: duration between two consecutive executions of get_status function :param exception_strategy: possible values ('raise', 'mark_as_true', 'mark_as_false'). If exception_strategy is 'raise' exception would be raised. If exception_strategy is 'mark_as_true', return value of get_status would marked as True, and in case of 'mark_as_false' - False. By default it's 'raise'. """ start_time = timeutils.utcnow() # We shouldn't raise TimeoutException if incorrect timeout specified and # status is ok now. In such way we should execute get_status at least once. at_least_once = True if not kwargs: kwargs = {} if not args: args = () while at_least_once or _get_consumed(start_time) < timeout: at_least_once = False try: status = get_status(*args, **kwargs) except BaseException: if exception_strategy == 'raise': raise elif exception_strategy == 'mark_as_true': status = True else: status = False if status: operation = "Operation" if operation_name: operation = "Operation with name {op_name}".format( op_name=operation_name) LOG.debug( '{operation_desc} was executed successfully in timeout ' '{timeout}' .format(operation_desc=operation, timeout=timeout)) return context.sleep(sleep) raise ex.TimeoutException(timeout, operation_name, timeout_name) def plugin_option_poll(cluster, get_status, option, operation_name, sleep_time, kwargs): def _get(n_cluster, n_kwargs): if not cluster_utils.check_cluster_exists(n_cluster): return True return get_status(**n_kwargs) poll_description = { 'get_status': _get, 'kwargs': {'n_cluster': cluster, 'n_kwargs': kwargs}, 'timeout': _get_current_value(cluster, option), 'operation_name': operation_name, 'sleep': sleep_time, 'timeout_name': option.name } poll(**poll_description) def poll_status(option, operation_name, sleep): def decorator(f): @functools.wraps(f) def handler(*args, **kwargs): poll_description = { 'get_status': f, 'kwargs': kwargs, 'args': args, 'timeout': getattr(CONF.timeouts, option), 'operation_name': operation_name, 'timeout_name': option, 'sleep': sleep, } poll(**poll_description) return handler return decorator sahara-8.0.0/sahara/utils/remote.py0000666000175100017510000001260113245514472017257 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # Copyright (c) 2013 Hortonworks, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc from oslo_config import cfg import six from sahara import exceptions as ex from sahara.i18n import _ # These options are for SSH remote only ssh_opts = [ cfg.IntOpt('global_remote_threshold', default=100, help='Maximum number of remote operations that will ' 'be running at the same time. Note that each ' 'remote operation requires its own process to ' 'run.'), cfg.IntOpt('cluster_remote_threshold', default=70, help='The same as global_remote_threshold, but for ' 'a single cluster.'), cfg.StrOpt('proxy_command', default='', help='Proxy command used to connect to instances. If set, this ' 'command should open a netcat socket, that Sahara will use for ' 'SSH and HTTP connections. Use {host} and {port} to describe ' 'the destination. Other available keywords: {tenant_id}, ' '{network_id}, {router_id}.'), cfg.BoolOpt('proxy_command_use_internal_ip', default=False, help='Force proxy_command usage to be consuming internal IP ' 'always, instead of management IP. Ignored if proxy_command ' 'is not set.') ] CONF = cfg.CONF CONF.register_opts(ssh_opts) DRIVER = None @six.add_metaclass(abc.ABCMeta) class RemoteDriver(object): @abc.abstractmethod def setup_remote(self, engine): """Performs driver initialization.""" @abc.abstractmethod def get_remote(self, instance): """Returns driver specific Remote.""" @abc.abstractmethod def get_userdata_template(self): """Returns userdata template preparing instance to work with driver.""" @abc.abstractmethod def get_type_and_version(self): """Returns engine type and version Result should be in the form 'type.major.minor'. """ @six.add_metaclass(abc.ABCMeta) class TerminalOnlyRemote(object): @abc.abstractmethod def execute_command(self, cmd, run_as_root=False, get_stderr=False, raise_when_error=True, timeout=300): """Execute specified command remotely using existing ssh connection. Return exit code, stdout data and stderr data of the executed command. """ @abc.abstractmethod def get_os_distrib(self): """Returns the OS distribution running on the target machine.""" @six.add_metaclass(abc.ABCMeta) class Remote(TerminalOnlyRemote): @abc.abstractmethod def get_neutron_info(self): """Returns dict which later could be passed to get_http_client.""" @abc.abstractmethod def get_http_client(self, port, info=None): """Returns HTTP client for a given instance's port.""" @abc.abstractmethod def close_http_session(self, port): """Closes cached HTTP session for a given instance's port.""" @abc.abstractmethod def write_file_to(self, remote_file, data, run_as_root=False, timeout=120): """Create remote file and write the given data to it. Uses existing ssh connection. """ @abc.abstractmethod def append_to_file(self, r_file, data, run_as_root=False, timeout=120): """Append the given data to remote file. Uses existing ssh connection. """ @abc.abstractmethod def write_files_to(self, files, run_as_root=False, timeout=120): """Copy file->data dictionary in a single ssh connection.""" @abc.abstractmethod def append_to_files(self, files, run_as_root=False, timeout=120): """Copy file->data dictionary in a single ssh connection.""" @abc.abstractmethod def read_file_from(self, remote_file, run_as_root=False, timeout=120): """Read remote file from the specified host and return given data.""" @abc.abstractmethod def replace_remote_string(self, remote_file, old_str, new_str, timeout=120): """Replaces strings in remote file using sed command.""" def setup_remote(driver, engine): global DRIVER DRIVER = driver DRIVER.setup_remote(engine) def get_remote_type_and_version(): return DRIVER.get_type_and_version() def _check_driver_is_loaded(): if not DRIVER: raise ex.SystemError(_('Remote driver is not loaded. Most probably ' 'you see this error because you are running ' 'Sahara in distributed mode and it is broken.' 'Try running sahara-all instead.')) def get_remote(instance): """Returns Remote for a given instance.""" _check_driver_is_loaded() return DRIVER.get_remote(instance) def get_userdata_template(): """Returns userdata template as a string.""" _check_driver_is_loaded() return DRIVER.get_userdata_template() sahara-8.0.0/sahara/utils/configs.py0000666000175100017510000000204413245514472017414 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. def merge_configs(*configs): """Merge configs in special format. It supports merging of configs in the following format: applicable_target -> config_name -> config_value """ result = {} for config in configs: if config: for a_target in config: if a_target not in result or not result[a_target]: result[a_target] = {} result[a_target].update(config[a_target]) return result sahara-8.0.0/sahara/utils/api.py0000666000175100017510000002664313245514472016550 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import traceback import flask from oslo_log import log as logging from oslo_middleware import request_id as oslo_req_id from werkzeug import datastructures from sahara import context from sahara import exceptions as ex from sahara.i18n import _ from sahara.utils import types from sahara.utils import wsgi LOG = logging.getLogger(__name__) class Rest(flask.Blueprint): def get(self, rule, status_code=200): return self._mroute('GET', rule, status_code) def post(self, rule, status_code=202): return self._mroute('POST', rule, status_code) def post_file(self, rule, status_code=202): return self._mroute('POST', rule, status_code, file_upload=True) def put(self, rule, status_code=202): return self._mroute('PUT', rule, status_code) def put_file(self, rule, status_code=202): return self._mroute('PUT', rule, status_code, file_upload=True) def delete(self, rule, status_code=204): return self._mroute('DELETE', rule, status_code) def patch(self, rule, status_code=202): return self._mroute('PATCH', rule, status_code) def _mroute(self, methods, rule, status_code=None, **kw): if type(methods) is str: methods = [methods] return self.route(rule, methods=methods, status_code=status_code, **kw) def route(self, rule, **options): status = options.pop('status_code', None) file_upload = options.pop('file_upload', False) def decorator(func): endpoint = options.pop('endpoint', func.__name__) def handler(**kwargs): context.set_ctx(None) LOG.debug("Rest.route.decorator.handler, kwargs={kwargs}" .format(kwargs=kwargs)) _init_resp_type(file_upload) # update status code if status: flask.request.status_code = status kwargs.pop("tenant_id", None) req_id = flask.request.environ.get(oslo_req_id.ENV_REQUEST_ID) auth_plugin = flask.request.environ.get('keystone.token_auth') ctx = context.Context( flask.request.headers['X-User-Id'], flask.request.headers['X-Tenant-Id'], flask.request.headers['X-Auth-Token'], flask.request.headers['X-Service-Catalog'], flask.request.headers['X-User-Name'], flask.request.headers['X-Tenant-Name'], flask.request.headers['X-Roles'].split(','), auth_plugin=auth_plugin, request_id=req_id) context.set_ctx(ctx) try: if flask.request.method in ['POST', 'PUT', 'PATCH']: kwargs['data'] = request_data() return func(**kwargs) except ex.Forbidden as e: return access_denied(e) except ex.SaharaException as e: return bad_request(e) except Exception as e: return internal_error(500, 'Internal Server Error', e) f_rule = "/" + rule self.add_url_rule(rule, endpoint, handler, **options) self.add_url_rule(rule + '.json', endpoint, handler, **options) self.add_url_rule(f_rule, endpoint, handler, **options) self.add_url_rule(f_rule + '.json', endpoint, handler, **options) return func return decorator class RestV2(Rest): def route(self, rule, **options): status = options.pop('status_code', None) file_upload = options.pop('file_upload', False) def decorator(func): endpoint = options.pop('endpoint', func.__name__) def handler(**kwargs): context.set_ctx(None) LOG.debug("Rest.route.decorator.handler, kwargs={kwargs}" .format(kwargs=kwargs)) _init_resp_type(file_upload) # update status code if status: flask.request.status_code = status kwargs.pop("tenant_id", None) req_id = flask.request.environ.get(oslo_req_id.ENV_REQUEST_ID) auth_plugin = flask.request.environ.get('keystone.token_auth') ctx = context.Context( flask.request.headers['X-User-Id'], flask.request.headers['X-Tenant-Id'], flask.request.headers['X-Auth-Token'], flask.request.headers['X-Service-Catalog'], flask.request.headers['X-User-Name'], flask.request.headers['X-Tenant-Name'], flask.request.headers['X-Roles'].split(','), auth_plugin=auth_plugin, request_id=req_id) context.set_ctx(ctx) try: if flask.request.method in ['POST', 'PUT', 'PATCH']: kwargs['data'] = request_data() return func(**kwargs) except ex.Forbidden as e: return access_denied(e) except ex.SaharaException as e: return bad_request(e) except Exception as e: return internal_error(500, 'Internal Server Error', e) f_rule = "/" + rule self.add_url_rule(rule, endpoint, handler, **options) self.add_url_rule(rule + '.json', endpoint, handler, **options) self.add_url_rule(f_rule, endpoint, handler, **options) self.add_url_rule(f_rule + '.json', endpoint, handler, **options) return func return decorator RT_JSON = datastructures.MIMEAccept([("application/json", 1)]) def _init_resp_type(file_upload): """Extracts response content type.""" # get content type from Accept header resp_type = flask.request.accept_mimetypes # url /foo.json if flask.request.path.endswith('.json'): resp_type = RT_JSON flask.request.resp_type = resp_type # set file upload flag flask.request.file_upload = file_upload def render(res=None, resp_type=None, status=None, name=None, **kwargs): if not res and type(res) is not types.Page: res = {} if type(res) is dict: res.update(kwargs) elif type(res) is types.Page: result = {name: [item.to_dict() for item in res]} result.update(kwargs) if res.prev or res.next or ('marker' in get_request_args()): result["markers"] = {"prev": res.prev, "next": res.next} res = result elif kwargs: # can't merge kwargs into the non-dict res abort_and_log(500, _("Non-dict and non-empty kwargs passed to render")) status_code = getattr(flask.request, 'status_code', None) if status: status_code = status if not status_code: status_code = 200 if not resp_type: resp_type = getattr(flask.request, 'resp_type', RT_JSON) if not resp_type: resp_type = RT_JSON serializer = None if "application/json" in resp_type: resp_type = RT_JSON serializer = wsgi.JSONDictSerializer() else: raise ex.InvalidDataException( _("Content type '%s' isn't supported") % resp_type) body = serializer.serialize(res) resp_type = str(resp_type) return flask.Response(response=body, status=status_code, mimetype=resp_type) def request_data(): if hasattr(flask.request, 'parsed_data'): return flask.request.parsed_data if not flask.request.content_length > 0: LOG.debug("Empty body provided in request") return dict() if flask.request.file_upload: return flask.request.data deserializer = None content_type = flask.request.mimetype if not content_type or content_type in RT_JSON: deserializer = wsgi.JSONDeserializer() else: raise ex.InvalidDataException( _("Content type '%s' isn't supported") % content_type) # parsed request data to avoid unwanted re-parsings parsed_data = deserializer.deserialize(flask.request.data)['body'] flask.request.parsed_data = parsed_data return flask.request.parsed_data def get_request_args(): return flask.request.args def abort_and_log(status_code, descr, exc=None): LOG.error("Request aborted with status code {code} and " "message '{message}'".format(code=status_code, message=descr)) if exc is not None: LOG.error(traceback.format_exc()) flask.abort(status_code, description=descr) def render_error_message(error_code, error_message, error_name): message = { "error_code": error_code, "error_message": error_message, "error_name": error_name } resp = render(message) resp.status_code = error_code return resp def internal_error(status_code, descr, exc=None): LOG.error("Request aborted with status code {code} and " "message '{message}'".format(code=status_code, message=descr)) if exc is not None: LOG.error(traceback.format_exc()) error_code = "INTERNAL_SERVER_ERROR" if status_code == 501: error_code = "NOT_IMPLEMENTED_ERROR" return render_error_message(status_code, descr, error_code) def bad_request(error): error_code = 400 LOG.error("Validation Error occurred: " "error_code={code}, error_message={message}, " "error_name={name}".format(code=error_code, message=error.message, name=error.code)) return render_error_message(error_code, error.message, error.code) def access_denied(error): error_code = 403 LOG.error("Access Denied: error_code={code}, error_message={message}, " "error_name={name}".format(code=error_code, message=error.message, name=error.code)) return render_error_message(error_code, error.message, error.code) def not_found(error): error_code = 404 LOG.error("Not Found exception occurred: " "error_code={code}, error_message={message}, " "error_name={name}".format(code=error_code, message=error.message, name=error.code)) return render_error_message(error_code, error.message, error.code) def to_wrapped_dict(func, id, *args, **kwargs): return render(to_wrapped_dict_no_render(func, id, *args, **kwargs)) def to_wrapped_dict_no_render(func, id, *args, **kwargs): obj = func(id, *args, **kwargs) if obj is None: e = ex.NotFoundException( {'id': id}, _('Object with %s not found')) return not_found(e) return obj.to_wrapped_dict() sahara-8.0.0/sahara/utils/tempfiles.py0000666000175100017510000000220213245514472017750 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import contextlib import shutil import tempfile from sahara import exceptions as ex from sahara.i18n import _ @contextlib.contextmanager def tempdir(**kwargs): argdict = kwargs.copy() if 'dir' not in argdict: argdict['dir'] = '/tmp/' tmpdir = tempfile.mkdtemp(**argdict) try: yield tmpdir finally: try: shutil.rmtree(tmpdir) except OSError as e: raise ex.SystemError( _("Failed to delete temp dir %(dir)s (reason: %(reason)s)") % {'dir': tmpdir, 'reason': e}) sahara-8.0.0/sahara/utils/ssh_remote.py0000666000175100017510000010167013245514472020141 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # Copyright (c) 2013 Hortonworks, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Helper methods for executing commands on nodes via SSH. The main access point is method get_remote(instance), it returns InstanceInteropHelper object which does the actual work. See the class for the list of available methods. It is a context manager, so it could be used with 'with' statement like that: with get_remote(instance) as r: r.execute_command(...) Note that the module offloads the ssh calls to a child process. It was implemented that way because we found no way to run paramiko and eventlet together. The private high-level module methods are implementations which are run in a separate process. """ import copy import os import shlex import sys import threading import time from eventlet.green import subprocess as e_subprocess from eventlet import semaphore from eventlet import timeout as e_timeout from oslo_config import cfg from oslo_log import log as logging from oslo_utils import excutils from oslo_utils import uuidutils import paramiko import requests from requests import adapters import six from sahara import context from sahara import exceptions as ex from sahara.i18n import _ from sahara.service import trusts from sahara.utils import crypto from sahara.utils import network as net_utils from sahara.utils.openstack import neutron from sahara.utils import procutils from sahara.utils import remote LOG = logging.getLogger(__name__) CONF = cfg.CONF ssh_config_options = [ cfg.IntOpt( 'ssh_timeout_common', default=300, min=1, help="Overrides timeout for common ssh operations, in seconds"), cfg.IntOpt( 'ssh_timeout_interactive', default=1800, min=1, help="Overrides timeout for interactive ssh operations, in seconds"), cfg.IntOpt( 'ssh_timeout_files', default=120, min=1, help="Overrides timeout for ssh operations with files, in seconds"), ] CONF.register_opts(ssh_config_options) _ssh = None _proxy_ssh = None _sessions = {} INFRA = None SSH_TIMEOUTS_MAPPING = { '_execute_command': 'ssh_timeout_common', '_execute_command_interactive': 'ssh_timeout_interactive' } _global_remote_semaphore = None def _get_access_ip(instance): if CONF.proxy_command and CONF.proxy_command_use_internal_ip: return instance.internal_ip return instance.management_ip def _default_timeout(func): timeout = SSH_TIMEOUTS_MAPPING.get(func.__name__, 'ssh_timeout_files') return getattr(CONF, timeout, CONF.ssh_timeout_common) def _get_ssh_timeout(func, timeout): return timeout if timeout else _default_timeout(func) def _connect(host, username, private_key, proxy_command=None, gateway_host=None, gateway_image_username=None): global _ssh global _proxy_ssh LOG.debug('Creating SSH connection') if isinstance(private_key, six.string_types): private_key = crypto.to_paramiko_private_key(private_key) _ssh = paramiko.SSHClient() _ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) proxy = None if proxy_command: LOG.debug('Creating proxy using command: {command}'.format( command=proxy_command)) proxy = paramiko.ProxyCommand(proxy_command) if gateway_host: _proxy_ssh = paramiko.SSHClient() _proxy_ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) LOG.debug('Connecting to proxy gateway at: {gateway}'.format( gateway=gateway_host)) _proxy_ssh.connect(gateway_host, username=gateway_image_username, pkey=private_key, sock=proxy) proxy = _proxy_ssh.get_transport().open_session() proxy.exec_command("nc {0} 22".format(host)) _ssh.connect(host, username=username, pkey=private_key, sock=proxy) def _cleanup(): global _ssh global _proxy_ssh _ssh.close() if _proxy_ssh: _proxy_ssh.close() def _read_paramimko_stream(recv_func): result = '' buf = recv_func(1024) while buf != '': result += buf buf = recv_func(1024) return result def _escape_quotes(command): command = command.replace('\\', '\\\\') command = command.replace('"', '\\"') command = command.replace('`', '\\`') return command def _execute_command(cmd, run_as_root=False, get_stderr=False, raise_when_error=True): global _ssh chan = _ssh.get_transport().open_session() if run_as_root: chan.exec_command('sudo bash -c "%s"' % _escape_quotes(cmd)) else: chan.exec_command(cmd) # TODO(dmitryme): that could hang if stderr buffer overflows stdout = _read_paramimko_stream(chan.recv) stderr = _read_paramimko_stream(chan.recv_stderr) ret_code = chan.recv_exit_status() if ret_code and raise_when_error: raise ex.RemoteCommandException(cmd=cmd, ret_code=ret_code, stdout=stdout, stderr=stderr) if get_stderr: return ret_code, stdout, stderr else: return ret_code, stdout def _execute_command_interactive(cmd, run_as_root=False): global _ssh chan = _ssh.get_transport().open_session() if run_as_root: chan.exec_command('sudo bash -c "%s"' % _escape_quotes(cmd)) else: chan.exec_command(cmd) _proxy_shell(chan) _ssh.close() def _proxy_shell(chan): def readall(): while True: d = sys.stdin.read(1) if not d or chan.exit_status_ready(): break chan.send(d) reader = threading.Thread(target=readall) reader.start() while True: data = chan.recv(256) if not data or chan.exit_status_ready(): break sys.stdout.write(data) sys.stdout.flush() def _get_http_client(host, port, proxy_command=None, gateway_host=None, gateway_username=None, gateway_private_key=None): global _sessions _http_session = _sessions.get((host, port), None) LOG.debug('Cached HTTP session for {host}:{port} is {session}'.format( host=host, port=port, session=_http_session)) if not _http_session: if gateway_host: _http_session = _get_proxy_gateway_http_session( gateway_host, gateway_username, gateway_private_key, host, port, proxy_command) LOG.debug('Created ssh proxied HTTP session for {host}:{port}' .format(host=host, port=port)) elif proxy_command: # can return a new session here because it actually uses # the same adapter (and same connection pools) for a given # host and port tuple _http_session = _get_proxied_http_session( proxy_command, host, port=port) LOG.debug('Created proxied HTTP session for {host}:{port}' .format(host=host, port=port)) else: # need to cache the sessions that are not proxied through # HTTPRemoteWrapper so that a new session with a new HTTPAdapter # and associated pools is not recreated for each HTTP invocation _http_session = requests.Session() LOG.debug('Created standard HTTP session for {host}:{port}' .format(host=host, port=port)) adapter = requests.adapters.HTTPAdapter() for prefix in ['http://', 'https://']: _http_session.mount(prefix + '%s:%s' % (host, port), adapter) LOG.debug('Caching session {session} for {host}:{port}' .format(session=_http_session, host=host, port=port)) _sessions[(host, port)] = _http_session return _http_session def _write_fl(sftp, remote_file, data): fl = sftp.file(remote_file, 'w') fl.write(data) fl.close() def _append_fl(sftp, remote_file, data): fl = sftp.file(remote_file, 'a') fl.write(data) fl.close() def _write_file(sftp, remote_file, data, run_as_root): if run_as_root: temp_file = 'temp-file-%s' % uuidutils.generate_uuid() _write_fl(sftp, temp_file, data) _execute_command( 'mv %s %s' % (temp_file, remote_file), run_as_root=True) else: _write_fl(sftp, remote_file, data) def _append_file(sftp, remote_file, data, run_as_root): if run_as_root: temp_file = 'temp-file-%s' % uuidutils.generate_uuid() _write_fl(sftp, temp_file, data) _execute_command( 'cat %s >> %s' % (temp_file, remote_file), run_as_root=True) _execute_command('rm -f %s' % temp_file) else: _append_fl(sftp, remote_file, data) def _write_file_to(remote_file, data, run_as_root=False): global _ssh _write_file(_ssh.open_sftp(), remote_file, data, run_as_root) def _write_files_to(files, run_as_root=False): global _ssh sftp = _ssh.open_sftp() for fl, data in six.iteritems(files): _write_file(sftp, fl, data, run_as_root) def _append_to_file(remote_file, data, run_as_root=False): global _ssh _append_file(_ssh.open_sftp(), remote_file, data, run_as_root) def _append_to_files(files, run_as_root=False): global _ssh sftp = _ssh.open_sftp() for fl, data in six.iteritems(files): _append_file(sftp, fl, data, run_as_root) def _read_file(sftp, remote_file): fl = sftp.file(remote_file, 'r') data = fl.read() fl.close() return data def _read_file_from(remote_file, run_as_root=False): global _ssh fl = remote_file if run_as_root: fl = 'temp-file-%s' % (uuidutils.generate_uuid()) _execute_command('cp %s %s' % (remote_file, fl), run_as_root=True) try: return _read_file(_ssh.open_sftp(), fl) except IOError: LOG.error("Can't read file {filename}".format(filename=remote_file)) raise finally: if run_as_root: _execute_command( 'rm %s' % fl, run_as_root=True, raise_when_error=False) def _get_python_to_execute(): try: _execute_command('python3 --version') except Exception: _execute_command('python2 --version') return 'python2' return 'python3' def _get_os_distrib(): python_version = _get_python_to_execute() return _execute_command( ('printf "import platform\nprint(platform.linux_distribution(' 'full_distribution_name=0)[0])" | {}'.format(python_version)), run_as_root=False)[1].lower().strip() def _get_os_version(): python_version = _get_python_to_execute() return _execute_command( ('printf "import platform\nprint(platform.linux_distribution()[1])"' ' | {}'.format(python_version)), run_as_root=False)[1].strip() def _install_packages(packages): distrib = _get_os_distrib() if distrib == 'ubuntu': cmd = 'RUNLEVEL=1 apt-get install -y %(pkgs)s' elif distrib == 'fedora': fversion = _get_os_version() if fversion >= 22: cmd = 'dnf install -y %(pkgs)s' else: cmd = 'yum install -y %(pkgs)s' elif distrib in ('redhat', 'centos'): cmd = 'yum install -y %(pkgs)s' else: raise ex.NotImplementedException( _('Package Installation'), _('%(fmt)s is not implemented for OS %(distrib)s') % { 'fmt': '%s', 'distrib': distrib}) cmd = cmd % {'pkgs': ' '.join(packages)} _execute_command(cmd, run_as_root=True) def _update_repository(): distrib = _get_os_distrib() if distrib == 'ubuntu': cmd = 'apt-get update' elif distrib == 'fedora': fversion = _get_os_version() if fversion >= 22: cmd = 'dnf clean all' else: cmd = 'yum clean all' elif distrib in ('redhat', 'centos'): cmd = 'yum clean all' else: raise ex.NotImplementedException( _('Repository Update'), _('%(fmt)s is not implemented for OS %(distrib)s') % { 'fmt': '%s', 'distrib': distrib}) _execute_command(cmd, run_as_root=True) def _replace_remote_string(remote_file, old_str, new_str): old_str = old_str.replace("\'", "\''") new_str = new_str.replace("\'", "\''") cmd = "sudo sed -i 's,%s,%s,g' %s" % (old_str, new_str, remote_file) _execute_command(cmd) def _execute_on_vm_interactive(cmd, matcher): global _ssh buf = '' channel = _ssh.invoke_shell() LOG.debug('Channel is {channel}'.format(channel=channel)) try: LOG.debug('Sending cmd {command}'.format(command=cmd)) channel.send(cmd + '\n') while not matcher.is_eof(buf): buf += channel.recv(4096) response = matcher.get_response(buf) if response is not None: channel.send(response + '\n') buf = '' finally: LOG.debug('Closing channel') channel.close() def _acquire_remote_semaphore(): context.current().remote_semaphore.acquire() _global_remote_semaphore.acquire() def _release_remote_semaphore(): _global_remote_semaphore.release() context.current().remote_semaphore.release() def _get_proxied_http_session(proxy_command, host, port=None): session = requests.Session() adapter = ProxiedHTTPAdapter( _simple_exec_func(shlex.split(proxy_command)), host, port) session.mount('http://{0}:{1}'.format(host, adapter.port), adapter) return session def _get_proxy_gateway_http_session(gateway_host, gateway_username, gateway_private_key, host, port=None, proxy_command=None): session = requests.Session() adapter = ProxiedHTTPAdapter( _proxy_gateway_func(gateway_host, gateway_username, gateway_private_key, host, port, proxy_command), host, port) session.mount('http://{0}:{1}'.format(host, port), adapter) return session def _simple_exec_func(cmd): def func(): return e_subprocess.Popen(cmd, stdin=e_subprocess.PIPE, stdout=e_subprocess.PIPE, stderr=e_subprocess.PIPE) return func def _proxy_gateway_func(gateway_host, gateway_username, gateway_private_key, host, port, proxy_command): def func(): proc = procutils.start_subprocess() try: conn_params = (gateway_host, gateway_username, gateway_private_key, proxy_command, None, None) procutils.run_in_subprocess(proc, _connect, conn_params) cmd = "nc {host} {port}".format(host=host, port=port) procutils.run_in_subprocess( proc, _execute_command_interactive, (cmd,), interactive=True) return proc except Exception: with excutils.save_and_reraise_exception(): procutils.shutdown_subprocess(proc, _cleanup) return func class ProxiedHTTPAdapter(adapters.HTTPAdapter): def __init__(self, create_process_func, host, port): super(ProxiedHTTPAdapter, self).__init__() LOG.debug('HTTP adapter created for {host}:{port}'.format(host=host, port=port)) self.create_process_func = create_process_func self.port = port self.host = host def get_connection(self, url, proxies=None): pool_conn = ( super(ProxiedHTTPAdapter, self).get_connection(url, proxies)) if hasattr(pool_conn, '_get_conn'): http_conn = pool_conn._get_conn() if http_conn.sock is None: if hasattr(http_conn, 'connect'): sock = self._connect() LOG.debug('HTTP connection {connection} getting new ' 'netcat socket {socket}'.format( connection=http_conn, socket=sock)) http_conn.sock = sock else: if hasattr(http_conn.sock, 'is_netcat_socket'): LOG.debug('Pooled http connection has existing ' 'netcat socket. resetting pipe') http_conn.sock.reset() pool_conn._put_conn(http_conn) return pool_conn def close(self): LOG.debug('Closing HTTP adapter for {host}:{port}' .format(host=self.host, port=self.port)) super(ProxiedHTTPAdapter, self).close() def _connect(self): LOG.debug('Returning netcat socket for {host}:{port}' .format(host=self.host, port=self.port)) rootwrap_command = CONF.rootwrap_command if CONF.use_rootwrap else '' return NetcatSocket(self.create_process_func, rootwrap_command) class NetcatSocket(object): def _create_process(self): self.process = self.create_process_func() def __init__(self, create_process_func, rootwrap_command=None): self.create_process_func = create_process_func self.rootwrap_command = rootwrap_command self._create_process() def send(self, content): try: self.process.stdin.write(content) self.process.stdin.flush() except IOError as e: raise ex.SystemError(e) return len(content) def sendall(self, content): return self.send(content) def makefile(self, mode, *arg): if mode.startswith('r'): return self.process.stdout if mode.startswith('w'): return self.process.stdin raise ex.IncorrectStateError(_("Unknown file mode %s") % mode) def recv(self, size): try: return os.read(self.process.stdout.fileno(), size) except IOError as e: raise ex.SystemError(e) def _terminate(self): if self.rootwrap_command: os.system('{0} kill {1}'.format(self.rootwrap_command, # nosec self.process.pid)) else: self.process.terminate() def close(self): LOG.debug('Socket close called') self._terminate() def settimeout(self, timeout): pass def fileno(self): return self.process.stdin.fileno() def is_netcat_socket(self): return True def reset(self): self._terminate() self._create_process() class InstanceInteropHelper(remote.Remote): def __init__(self, instance): self.instance = instance def __enter__(self): _acquire_remote_semaphore() try: self.bulk = BulkInstanceInteropHelper(self.instance) return self.bulk except Exception: with excutils.save_and_reraise_exception(): _release_remote_semaphore() def __exit__(self, *exc_info): try: self.bulk.close() finally: _release_remote_semaphore() def get_neutron_info(self, instance=None): if not instance: instance = self.instance neutron_info = dict() neutron_info['network'] = instance.cluster.neutron_management_network ctx = context.current() neutron_info['token'] = context.get_auth_token() neutron_info['tenant'] = ctx.tenant_name neutron_info['host'] = _get_access_ip(instance) log_info = copy.deepcopy(neutron_info) del log_info['token'] LOG.debug('Returning neutron info: {info}'.format(info=log_info)) return neutron_info def _build_proxy_command(self, command, instance=None, port=None, info=None, rootwrap_command=None): # Accepted keywords in the proxy command template: # {host}, {port}, {tenant_id}, {network_id}, {router_id} keywords = {} if not info: info = self.get_neutron_info(instance) keywords['tenant_id'] = context.current().tenant_id keywords['network_id'] = info['network'] # Query Neutron only if needed if '{router_id}' in command: auth = trusts.get_os_admin_auth_plugin(instance.cluster) client = neutron.NeutronClient(info['network'], info['token'], info['tenant'], auth=auth) keywords['router_id'] = client.get_router() keywords['host'] = _get_access_ip(instance) keywords['port'] = port try: command = command.format(**keywords) except KeyError as e: LOG.error('Invalid keyword in proxy_command: {result}'.format( result=e)) # Do not give more details to the end-user raise ex.SystemError('Misconfiguration') if rootwrap_command: command = '{0} {1}'.format(rootwrap_command, command) return command def _get_conn_params(self): host_ng = self.instance.node_group cluster = host_ng.cluster access_instance = self.instance proxy_gateway_node = cluster.get_proxy_gateway_node() gateway_host = None gateway_image_username = None if proxy_gateway_node and not host_ng.is_proxy_gateway: # tmckay-fp in other words, if we are going to connect # through the proxy instead of the node we are actually # trying to reach # okay, the node group that supplies the proxy gateway # must have fps, but if a proxy is used the other # nodes are not required to have an fp. # so, this instance is assumed not to have a floating # ip and we are going to get to it through the proxy access_instance = proxy_gateway_node gateway_host = proxy_gateway_node.management_ip ng = proxy_gateway_node.node_group gateway_image_username = ng.image_username proxy_command = None if CONF.proxy_command: # Build a session through a user-defined socket proxy_command = CONF.proxy_command # tmckay-fp we have the node_group for the instance right here # okay, this test here whether access_instance.management_ip is an # fp -- just compare to internal? # in the neutron case, we check the node group for the # access_instance and look for fp elif CONF.use_namespaces and not net_utils.has_floating_ip( access_instance): # Build a session through a netcat socket in the Neutron namespace proxy_command = ( 'ip netns exec qrouter-{router_id} nc {host} {port}') # proxy_command is currently a template, turn it into a real command # i.e. dereference {host}, {port}, etc. if proxy_command: rootwrap = CONF.rootwrap_command if CONF.use_rootwrap else '' proxy_command = self._build_proxy_command( proxy_command, instance=access_instance, port=22, info=None, rootwrap_command=rootwrap) host_ip = _get_access_ip(self.instance) return (host_ip, host_ng.image_username, cluster.management_private_key, proxy_command, gateway_host, gateway_image_username) def _run(self, func, *args, **kwargs): proc = procutils.start_subprocess() try: procutils.run_in_subprocess(proc, _connect, self._get_conn_params()) return procutils.run_in_subprocess(proc, func, args, kwargs) except Exception: with excutils.save_and_reraise_exception(): procutils.shutdown_subprocess(proc, _cleanup) finally: procutils.shutdown_subprocess(proc, _cleanup) def _run_with_log(self, func, timeout, description, *args, **kwargs): start_time = time.time() try: with e_timeout.Timeout(timeout, ex.TimeoutException(timeout, op_name=description)): return self._run(func, *args, **kwargs) finally: self._log_command('"%s" took %.1f seconds to complete' % ( description, time.time() - start_time)) def _run_s(self, func, timeout, description, *args, **kwargs): timeout = _get_ssh_timeout(func, timeout) _acquire_remote_semaphore() try: return self._run_with_log(func, timeout, description, *args, **kwargs) finally: _release_remote_semaphore() def get_http_client(self, port, info=None): self._log_command('Retrieving HTTP session for {0}:{1}'.format( _get_access_ip(self.instance), port)) host_ng = self.instance.node_group cluster = host_ng.cluster access_instance = self.instance access_port = port proxy_gateway_node = cluster.get_proxy_gateway_node() gateway_host = None gateway_username = None gateway_private_key = None if proxy_gateway_node and not host_ng.is_proxy_gateway: access_instance = proxy_gateway_node access_port = 22 gateway_host = proxy_gateway_node.management_ip gateway_username = proxy_gateway_node.node_group.image_username gateway_private_key = cluster.management_private_key proxy_command = None if CONF.proxy_command: # Build a session through a user-defined socket proxy_command = CONF.proxy_command # tmckay-fp again we can check the node group for the instance # what are the implications for nova here? None. # This is a test on whether access_instance has a floating_ip # in the neutron case, we check the node group for the # access_instance and look for fp elif (CONF.use_namespaces and not net_utils.has_floating_ip( access_instance)): # need neutron info if not info: info = self.get_neutron_info(access_instance) # Build a session through a netcat socket in the Neutron namespace proxy_command = ( 'ip netns exec qrouter-{router_id} nc {host} {port}') # proxy_command is currently a template, turn it into a real command # i.e. dereference {host}, {port}, etc. if proxy_command: rootwrap = CONF.rootwrap_command if CONF.use_rootwrap else '' proxy_command = self._build_proxy_command( proxy_command, instance=access_instance, port=access_port, info=info, rootwrap_command=rootwrap) return _get_http_client(_get_access_ip(self.instance), port, proxy_command, gateway_host, gateway_username, gateway_private_key) def close_http_session(self, port): global _sessions host = _get_access_ip(self.instance) self._log_command(_("Closing HTTP session for %(host)s:%(port)s") % { 'host': host, 'port': port}) session = _sessions.get((host, port), None) if session is None: raise ex.NotFoundException( {'host': host, 'port': port}, _('Session for %(host)s:%(port)s not cached')) session.close() del _sessions[(host, port)] def execute_command(self, cmd, run_as_root=False, get_stderr=False, raise_when_error=True, timeout=None): description = _('Executing "%s"') % cmd self._log_command(description) return self._run_s(_execute_command, timeout, description, cmd, run_as_root, get_stderr, raise_when_error) def write_file_to(self, remote_file, data, run_as_root=False, timeout=None): description = _('Writing file "%s"') % remote_file self._log_command(description) self._run_s(_write_file_to, timeout, description, remote_file, data, run_as_root) def write_files_to(self, files, run_as_root=False, timeout=None): description = _('Writing files "%s"') % list(files) self._log_command(description) self._run_s(_write_files_to, timeout, description, files, run_as_root) def append_to_file(self, r_file, data, run_as_root=False, timeout=None): description = _('Appending to file "%s"') % r_file self._log_command(description) self._run_s(_append_to_file, timeout, description, r_file, data, run_as_root) def append_to_files(self, files, run_as_root=False, timeout=None): description = _('Appending to files "%s"') % list(files) self._log_command(description) self._run_s(_append_to_files, timeout, description, files, run_as_root) def read_file_from(self, remote_file, run_as_root=False, timeout=None): description = _('Reading file "%s"') % remote_file self._log_command(description) return self._run_s(_read_file_from, timeout, description, remote_file, run_as_root) def get_python_version(self, timeout=None): return self._run_s( _get_python_to_execute, timeout, "get_python_version") def get_os_distrib(self, timeout=None): return self._run_s(_get_os_distrib, timeout, "get_os_distrib") def get_os_version(self, timeout=None): return self._run_s(_get_os_version, timeout, "get_os_version") def install_packages(self, packages, timeout=None): description = _('Installing packages "%s"') % list(packages) self._log_command(description) self._run_s(_install_packages, timeout, description, packages) def update_repository(self, timeout=None): description = _('Updating repository') self._log_command(description) self._run_s(_update_repository, timeout, description) def replace_remote_string(self, remote_file, old_str, new_str, timeout=None): description = _('In file "%(file)s" replacing string ' '"%(old_string)s" with "%(new_string)s"') % { "file": remote_file, "old_string": old_str, "new_string": new_str} self._log_command(description) self._run_s(_replace_remote_string, timeout, description, remote_file, old_str, new_str) def execute_on_vm_interactive(self, cmd, matcher, timeout=None): """Runs given command and responds to prompts. 'cmd' is a command to execute. 'matcher' is an object which provides responses on command's prompts. It should have two methods implemented: * get_response(buf) - returns response on prompt if it is found in 'buf' string, which is a part of command output. If no prompt is found, the method should return None. * is_eof(buf) - returns True if current 'buf' indicates that the command is finished. False should be returned otherwise. """ description = _('Executing interactively "%s"') % cmd self._log_command(description) self._run_s(_execute_on_vm_interactive, timeout, description, cmd, matcher) def _log_command(self, str): with context.set_current_instance_id(self.instance.instance_id): LOG.debug(str) class BulkInstanceInteropHelper(InstanceInteropHelper): def __init__(self, instance): super(BulkInstanceInteropHelper, self).__init__(instance) self.proc = procutils.start_subprocess() try: procutils.run_in_subprocess(self.proc, _connect, self._get_conn_params()) except Exception: with excutils.save_and_reraise_exception(): procutils.shutdown_subprocess(self.proc, _cleanup) def close(self): procutils.shutdown_subprocess(self.proc, _cleanup) def _run(self, func, *args, **kwargs): return procutils.run_in_subprocess(self.proc, func, args, kwargs) def _run_s(self, func, timeout, description, *args, **kwargs): timeout = _get_ssh_timeout(func, timeout) return self._run_with_log(func, timeout, description, *args, **kwargs) class SshRemoteDriver(remote.RemoteDriver): def get_type_and_version(self): return "ssh.1.0" def setup_remote(self, engine): global _global_remote_semaphore global INFRA _global_remote_semaphore = semaphore.Semaphore( CONF.global_remote_threshold) INFRA = engine def get_remote(self, instance): return InstanceInteropHelper(instance) def get_userdata_template(self): # SSH does not need any instance customization return "" sahara-8.0.0/sahara/utils/__init__.py0000666000175100017510000000000013245514472017511 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/files.py0000666000175100017510000000225413245514472017071 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from os import path import pkg_resources as pkg from sahara import version def get_file_text(file_name): full_name = pkg.resource_filename( version.version_info.package, file_name) return open(full_name).read() def get_file_binary(file_name): full_name = pkg.resource_filename( version.version_info.package, file_name) return open(full_name, "rb").read() def try_get_file_text(file_name): full_name = pkg.resource_filename( version.version_info.package, file_name) return ( open(full_name, "rb").read() if path.isfile(full_name) else False) sahara-8.0.0/sahara/utils/patches.py0000666000175100017510000000667413245514472017430 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import eventlet EVENTLET_MONKEY_PATCH_MODULES = dict(os=True, select=True, socket=True, thread=True, time=True) def patch_all(): """Apply all patches. List of patches: * eventlet's monkey patch for all cases; * minidom's writexml patch for py < 2.7.3 only. """ eventlet_monkey_patch() patch_minidom_writexml() def eventlet_monkey_patch(): """Apply eventlet's monkey patch. This call should be the first call in application. It's safe to call monkey_patch multiple times. """ eventlet.monkey_patch(**EVENTLET_MONKEY_PATCH_MODULES) def eventlet_import_monkey_patched(module): """Returns module monkey patched by eventlet. It's needed for some tests, for example, context test. """ return eventlet.import_patched(module, **EVENTLET_MONKEY_PATCH_MODULES) def patch_minidom_writexml(): """Patch for xml.dom.minidom toprettyxml bug with whitespaces around text We apply the patch to avoid excess whitespaces in generated xml configuration files that brakes Hadoop. (This patch will be applied for all Python versions < 2.7.3) Issue: http://bugs.python.org/issue4147 Patch: http://hg.python.org/cpython/rev/cb6614e3438b/ Description: http://ronrothman.com/public/leftbraned/xml-dom-minidom-\ toprettyxml-and-silly-whitespace/#best-solution """ import sys if sys.version_info >= (2, 7, 3): return import xml.dom.minidom as md def element_writexml(self, writer, indent="", addindent="", newl=""): # indent = current indentation # addindent = indentation to add to higher levels # newl = newline string writer.write(indent + "<" + self.tagName) attrs = self._get_attributes() a_names = list(attrs.keys()) a_names.sort() for a_name in a_names: writer.write(" %s=\"" % a_name) md._write_data(writer, attrs[a_name].value) writer.write("\"") if self.childNodes: writer.write(">") if (len(self.childNodes) == 1 and self.childNodes[0].nodeType == md.Node.TEXT_NODE): self.childNodes[0].writexml(writer, '', '', '') else: writer.write(newl) for node in self.childNodes: node.writexml(writer, indent + addindent, addindent, newl) writer.write(indent) writer.write("%s" % (self.tagName, newl)) else: writer.write("/>%s" % (newl)) md.Element.writexml = element_writexml def text_writexml(self, writer, indent="", addindent="", newl=""): md._write_data(writer, "%s%s%s" % (indent, self.data, newl)) md.Text.writexml = text_writexml sahara-8.0.0/sahara/utils/edp.py0000666000175100017510000001110313245514472016530 0ustar zuulzuul00000000000000# Copyright (c) 2014 Red Hat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils from sahara.utils import files # job execution status JOB_STATUS_DONEWITHERROR = 'DONEWITHERROR' JOB_STATUS_FAILED = 'FAILED' JOB_STATUS_KILLED = 'KILLED' JOB_STATUS_PENDING = 'PENDING' JOB_STATUS_READYTORUN = 'READYTORUN' JOB_STATUS_RUNNING = 'RUNNING' JOB_STATUS_SUCCEEDED = 'SUCCEEDED' JOB_STATUS_TOBEKILLED = 'TOBEKILLED' JOB_STATUS_TOBESUSPENDED = 'TOBESUSPENDED' JOB_STATUS_PREP = 'PREP' JOB_STATUS_PREPSUSPENDED = 'PREPSUSPENDED' JOB_STATUS_SUSPENDED = 'SUSPENDED' JOB_STATUS_SUSPEND_FAILED = 'SUSPENDFAILED' # statuses for suspended jobs JOB_STATUSES_SUSPENDIBLE = [ JOB_STATUS_PREP, JOB_STATUS_RUNNING ] # statuses for terminated jobs JOB_STATUSES_TERMINATED = [ JOB_STATUS_DONEWITHERROR, JOB_STATUS_FAILED, JOB_STATUS_KILLED, JOB_STATUS_SUCCEEDED, JOB_STATUS_SUSPEND_FAILED ] # job type separator character JOB_TYPE_SEP = '.' # job sub types available JOB_SUBTYPE_STREAMING = 'Streaming' JOB_SUBTYPE_NONE = '' # job types available JOB_TYPE_HIVE = 'Hive' JOB_TYPE_JAVA = 'Java' JOB_TYPE_MAPREDUCE = 'MapReduce' JOB_TYPE_SPARK = 'Spark' JOB_TYPE_STORM = 'Storm' JOB_TYPE_PYLEUS = 'Storm.Pyleus' JOB_TYPE_MAPREDUCE_STREAMING = (JOB_TYPE_MAPREDUCE + JOB_TYPE_SEP + JOB_SUBTYPE_STREAMING) JOB_TYPE_PIG = 'Pig' JOB_TYPE_SHELL = 'Shell' # job type groupings available JOB_TYPES_ALL = [ JOB_TYPE_HIVE, JOB_TYPE_JAVA, JOB_TYPE_MAPREDUCE, JOB_TYPE_MAPREDUCE_STREAMING, JOB_TYPE_PIG, JOB_TYPE_SHELL, JOB_TYPE_SPARK, JOB_TYPE_STORM, JOB_TYPE_PYLEUS ] JOB_TYPES_ACCEPTABLE_CONFIGS = { JOB_TYPE_HIVE: {"configs", "params"}, JOB_TYPE_PIG: {"configs", "params", "args"}, JOB_TYPE_MAPREDUCE: {"configs"}, JOB_TYPE_MAPREDUCE_STREAMING: {"configs"}, JOB_TYPE_JAVA: {"configs", "args"}, JOB_TYPE_SHELL: {"configs", "params", "args"}, JOB_TYPE_SPARK: {"configs", "args"}, JOB_TYPE_STORM: {"args"}, JOB_TYPE_PYLEUS: {} } # job actions JOB_ACTION_SUSPEND = 'suspend' JOB_ACTION_CANCEL = 'cancel' JOB_ACTION_TYPES_ACCEPTABLE = [ JOB_ACTION_SUSPEND, JOB_ACTION_CANCEL ] ADAPT_FOR_OOZIE = 'edp.java.adapt_for_oozie' SPARK_DRIVER_CLASSPATH = 'edp.spark.driver.classpath' ADAPT_SPARK_FOR_SWIFT = 'edp.spark.adapt_for_swift' def split_job_type(job_type): '''Split a job type string into a type and subtype The split is done on the first '.'. A subtype will always be returned, even if it is empty. ''' type_info = job_type.split(JOB_TYPE_SEP, 1) if len(type_info) == 1: type_info.append('') return type_info def compare_job_type(job_type, *args, **kwargs): '''Compare a job type against a list of job types :param job_type: The job type being compared :param *args: A list of types to compare against :param strict: Passed as a keyword arg. Default is False. If strict is False, job_type will be compared with and without its subtype indicator. :returns: True if job_type is present in the list, False otherwise ''' strict = kwargs.get('strict', False) res = job_type in args if res or strict or JOB_TYPE_SEP not in job_type: return res jtype, jsubtype = split_job_type(job_type) return jtype in args def get_hive_shared_conf_path(hdfs_user): return "/user/%s/conf/hive-site.xml" % hdfs_user def is_adapt_for_oozie_enabled(configs): return configs.get(ADAPT_FOR_OOZIE, False) def is_adapt_spark_for_swift_enabled(configs): return configs.get(ADAPT_SPARK_FOR_SWIFT, False) def spark_driver_classpath(configs): # Return None in case when you need to use default value return configs.get(SPARK_DRIVER_CLASSPATH) def get_builtin_binaries(job, configs): if job.type == JOB_TYPE_JAVA: if is_adapt_for_oozie_enabled(configs): path = 'service/edp/resources/edp-main-wrapper.jar' name = 'builtin-%s.jar' % uuidutils.generate_uuid() return [{'raw': files.get_file_binary(path), 'name': name}] return [] sahara-8.0.0/sahara/utils/types.py0000666000175100017510000000530613245514472017134 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import exceptions as ex class FrozenList(list): def append(self, p_object): raise ex.FrozenClassError(self) def extend(self, iterable): raise ex.FrozenClassError(self) def insert(self, index, p_object): raise ex.FrozenClassError(self) def pop(self, index=None): raise ex.FrozenClassError(self) def remove(self, value): raise ex.FrozenClassError(self) def reverse(self): raise ex.FrozenClassError(self) def sort(self, cmp=None, key=None, reverse=False): raise ex.FrozenClassError(self) def __add__(self, y): raise ex.FrozenClassError(self) def __delitem__(self, y): raise ex.FrozenClassError(self) def __delslice__(self, i, j): raise ex.FrozenClassError(self) def __iadd__(self, y): raise ex.FrozenClassError(self) def __imul__(self, y): raise ex.FrozenClassError(self) def __setitem__(self, i, y): raise ex.FrozenClassError(self) def __setslice__(self, i, j, y): raise ex.FrozenClassError(self) class FrozenDict(dict): def clear(self): raise ex.FrozenClassError(self) def pop(self, k, d=None, force=False): if force: return super(FrozenDict, self).pop(k, d) raise ex.FrozenClassError(self) def popitem(self): raise ex.FrozenClassError(self) def setdefault(self, k, d=None): raise ex.FrozenClassError(self) def update(self, E=None, **F): raise ex.FrozenClassError(self) def __delitem__(self, y): raise ex.FrozenClassError(self) def __setitem__(self, i, y): raise ex.FrozenClassError(self) def is_int(s): try: int(s) return True except Exception: return False def transform_to_num(s): # s can be a string or non-string. try: return int(str(s)) except ValueError: try: return float(str(s)) except ValueError: return s class Page(list): def __init__(self, l, prev=None, next=None): super(Page, self).__init__(l) self.prev = prev self.next = next sahara-8.0.0/sahara/utils/network.py0000666000175100017510000000310113245514472017450 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg CONF = cfg.CONF def has_floating_ip(instance): # Alternatively in each of these cases # we could use the nova client to look up the # ips for the instance and check the attributes # to ensure that the management_ip is a floating # ip, but a simple comparison with the internal_ip # corresponds with the logic in # sahara.service.networks.init_instances_ips if not instance.node_group.floating_ip_pool: return False # in the neutron case comparing ips is an extra simple check ... # maybe allocation of a floating ip failed for some reason # (Alternatively in each of these cases # we could use the nova client to look up the # ips for the instance and check the attributes # to ensure that the management_ip is a floating # ip, but a simple comparison with the internal_ip # corresponds with the logic in # sahara.service.networks.init_instances_ips) return instance.management_ip != instance.internal_ip sahara-8.0.0/sahara/utils/cluster.py0000666000175100017510000001430613245514472017451 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import socket from keystoneauth1 import exceptions as keystone_ex from oslo_config import cfg from oslo_log import log as logging from six.moves.urllib import parse from sahara import conductor as c from sahara import context from sahara import exceptions as e from sahara.utils.notification import sender from sahara.utils.openstack import base as auth_base conductor = c.API LOG = logging.getLogger(__name__) CONF = cfg.CONF # cluster status CLUSTER_STATUS_VALIDATING = "Validating" CLUSTER_STATUS_INFRAUPDATING = "InfraUpdating" CLUSTER_STATUS_SPAWNING = "Spawning" CLUSTER_STATUS_WAITING = "Waiting" CLUSTER_STATUS_PREPARING = "Preparing" CLUSTER_STATUS_CONFIGURING = "Configuring" CLUSTER_STATUS_STARTING = "Starting" CLUSTER_STATUS_ACTIVE = "Active" CLUSTER_STATUS_DECOMMISSIONING = "Decommissioning" CLUSTER_STATUS_ERROR = "Error" CLUSTER_STATUS_DELETING = "Deleting" CLUSTER_STATUS_AWAITINGTERMINATION = "AwaitingTermination" # cluster status -- Instances CLUSTER_STATUS_DELETING_INSTANCES = "Deleting Instances" CLUSTER_STATUS_ADDING_INSTANCES = "Adding Instances" # Scaling status CLUSTER_STATUS_SCALING = "Scaling" CLUSTER_STATUS_SCALING_SPAWNING = (CLUSTER_STATUS_SCALING + ": " + CLUSTER_STATUS_SPAWNING) CLUSTER_STATUS_SCALING_WAITING = (CLUSTER_STATUS_SCALING + ": " + CLUSTER_STATUS_WAITING) CLUSTER_STATUS_SCALING_PREPARING = (CLUSTER_STATUS_SCALING + ": " + CLUSTER_STATUS_PREPARING) # Rollback status CLUSTER_STATUS_ROLLBACK = "Rollback" CLUSTER_STATUS_ROLLBACK_SPAWNING = (CLUSTER_STATUS_ROLLBACK + ": " + CLUSTER_STATUS_SPAWNING) CLUSTER_STATUS_ROLLBACK_WAITING = (CLUSTER_STATUS_ROLLBACK + ": " + CLUSTER_STATUS_WAITING) CLUSTER_STATUS_ROLLBACK__PREPARING = (CLUSTER_STATUS_ROLLBACK + ": " + CLUSTER_STATUS_PREPARING) def change_cluster_status_description(cluster, status_description): try: ctx = context.ctx() return conductor.cluster_update( ctx, cluster, {'status_description': status_description}) except e.NotFoundException: return None def change_cluster_status(cluster, status, status_description=None): ctx = context.ctx() # Update cluster status. Race conditions with deletion are still possible, # but this reduces probability at least. cluster = conductor.cluster_get(ctx, cluster) if cluster else None if status_description is not None: change_cluster_status_description(cluster, status_description) # 'Deleting' is final and can't be changed if cluster is None or cluster.status == CLUSTER_STATUS_DELETING: return cluster update_dict = {"status": status} cluster = conductor.cluster_update(ctx, cluster, update_dict) conductor.cluster_provision_progress_update(ctx, cluster.id) LOG.info("Cluster status has been changed. New status=" "{status}".format(status=cluster.status)) sender.status_notify(cluster.id, cluster.name, cluster.status, "update") return cluster def count_instances(cluster): return sum([node_group.count for node_group in cluster.node_groups]) def check_cluster_exists(cluster): ctx = context.ctx() # check if cluster still exists (it might have been removed) cluster = conductor.cluster_get(ctx, cluster) return cluster is not None def get_instances(cluster, instances_ids=None): inst_map = {} for node_group in cluster.node_groups: for instance in node_group.instances: inst_map[instance.id] = instance if instances_ids is not None: return [inst_map[id] for id in instances_ids] else: return [v for v in inst_map.values()] def clean_cluster_from_empty_ng(cluster): ctx = context.ctx() for ng in cluster.node_groups: if ng.count == 0: conductor.node_group_remove(ctx, ng) def etc_hosts_entry_for_service(service): result = "" try: hostname = parse.urlparse( auth_base.url_for(service_type=service, endpoint_type="publicURL")).hostname except keystone_ex.EndpointNotFound: LOG.debug("Endpoint not found for service: '{}'".format(service)) return result try: result = "%s %s\n" % (socket.gethostbyname(hostname), hostname) except socket.gaierror: LOG.warning("Failed to resolve hostname of service: '{}'" .format(service)) result = "# Failed to resolve {} during deployment\n".format(hostname) return result def _etc_hosts_for_services(hosts): # add alias for keystone and swift for service in ["identity", "object-store"]: hosts += etc_hosts_entry_for_service(service) return hosts def _etc_hosts_for_instances(hosts, cluster): for node_group in cluster.node_groups: for instance in node_group.instances: hosts += "%s %s %s\n" % (instance.internal_ip, instance.fqdn(), instance.hostname()) return hosts def generate_etc_hosts(cluster): hosts = "127.0.0.1 localhost\n" if not cluster.use_designate_feature(): hosts = _etc_hosts_for_instances(hosts, cluster) hosts = _etc_hosts_for_services(hosts) return hosts def generate_resolv_conf_diff(curr_resolv_conf): # returns string that contains nameservers # which are lacked in the 'curr_resolve_conf' resolv_conf = "" for ns in CONF.nameservers: if ns not in curr_resolv_conf: resolv_conf += "nameserver {}\n".format(ns) return resolv_conf sahara-8.0.0/sahara/utils/notification/0000775000175100017510000000000013245515027020073 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/notification/__init__.py0000666000175100017510000000000013245514472022177 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/utils/notification/sender.py0000666000175100017510000000574713245514472021747 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging from sahara import context from sahara.utils import rpc as messaging LOG = logging.getLogger(__name__) SERVICE = 'sahara' CLUSTER_EVENT_TEMPLATE = "sahara.cluster.%s" HEALTH_EVENT_TYPE = CLUSTER_EVENT_TEMPLATE % "health" notifier_opts = [ cfg.StrOpt('level', default='INFO', deprecated_name='notification_level', deprecated_group='DEFAULT', help='Notification level for outgoing notifications'), cfg.StrOpt('publisher_id', deprecated_name='notification_publisher_id', deprecated_group='DEFAULT', help='Identifier of the publisher') ] notifier_opts_group = 'oslo_messaging_notifications' CONF = cfg.CONF CONF.register_opts(notifier_opts, group=notifier_opts_group) def _get_publisher(): publisher_id = CONF.oslo_messaging_notifications.publisher_id if publisher_id is None: publisher_id = SERVICE return publisher_id def _notify(event_type, body): LOG.debug("Notification about cluster is going to be sent. Notification " "type={type}".format(type=event_type)) ctx = context.ctx() level = CONF.oslo_messaging_notifications.level body.update({'project_id': ctx.tenant_id, 'user_id': ctx.user_id}) client = messaging.get_notifier(_get_publisher()) method = getattr(client, level.lower()) method(ctx, event_type, body) def _health_notification_body(cluster, health_check): verification = cluster.verification return { 'cluster_id': cluster.id, 'cluster_name': cluster.name, 'verification_id': verification['id'], 'health_check_status': health_check['status'], 'health_check_name': health_check['name'], 'health_check_description': health_check['description'], 'created_at': health_check['created_at'], 'updated_at': health_check['updated_at'] } def status_notify(cluster_id, cluster_name, cluster_status, ev_type): """Sends notification about creating/updating/deleting cluster.""" _notify(CLUSTER_EVENT_TEMPLATE % ev_type, { 'cluster_id': cluster_id, 'cluster_name': cluster_name, 'cluster_status': cluster_status}) def health_notify(cluster, health_check): """Sends notification about current cluster health.""" _notify(HEALTH_EVENT_TYPE, _health_notification_body(cluster, health_check)) sahara-8.0.0/sahara/utils/api_validator.py0000666000175100017510000001316713245514472020612 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re import jsonschema from oslo_utils import uuidutils import six from sahara.service.edp.job_binaries import manager as jb_manager @jsonschema.FormatChecker.cls_checks('valid_name_hostname') def validate_name_hostname_format(entry): if not isinstance(entry, six.string_types) or not entry: # should fail type or length validation return True res = re.match(r"^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]" r"*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z]" r"[A-Za-z0-9\-]*[A-Za-z0-9])$", entry) return res is not None @jsonschema.FormatChecker.cls_checks('valid_name') def validate_name_format(entry): if not isinstance(entry, six.string_types): # should fail type validation return True res = re.match(r"^[a-zA-Z0-9][a-zA-Z0-9\-_\.]*$", entry) return res is not None @jsonschema.FormatChecker.cls_checks('valid_keypair_name') def validate_keypair_name_format(entry): if not isinstance(entry, six.string_types): # should fail type validation return True # this follows the validation put forth by nova for keypair names res = re.match(r'^[a-zA-Z0-9\-_ ]+$', entry) return res is not None @jsonschema.FormatChecker.cls_checks('valid_job_location') def validate_job_location_format(entry): if not isinstance(entry, six.string_types): # should fail type validation return True return jb_manager.JOB_BINARIES \ .get_job_binary_by_url(entry) \ .validate_job_location_format(entry) @jsonschema.FormatChecker.cls_checks('valid_tag') def validate_valid_tag_format(entry): if not isinstance(entry, six.string_types): # should fail type validation return True res = re.match(r"^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-_]" r"*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9]" r"[A-Za-z0-9\-_]*[A-Za-z0-9])$", entry) return res is not None @jsonschema.FormatChecker.cls_checks('uuid') def validate_uuid_format(entry): if not isinstance(entry, six.string_types): # should fail type validation return True return uuidutils.is_uuid_like(entry) @jsonschema.FormatChecker.cls_checks('posix_path') def validate_posix_path(entry): if not isinstance(entry, six.string_types): # should fail type validation return True res = re.match("^(/([A-Z]|[a-z]|[0-9]|\-|_)+)+$", entry) return res is not None class ConfigTypeMeta(type): def __instancecheck__(cls, instance): # configs should be dict if not isinstance(instance, dict): return False # check dict content for applicable_target, configs in six.iteritems(instance): # upper-level dict keys (applicable targets) should be strings if not isinstance(applicable_target, six.string_types): return False # upper-level dict values should be dicts if not isinstance(configs, dict): return False # check internal dict content for config_name, config_value in six.iteritems(configs): # internal dict keys should be strings if not isinstance(config_name, six.string_types): return False # internal dict values should be strings or integers or bools if not isinstance(config_value, (six.string_types, six.integer_types)): return False return True class SimpleConfigTypeMeta(type): def __instancecheck__(cls, instance): # configs should be dict if not isinstance(instance, dict): return False # check dict content for conf_name, conf_value in six.iteritems(instance): # keys should be strings, values should be int, string or bool if not isinstance(conf_name, six.string_types): return False if not isinstance(conf_value, (six.string_types, six.integer_types)): return False return True @six.add_metaclass(ConfigTypeMeta) class ConfigsType(dict): pass @six.add_metaclass(SimpleConfigTypeMeta) class SimpleConfigsType(dict): pass class FlavorTypeMeta(type): def __instancecheck__(cls, instance): try: int(instance) except (ValueError, TypeError): return (isinstance(instance, six.string_types) and uuidutils.is_uuid_like(instance)) return (isinstance(instance, six.integer_types + six.string_types) and type(instance) != bool) @six.add_metaclass(FlavorTypeMeta) class FlavorType(object): pass class ApiValidator(jsonschema.Draft4Validator): def __init__(self, schema): format_checker = jsonschema.FormatChecker() super(ApiValidator, self).__init__( schema, format_checker=format_checker, types={ "configs": ConfigsType, "flavor": FlavorType, "simple_config": SimpleConfigsType, }) sahara-8.0.0/sahara/plugins/0000775000175100017510000000000013245515026015725 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/ambari/0000775000175100017510000000000013245515026017160 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/ambari/deploy.py0000666000175100017510000006251113245514472021041 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools import telnetlib # nosec from oslo_log import log as logging from oslo_utils import uuidutils from sahara import conductor from sahara import context from sahara.i18n import _ from sahara.plugins.ambari import client as ambari_client from sahara.plugins.ambari import common as p_common from sahara.plugins.ambari import configs from sahara.plugins.ambari import ha_helper from sahara.plugins import kerberos from sahara.plugins import utils as plugin_utils from sahara.topology import topology_helper as t_helper from sahara.utils import cluster_progress_ops as cpo from sahara.utils import poll_utils LOG = logging.getLogger(__name__) conductor = conductor.API repo_id_map = { "2.3": { "HDP": "HDP-2.3", "HDP-UTILS": "HDP-UTILS-1.1.0.20" }, "2.4": { "HDP": "HDP-2.4", "HDP-UTILS": "HDP-UTILS-1.1.0.20" }, "2.5": { "HDP": "HDP-2.5", "HDP-UTILS": "HDP-UTILS-1.1.0.20" }, } os_type_map = { "centos6": "redhat6", "redhat6": "redhat6", "centos7": "redhat7", "redhat7": "redhat7", "ubuntu14": "ubuntu14" } @cpo.event_wrapper(True, step=_("Set up Ambari management console"), param=('cluster', 0)) def setup_ambari(cluster): LOG.debug("Set up Ambari management console") ambari = plugin_utils.get_instance(cluster, p_common.AMBARI_SERVER) with ambari.remote() as r: sudo = functools.partial(r.execute_command, run_as_root=True) sudo("rngd -r /dev/urandom -W 4096") sudo("ambari-server setup -s -j" " `cut -f2 -d \"=\" /etc/profile.d/99-java.sh`", timeout=1800) redirect_file = "/tmp/%s" % uuidutils.generate_uuid() sudo("service ambari-server start >{rfile} && " "cat {rfile} && rm {rfile}".format(rfile=redirect_file)) LOG.debug("Ambari management console installed") def setup_agents(cluster, instances=None): LOG.debug("Set up Ambari agents") manager_address = plugin_utils.get_instance( cluster, p_common.AMBARI_SERVER).fqdn() if not instances: instances = plugin_utils.get_instances(cluster) _setup_agents(instances, manager_address) def _setup_agents(instances, manager_address): cpo.add_provisioning_step( instances[0].cluster.id, _("Set up Ambari agents"), len(instances)) with context.ThreadGroup() as tg: for inst in instances: tg.spawn("hwx-agent-setup-%s" % inst.id, _setup_agent, inst, manager_address) LOG.debug("Ambari agents have been installed") def _disable_repos_on_inst(instance): with context.set_current_instance_id(instance_id=instance.instance_id): with instance.remote() as r: sudo = functools.partial(r.execute_command, run_as_root=True) if r.get_os_distrib() == "ubuntu": sudo("mv /etc/apt/sources.list /etc/apt/sources.list.tmp") else: tmp_name = "/tmp/yum.repos.d-%s" % instance.instance_id[:8] # moving to other folder sudo("mv /etc/yum.repos.d/ {fold_name}".format( fold_name=tmp_name)) sudo("mkdir /etc/yum.repos.d") def disable_repos(cluster): if configs.use_base_repos_needed(cluster): LOG.debug("Using base repos") return instances = plugin_utils.get_instances(cluster) with context.ThreadGroup() as tg: for inst in instances: tg.spawn("disable-repos-%s" % inst.instance_name, _disable_repos_on_inst, inst) @cpo.event_wrapper(True) def _setup_agent(instance, ambari_address): with instance.remote() as r: sudo = functools.partial(r.execute_command, run_as_root=True) r.replace_remote_string("/etc/ambari-agent/conf/ambari-agent.ini", "localhost", ambari_address) try: sudo("ambari-agent start") except Exception as e: # workaround for ubuntu, because on ubuntu the ambari agent # starts automatically after image boot msg = _("Restart of ambari-agent is needed for host {}, " "reason: {}").format(instance.fqdn(), e) LOG.exception(msg) sudo("ambari-agent restart") # for correct installing packages r.update_repository() @cpo.event_wrapper(True, step=_("Wait Ambari accessible"), param=('cluster', 0)) def wait_ambari_accessible(cluster): ambari = plugin_utils.get_instance(cluster, p_common.AMBARI_SERVER) kwargs = {"host": ambari.management_ip, "port": 8080} poll_utils.poll(_check_port_accessible, kwargs=kwargs, timeout=300) def _check_port_accessible(host, port): try: conn = telnetlib.Telnet(host, port) conn.close() return True except IOError: return False def resolve_package_conflicts(cluster, instances=None): if not instances: instances = plugin_utils.get_instances(cluster) for instance in instances: with instance.remote() as r: if r.get_os_distrib() == 'ubuntu': try: r.execute_command( "apt-get remove -y libmysql-java", run_as_root=True) except Exception: LOG.warning("Can't remove libmysql-java, " "it's probably not installed") def _prepare_ranger(cluster): ranger = plugin_utils.get_instance(cluster, p_common.RANGER_ADMIN) if not ranger: return ambari = plugin_utils.get_instance(cluster, p_common.AMBARI_SERVER) with ambari.remote() as r: sudo = functools.partial(r.execute_command, run_as_root=True) sudo("ambari-server setup --jdbc-db=mysql " "--jdbc-driver=/usr/share/java/mysql-connector-java.jar") init_db_template = ( "create user 'root'@'%' identified by '{password}';\n" "set password for 'root'@'localhost' = password('{password}');") password = uuidutils.generate_uuid() extra = cluster.extra.to_dict() if cluster.extra else {} extra["ranger_db_password"] = password ctx = context.ctx() conductor.cluster_update(ctx, cluster, {"extra": extra}) with ranger.remote() as r: sudo = functools.partial(r.execute_command, run_as_root=True) # TODO(sreshetnyak): add ubuntu support sudo("yum install -y mysql-server") sudo("service mysqld start") r.write_file_to("/tmp/init.sql", init_db_template.format(password=password)) sudo("mysql < /tmp/init.sql") sudo("rm /tmp/init.sql") @cpo.event_wrapper(True, step=_("Prepare Hive"), param=('cluster', 0)) def prepare_hive(cluster): hive = plugin_utils.get_instance(cluster, p_common.HIVE_SERVER) if not hive: return with hive.remote() as r: r.execute_command( 'sudo su - -c "hadoop fs -mkdir /user/oozie/conf" hdfs') r.execute_command( 'sudo su - -c "hadoop fs -copyFromLocal ' '/etc/hive/conf/hive-site.xml ' '/user/oozie/conf/hive-site.xml" hdfs') @cpo.event_wrapper(True, step=_("Update default Ambari password"), param=('cluster', 0)) def update_default_ambari_password(cluster): ambari = plugin_utils.get_instance(cluster, p_common.AMBARI_SERVER) new_password = uuidutils.generate_uuid() with ambari_client.AmbariClient(ambari) as client: client.update_user_password("admin", "admin", new_password) extra = cluster.extra.to_dict() if cluster.extra else {} extra["ambari_password"] = new_password ctx = context.ctx() conductor.cluster_update(ctx, cluster, {"extra": extra}) cluster = conductor.cluster_get(ctx, cluster.id) @cpo.event_wrapper(True, step=_("Wait registration of hosts"), param=('cluster', 0)) def wait_host_registration(cluster, instances): with _get_ambari_client(cluster) as client: kwargs = {"client": client, "instances": instances} poll_utils.poll(_check_host_registration, kwargs=kwargs, timeout=600) def _check_host_registration(client, instances): hosts = client.get_registered_hosts() registered_host_names = [h["Hosts"]["host_name"] for h in hosts] for instance in instances: if instance.fqdn() not in registered_host_names: return False return True @cpo.event_wrapper(True, step=_("Set up HDP repositories"), param=('cluster', 0)) def _set_up_hdp_repos(cluster, hdp_repo, hdp_utils_repo): ambari = plugin_utils.get_instance(cluster, p_common.AMBARI_SERVER) pv = cluster.hadoop_version repos = repo_id_map[pv] with _get_ambari_client(cluster) as client: os_type = os_type_map[client.get_host_info(ambari.fqdn())["os_type"]] if hdp_repo: client.set_up_mirror(pv, os_type, repos["HDP"], hdp_repo) if hdp_utils_repo: client.set_up_mirror(pv, os_type, repos["HDP-UTILS"], hdp_utils_repo) def set_up_hdp_repos(cluster): hdp_repo = configs.get_hdp_repo_url(cluster) hdp_utils_repo = configs.get_hdp_utils_repo_url(cluster) if hdp_repo or hdp_utils_repo: _set_up_hdp_repos(cluster, hdp_repo, hdp_utils_repo) def get_kdc_server(cluster): return plugin_utils.get_instance( cluster, p_common.AMBARI_SERVER) def _prepare_kerberos(cluster, instances=None): if instances is None: kerberos.deploy_infrastructure(cluster, get_kdc_server(cluster)) kerberos.prepare_policy_files(cluster) else: server = None if not kerberos.using_existing_kdc(cluster): server = get_kdc_server(cluster) kerberos.setup_clients(cluster, server) kerberos.prepare_policy_files(cluster) def prepare_kerberos(cluster, instances=None): if kerberos.is_kerberos_security_enabled(cluster): _prepare_kerberos(cluster, instances) def _serialize_mit_kdc_kerberos_env(cluster): return { 'kerberos-env': { "realm": kerberos.get_realm_name(cluster), "kdc_type": "mit-kdc", "kdc_host": kerberos.get_kdc_host( cluster, get_kdc_server(cluster)), "admin_server_host": kerberos.get_kdc_host( cluster, get_kdc_server(cluster)), 'encryption_types': 'aes256-cts-hmac-sha1-96', 'ldap_url': '', 'container_dn': '', } } def _serialize_krb5_configs(cluster): return { "krb5-conf": { "properties_attributes": {}, "properties": { "manage_krb5_conf": "false" } } } def _get_credentials(cluster): return [{ "alias": "kdc.admin.credential", "principal": kerberos.get_admin_principal(cluster), "key": kerberos.get_server_password(cluster), "type": "TEMPORARY" }] def get_host_group_components(cluster, processes): result = [] for proc in processes: result.append({'name': proc}) return result @cpo.event_wrapper(True, step=_("Create Ambari blueprint"), param=('cluster', 0)) def create_blueprint(cluster): _prepare_ranger(cluster) cluster = conductor.cluster_get(context.ctx(), cluster.id) host_groups = [] for ng in cluster.node_groups: procs = p_common.get_ambari_proc_list(ng) procs.extend(p_common.get_clients(cluster)) for instance in ng.instances: hg = { "name": instance.instance_name, "configurations": configs.get_instance_params(instance), "components": get_host_group_components(cluster, procs) } host_groups.append(hg) bp = { "Blueprints": { "stack_name": "HDP", "stack_version": cluster.hadoop_version, }, "host_groups": host_groups, "configurations": configs.get_cluster_params(cluster) } if kerberos.is_kerberos_security_enabled(cluster): bp['configurations'].extend([ _serialize_mit_kdc_kerberos_env(cluster), _serialize_krb5_configs(cluster) ]) bp['Blueprints']['security'] = {'type': 'KERBEROS'} general_configs = cluster.cluster_configs.get("general", {}) if (general_configs.get(p_common.NAMENODE_HA) or general_configs.get(p_common.RESOURCEMANAGER_HA) or general_configs.get(p_common.HBASE_REGIONSERVER_HA)): bp = ha_helper.update_bp_ha_common(cluster, bp) if general_configs.get(p_common.NAMENODE_HA): bp = ha_helper.update_bp_for_namenode_ha(cluster, bp) if general_configs.get(p_common.RESOURCEMANAGER_HA): bp = ha_helper.update_bp_for_resourcemanager_ha(cluster, bp) if general_configs.get(p_common.HBASE_REGIONSERVER_HA): bp = ha_helper.update_bp_for_hbase_ha(cluster, bp) with _get_ambari_client(cluster) as client: return client.create_blueprint(cluster.name, bp) def _build_ambari_cluster_template(cluster): cl_tmpl = { "blueprint": cluster.name, "default_password": uuidutils.generate_uuid(), "host_groups": [] } if cluster.use_autoconfig: strategy = configs.get_auto_configuration_strategy(cluster) cl_tmpl["config_recommendation_strategy"] = strategy if kerberos.is_kerberos_security_enabled(cluster): cl_tmpl["credentials"] = _get_credentials(cluster) cl_tmpl["security"] = {"type": "KERBEROS"} topology = _get_topology_data(cluster) for ng in cluster.node_groups: for instance in ng.instances: host = {"fqdn": instance.fqdn()} if t_helper.is_data_locality_enabled(): host["rack_info"] = topology[instance.instance_name] cl_tmpl["host_groups"].append({ "name": instance.instance_name, "hosts": [host] }) return cl_tmpl @cpo.event_wrapper(True, step=_("Start cluster"), param=('cluster', 0)) def start_cluster(cluster): ambari_template = _build_ambari_cluster_template(cluster) with _get_ambari_client(cluster) as client: req_id = client.create_cluster(cluster.name, ambari_template)["id"] client.wait_ambari_request(req_id, cluster.name) @cpo.event_wrapper(True) def _add_host_to_cluster(instance, client): client.add_host_to_cluster(instance) def add_new_hosts(cluster, instances): with _get_ambari_client(cluster) as client: cpo.add_provisioning_step( cluster.id, _("Add new hosts"), len(instances)) for inst in instances: _add_host_to_cluster(inst, client) @cpo.event_wrapper(True, step=_("Generate config groups"), param=('cluster', 0)) def manage_config_groups(cluster, instances): groups = [] for instance in instances: groups.extend(configs.get_config_group(instance)) with _get_ambari_client(cluster) as client: client.create_config_group(cluster, groups) @cpo.event_wrapper(True, step=_("Cleanup config groups"), param=('cluster', 0)) def cleanup_config_groups(cluster, instances): to_remove = set() for instance in instances: cfg_name = "%s:%s" % (cluster.name, instance.instance_name) to_remove.add(cfg_name) with _get_ambari_client(cluster) as client: config_groups = client.get_config_groups(cluster) for group in config_groups['items']: cfg_id = group['ConfigGroup']['id'] detailed = client.get_detailed_config_group(cluster, cfg_id) cfg_name = detailed['ConfigGroup']['group_name'] # we have config group per host if cfg_name in to_remove: client.remove_config_group(cluster, cfg_id) @cpo.event_wrapper(True, step=_("Regenerate keytabs for Kerberos"), param=('cluster', 0)) def _regenerate_keytabs(cluster): with _get_ambari_client(cluster) as client: alias = "kdc.admin.credential" try: client.get_credential(cluster.name, alias) except ambari_client.AmbariNotFound: # credentials are missing data = { 'Credential': { "principal": kerberos.get_admin_principal(cluster), "key": kerberos.get_server_password(cluster), "type": "TEMPORARY" } } client.import_credential(cluster.name, alias, data) req_id = client.regenerate_keytabs(cluster.name) client.wait_ambari_request(req_id, cluster.name) @cpo.event_wrapper(True, step=_("Install services on hosts"), param=('cluster', 0)) def _install_services_to_hosts(cluster, instances): requests_ids = [] with _get_ambari_client(cluster) as client: clients = p_common.get_clients(cluster) for instance in instances: services = p_common.get_ambari_proc_list(instance.node_group) services.extend(clients) for service in services: client.add_service_to_host(instance, service) requests_ids.append( client.start_service_on_host( instance, service, 'INSTALLED')) client.wait_ambari_requests(requests_ids, cluster.name) @cpo.event_wrapper(True, step=_("Start services on hosts"), param=('cluster', 0)) def _start_services_on_hosts(cluster, instances): with _get_ambari_client(cluster) as client: # all services added and installed, let's start them requests_ids = [] for instance in instances: services = p_common.get_ambari_proc_list(instance.node_group) services.extend(p_common.ALL_LIST) for service in services: requests_ids.append( client.start_service_on_host( instance, service, 'STARTED')) client.wait_ambari_requests(requests_ids, cluster.name) def manage_host_components(cluster, instances): _install_services_to_hosts(cluster, instances) if kerberos.is_kerberos_security_enabled(cluster): _regenerate_keytabs(cluster) _start_services_on_hosts(cluster, instances) @cpo.event_wrapper(True, step=_("Decommission NodeManagers and DataNodes"), param=('cluster', 0)) def decommission_hosts(cluster, instances): nodemanager_instances = filter( lambda i: p_common.NODEMANAGER in i.node_group.node_processes, instances) if len(nodemanager_instances) > 0: decommission_nodemanagers(cluster, nodemanager_instances) datanode_instances = filter( lambda i: p_common.DATANODE in i.node_group.node_processes, instances) if len(datanode_instances) > 0: decommission_datanodes(cluster, datanode_instances) def decommission_nodemanagers(cluster, instances): with _get_ambari_client(cluster) as client: client.decommission_nodemanagers(cluster.name, instances) def decommission_datanodes(cluster, instances): with _get_ambari_client(cluster) as client: client.decommission_datanodes(cluster.name, instances) def restart_namenode(cluster, instance): with _get_ambari_client(cluster) as client: client.restart_namenode(cluster.name, instance) def restart_resourcemanager(cluster, instance): with _get_ambari_client(cluster) as client: client.restart_resourcemanager(cluster.name, instance) @cpo.event_wrapper(True, step=_("Restart NameNodes and ResourceManagers"), param=('cluster', 0)) def restart_nns_and_rms(cluster): nns = plugin_utils.get_instances(cluster, p_common.NAMENODE) for nn in nns: restart_namenode(cluster, nn) rms = plugin_utils.get_instances(cluster, p_common.RESOURCEMANAGER) for rm in rms: restart_resourcemanager(cluster, rm) def restart_service(cluster, service_name): with _get_ambari_client(cluster) as client: client.restart_service(cluster.name, service_name) @cpo.event_wrapper(True, step=_("Remove hosts"), param=('cluster', 0)) def remove_services_from_hosts(cluster, instances): for inst in instances: LOG.debug("Stopping and removing processes from host %s", inst.fqdn()) _remove_services_from_host(cluster, inst) LOG.debug("Removing the host %s", inst.fqdn()) _remove_host(cluster, inst) def _remove_services_from_host(cluster, instance): with _get_ambari_client(cluster) as client: hdp_processes = client.list_host_processes(cluster.name, instance) for proc in hdp_processes: LOG.debug("Stopping process %(proc)s on host %(fqdn)s ", {'proc': proc, 'fqdn': instance.fqdn()}) client.stop_process_on_host(cluster.name, instance, proc) LOG.debug("Removing process %(proc)s from host %(fqdn)s ", {'proc': proc, 'fqdn': instance.fqdn()}) client.remove_process_from_host(cluster.name, instance, proc) _wait_all_processes_removed(cluster, instance) def _remove_host(cluster, inst): with _get_ambari_client(cluster) as client: client.delete_host(cluster.name, inst) def _wait_all_processes_removed(cluster, instance): with _get_ambari_client(cluster) as client: while True: hdp_processes = client.list_host_processes(cluster.name, instance) if not hdp_processes: return context.sleep(5) def _get_ambari_client(cluster): ambari = plugin_utils.get_instance(cluster, p_common.AMBARI_SERVER) password = cluster.extra["ambari_password"] return ambari_client.AmbariClient(ambari, password=password) def _get_topology_data(cluster): if not t_helper.is_data_locality_enabled(): return {} LOG.warning("Node group awareness is not implemented in YARN yet " "so enable_hypervisor_awareness set to False " "explicitly") return t_helper.generate_topology_map(cluster, is_node_awareness=False) @cpo.event_wrapper(True) def _configure_topology_data(cluster, inst, client): topology = _get_topology_data(cluster) client.set_rack_info_for_instance( cluster.name, inst, topology[inst.instance_name]) @cpo.event_wrapper(True, step=_("Restart HDFS and MAPREDUCE2 services"), param=('cluster', 0)) def _restart_hdfs_and_mapred_services(cluster, client): client.restart_service(cluster.name, p_common.HDFS_SERVICE) client.restart_service(cluster.name, p_common.MAPREDUCE2_SERVICE) def configure_rack_awareness(cluster, instances): if not t_helper.is_data_locality_enabled(): return with _get_ambari_client(cluster) as client: cpo.add_provisioning_step( cluster.id, _("Configure rack awareness"), len(instances)) for inst in instances: _configure_topology_data(cluster, inst, client) _restart_hdfs_and_mapred_services(cluster, client) @cpo.event_wrapper(True) def _add_hadoop_swift_jar(instance, new_jar): with instance.remote() as r: code, out = r.execute_command( "test -f %s" % new_jar, raise_when_error=False) if code == 0: # get ambari hadoop version (e.g.: 2.7.1.2.3.4.0-3485) code, amb_hadoop_version = r.execute_command( "sudo hadoop version | grep 'Hadoop' | awk '{print $2}'") amb_hadoop_version = amb_hadoop_version.strip() # get special code of ambari hadoop version(e.g.:2.3.4.0-3485) amb_code = '.'.join(amb_hadoop_version.split('.')[3:]) origin_jar = ( "/usr/hdp/{}/hadoop-mapreduce/hadoop-openstack-{}.jar".format( amb_code, amb_hadoop_version)) r.execute_command("sudo cp {} {}".format(new_jar, origin_jar)) else: LOG.warning("The {jar_file} file cannot be found " "in the {dir} directory so Keystone API v3 " "is not enabled for this cluster." .format(jar_file="hadoop-openstack.jar", dir="/opt")) def add_hadoop_swift_jar(instances): new_jar = "/opt/hadoop-openstack.jar" cpo.add_provisioning_step(instances[0].cluster.id, _("Add Hadoop Swift jar to instances"), len(instances)) for inst in instances: _add_hadoop_swift_jar(inst, new_jar) def deploy_kerberos_principals(cluster, instances=None): if not kerberos.is_kerberos_security_enabled(cluster): return if instances is None: instances = plugin_utils.get_instances(cluster) mapper = { 'hdfs': plugin_utils.instances_with_services( instances, [p_common.SECONDARY_NAMENODE, p_common.NAMENODE, p_common.DATANODE, p_common.JOURNAL_NODE]), 'spark': plugin_utils.instances_with_services( instances, [p_common.SPARK_JOBHISTORYSERVER]), 'oozie': plugin_utils.instances_with_services( instances, [p_common.OOZIE_SERVER]), } kerberos.create_keytabs_for_map(cluster, mapper) sahara-8.0.0/sahara/plugins/ambari/client.py0000666000175100017510000003504713245514472021027 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_log import log as logging from oslo_serialization import jsonutils from requests import auth from sahara import context from sahara.i18n import _ from sahara.plugins.ambari import requests_helper as r_helper from sahara.plugins import exceptions as p_exc LOG = logging.getLogger(__name__) class AmbariNotFound(Exception): pass class AmbariClient(object): def __init__(self, instance, port="8080", **kwargs): kwargs.setdefault("username", "admin") kwargs.setdefault("password", "admin") self._port = port self._base_url = "http://{host}:{port}/api/v1".format( host=instance.management_ip, port=port) self._instance = instance self._http_client = instance.remote().get_http_client(port) self._headers = {"X-Requested-By": "sahara"} self._auth = auth.HTTPBasicAuth(kwargs["username"], kwargs["password"]) self._default_client_args = {"verify": False, "auth": self._auth, "headers": self._headers} def __enter__(self): return self def __exit__(self, type, value, traceback): self.close() def close(self): self._instance.remote().close_http_session(self._port) def get(self, *args, **kwargs): kwargs.update(self._default_client_args) return self._http_client.get(*args, **kwargs) def post(self, *args, **kwargs): kwargs.update(self._default_client_args) return self._http_client.post(*args, **kwargs) def put(self, *args, **kwargs): kwargs.update(self._default_client_args) return self._http_client.put(*args, **kwargs) def delete(self, *args, **kwargs): kwargs.update(self._default_client_args) return self._http_client.delete(*args, **kwargs) def get_alerts_data(self, cluster): url = self._base_url + "/clusters/%s/alerts?fields=*" % cluster.name resp = self.get(url) data = self.check_response(resp) return data.get('items', []) @staticmethod def check_response(resp, handle_not_found=False): if handle_not_found and resp.status_code == 404: raise AmbariNotFound() resp.raise_for_status() if resp.text: return jsonutils.loads(resp.text) @staticmethod def req_id(response): if not response.text: raise p_exc.HadoopProvisionError("Cannot find request id. " "No response body") body = jsonutils.loads(response.text) if "Requests" not in body or "id" not in body["Requests"]: raise p_exc.HadoopProvisionError("Cannot find request id. " "Unexpected response format") return body["Requests"]["id"] def import_credential(self, cl_name, alias, data): url = self._base_url + "/clusters/%s/credentials/%s" % (cl_name, alias) resp = self.post(url, data=jsonutils.dumps(data)) self.check_response(resp) def get_credential(self, cl_name, alias): url = self._base_url + "/clusters/%s/credentials/%s" % (cl_name, alias) resp = self.get(url) self.check_response(resp, handle_not_found=True) def regenerate_keytabs(self, cl_name): url = (self._base_url + "/clusters/%s?regenerate_keytabs=missing" % cl_name) data = jsonutils.dumps({"Clusters": {"security_type": "KERBEROS"}}) resp = self.put(url, data=data) self.check_response(resp) return self.req_id(resp) def get_registered_hosts(self): url = self._base_url + "/hosts" resp = self.get(url) data = self.check_response(resp) return data.get("items", []) def get_host_info(self, host): url = self._base_url + "/hosts/%s" % host resp = self.get(url) data = self.check_response(resp) return data.get("Hosts", {}) def update_user_password(self, user, old_password, new_password): url = self._base_url + "/users/%s" % user data = jsonutils.dumps({ "Users": { "old_password": old_password, "password": new_password } }) resp = self.put(url, data=data) self.check_response(resp) def create_blueprint(self, name, data): url = self._base_url + "/blueprints/%s" % name resp = self.post(url, data=jsonutils.dumps(data)) return self.check_response(resp) def create_cluster(self, name, data): url = self._base_url + "/clusters/%s" % name resp = self.post(url, data=jsonutils.dumps(data)) return self.check_response(resp).get("Requests") def add_host_to_cluster(self, instance): cluster_name = instance.cluster.name hostname = instance.fqdn() url = self._base_url + "/clusters/{cluster}/hosts/{hostname}".format( cluster=cluster_name, hostname=hostname) resp = self.post(url) self.check_response(resp) def get_config_groups(self, cluster): url = self._base_url + "/clusters/%s/config_groups" % cluster.name resp = self.get(url) return self.check_response(resp) def get_detailed_config_group(self, cluster, cfg_id): url = self._base_url + "/clusters/%s/config_groups/%s" % ( cluster.name, cfg_id) resp = self.get(url) return self.check_response(resp) def remove_config_group(self, cluster, cfg_id): url = self._base_url + "/clusters/%s/config_groups/%s" % ( cluster.name, cfg_id) resp = self.delete(url) return self.check_response(resp) def create_config_group(self, cluster, data): url = self._base_url + "/clusters/%s/config_groups" % cluster.name resp = self.post(url, data=jsonutils.dumps(data)) return self.check_response(resp) def add_service_to_host(self, inst, service): url = "{pref}/clusters/{cluster}/hosts/{host}/host_components/{proc}" url = url.format(pref=self._base_url, cluster=inst.cluster.name, host=inst.fqdn(), proc=service) self.check_response(self.post(url)) def start_service_on_host(self, inst, service, final_state): url = "{pref}/clusters/{cluster}/hosts/{host}/host_components/{proc}" url = url.format( pref=self._base_url, cluster=inst.cluster.name, host=inst.fqdn(), proc=service) data = { 'HostRoles': { 'state': final_state }, 'RequestInfo': { 'context': "Starting service {service}, moving to state " "{state}".format(service=service, state=final_state) } } resp = self.put(url, data=jsonutils.dumps(data)) self.check_response(resp) # return req_id to check health of request return self.req_id(resp) def decommission_nodemanagers(self, cluster_name, instances): url = self._base_url + "/clusters/%s/requests" % cluster_name data = r_helper.build_nodemanager_decommission_request(cluster_name, instances) resp = self.post(url, data=jsonutils.dumps(data)) self.wait_ambari_request(self.req_id(resp), cluster_name) def decommission_datanodes(self, cluster_name, instances): url = self._base_url + "/clusters/%s/requests" % cluster_name data = r_helper.build_datanode_decommission_request(cluster_name, instances) resp = self.post(url, data=jsonutils.dumps(data)) self.wait_ambari_request(self.req_id(resp), cluster_name) def remove_process_from_host(self, cluster_name, instance, process): url = self._base_url + "/clusters/%s/hosts/%s/host_components/%s" % ( cluster_name, instance.fqdn(), process) resp = self.delete(url) return self.check_response(resp) def stop_process_on_host(self, cluster_name, instance, process): url = self._base_url + "/clusters/%s/hosts/%s/host_components/%s" % ( cluster_name, instance.fqdn(), process) check_installed_resp = self.check_response(self.get(url)) if check_installed_resp["HostRoles"]["state"] != "INSTALLED": data = {"HostRoles": {"state": "INSTALLED"}, "RequestInfo": {"context": "Stopping %s" % process}} resp = self.put(url, data=jsonutils.dumps(data)) self.wait_ambari_request(self.req_id(resp), cluster_name) def restart_namenode(self, cluster_name, instance): url = self._base_url + "/clusters/%s/requests" % cluster_name data = r_helper.build_namenode_restart_request(cluster_name, instance) resp = self.post(url, data=jsonutils.dumps(data)) self.wait_ambari_request(self.req_id(resp), cluster_name) def restart_resourcemanager(self, cluster_name, instance): url = self._base_url + "/clusters/%s/requests" % cluster_name data = r_helper.build_resourcemanager_restart_request(cluster_name, instance) resp = self.post(url, data=jsonutils.dumps(data)) self.wait_ambari_request(self.req_id(resp), cluster_name) def restart_service(self, cluster_name, service_name): url = self._base_url + "/clusters/{}/services/{}".format( cluster_name, service_name) data = r_helper.build_stop_service_request(service_name) resp = self.put(url, data=jsonutils.dumps(data)) self.wait_ambari_request(self.req_id(resp), cluster_name) data = r_helper.build_start_service_request(service_name) resp = self.put(url, data=jsonutils.dumps(data)) self.wait_ambari_request(self.req_id(resp), cluster_name) def delete_host(self, cluster_name, instance): url = self._base_url + "/clusters/%s/hosts/%s" % (cluster_name, instance.fqdn()) resp = self.delete(url) return self.check_response(resp) def check_request_status(self, cluster_name, req_id): url = self._base_url + "/clusters/%s/requests/%d" % (cluster_name, req_id) resp = self.get(url) return self.check_response(resp).get("Requests") def list_host_processes(self, cluster_name, instance): url = self._base_url + "/clusters/%s/hosts/%s" % ( cluster_name, instance.fqdn()) resp = self.get(url) body = jsonutils.loads(resp.text) procs = [p["HostRoles"]["component_name"] for p in body["host_components"]] return procs def set_up_mirror(self, stack_version, os_type, repo_id, repo_url): url = self._base_url + ( "/stacks/HDP/versions/%s/operating_systems/%s/repositories/%s") % ( stack_version, os_type, repo_id) data = { "Repositories": { "base_url": repo_url, "verify_base_url": True } } resp = self.put(url, data=jsonutils.dumps(data)) self.check_response(resp) def set_rack_info_for_instance(self, cluster_name, instance, rack_name): url = self._base_url + "/clusters/%s/hosts/%s" % ( cluster_name, instance.fqdn()) data = { "Hosts": { "rack_info": rack_name } } resp = self.put(url, data=jsonutils.dumps(data)) self.check_response(resp) def get_request_info(self, cluster_name, request_id): url = self._base_url + ("/clusters/%s/requests/%s" % (cluster_name, request_id)) resp = self.check_response(self.get(url)) return resp.get('Requests') def wait_ambari_requests(self, requests, cluster_name): requests = set(requests) failed = [] context.sleep(20) while len(requests) > 0: completed, not_completed = set(), set() for req_id in requests: request = self.get_request_info(cluster_name, req_id) status = request.get("request_status") if status == 'COMPLETED': completed.add(req_id) elif status in ['IN_PROGRESS', 'PENDING']: not_completed.add(req_id) else: failed.append(request) if failed: msg = _("Some Ambari request(s) " "not in COMPLETED state: %(description)s.") descrs = [] for req in failed: descr = _( "request %(id)d: %(name)s - in status %(status)s") descrs.append(descr % {'id': req.get("id"), 'name': req.get("request_context"), 'status': req.get("request_status")}) raise p_exc.HadoopProvisionError(msg % {'description': descrs}) requests = not_completed context.sleep(5) LOG.debug("Waiting for %d ambari request(s) to be completed", len(not_completed)) LOG.debug("All ambari requests have been completed") def wait_ambari_request(self, request_id, cluster_name): context.sleep(20) while True: status = self.check_request_status(cluster_name, request_id) LOG.debug("Task %(context)s in %(status)s state. " "Completed %(percent).1f%%", {'context': status["request_context"], 'status': status["request_status"], 'percent': status["progress_percent"]}) if status["request_status"] == "COMPLETED": return if status["request_status"] in ["IN_PROGRESS", "PENDING"]: context.sleep(5) else: raise p_exc.HadoopProvisionError( _("Ambari request in %s state") % status["request_status"]) sahara-8.0.0/sahara/plugins/ambari/validation.py0000666000175100017510000002222013245514472021670 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor from sahara import context from sahara.i18n import _ from sahara.plugins.ambari import common from sahara.plugins import exceptions as ex from sahara.plugins import utils conductor = conductor.API def validate(cluster_id): ctx = context.ctx() cluster = conductor.cluster_get(ctx, cluster_id) _check_ambari(cluster) _check_hdfs(cluster) _check_yarn(cluster) _check_oozie(cluster) _check_hive(cluster) _check_hbase(cluster) _check_spark(cluster) _check_ranger(cluster) _check_storm(cluster) def _check_ambari(cluster): am_count = utils.get_instances_count(cluster, common.AMBARI_SERVER) zk_count = utils.get_instances_count(cluster, common.ZOOKEEPER_SERVER) if am_count != 1: raise ex.InvalidComponentCountException(common.AMBARI_SERVER, 1, am_count) if zk_count == 0: raise ex.InvalidComponentCountException(common.ZOOKEEPER_SERVER, _("1 or more"), zk_count) def _check_hdfs(cluster): nn_count = utils.get_instances_count(cluster, common.NAMENODE) dn_count = utils.get_instances_count(cluster, common.DATANODE) snn_count = utils.get_instances_count(cluster, common.SECONDARY_NAMENODE) if cluster.cluster_configs.get("general", {}).get(common.NAMENODE_HA): _check_zk_ha(cluster) _check_jn_ha(cluster) if nn_count != 2: raise ex.InvalidComponentCountException(common.NAMENODE, 2, nn_count) else: if nn_count != 1: raise ex.InvalidComponentCountException(common.NAMENODE, 1, nn_count) if snn_count != 1: raise ex.InvalidComponentCountException(common.SECONDARY_NAMENODE, 1, snn_count) if dn_count == 0: raise ex.InvalidComponentCountException( common.DATANODE, _("1 or more"), dn_count) def _check_yarn(cluster): rm_count = utils.get_instances_count(cluster, common.RESOURCEMANAGER) nm_count = utils.get_instances_count(cluster, common.NODEMANAGER) hs_count = utils.get_instances_count(cluster, common.HISTORYSERVER) at_count = utils.get_instances_count(cluster, common.APP_TIMELINE_SERVER) if cluster.cluster_configs.get("general", {}).get( common.RESOURCEMANAGER_HA): _check_zk_ha(cluster) if rm_count != 2: raise ex.InvalidComponentCountException(common.RESOURCEMANAGER, 2, rm_count) else: if rm_count != 1: raise ex.InvalidComponentCountException(common.RESOURCEMANAGER, 1, rm_count) if hs_count != 1: raise ex.InvalidComponentCountException(common.HISTORYSERVER, 1, hs_count) if at_count != 1: raise ex.InvalidComponentCountException(common.APP_TIMELINE_SERVER, 1, at_count) if nm_count == 0: raise ex.InvalidComponentCountException(common.NODEMANAGER, _("1 or more"), nm_count) def _check_zk_ha(cluster): zk_count = utils.get_instances_count(cluster, common.ZOOKEEPER_SERVER) if zk_count < 3: raise ex.InvalidComponentCountException( common.ZOOKEEPER_SERVER, _("3 or more. Odd number"), zk_count, _("At least 3 ZooKeepers are required for HA")) if zk_count % 2 != 1: raise ex.InvalidComponentCountException( common.ZOOKEEPER_SERVER, _("Odd number"), zk_count, _("Odd number of ZooKeepers are required for HA")) def _check_jn_ha(cluster): jn_count = utils.get_instances_count(cluster, common.JOURNAL_NODE) if jn_count < 3: raise ex.InvalidComponentCountException( common.JOURNAL_NODE, _("3 or more. Odd number"), jn_count, _("At least 3 JournalNodes are required for HA")) if jn_count % 2 != 1: raise ex.InvalidComponentCountException( common.JOURNAL_NODE, _("Odd number"), jn_count, _("Odd number of JournalNodes are required for HA")) def _check_oozie(cluster): count = utils.get_instances_count(cluster, common.OOZIE_SERVER) if count > 1: raise ex.InvalidComponentCountException(common.OOZIE_SERVER, _("0 or 1"), count) def _check_hive(cluster): hs_count = utils.get_instances_count(cluster, common.HIVE_SERVER) hm_count = utils.get_instances_count(cluster, common.HIVE_METASTORE) if hs_count > 1: raise ex.InvalidComponentCountException(common.HIVE_SERVER, _("0 or 1"), hs_count) if hm_count > 1: raise ex.InvalidComponentCountException(common.HIVE_METASTORE, _("0 or 1"), hm_count) if hs_count == 0 and hm_count == 1: raise ex.RequiredServiceMissingException( common.HIVE_SERVER, required_by=common.HIVE_METASTORE) if hs_count == 1 and hm_count == 0: raise ex.RequiredServiceMissingException( common.HIVE_METASTORE, required_by=common.HIVE_SERVER) def _check_hbase(cluster): hm_count = utils.get_instances_count(cluster, common.HBASE_MASTER) hr_count = utils.get_instances_count(cluster, common.HBASE_REGIONSERVER) if hm_count > 1: raise ex.InvalidComponentCountException(common.HBASE_MASTER, _("0 or 1"), hm_count) if hm_count == 1 and hr_count == 0: raise ex.RequiredServiceMissingException( common.HBASE_REGIONSERVER, required_by=common.HBASE_MASTER) if hr_count > 0 and hm_count == 0: raise ex.RequiredServiceMissingException( common.HBASE_MASTER, required_by=common.HBASE_REGIONSERVER) def _check_spark(cluster): count = utils.get_instances_count(cluster, common.SPARK_JOBHISTORYSERVER) if count > 1: raise ex.InvalidComponentCountException(common.SPARK_JOBHISTORYSERVER, _("0 or 1"), count) def _check_ranger(cluster): ra_count = utils.get_instances_count(cluster, common.RANGER_ADMIN) ru_count = utils.get_instances_count(cluster, common.RANGER_USERSYNC) if ra_count > 1: raise ex.InvalidComponentCountException(common.RANGER_ADMIN, _("0 or 1"), ra_count) if ru_count > 1: raise ex.InvalidComponentCountException(common.RANGER_USERSYNC, _("0 or 1"), ru_count) if ra_count == 1 and ru_count == 0: raise ex.RequiredServiceMissingException( common.RANGER_USERSYNC, required_by=common.RANGER_ADMIN) if ra_count == 0 and ru_count == 1: raise ex.RequiredServiceMissingException( common.RANGER_ADMIN, required_by=common.RANGER_USERSYNC) def _check_storm(cluster): dr_count = utils.get_instances_count(cluster, common.DRPC_SERVER) ni_count = utils.get_instances_count(cluster, common.NIMBUS) su_count = utils.get_instances_count(cluster, common.STORM_UI_SERVER) sv_count = utils.get_instances_count(cluster, common.SUPERVISOR) if dr_count > 1: raise ex.InvalidComponentCountException(common.DRPC_SERVER, _("0 or 1"), dr_count) if ni_count > 1: raise ex.InvalidComponentCountException(common.NIMBUS, _("0 or 1"), ni_count) if su_count > 1: raise ex.InvalidComponentCountException(common.STORM_UI_SERVER, _("0 or 1"), su_count) if dr_count == 0 and ni_count == 1: raise ex.RequiredServiceMissingException( common.DRPC_SERVER, required_by=common.NIMBUS) if dr_count == 1 and ni_count == 0: raise ex.RequiredServiceMissingException( common.NIMBUS, required_by=common.DRPC_SERVER) if su_count == 1 and (dr_count == 0 or ni_count == 0): raise ex.RequiredServiceMissingException( common.NIMBUS, required_by=common.STORM_UI_SERVER) if dr_count == 1 and sv_count == 0: raise ex.RequiredServiceMissingException( common.SUPERVISOR, required_by=common.DRPC_SERVER) if sv_count > 0 and dr_count == 0: raise ex.RequiredServiceMissingException( common.DRPC_SERVER, required_by=common.SUPERVISOR) sahara-8.0.0/sahara/plugins/ambari/plugin.py0000666000175100017510000003101013245514472021031 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor from sahara import context from sahara.i18n import _ from sahara.plugins.ambari import common as p_common from sahara.plugins.ambari import configs from sahara.plugins.ambari import deploy from sahara.plugins.ambari import edp_engine from sahara.plugins.ambari import health from sahara.plugins.ambari import validation from sahara.plugins import images from sahara.plugins import kerberos from sahara.plugins import provisioning as p from sahara.plugins import utils as plugin_utils from sahara.swift import swift_helper conductor = conductor.API class AmbariPluginProvider(p.ProvisioningPluginBase): def get_title(self): return "HDP Plugin" def get_description(self): return _("The Ambari Sahara plugin provides the ability to launch " "clusters with Hortonworks Data Platform (HDP) on OpenStack " "using Apache Ambari") def get_versions(self): return ["2.3", "2.4", "2.5"] def get_node_processes(self, hadoop_version): return { p_common.AMBARI_SERVICE: [p_common.AMBARI_SERVER], p_common.FALCON_SERVICE: [p_common.FALCON_SERVER], p_common.FLUME_SERVICE: [p_common.FLUME_HANDLER], p_common.HBASE_SERVICE: [p_common.HBASE_MASTER, p_common.HBASE_REGIONSERVER], p_common.HDFS_SERVICE: [p_common.DATANODE, p_common.NAMENODE, p_common.SECONDARY_NAMENODE, p_common.JOURNAL_NODE], p_common.HIVE_SERVICE: [p_common.HIVE_METASTORE, p_common.HIVE_SERVER], p_common.KAFKA_SERVICE: [p_common.KAFKA_BROKER], p_common.KNOX_SERVICE: [p_common.KNOX_GATEWAY], p_common.OOZIE_SERVICE: [p_common.OOZIE_SERVER], p_common.RANGER_SERVICE: [p_common.RANGER_ADMIN, p_common.RANGER_USERSYNC], p_common.SLIDER_SERVICE: [p_common.SLIDER], p_common.SPARK_SERVICE: [p_common.SPARK_JOBHISTORYSERVER], p_common.SQOOP_SERVICE: [p_common.SQOOP], p_common.STORM_SERVICE: [ p_common.DRPC_SERVER, p_common.NIMBUS, p_common.STORM_UI_SERVER, p_common.SUPERVISOR], p_common.YARN_SERVICE: [ p_common.APP_TIMELINE_SERVER, p_common.HISTORYSERVER, p_common.NODEMANAGER, p_common.RESOURCEMANAGER], p_common.ZOOKEEPER_SERVICE: [p_common.ZOOKEEPER_SERVER], 'Kerberos': [], } def get_configs(self, hadoop_version): cfgs = kerberos.get_config_list() cfgs.extend(configs.load_configs(hadoop_version)) return cfgs def configure_cluster(self, cluster): deploy.disable_repos(cluster) deploy.setup_ambari(cluster) deploy.setup_agents(cluster) deploy.wait_ambari_accessible(cluster) deploy.update_default_ambari_password(cluster) cluster = conductor.cluster_get(context.ctx(), cluster.id) deploy.wait_host_registration(cluster, plugin_utils.get_instances(cluster)) deploy.prepare_kerberos(cluster) deploy.set_up_hdp_repos(cluster) deploy.resolve_package_conflicts(cluster) deploy.create_blueprint(cluster) def start_cluster(self, cluster): self._set_cluster_info(cluster) deploy.start_cluster(cluster) cluster_instances = plugin_utils.get_instances(cluster) swift_helper.install_ssl_certs(cluster_instances) deploy.add_hadoop_swift_jar(cluster_instances) deploy.prepare_hive(cluster) deploy.deploy_kerberos_principals(cluster) def _set_cluster_info(self, cluster): ambari_ip = plugin_utils.get_instance( cluster, p_common.AMBARI_SERVER).get_ip_or_dns_name() ambari_port = "8080" info = { p_common.AMBARI_SERVER: { "Web UI": "http://{host}:{port}".format(host=ambari_ip, port=ambari_port), "Username": "admin", "Password": cluster.extra["ambari_password"] } } nns = plugin_utils.get_instances(cluster, p_common.NAMENODE) info[p_common.NAMENODE] = {} for idx, namenode in enumerate(nns): info[p_common.NAMENODE][ "Web UI %s" % (idx + 1)] = ( "http://%s:50070" % namenode.get_ip_or_dns_name()) rms = plugin_utils.get_instances(cluster, p_common.RESOURCEMANAGER) info[p_common.RESOURCEMANAGER] = {} for idx, resourcemanager in enumerate(rms): info[p_common.RESOURCEMANAGER][ "Web UI %s" % (idx + 1)] = ( "http://%s:8088" % resourcemanager.get_ip_or_dns_name()) historyserver = plugin_utils.get_instance(cluster, p_common.HISTORYSERVER) if historyserver: info[p_common.HISTORYSERVER] = { "Web UI": "http://%s:19888" % historyserver.get_ip_or_dns_name() } atlserver = plugin_utils.get_instance(cluster, p_common.APP_TIMELINE_SERVER) if atlserver: info[p_common.APP_TIMELINE_SERVER] = { "Web UI": "http://%s:8188" % atlserver.get_ip_or_dns_name() } oozie = plugin_utils.get_instance(cluster, p_common.OOZIE_SERVER) if oozie: info[p_common.OOZIE_SERVER] = { "Web UI": "http://%s:11000/oozie" % oozie.get_ip_or_dns_name() } hbase_master = plugin_utils.get_instance(cluster, p_common.HBASE_MASTER) if hbase_master: info[p_common.HBASE_MASTER] = { "Web UI": "http://%s:60010" % hbase_master.get_ip_or_dns_name() } falcon = plugin_utils.get_instance(cluster, p_common.FALCON_SERVER) if falcon: info[p_common.FALCON_SERVER] = { "Web UI": "http://%s:15000" % falcon.get_ip_or_dns_name() } storm_ui = plugin_utils.get_instance(cluster, p_common.STORM_UI_SERVER) if storm_ui: info[p_common.STORM_UI_SERVER] = { "Web UI": "http://%s:8744" % storm_ui.get_ip_or_dns_name() } ranger_admin = plugin_utils.get_instance(cluster, p_common.RANGER_ADMIN) if ranger_admin: info[p_common.RANGER_ADMIN] = { "Web UI": "http://%s:6080" % ranger_admin.get_ip_or_dns_name(), "Username": "admin", "Password": "admin" } spark_hs = plugin_utils.get_instance(cluster, p_common.SPARK_JOBHISTORYSERVER) if spark_hs: info[p_common.SPARK_JOBHISTORYSERVER] = { "Web UI": "http://%s:18080" % spark_hs.get_ip_or_dns_name() } info.update(cluster.info.to_dict()) ctx = context.ctx() conductor.cluster_update(ctx, cluster, {"info": info}) cluster = conductor.cluster_get(ctx, cluster.id) def validate(self, cluster): validation.validate(cluster.id) def scale_cluster(self, cluster, instances): deploy.prepare_kerberos(cluster, instances) deploy.setup_agents(cluster, instances) cluster = conductor.cluster_get(context.ctx(), cluster.id) deploy.wait_host_registration(cluster, instances) deploy.resolve_package_conflicts(cluster, instances) deploy.add_new_hosts(cluster, instances) deploy.manage_config_groups(cluster, instances) deploy.manage_host_components(cluster, instances) deploy.configure_rack_awareness(cluster, instances) swift_helper.install_ssl_certs(instances) deploy.add_hadoop_swift_jar(instances) deploy.deploy_kerberos_principals(cluster, instances) def decommission_nodes(self, cluster, instances): deploy.decommission_hosts(cluster, instances) deploy.remove_services_from_hosts(cluster, instances) deploy.restart_nns_and_rms(cluster) deploy.cleanup_config_groups(cluster, instances) def validate_scaling(self, cluster, existing, additional): validation.validate(cluster.id) def get_edp_engine(self, cluster, job_type): if job_type in edp_engine.EDPSparkEngine.get_supported_job_types(): return edp_engine.EDPSparkEngine(cluster) if job_type in edp_engine.EDPOozieEngine.get_supported_job_types(): return edp_engine.EDPOozieEngine(cluster) return None def get_edp_job_types(self, versions=None): res = {} for version in self.get_versions(): if not versions or version in versions: oozie_engine = edp_engine.EDPOozieEngine spark_engine = edp_engine.EDPSparkEngine res[version] = (oozie_engine.get_supported_job_types() + spark_engine.get_supported_job_types()) return res def get_edp_config_hints(self, job_type, version): if job_type in edp_engine.EDPSparkEngine.get_supported_job_types(): return edp_engine.EDPSparkEngine.get_possible_job_config(job_type) if job_type in edp_engine.EDPOozieEngine.get_supported_job_types(): return edp_engine.EDPOozieEngine.get_possible_job_config(job_type) def get_open_ports(self, node_group): ports_map = { p_common.AMBARI_SERVER: [8080], p_common.APP_TIMELINE_SERVER: [8188, 8190, 10200], p_common.DATANODE: [50075, 50475], p_common.DRPC_SERVER: [3772, 3773], p_common.FALCON_SERVER: [15000], p_common.FLUME_HANDLER: [8020, 41414], p_common.HBASE_MASTER: [60000, 60010], p_common.HBASE_REGIONSERVER: [60020, 60030], p_common.HISTORYSERVER: [10020, 19888], p_common.HIVE_METASTORE: [9933], p_common.HIVE_SERVER: [9999, 10000], p_common.KAFKA_BROKER: [6667], p_common.NAMENODE: [8020, 9000, 50070, 50470], p_common.NIMBUS: [6627], p_common.NODEMANAGER: [8042, 8044, 45454], p_common.OOZIE_SERVER: [11000, 11443], p_common.RANGER_ADMIN: [6080], p_common.RESOURCEMANAGER: [8025, 8030, 8050, 8088, 8141], p_common.SECONDARY_NAMENODE: [50090], p_common.SPARK_JOBHISTORYSERVER: [18080], p_common.STORM_UI_SERVER: [8000, 8080, 8744], p_common.ZOOKEEPER_SERVER: [2181], } ports = [] for service in node_group.node_processes: ports.extend(ports_map.get(service, [])) return ports def get_health_checks(self, cluster): return health.get_health_checks(cluster) validator = images.SaharaImageValidator.from_yaml( 'plugins/ambari/resources/images/image.yaml', resource_roots=['plugins/ambari/resources/images']) def get_image_arguments(self, hadoop_version): if hadoop_version != '2.4': return NotImplemented return self.validator.get_argument_list() def pack_image(self, hadoop_version, remote, test_only=False, image_arguments=None): self.validator.validate(remote, test_only=test_only, image_arguments=image_arguments) def validate_images(self, cluster, test_only=False, image_arguments=None): image_arguments = self.get_image_arguments(cluster['hadoop_version']) if not test_only: instances = plugin_utils.get_instances(cluster) else: instances = plugin_utils.get_instances(cluster)[0] for instance in instances: with instance.remote() as r: self.validator.validate(r, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/ambari/common.py0000666000175100017510000001103213245514472021025 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import kerberos # define service names AMBARI_SERVICE = "Ambari" FALCON_SERVICE = "Falcon" FLUME_SERVICE = "Flume" HBASE_SERVICE = "HBase" HDFS_SERVICE = "HDFS" HIVE_SERVICE = "Hive" KAFKA_SERVICE = "Kafka" KNOX_SERVICE = "Knox" MAPREDUCE2_SERVICE = "MAPREDUCE2" OOZIE_SERVICE = "Oozie" RANGER_SERVICE = "Ranger" SLIDER_SERVICE = "Slider" SPARK_SERVICE = "Spark" SQOOP_SERVICE = "Sqoop" STORM_SERVICE = "Storm" YARN_SERVICE = "YARN" ZOOKEEPER_SERVICE = "ZooKeeper" # define process names AMBARI_SERVER = "Ambari" APP_TIMELINE_SERVER = "YARN Timeline Server" DATANODE = "DataNode" DRPC_SERVER = "DRPC Server" FALCON_SERVER = "Falcon Server" FLUME_HANDLER = "Flume" HBASE_MASTER = "HBase Master" HBASE_REGIONSERVER = "HBase RegionServer" HISTORYSERVER = "MapReduce History Server" HIVE_METASTORE = "Hive Metastore" HIVE_SERVER = "HiveServer" KAFKA_BROKER = "Kafka Broker" KNOX_GATEWAY = "Knox Gateway" NAMENODE = "NameNode" NIMBUS = "Nimbus" NODEMANAGER = "NodeManager" OOZIE_SERVER = "Oozie" RANGER_ADMIN = "Ranger Admin" RANGER_USERSYNC = "Ranger Usersync" RESOURCEMANAGER = "ResourceManager" SECONDARY_NAMENODE = "SecondaryNameNode" SLIDER = "Slider" SPARK_JOBHISTORYSERVER = "Spark History Server" SQOOP = "Sqoop" STORM_UI_SERVER = "Storm UI Server" SUPERVISOR = "Supervisor" ZOOKEEPER_SERVER = "ZooKeeper" JOURNAL_NODE = "JournalNode" PROC_MAP = { AMBARI_SERVER: ["METRICS_COLLECTOR"], APP_TIMELINE_SERVER: ["APP_TIMELINE_SERVER"], DATANODE: ["DATANODE"], DRPC_SERVER: ["DRPC_SERVER"], FALCON_SERVER: ["FALCON_SERVER"], HBASE_MASTER: ["HBASE_MASTER"], HBASE_REGIONSERVER: ["HBASE_REGIONSERVER"], HISTORYSERVER: ["HISTORYSERVER"], HIVE_METASTORE: ["HIVE_METASTORE"], HIVE_SERVER: ["HIVE_SERVER", "MYSQL_SERVER", "WEBHCAT_SERVER"], KAFKA_BROKER: ["KAFKA_BROKER"], KNOX_GATEWAY: ["KNOX_GATEWAY"], NAMENODE: ["NAMENODE"], NIMBUS: ["NIMBUS"], NODEMANAGER: ["NODEMANAGER"], OOZIE_SERVER: ["OOZIE_SERVER", "PIG"], RANGER_ADMIN: ["RANGER_ADMIN"], RANGER_USERSYNC: ["RANGER_USERSYNC"], RESOURCEMANAGER: ["RESOURCEMANAGER"], SECONDARY_NAMENODE: ["SECONDARY_NAMENODE"], SLIDER: ["SLIDER"], SPARK_JOBHISTORYSERVER: ["SPARK_JOBHISTORYSERVER"], SQOOP: ["SQOOP"], STORM_UI_SERVER: ["STORM_UI_SERVER"], SUPERVISOR: ["SUPERVISOR"], ZOOKEEPER_SERVER: ["ZOOKEEPER_SERVER"], JOURNAL_NODE: ["JOURNALNODE"] } CLIENT_MAP = { APP_TIMELINE_SERVER: ["MAPREDUCE2_CLIENT", "YARN_CLIENT"], DATANODE: ["HDFS_CLIENT"], FALCON_SERVER: ["FALCON_CLIENT"], FLUME_HANDLER: ["FLUME_HANDLER"], HBASE_MASTER: ["HBASE_CLIENT"], HBASE_REGIONSERVER: ["HBASE_CLIENT"], HISTORYSERVER: ["MAPREDUCE2_CLIENT", "YARN_CLIENT"], HIVE_METASTORE: ["HIVE_CLIENT"], HIVE_SERVER: ["HIVE_CLIENT"], NAMENODE: ["HDFS_CLIENT"], NODEMANAGER: ["MAPREDUCE2_CLIENT", "YARN_CLIENT"], OOZIE_SERVER: ["OOZIE_CLIENT", "TEZ_CLIENT"], RESOURCEMANAGER: ["MAPREDUCE2_CLIENT", "YARN_CLIENT"], SECONDARY_NAMENODE: ["HDFS_CLIENT"], SPARK_JOBHISTORYSERVER: ["SPARK_CLIENT"], ZOOKEEPER_SERVER: ["ZOOKEEPER_CLIENT"] } KERBEROS_CLIENT = 'KERBEROS_CLIENT' ALL_LIST = ["METRICS_MONITOR"] # types of HA NAMENODE_HA = "NameNode HA" RESOURCEMANAGER_HA = "ResourceManager HA" HBASE_REGIONSERVER_HA = "HBase RegionServer HA" def get_ambari_proc_list(node_group): procs = [] for sp in node_group.node_processes: procs.extend(PROC_MAP.get(sp, [])) return procs def get_clients(cluster): procs = [] for ng in cluster.node_groups: procs.extend(ng.node_processes) clients = [] for proc in procs: clients.extend(CLIENT_MAP.get(proc, [])) clients = list(set(clients)) clients.extend(ALL_LIST) if kerberos.is_kerberos_security_enabled(cluster): clients.append(KERBEROS_CLIENT) return clients def instances_have_process(instances, process): for i in instances: if process in i.node_group.node_processes: return True return False sahara-8.0.0/sahara/plugins/ambari/configs.py0000666000175100017510000002616513245514472021202 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils import six from sahara.i18n import _ from sahara.plugins.ambari import common from sahara.plugins import provisioning from sahara.plugins import utils from sahara.swift import swift_helper from sahara.utils import files CONFIGS = {} OBJ_CONFIGS = {} CFG_PROCESS_MAP = { "admin-properties": common.RANGER_SERVICE, "ams-env": common.AMBARI_SERVICE, "ams-hbase-env": common.AMBARI_SERVICE, "ams-hbase-policy": common.AMBARI_SERVICE, "ams-hbase-security-site": common.AMBARI_SERVICE, "ams-hbase-site": common.AMBARI_SERVICE, "ams-site": common.AMBARI_SERVICE, "capacity-scheduler": common.YARN_SERVICE, "cluster-env": "general", "core-site": common.HDFS_SERVICE, "falcon-env": common.FALCON_SERVICE, "falcon-runtime.properties": common.FALCON_SERVICE, "falcon-startup.properties": common.FALCON_SERVICE, "flume-env": common.FLUME_SERVICE, "gateway-site": common.KNOX_SERVICE, "hadoop-env": common.HDFS_SERVICE, "hadoop-policy": common.HDFS_SERVICE, "hbase-env": common.HBASE_SERVICE, "hbase-policy": common.HBASE_SERVICE, "hbase-site": common.HBASE_SERVICE, "hdfs-site": common.HDFS_SERVICE, "hive-env": common.HIVE_SERVICE, "hive-site": common.HIVE_SERVICE, "hiveserver2-site": common.HIVE_SERVICE, "kafka-broker": common.KAFKA_SERVICE, "kafka-env": common.KAFKA_SERVICE, "knox-env": common.KNOX_SERVICE, "mapred-env": common.YARN_SERVICE, "mapred-site": common.YARN_SERVICE, "oozie-env": common.OOZIE_SERVICE, "oozie-site": common.OOZIE_SERVICE, "ranger-env": common.RANGER_SERVICE, "ranger-hbase-plugin-properties": common.HBASE_SERVICE, "ranger-hdfs-plugin-properties": common.HDFS_SERVICE, "ranger-hive-plugin-properties": common.HIVE_SERVICE, "ranger-knox-plugin-properties": common.KNOX_SERVICE, "ranger-site": common.RANGER_SERVICE, "ranger-storm-plugin-properties": common.STORM_SERVICE, "spark-defaults": common.SPARK_SERVICE, "spark-env": common.SPARK_SERVICE, "sqoop-env": common.SQOOP_SERVICE, "storm-env": common.STORM_SERVICE, "storm-site": common.STORM_SERVICE, "tez-site": common.OOZIE_SERVICE, "usersync-properties": common.RANGER_SERVICE, "yarn-env": common.YARN_SERVICE, "yarn-site": common.YARN_SERVICE, "zoo.cfg": common.ZOOKEEPER_SERVICE, "zookeeper-env": common.ZOOKEEPER_SERVICE } SERVICES_TO_CONFIGS_MAP = None def get_service_to_configs_map(): global SERVICES_TO_CONFIGS_MAP if SERVICES_TO_CONFIGS_MAP: return SERVICES_TO_CONFIGS_MAP data = {} for (key, item) in six.iteritems(CFG_PROCESS_MAP): if item not in data: data[item] = [] data[item].append(key) SERVICES_TO_CONFIGS_MAP = data return SERVICES_TO_CONFIGS_MAP ng_confs = [ "dfs.datanode.data.dir", "dtnode_heapsize", "mapreduce.map.java.opts", "mapreduce.map.memory.mb", "mapreduce.reduce.java.opts", "mapreduce.reduce.memory.mb", "mapreduce.task.io.sort.mb", "nodemanager_heapsize", "yarn.app.mapreduce.am.command-opts", "yarn.app.mapreduce.am.resource.mb", "yarn.nodemanager.resource.cpu-vcores", "yarn.nodemanager.resource.memory-mb", "yarn.scheduler.maximum-allocation-mb", "yarn.scheduler.minimum-allocation-mb" ] use_base_repos_cfg = provisioning.Config( "Enable external repos on instances", 'general', 'cluster', priority=1, default_value=True, config_type="bool") hdp_repo_cfg = provisioning.Config( "HDP repo URL", "general", "cluster", priority=1, default_value="") hdp_utils_repo_cfg = provisioning.Config( "HDP-UTILS repo URL", "general", "cluster", priority=1, default_value="") autoconfigs_strategy = provisioning.Config( "Auto-configuration strategy", 'general', 'cluster', priority=1, config_type='dropdown', default_value='NEVER_APPLY', config_values=[(v, v) for v in [ 'NEVER_APPLY', 'ALWAYS_APPLY', 'ONLY_STACK_DEFAULTS_APPLY', ]], ) def _get_service_name(service): return CFG_PROCESS_MAP.get(service, service) def _get_config_group(group, param, plugin_version): if not CONFIGS or plugin_version not in CONFIGS: load_configs(plugin_version) for section, process in six.iteritems(CFG_PROCESS_MAP): if process == group and param in CONFIGS[plugin_version][section]: return section def _get_param_scope(param): if param in ng_confs: return "node" else: return "cluster" def _get_ha_params(): enable_namenode_ha = provisioning.Config( name=common.NAMENODE_HA, applicable_target="general", scope="cluster", config_type="bool", default_value=False, is_optional=True, description=_("Enable NameNode HA"), priority=1) enable_resourcemanager_ha = provisioning.Config( name=common.RESOURCEMANAGER_HA, applicable_target="general", scope="cluster", config_type="bool", default_value=False, is_optional=True, description=_("Enable ResourceManager HA"), priority=1) enable_regionserver_ha = provisioning.Config( name=common.HBASE_REGIONSERVER_HA, applicable_target="general", scope="cluster", config_type="bool", default_value=False, is_optional=True, description=_("Enable HBase RegionServer HA"), priority=1) return [enable_namenode_ha, enable_resourcemanager_ha, enable_regionserver_ha] def load_configs(version): if OBJ_CONFIGS.get(version): return OBJ_CONFIGS[version] cfg_path = "plugins/ambari/resources/configs-%s.json" % version vanilla_cfg = jsonutils.loads(files.get_file_text(cfg_path)) CONFIGS[version] = vanilla_cfg sahara_cfg = [hdp_repo_cfg, hdp_utils_repo_cfg, use_base_repos_cfg, autoconfigs_strategy] for service, confs in vanilla_cfg.items(): for k, v in confs.items(): sahara_cfg.append(provisioning.Config( k, _get_service_name(service), _get_param_scope(k), default_value=v)) sahara_cfg.extend(_get_ha_params()) OBJ_CONFIGS[version] = sahara_cfg return sahara_cfg def _get_config_value(cluster, key): return cluster.cluster_configs.get("general", {}).get(key.name, key.default_value) def use_base_repos_needed(cluster): return _get_config_value(cluster, use_base_repos_cfg) def get_hdp_repo_url(cluster): return _get_config_value(cluster, hdp_repo_cfg) def get_hdp_utils_repo_url(cluster): return _get_config_value(cluster, hdp_utils_repo_cfg) def get_auto_configuration_strategy(cluster): return _get_config_value(cluster, autoconfigs_strategy) def _serialize_ambari_configs(configs): return list(map(lambda x: {x: configs[x]}, configs)) def _create_ambari_configs(sahara_configs, plugin_version): configs = {} for service, params in six.iteritems(sahara_configs): if service == "general" or service == "Kerberos": # General and Kerberos configs are designed for Sahara, not for # the plugin continue for k, v in six.iteritems(params): group = _get_config_group(service, k, plugin_version) configs.setdefault(group, {}) configs[group].update({k: v}) return configs def _make_paths(dirs, suffix): return ",".join([d + suffix for d in dirs]) def get_instance_params_mapping(inst): configs = _create_ambari_configs(inst.node_group.node_configs, inst.node_group.cluster.hadoop_version) storage_paths = inst.storage_paths() configs.setdefault("hdfs-site", {}) configs["hdfs-site"]["dfs.datanode.data.dir"] = _make_paths( storage_paths, "/hdfs/data") configs["hdfs-site"]["dfs.journalnode.edits.dir"] = _make_paths( storage_paths, "/hdfs/journalnode") configs["hdfs-site"]["dfs.namenode.checkpoint.dir"] = _make_paths( storage_paths, "/hdfs/namesecondary") configs["hdfs-site"]["dfs.namenode.name.dir"] = _make_paths( storage_paths, "/hdfs/namenode") configs.setdefault("yarn-site", {}) configs["yarn-site"]["yarn.nodemanager.local-dirs"] = _make_paths( storage_paths, "/yarn/local") configs["yarn-site"]["yarn.nodemanager.log-dirs"] = _make_paths( storage_paths, "/yarn/log") configs["yarn-site"][ "yarn.timeline-service.leveldb-timeline-store.path"] = _make_paths( storage_paths, "/yarn/timeline") configs.setdefault("oozie-site", {}) configs["oozie-site"][ "oozie.service.AuthorizationService.security.enabled"] = "false" return configs def get_instance_params(inst): return _serialize_ambari_configs(get_instance_params_mapping(inst)) def get_cluster_params(cluster): configs = _create_ambari_configs(cluster.cluster_configs, cluster.hadoop_version) swift_configs = {x["name"]: x["value"] for x in swift_helper.get_swift_configs()} configs.setdefault("core-site", {}) configs["core-site"].update(swift_configs) if utils.get_instance(cluster, common.RANGER_ADMIN): configs.setdefault("admin-properties", {}) configs["admin-properties"]["db_root_password"] = ( cluster.extra["ranger_db_password"]) return _serialize_ambari_configs(configs) def get_config_group(instance): params = get_instance_params_mapping(instance) groups = [] for (service, targets) in six.iteritems(get_service_to_configs_map()): current_group = { 'cluster_name': instance.cluster.name, 'group_name': "%s:%s" % ( instance.cluster.name, instance.instance_name), 'tag': service, 'description': "Config group for scaled " "node %s" % instance.instance_name, 'hosts': [ { 'host_name': instance.fqdn() } ], 'desired_configs': [] } at_least_one_added = False for target in targets: configs = params.get(target, {}) if configs: current_group['desired_configs'].append({ 'type': target, 'properties': configs, 'tag': instance.instance_name }) at_least_one_added = True if at_least_one_added: # Config Group without overridden data is not interesting groups.append({'ConfigGroup': current_group}) return groups sahara-8.0.0/sahara/plugins/ambari/__init__.py0000666000175100017510000000000013245514472021265 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/ambari/requests_helper.py0000666000175100017510000001031313245514472022750 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy _COMMON_DECOMMISSION_TEMPLATE = { "RequestInfo": { "context": "", "command": "DECOMMISSION", "parameters": { "slave_type": "", "excluded_hosts": "" }, "operation_level": { "level": "HOST_COMPONENT", "cluster_name": "" } }, "Requests/resource_filters": [ { "service_name": "", "component_name": "" } ] } _COMMON_RESTART_TEMPLATE = { "RequestInfo": { "context": "", "command": "RESTART", "operation_level": { "level": "HOST", "cluster_name": "" } }, "Requests/resource_filters": [ { "service_name": "", "component_name": "", "hosts": "" } ] } _COMMON_RESTART_SERVICE_TEMPLATE = { "RequestInfo": { "context": "", }, "Body": { "ServiceInfo": { "state": "" } } } def build_datanode_decommission_request(cluster_name, instances): tmpl = copy.deepcopy(_COMMON_DECOMMISSION_TEMPLATE) tmpl["RequestInfo"]["context"] = "Decommission DataNodes" tmpl["RequestInfo"]["parameters"]["slave_type"] = "DATANODE" tmpl["RequestInfo"]["parameters"]["excluded_hosts"] = ",".join( [i.fqdn() for i in instances]) tmpl["RequestInfo"]["operation_level"]["cluster_name"] = cluster_name tmpl["Requests/resource_filters"][0]["service_name"] = "HDFS" tmpl["Requests/resource_filters"][0]["component_name"] = "NAMENODE" return tmpl def build_nodemanager_decommission_request(cluster_name, instances): tmpl = copy.deepcopy(_COMMON_DECOMMISSION_TEMPLATE) tmpl["RequestInfo"]["context"] = "Decommission NodeManagers" tmpl["RequestInfo"]["parameters"]["slave_type"] = "NODEMANAGER" tmpl["RequestInfo"]["parameters"]["excluded_hosts"] = ",".join( [i.fqdn() for i in instances]) tmpl["RequestInfo"]["operation_level"]["cluster_name"] = cluster_name tmpl["Requests/resource_filters"][0]["service_name"] = "YARN" tmpl["Requests/resource_filters"][0]["component_name"] = "RESOURCEMANAGER" return tmpl def build_namenode_restart_request(cluster_name, nn_instance): tmpl = copy.deepcopy(_COMMON_RESTART_TEMPLATE) tmpl["RequestInfo"]["context"] = "Restart NameNode" tmpl["RequestInfo"]["operation_level"]["cluster_name"] = cluster_name tmpl["Requests/resource_filters"][0]["service_name"] = "HDFS" tmpl["Requests/resource_filters"][0]["component_name"] = "NAMENODE" tmpl["Requests/resource_filters"][0]["hosts"] = nn_instance.fqdn() return tmpl def build_resourcemanager_restart_request(cluster_name, rm_instance): tmpl = copy.deepcopy(_COMMON_RESTART_TEMPLATE) tmpl["RequestInfo"]["context"] = "Restart ResourceManager" tmpl["RequestInfo"]["operation_level"]["cluster_name"] = cluster_name tmpl["Requests/resource_filters"][0]["service_name"] = "YARN" tmpl["Requests/resource_filters"][0]["component_name"] = "RESOURCEMANAGER" tmpl["Requests/resource_filters"][0]["hosts"] = rm_instance.fqdn() return tmpl def build_stop_service_request(service_name): tmpl = copy.deepcopy(_COMMON_RESTART_SERVICE_TEMPLATE) tmpl["RequestInfo"]["context"] = ( "Restart %s service (stopping)" % service_name) tmpl["Body"]["ServiceInfo"]["state"] = "INSTALLED" return tmpl def build_start_service_request(service_name): tmpl = copy.deepcopy(_COMMON_RESTART_SERVICE_TEMPLATE) tmpl["RequestInfo"]["context"] = ( "Restart %s service (starting)" % service_name) tmpl["Body"]["ServiceInfo"]["state"] = "STARTED" return tmpl sahara-8.0.0/sahara/plugins/ambari/ha_helper.py0000666000175100017510000002152213245514472021471 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.ambari import common as p_common from sahara.plugins import utils CORE_SITE = "core-site" YARN_SITE = "yarn-site" HBASE_SITE = "hbase-site" HDFS_SITE = "hdfs-site" HADOOP_ENV = "hadoop-env" ZOO_CFG = "zoo.cfg" def update_bp_ha_common(cluster, blueprint): blueprint = _set_default_fs(cluster, blueprint, p_common.NAMENODE_HA) blueprint = _set_high_zk_limits(blueprint) return blueprint def update_bp_for_namenode_ha(cluster, blueprint): blueprint = _add_zkfc_to_namenodes(blueprint) blueprint = _set_zk_quorum(cluster, blueprint, CORE_SITE) blueprint = _configure_hdfs_site(cluster, blueprint) return blueprint def update_bp_for_resourcemanager_ha(cluster, blueprint): blueprint = _configure_yarn_site(cluster, blueprint) blueprint = _set_zk_quorum(cluster, blueprint, YARN_SITE) blueprint = _set_default_fs(cluster, blueprint, p_common.RESOURCEMANAGER_HA) return blueprint def update_bp_for_hbase_ha(cluster, blueprint): return _confgure_hbase_site(cluster, blueprint) def _add_zkfc_to_namenodes(blueprint): for hg in blueprint["host_groups"]: if {"name": "NAMENODE"} in hg["components"]: hg["components"].append({"name": "ZKFC"}) return blueprint def _find_create_properties_section(blueprint, section_name): for conf_group in blueprint["configurations"]: if section_name in conf_group: return conf_group[section_name] new_group = {section_name: {}} blueprint["configurations"].append(new_group) return new_group[section_name] def _find_hdfs_site(blueprint): return _find_create_properties_section(blueprint, HDFS_SITE) def _find_yarn_site(blueprint): return _find_create_properties_section(blueprint, YARN_SITE) def _find_core_site(blueprint): return _find_create_properties_section(blueprint, CORE_SITE) def _find_hadoop_env(blueprint): return _find_create_properties_section(blueprint, HADOOP_ENV) def _find_zoo_cfg(blueprint): return _find_create_properties_section(blueprint, ZOO_CFG) def _find_hbase_site(blueprint): return _find_create_properties_section(blueprint, HBASE_SITE) def _set_default_fs(cluster, blueprint, ha_type): if ha_type == p_common.NAMENODE_HA: _find_core_site(blueprint)["fs.defaultFS"] = "hdfs://hdfs-ha" elif ha_type == p_common.RESOURCEMANAGER_HA: nn_instance = utils.get_instances(cluster, p_common.NAMENODE)[0] _find_core_site(blueprint)["fs.defaultFS"] = ( "hdfs://%s:8020" % nn_instance.fqdn()) return blueprint def _set_zk_quorum(cluster, blueprint, conf_type): zk_instances = utils.get_instances(cluster, p_common.ZOOKEEPER_SERVER) value = ",".join(["%s:2181" % i.fqdn() for i in zk_instances]) if conf_type == CORE_SITE: _find_core_site(blueprint)["ha.zookeeper.quorum"] = value elif conf_type == YARN_SITE: _find_yarn_site(blueprint)["hadoop.registry.zk.quorum"] = value return blueprint def _set_high_zk_limits(blueprint): props = _find_zoo_cfg(blueprint) props["tickTime"] = "10000" return blueprint def _set_primary_and_standby_namenode(cluster, blueprint): props = _find_hadoop_env(blueprint) nns = utils.get_instances(cluster, p_common.NAMENODE) props["dfs_ha_initial_namenode_active"] = nns[0].fqdn() props["dfs_ha_initial_namenode_standby"] = nns[1].fqdn() return blueprint def _configure_hdfs_site(cluster, blueprint): props = _find_hdfs_site(blueprint) props["dfs.client.failover.proxy.provider.hdfs-ha"] = ( "org.apache.hadoop.hdfs.server.namenode.ha." "ConfiguredFailoverProxyProvider") props["dfs.ha.automatic-failover.enabled"] = "true" props["dfs.ha.fencing.methods"] = "shell(/bin/true)" props["dfs.nameservices"] = "hdfs-ha" jns = utils.get_instances(cluster, p_common.JOURNAL_NODE) journalnodes_concat = ";".join( ["%s:8485" % i.fqdn() for i in jns]) journalnodes_value = "qjournal://%s/hdfs-ha" % journalnodes_concat props["dfs.namenode.shared.edits.dir"] = journalnodes_value nns = utils.get_instances(cluster, p_common.NAMENODE) nn_id_concat = ",".join([i.instance_name for i in nns]) props["dfs.ha.namenodes.hdfs-ha"] = nn_id_concat props["dfs.namenode.http-address"] = "%s:50070" % nns[0].fqdn() props["dfs.namenode.https-address"] = "%s:50470" % nns[0].fqdn() for i in nns: props["dfs.namenode.http-address.hdfs-ha.%s" % i.instance_name] = ( "%s:50070" % i.fqdn()) props["dfs.namenode.https-address.hdfs-ha.%s" % i.instance_name] = ( "%s:50470" % i.fqdn()) props["dfs.namenode.rpc-address.hdfs-ha.%s" % i.instance_name] = ( "%s:8020" % i.fqdn()) return blueprint def _configure_yarn_site(cluster, blueprint): props = _find_yarn_site(blueprint) name = cluster.name rm_instances = utils.get_instances(cluster, p_common.RESOURCEMANAGER) props["hadoop.registry.rm.enabled"] = "false" zk_instances = utils.get_instances(cluster, p_common.ZOOKEEPER_SERVER) zks = ",".join(["%s:2181" % i.fqdn() for i in zk_instances]) props["yarn.resourcemanager.zk-address"] = zks hs = utils.get_instance(cluster, p_common.HISTORYSERVER) props["yarn.log.server.url"] = "%s:19888/jobhistory/logs/" % hs.fqdn() props["yarn.resourcemanager.address"] = "%s:8050" % rm_instances[0].fqdn() props["yarn.resourcemanager.admin.address"] = ("%s:8141" % rm_instances[0].fqdn()) props["yarn.resourcemanager.cluster-id"] = name props["yarn.resourcemanager.ha.automatic-failover.zk-base-path"] = ( "/yarn-leader-election") props["yarn.resourcemanager.ha.enabled"] = "true" rm_id_concat = ",".join([i.instance_name for i in rm_instances]) props["yarn.resourcemanager.ha.rm-ids"] = rm_id_concat for i in rm_instances: props["yarn.resourcemanager.hostname.%s" % i.instance_name] = i.fqdn() props["yarn.resourcemanager.webapp.address.%s" % i.instance_name] = "%s:8088" % i.fqdn() props["yarn.resourcemanager.webapp.https.address.%s" % i.instance_name] = "%s:8090" % i.fqdn() props["yarn.resourcemanager.hostname"] = rm_instances[0].fqdn() props["yarn.resourcemanager.recovery.enabled"] = "true" props["yarn.resourcemanager.resource-tracker.address"] = ( "%s:8025" % rm_instances[0].fqdn()) props["yarn.resourcemanager.scheduler.address"] = ( "%s:8030" % rm_instances[0].fqdn()) props["yarn.resourcemanager.store.class"] = ( "org.apache.hadoop.yarn.server.resourcemanager.recovery." "ZKRMStateStore") props["yarn.resourcemanager.webapp.address"] = ( "%s:8088" % rm_instances[0].fqdn()) props["yarn.resourcemanager.webapp.https.address"] = ( "%s:8090" % rm_instances[0].fqdn()) tls_instance = utils.get_instance(cluster, p_common.APP_TIMELINE_SERVER) props["yarn.timeline-service.address"] = "%s:10200" % tls_instance.fqdn() props["yarn.timeline-service.webapp.address"] = ( "%s:8188" % tls_instance.fqdn()) props["yarn.timeline-service.webapp.https.address"] = ( "%s:8190" % tls_instance.fqdn()) return blueprint def _confgure_hbase_site(cluster, blueprint): props = _find_hbase_site(blueprint) props["hbase.regionserver.global.memstore.lowerLimit"] = "0.38" props["hbase.regionserver.global.memstore.upperLimit"] = "0.4" props["hbase.regionserver.handler.count"] = "60" props["hbase.regionserver.info.port"] = "60030" props["hbase.regionserver.storefile.refresh.period"] = "20" props["hbase.rootdir"] = "hdfs://hdfs-ha/apps/hbase/data" props["hbase.security.authentication"] = "simple" props["hbase.security.authorization"] = "false" props["hbase.superuser"] = "hbase" props["hbase.tmp.dir"] = "/hadoop/hbase" props["hbase.zookeeper.property.clientPort"] = "2181" zk_instances = utils.get_instances(cluster, p_common.ZOOKEEPER_SERVER) zk_quorum_value = ",".join([i.fqdn() for i in zk_instances]) props["hbase.zookeeper.quorum"] = zk_quorum_value props["hbase.zookeeper.useMulti"] = "true" props["hfile.block.cache.size"] = "0.40" props["zookeeper.session.timeout"] = "30000" props["zookeeper.znode.parent"] = "/hbase-unsecure" return blueprint sahara-8.0.0/sahara/plugins/ambari/health.py0000666000175100017510000001304613245514472021011 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import collections import functools from oslo_log import log as logging import six from sahara.i18n import _ from sahara.plugins.ambari import client from sahara.plugins.ambari import common as p_common from sahara.plugins import utils as plugin_utils from sahara.service.health import health_check_base LOG = logging.getLogger(__name__) class AlertsProvider(object): def __init__(self, cluster): self._data = None self._cluster_services = None self._exception_store = None self.cluster = cluster # calling to cache all data self.get_alerts_data() def get_cluster_services(self): return self._cluster_services def is_ambari_active(self): if self._exception_store: raise health_check_base.RedHealthError(self._exception_store) return _("Ambari Monitor is healthy") def get_alerts_data(self, service=None): if self._data is not None: # return cached data return self._data.get(service, []) if service else self._data self._data = {} self._cluster_services = [] try: ambari = plugin_utils.get_instance( self.cluster, p_common.AMBARI_SERVER) password = self.cluster.extra.get("ambari_password") with client.AmbariClient(ambari, password=password) as ambari: resp = ambari.get_alerts_data(self.cluster) for alert in resp: alert = alert.get('Alert', {}) service = alert.get('service_name').lower() if service not in self._data: self._data[service] = [] self._cluster_services.append(service) self._data[service].append(alert) except Exception as e: prefix = _("Can't get response from Ambari Monitor") msg = _("%(problem)s: %(description)s") % { 'problem': prefix, 'description': six.text_type(e)} # don't put in exception to logs, it will be done by log.exception LOG.exception(prefix) self._exception_store = msg class AmbariHealthCheck(health_check_base.BasicHealthCheck): def __init__(self, cluster, provider): self.provider = provider super(AmbariHealthCheck, self).__init__(cluster) def get_health_check_name(self): return "Ambari alerts health check" def is_available(self): return self.cluster.plugin_name == 'ambari' def check_health(self): return self.provider.is_ambari_active() class AmbariServiceHealthCheck(health_check_base.BasicHealthCheck): def __init__(self, cluster, provider, service): self.provider = provider self.service = service.lower() super(AmbariServiceHealthCheck, self).__init__(cluster) def get_health_check_name(self): return "Ambari alerts for %s Service" % self.service def is_available(self): return self.cluster.plugin_name == 'ambari' def get_important_services(self): return [ p_common.HDFS_SERVICE.lower(), p_common.YARN_SERVICE.lower(), p_common.OOZIE_SERVICE.lower(), p_common.ZOOKEEPER_SERVICE.lower() ] def check_health(self): imp_map = {'OK': 'GREEN', 'WARNING': 'YELLOW', 'CRITICAL': 'RED'} other_map = {'OK': 'GREEN'} color_counter = collections.Counter() important_services = self.get_important_services() for alert in self.provider.get_alerts_data(self.service): alert_summary = alert.get('state', 'UNKNOWN') if self.service in important_services: target = imp_map.get(alert_summary, 'RED') else: target = other_map.get(alert_summary, 'YELLOW') color_counter[target] += 1 if color_counter['RED'] > 0 and color_counter['YELLOW'] > 0: raise health_check_base.RedHealthError( _("Ambari Monitor has responded that cluster has " "%(red)d critical and %(yellow)d warning alert(s)") % {'red': color_counter['RED'], 'yellow': color_counter['YELLOW']}) elif color_counter['RED'] > 0: raise health_check_base.RedHealthError( _("Ambari Monitor has responded that cluster has " "%(red)d critical alert(s)") % {'red': color_counter['RED']}) elif color_counter['YELLOW'] > 0: raise health_check_base.YellowHealthError( _("Ambari Monitor has responded that cluster " "has %d warning alert(s)") % color_counter['YELLOW']) return _("No alerts found") def get_health_checks(cluster): provider = AlertsProvider(cluster) checks = [functools.partial(AmbariHealthCheck, provider=provider)] for service in provider.get_cluster_services(): checks.append(functools.partial( AmbariServiceHealthCheck, provider=provider, service=service)) return checks sahara-8.0.0/sahara/plugins/ambari/edp_engine.py0000666000175100017510000001176713245514472021651 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import exceptions as exc from sahara.i18n import _ from sahara.plugins.ambari import common as p_common from sahara.plugins import exceptions as pex from sahara.plugins import kerberos from sahara.plugins import utils as plugin_utils from sahara.service.edp import hdfs_helper from sahara.service.edp.oozie import engine as oozie_engine from sahara.service.edp.spark import engine as spark_engine def _get_lib_location(instance, lib_name): with instance.remote() as r: code, jar_path = r.execute_command( ('find /usr/hdp -name "{lib_name}" 2>/dev/null ' '-print | head -n 1'.format(lib_name=lib_name)), run_as_root=True) # drop last whitespace character return jar_path.rstrip() def _get_hadoop_openstack_jar_location(instance): return _get_lib_location(instance, "hadoop-openstack*.jar") def _get_jackson_core(instance): return _get_lib_location(instance, "jackson-core-asl-1.9*.jar") class EDPOozieEngine(oozie_engine.OozieJobEngine): def get_hdfs_user(self): return "oozie" def get_client(self): if kerberos.is_kerberos_security_enabled(self.cluster): return super(EDPOozieEngine, self).get_remote_client() return super(EDPOozieEngine, self).get_client() def create_hdfs_dir(self, remote, dir_name): hdfs_helper.create_dir_hadoop2(remote, dir_name, self.get_hdfs_user()) def get_oozie_server_uri(self, cluster): oozie = plugin_utils.get_instance(cluster, p_common.OOZIE_SERVER) return "http://%s:11000/oozie" % oozie.management_ip def get_name_node_uri(self, cluster): namenodes = plugin_utils.get_instances(cluster, p_common.NAMENODE) if len(namenodes) == 1: return "hdfs://%s:8020" % namenodes[0].fqdn() else: return "hdfs://hdfs-ha" def get_resource_manager_uri(self, cluster): resourcemanagers = plugin_utils.get_instances(cluster, p_common.RESOURCEMANAGER) return "%s:8050" % resourcemanagers[0].fqdn() def get_oozie_server(self, cluster): return plugin_utils.get_instance(cluster, p_common.OOZIE_SERVER) def validate_job_execution(self, cluster, job, data): oozie_count = plugin_utils.get_instances_count(cluster, p_common.OOZIE_SERVER) if oozie_count != 1: raise pex.InvalidComponentCountException( p_common.OOZIE_SERVER, "1", oozie_count) super(EDPOozieEngine, self).validate_job_execution(cluster, job, data) @staticmethod def get_possible_job_config(job_type): return {"job_config": []} class EDPSparkEngine(spark_engine.SparkJobEngine): edp_base_version = "2.2" def __init__(self, cluster): super(EDPSparkEngine, self).__init__(cluster) # searching for spark instance self.master = plugin_utils.get_instance( cluster, p_common.SPARK_JOBHISTORYSERVER) self.plugin_params["spark-user"] = "sudo -u spark " self.plugin_params["spark-submit"] = "spark-submit" self.plugin_params["deploy-mode"] = "cluster" self.plugin_params["master"] = "yarn-cluster" @staticmethod def edp_supported(version): return version >= EDPSparkEngine.edp_base_version def run_job(self, job_execution): # calculate class-path dynamically driver_classpath = [ _get_hadoop_openstack_jar_location(self.master), _get_jackson_core(self.master)] self.plugin_params['driver-class-path'] = ":".join(driver_classpath) self.plugin_params['drivers-to-jars'] = driver_classpath return super(EDPSparkEngine, self).run_job(job_execution) def validate_job_execution(self, cluster, job, data): if not self.edp_supported(cluster.hadoop_version): raise exc.InvalidDataException( _('Ambari plugin of {base} or higher required to run {type} ' 'jobs').format( base=EDPSparkEngine.edp_base_version, type=job.type)) spark_nodes_count = plugin_utils.get_instances_count( cluster, p_common.SPARK_JOBHISTORYSERVER) if spark_nodes_count != 1: raise pex.InvalidComponentCountException( p_common.SPARK_JOBHISTORYSERVER, '1', spark_nodes_count) super(EDPSparkEngine, self).validate_job_execution( cluster, job, data) sahara-8.0.0/sahara/plugins/ambari/resources/0000775000175100017510000000000013245515026021172 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/ambari/resources/generate_config.py0000777000175100017510000000477513245514472024711 0ustar zuulzuul00000000000000#!/usr/bin/env python # Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import print_function import argparse import sys from oslo_serialization import jsonutils import requests def get_blueprint(ambari_address, username, password, cluster_name): url = "http://%s:8080/api/v1/clusters/%s?format=blueprint" % ( ambari_address, cluster_name) resp = requests.get(url, auth=(username, password)) resp.raise_for_status() if resp.text: return jsonutils.loads(resp.text) def generate_config(blueprint): configs = {} for entity in blueprint["configurations"]: for cfg in entity: p = entity[cfg]["properties"] if not p: continue if "content" in p: del p["content"] for k, v in p.items(): p[k] = " ".join(v.split()) if p: configs[cfg] = p return configs def write_config(cfg, version): with open("sahara/plugins/ambari/resources/configs-%s.json" % version, "w") as fp: jsonutils.dump(cfg, fp, indent=4, sort_keys=True, separators=(",", ": ")) def main(): parser = argparse.ArgumentParser( description="Ambari sample config generator") parser.add_argument("--address", help="Ambari address", default="localhost") parser.add_argument("--username", help="Ambari username", default="admin") parser.add_argument("--password", help="Ambari password", default="admin") parser.add_argument("--cluster-name", help="Name of cluster", default="cluster") ns = parser.parse_args(sys.argv[1:]) bp = get_blueprint(ns.address, ns.username, ns.password, ns.cluster_name) cfg = generate_config(bp) write_config(cfg, bp["Blueprints"]["stack_version"]) if __name__ == "__main__": main() sahara-8.0.0/sahara/plugins/ambari/resources/configs-2.5.json0000666000175100017510000043771313245514472024044 0ustar zuulzuul00000000000000{ "accumulo-env": { "accumulo_gc_heapsize": "256", "accumulo_instance_name": "hdp-accumulo-instance", "accumulo_log_dir": "/var/log/accumulo", "accumulo_master_heapsize": "1024", "accumulo_monitor_bind_all": "false", "accumulo_monitor_heapsize": "1024", "accumulo_other_heapsize": "1024", "accumulo_pid_dir": "/var/run/accumulo", "accumulo_tserver_heapsize": "1536", "accumulo_user": "accumulo", "server_content": "#! /usr/bin/env bash export HADOOP_PREFIX={{hadoop_prefix}} export HADOOP_CONF_DIR={{hadoop_conf_dir}} export JAVA_HOME={{java64_home}} export ZOOKEEPER_HOME={{zookeeper_home}} export ACCUMULO_PID_DIR={{pid_dir}} export ACCUMULO_LOG_DIR={{log_dir}} export ACCUMULO_CONF_DIR={{server_conf_dir}} export ACCUMULO_TSERVER_OPTS=\"-Xmx{{accumulo_tserver_heapsize}}m -Xms{{accumulo_tserver_heapsize}}m\" export ACCUMULO_MASTER_OPTS=\"-Xmx{{accumulo_master_heapsize}}m -Xms{{accumulo_master_heapsize}}m\" export ACCUMULO_MONITOR_OPTS=\"-Xmx{{accumulo_monitor_heapsize}}m -Xms{{accumulo_monitor_heapsize}}m\" export ACCUMULO_GC_OPTS=\"-Xmx{{accumulo_gc_heapsize}}m -Xms{{accumulo_gc_heapsize}}m\" export ACCUMULO_GENERAL_OPTS=\"-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -Djava.net.preferIPv4Stack=true ${ACCUMULO_GENERAL_OPTS}\" export ACCUMULO_OTHER_OPTS=\"-Xmx{{accumulo_other_heapsize}}m -Xms{{accumulo_other_heapsize}}m ${ACCUMULO_OTHER_OPTS}\" export ACCUMULO_MONITOR_BIND_ALL={{monitor_bind_str}} # what do when the JVM runs out of heap memory export ACCUMULO_KILL_CMD='kill -9 %p'" }, "accumulo-log4j": { "audit_log_level": "OFF", "debug_log_size": "512M", "debug_num_logs": "10", "info_log_size": "512M", "info_num_logs": "10", "monitor_forwarding_log_level": "WARN" }, "accumulo-site": { "gc.port.client": "50092", "general.classpaths": "$ACCUMULO_HOME/lib/accumulo-server.jar, $ACCUMULO_HOME/lib/accumulo-core.jar, $ACCUMULO_HOME/lib/accumulo-start.jar, $ACCUMULO_HOME/lib/accumulo-fate.jar, $ACCUMULO_HOME/lib/accumulo-proxy.jar, $ACCUMULO_HOME/lib/[^.].*.jar, $ZOOKEEPER_HOME/zookeeper[^.].*.jar, $HADOOP_CONF_DIR, /usr/hdp/current/hadoop-client/[^.].*.jar, /usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar, /usr/hdp/current/hadoop-hdfs-client/[^.].*.jar, /usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar, /usr/hdp/current/hadoop-yarn-client/[^.].*.jar, /usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar, /usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar", "instance.volumes": "hdfs://%HOSTGROUP::host_group_1%:8020/apps/accumulo/data", "instance.zookeeper.host": "%HOSTGROUP::host_group_1%:2181", "instance.zookeeper.timeout": "30s", "master.port.client": "9999", "monitor.port.client": "50095", "monitor.port.log4j": "4560", "trace.port.client": "12234", "trace.user": "trace", "tserver.cache.data.size": "128M", "tserver.cache.index.size": "256M", "tserver.memory.maps.max": "1G", "tserver.memory.maps.native.enabled": "true", "tserver.port.client": "9997", "tserver.sort.buffer.size": "200M", "tserver.walog.max.size": "1G" }, "activity-conf": { "activity.explorer.user": "activity_explorer", "global.activity.analyzer.user": "activity_analyzer", "global.activity.processing.parallelism": "8", "global.activity.processor.pool.max.wait.seconds": "60", "hdfs.activity.watcher.enabled": "true", "mr_job.activity.watcher.enabled": "true", "mr_job.max.job.size.mb.for.parallel.execution": "500", "phoenix.sink.batch.size": "100", "phoenix.sink.flush.interval.seconds": "30", "tez_job.activity.watcher.enabled": "true", "tez_job.tmp.dir": "/var/lib/smartsense/activity-analyzer/tez/tmp/", "yarn_app.activity.watcher.enabled": "true" }, "activity-env": { "activity-env-content": "#!/bin/bash # Copyright 2014 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # \"License\"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an \"AS IS\" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Enable verbose shell execution #set -xv ## Set HOME for various components export HADOOP_HOME=/usr/hdp/current/hadoop-client export HDFS_HOME=/usr/hdp/current/hadoop-hdfs-client export MAPREDUCE_HOME=/usr/hdp/current/hadoop-mapreduce-client export YARN_HOME=/usr/hdp/current/hadoop-yarn-client export HIVE_HOME=/usr/hdp/current/hive-client export HCAT_HOME=/usr/hdp/current/hive-webhcat export TEZ_HOME=/usr/hdp/current/tez-client export HBASE_HOME=/usr/hdp/current/hbase-client export PHOENIX_HOME=/usr/hdp/current/phoenix-client export ACTIVITY_ANALYZER_HOME=/usr/hdp/share/hst/activity-analyzer export AMS_COLLECTOR_HOME=/usr/lib/ambari-metrics-collector ## Set conf dir for various components export HADOOP_CONF_DIR=/etc/hadoop/conf/ export HIVE_CONF_DIR=/etc/hive/conf/ export HBASE_CONF_DIR=/etc/hbase/conf/ export TEZ_CONF_DIR=/etc/tez/conf/ export ACTIVITY_ANALYZER_CONF_DIR=/etc/smartsense-activity/conf/ export AMS_HBASE_CONF=/etc/ams-hbase/conf export DEBUG_ENABLED=false" }, "activity-log4j": { "activity-log4j-content": "# Copyright 2014 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # \"License\"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an \"AS IS\" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Define some default values that can be overridden by system properties # Root logger option log4j.rootLogger=INFO,file log4j.appender.file=org.apache.log4j.RollingFileAppender log4j.appender.file.File={{activity_log_dir}}/${log.file.name} log4j.appender.file.MaxFileSize=30MB log4j.appender.file.MaxBackupIndex=10 log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%d{DATE} %5p [%t] %c{1}:%L - %m%n", "activity_log_dir": "/var/log/smartsense-activity" }, "activity-zeppelin-env": { "activity-zeppelin-env-content": "#!/bin/bash # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the \"License\"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an \"AS IS\" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # export JAVA_HOME={{java_home}} # Additional jvm options. for example, export ZEPPELIN_JAVA_OPTS=\"-Dspark.executor.memory=8g -Dspark.cores.max=16\" export ZEPPELIN_JAVA_OPTS=\"-Dhdp.version={{hdp_version}} -Dlog.file.name=activity-explorer.log -DSmartSenseActivityExplorer\" # export ZEPPELIN_MEM # Zeppelin jvm mem options Default -Xmx1024m -XX:MaxPermSize=512m # export ZEPPELIN_INTP_MEM # zeppelin interpreter process jvm mem options. Default = ZEPPELIN_MEM # export ZEPPELIN_INTP_JAVA_OPTS # zeppelin interpreter process jvm options. Default = ZEPPELIN_JAVA_OPTS export ZEPPELIN_LOG_DIR={{activity_log_dir}} export ZEPPELIN_PID_DIR=/var/run/smartsense-activity-explorer export ZEPPELIN_WAR_TEMPDIR=/var/lib/smartsense/activity-explorer/webapp export ZEPPELIN_NOTEBOOK_DIR=/var/lib/smartsense/activity-explorer/notebook export ZEPPELIN_CLASSPATH=\"/etc/ams-hbase/conf:${ZEPPELIN_CLASSPATH}\" export CLASSPATH=${ZEPPELIN_CLASSPATH}" }, "activity-zeppelin-interpreter": { "activity-zeppelin-interpreter-content": "{ \"interpreterSettings\": { \"2BJB693M8\": { \"id\": \"2BJB693M8\", \"name\": \"phoenix\", \"group\": \"phoenix\", \"properties\": { \"phoenix.jdbc.url\": \"{{activity_explorer_jdbc_url}}\", \"phoenix.user\": \"\", \"phoenix.password\": \"\", \"phoenix.max.result\": \"1000\", \"phoenix.driver.name\": \"org.apache.phoenix.jdbc.PhoenixDriver\" }, \"interpreterGroup\": [ { \"class\": \"org.apache.zeppelin.phoenix.PhoenixInterpreter\", \"name\": \"sql\" } ], \"dependencies\": [], \"option\": { \"remote\": true, \"perNoteSession\": false } } }, \"interpreterBindings\": { \"2BNVQJUBK\": [ \"2BJB693M8\" ], \"2BPD7951H\": [ \"2BJB693M8\" ], \"2BQH91X36\": [ \"2BJB693M8\" ], \"2BTCVPTMH\": [ \"2BJB693M8\" ] }, \"interpreterRepositories\": [ { \"id\": \"central\", \"type\": \"default\", \"url\": \"http://repo1.maven.org/maven2/\", \"releasePolicy\": { \"enabled\": true, \"updatePolicy\": \"daily\", \"checksumPolicy\": \"warn\" }, \"snapshotPolicy\": { \"enabled\": true, \"updatePolicy\": \"daily\", \"checksumPolicy\": \"warn\" }, \"mirroredRepositories\": [], \"repositoryManager\": false } ] }" }, "activity-zeppelin-shiro": { "main.securityManager.sessionManager": "$sessionManager", "main.sessionManager": "org.apache.shiro.web.session.mgt.DefaultWebSessionManager", "securityManager.sessionManager.globalSessionTimeout": "86400000" }, "activity-zeppelin-site": { "zeppelin.anonymous.allowed": "false", "zeppelin.interpreter.connect.timeout": "30000", "zeppelin.interpreter.dir": "/usr/hdp/share/hst/activity-explorer/interpreter", "zeppelin.interpreters": "org.apache.zeppelin.phoenix.PhoenixInterpreter", "zeppelin.notebook.dir": "/var/lib/smartsense/activity-explorer/notebook", "zeppelin.notebook.homescreen.hide": "false", "zeppelin.notebook.storage": "org.apache.zeppelin.notebook.repo.VFSNotebookRepo", "zeppelin.server.addr": "0.0.0.0", "zeppelin.server.allowed.origins": "*", "zeppelin.server.context.path": "/", "zeppelin.server.port": "9060", "zeppelin.ssl": "false", "zeppelin.ssl.client.auth": "false", "zeppelin.ssl.keystore.path": "/var/lib/smartsense/activity-explorer/keystore", "zeppelin.ssl.keystore.type": "JKS", "zeppelin.ssl.truststore.path": "/var/lib/smartsense/activity-explorer/truststore", "zeppelin.ssl.truststore.type": "JKS", "zeppelin.war.tempdir": "/var/lib/smartsense/activity-explorer/webapp", "zeppelin.websocket.max.text.message.size": "1024000" }, "ams-env": { "ambari_metrics_user": "ams", "metrics_collector_heapsize": "512", "metrics_collector_log_dir": "/var/log/ambari-metrics-collector", "metrics_collector_pid_dir": "/var/run/ambari-metrics-collector", "metrics_monitor_log_dir": "/var/log/ambari-metrics-monitor", "metrics_monitor_pid_dir": "/var/run/ambari-metrics-monitor" }, "ams-grafana-env": { "metrics_grafana_data_dir": "/var/lib/ambari-metrics-grafana", "metrics_grafana_log_dir": "/var/log/ambari-metrics-grafana", "metrics_grafana_pid_dir": "/var/run/ambari-metrics-grafana", "metrics_grafana_username": "admin" }, "ams-grafana-ini": { "cert_file": "/etc/ambari-metrics-grafana/conf/ams-grafana.crt", "cert_key": "/etc/ambari-metrics-grafana/conf/ams-grafana.key", "port": "3000", "protocol": "http" }, "ams-hbase-env": { "hbase_classpath_additional": "", "hbase_log_dir": "/var/log/ambari-metrics-collector", "hbase_master_heapsize": "768", "hbase_master_maxperm_size": "128", "hbase_master_xmn_size": "256", "hbase_pid_dir": "/var/run/ambari-metrics-collector/", "hbase_regionserver_heapsize": "768", "hbase_regionserver_shutdown_timeout": "30", "hbase_regionserver_xmn_ratio": "0.2", "max_open_files_limit": "32768", "regionserver_xmn_size": "128" }, "ams-hbase-policy": { "security.admin.protocol.acl": "*", "security.client.protocol.acl": "*", "security.masterregion.protocol.acl": "*" }, "ams-hbase-security-site": { "ams.zookeeper.keytab": "", "ams.zookeeper.principal": "", "hadoop.security.authentication": "", "hbase.coprocessor.master.classes": "", "hbase.coprocessor.region.classes": "", "hbase.master.kerberos.principal": "", "hbase.master.keytab.file": "", "hbase.myclient.keytab": "", "hbase.myclient.principal": "", "hbase.regionserver.kerberos.principal": "", "hbase.regionserver.keytab.file": "", "hbase.security.authentication": "", "hbase.security.authorization": "", "hbase.zookeeper.property.authProvider.1": "", "hbase.zookeeper.property.jaasLoginRenew": "", "hbase.zookeeper.property.kerberos.removeHostFromPrincipal": "", "hbase.zookeeper.property.kerberos.removeRealmFromPrincipal": "" }, "ams-hbase-site": { "dfs.client.read.shortcircuit": "true", "hbase.client.scanner.caching": "10000", "hbase.client.scanner.timeout.period": "300000", "hbase.cluster.distributed": "false", "hbase.hregion.majorcompaction": "0", "hbase.hregion.max.filesize": "4294967296", "hbase.hregion.memstore.block.multiplier": "4", "hbase.hregion.memstore.flush.size": "134217728", "hbase.hstore.blockingStoreFiles": "200", "hbase.hstore.flusher.count": "2", "hbase.local.dir": "${hbase.tmp.dir}/local", "hbase.master.info.bindAddress": "0.0.0.0", "hbase.master.info.port": "61310", "hbase.master.normalizer.class": "org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer", "hbase.master.port": "61300", "hbase.master.wait.on.regionservers.mintostart": "1", "hbase.normalizer.enabled": "false", "hbase.normalizer.period": "600000", "hbase.regionserver.global.memstore.lowerLimit": "0.3", "hbase.regionserver.global.memstore.upperLimit": "0.35", "hbase.regionserver.info.port": "61330", "hbase.regionserver.port": "61320", "hbase.regionserver.thread.compaction.large": "2", "hbase.regionserver.thread.compaction.small": "3", "hbase.replication": "false", "hbase.rootdir": "file:///var/lib/ambari-metrics-collector/hbase", "hbase.rpc.timeout": "300000", "hbase.snapshot.enabled": "false", "hbase.superuser": "activity_explorer,activity_analyzer", "hbase.tmp.dir": "/var/lib/ambari-metrics-collector/hbase-tmp", "hbase.zookeeper.leaderport": "61388", "hbase.zookeeper.peerport": "61288", "hbase.zookeeper.property.clientPort": "{{zookeeper_clientPort}}", "hbase.zookeeper.property.dataDir": "${hbase.tmp.dir}/zookeeper", "hbase.zookeeper.property.tickTime": "6000", "hbase.zookeeper.quorum": "{{zookeeper_quorum_hosts}}", "hfile.block.cache.size": "0.3", "phoenix.coprocessor.maxMetaDataCacheSize": "20480000", "phoenix.coprocessor.maxServerCacheTimeToLiveMs": "60000", "phoenix.groupby.maxCacheSize": "307200000", "phoenix.mutate.batchSize": "10000", "phoenix.query.keepAliveMs": "300000", "phoenix.query.maxGlobalMemoryPercentage": "15", "phoenix.query.rowKeyOrderSaltedTable": "true", "phoenix.query.spoolThresholdBytes": "20971520", "phoenix.query.timeoutMs": "300000", "phoenix.sequence.saltBuckets": "2", "phoenix.spool.directory": "${hbase.tmp.dir}/phoenix-spool", "zookeeper.session.timeout": "120000", "zookeeper.session.timeout.localHBaseCluster": "120000", "zookeeper.znode.parent": "/ams-hbase-unsecure" }, "ams-site": { "phoenix.query.maxGlobalMemoryPercentage": "25", "phoenix.spool.directory": "/tmp", "timeline.metrics.aggregator.checkpoint.dir": "/var/lib/ambari-metrics-collector/checkpoint", "timeline.metrics.aggregators.skip.blockcache.enabled": "false", "timeline.metrics.cache.commit.interval": "3", "timeline.metrics.cache.enabled": "true", "timeline.metrics.cache.size": "150", "timeline.metrics.cluster.aggregate.splitpoints": "mapred.ShuffleMetrics.ShuffleOutputsFailed", "timeline.metrics.cluster.aggregator.daily.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.daily.disabled": "false", "timeline.metrics.cluster.aggregator.daily.interval": "86400", "timeline.metrics.cluster.aggregator.daily.ttl": "63072000", "timeline.metrics.cluster.aggregator.hourly.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.hourly.disabled": "false", "timeline.metrics.cluster.aggregator.hourly.interval": "3600", "timeline.metrics.cluster.aggregator.hourly.ttl": "31536000", "timeline.metrics.cluster.aggregator.interpolation.enabled": "true", "timeline.metrics.cluster.aggregator.minute.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.minute.disabled": "false", "timeline.metrics.cluster.aggregator.minute.interval": "300", "timeline.metrics.cluster.aggregator.minute.ttl": "2592000", "timeline.metrics.cluster.aggregator.second.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.second.disabled": "false", "timeline.metrics.cluster.aggregator.second.interval": "120", "timeline.metrics.cluster.aggregator.second.timeslice.interval": "30", "timeline.metrics.cluster.aggregator.second.ttl": "259200", "timeline.metrics.daily.aggregator.minute.interval": "86400", "timeline.metrics.hbase.compression.scheme": "SNAPPY", "timeline.metrics.hbase.data.block.encoding": "FAST_DIFF", "timeline.metrics.hbase.fifo.compaction.enabled": "true", "timeline.metrics.hbase.init.check.enabled": "true", "timeline.metrics.host.aggregate.splitpoints": "mapred.ShuffleMetrics.ShuffleOutputsFailed", "timeline.metrics.host.aggregator.daily.checkpointCutOffMultiplier": "2", "timeline.metrics.host.aggregator.daily.disabled": "false", "timeline.metrics.host.aggregator.daily.ttl": "31536000", "timeline.metrics.host.aggregator.hourly.checkpointCutOffMultiplier": "2", "timeline.metrics.host.aggregator.hourly.disabled": "false", "timeline.metrics.host.aggregator.hourly.interval": "3600", "timeline.metrics.host.aggregator.hourly.ttl": "2592000", "timeline.metrics.host.aggregator.minute.checkpointCutOffMultiplier": "2", "timeline.metrics.host.aggregator.minute.disabled": "false", "timeline.metrics.host.aggregator.minute.interval": "300", "timeline.metrics.host.aggregator.minute.ttl": "604800", "timeline.metrics.host.aggregator.ttl": "86400", "timeline.metrics.service.checkpointDelay": "60", "timeline.metrics.service.cluster.aggregator.appIds": "datanode,nodemanager,hbase", "timeline.metrics.service.default.result.limit": "15840", "timeline.metrics.service.handler.thread.count": "20", "timeline.metrics.service.http.policy": "HTTP_ONLY", "timeline.metrics.service.operation.mode": "embedded", "timeline.metrics.service.resultset.fetchSize": "2000", "timeline.metrics.service.rpc.address": "0.0.0.0:60200", "timeline.metrics.service.use.groupBy.aggregators": "true", "timeline.metrics.service.watcher.delay": "30", "timeline.metrics.service.watcher.disabled": "false", "timeline.metrics.service.watcher.initial.delay": "600", "timeline.metrics.service.watcher.timeout": "30", "timeline.metrics.service.webapp.address": "%HOSTGROUP::host_group_1%:6188", "timeline.metrics.sink.collection.period": "10", "timeline.metrics.sink.report.interval": "60" }, "ams-ssl-client": { "ssl.client.truststore.location": "/etc/security/clientKeys/all.jks", "ssl.client.truststore.type": "jks" }, "ams-ssl-server": { "ssl.server.keystore.location": "/etc/security/serverKeys/keystore.jks", "ssl.server.keystore.type": "jks", "ssl.server.truststore.location": "/etc/security/serverKeys/all.jks", "ssl.server.truststore.reload.interval": "10000", "ssl.server.truststore.type": "jks" }, "anonymization-rules": { "anonymization-rules-content": "{ \"rules\":[ { \"name\":\"ip_address\", \"ruleId\": \"Pattern\", \"path\":null, \"pattern\": \"([^a-z0-9\\\\.]|^)[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}([^a-z0-9\\\\.\\\\-]|(\\\\.[^0-9])|$)\", \"extract\": \"[ :\\\\/]?([0-9\\\\.]+)[ :\\\\/]?\", \"excludes\": [\"hdp-select*.*\", \"*version.txt\"], \"shared\":true }, { \"name\":\"domain\", \"ruleId\": \"Domain\", \"path\":null, \"pattern\": \"$DOMAIN_RULE$\", \"shared\":true }, { \"name\":\"delete_oozie_jdbc_password\", \"ruleId\": \"Property\", \"path\":\"oozie-site.xml\", \"property\": \"oozie.service.JPAService.jdbc.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_sqoop_metastore_password\", \"ruleId\": \"Property\", \"path\":\"sqoop-site.xml\", \"property\": \"sqoop.metastore.client.autoconnect.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_hive_metastore_password\", \"ruleId\": \"Property\", \"path\":\"hive-site.xml\", \"property\": \"javax.jdo.option.ConnectionPassword\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_s3_accesskey\", \"ruleId\": \"Property\", \"path\":\"core-site.xml\", \"property\": \"fs.s3.awsAccessKeyId\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_s3_secret_accesskey\", \"ruleId\": \"Property\", \"path\":\"core-site.xml\", \"property\": \"fs.s3.awsSecretAccessKey\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_s3n_accesskey\", \"ruleId\": \"Property\", \"path\":\"core-site.xml\", \"property\": \"fs.s3n.awsAccessKeyId\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_s3n_secret_accesskey\", \"ruleId\": \"Property\", \"path\":\"core-site.xml\", \"property\": \"fs.s3n.awsSecretAccessKey\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_azure_account_key\", \"ruleId\": \"Property\", \"path\":\"core-site.xml\", \"property\": \"fs.azure.account.key.*\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_ldap_password\", \"ruleId\": \"Property\", \"path\":\"core-site.xml\", \"property\": \"hadoop.security.group.mapping.ldap.bind.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"hide_ssl_client_keystore_pwd\", \"ruleId\": \"Property\", \"path\":\"ssl-client.xml\", \"property\": \"ssl.client.keystore.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"hide_ssl_client_truststore_pwd\", \"ruleId\": \"Property\", \"path\":\"ssl-client.xml\", \"property\": \"ssl.client.truststore.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"hide_ssl_server_keystore_keypwd\", \"ruleId\": \"Property\", \"path\":\"ssl-server.xml\", \"property\": \"ssl.server.keystore.keypassword\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"hide_ssl_server_keystore_pwd\", \"ruleId\": \"Property\", \"path\":\"ssl-server.xml\", \"property\": \"ssl.server.keystore.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"hide_ssl_server_truststore_pwd\", \"ruleId\": \"Property\", \"path\":\"ssl-server.xml\", \"property\": \"ssl.server.truststore.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"hide_oozie_pwd_in_java_process_info\", \"ruleId\": \"Pattern\", \"path\":\"java_process.txt\", \"pattern\": \"oozie.https.keystore.pass=([^ ]*)\", \"extract\": \"=([^ ]*)\", \"shared\":false }, { \"name\":\"hide_oozie_pwd_in_process_info\", \"ruleId\": \"Pattern\", \"path\":\"pid.txt\", \"pattern\": \"oozie.https.keystore.pass=([^ ]*)\", \"extract\": \"=([^ ]*)\", \"shared\":false }, { \"name\":\"hide_oozie_pwd_in_ambariagent_log\", \"ruleId\": \"Pattern\", \"path\":\"ambari-agent.log\", \"pattern\": \"oozie.https.keystore.pass=([^ ]*)\", \"extract\": \"=([^ ]*)\", \"shared\":false }, { \"name\":\"delete_oozie_https_keystore_pass\", \"ruleId\": \"Pattern\", \"path\":\"oozie-env.cmd\", \"pattern\":\"OOZIE_HTTPS_KEYSTORE_PASS=([^ ]*)\", \"extract\": \"=([^ ]*)\", \"shared\":false }, { \"name\":\"java_process_ganglia_password\", \"ruleId\": \"Pattern\", \"path\":\"java_process.txt\", \"pattern\":\"ganglia_password=([^ ]*)\", \"extract\": \"=([^ ]*)\", \"shared\":false }, { \"name\":\"hide_ssn_from_logs\", \"ruleId\": \"Pattern\", \"path\":\"*\\\\.log*\", \"pattern\": \"(^|[^0-9x])[0-9x]{3}-[0-9x]{2}-[0-9]{4}($|[^0-9x])\", \"extract\": \"(? 7 %}-XX:+UseG1GC -XX:TLABSize=8m -XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts -XX:MetaspaceSize=1024m -XX:InitiatingHeapOccupancyPercent=80 -XX:MaxGCPauseMillis=200{% else %}-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC{% endif %}", "llap_log_level": "INFO", "llap_queue_capacity": "0", "num_llap_nodes": "1", "num_retries_for_checking_llap_status": "10", "slider_am_container_mb": "341" }, "hive-interactive-site": { "hive.driver.parallel.compilation": "true", "hive.exec.orc.split.strategy": "HYBRID", "hive.execution.engine": "tez", "hive.execution.mode": "llap", "hive.llap.auto.allow.uber": "false", "hive.llap.client.consistent.splits": "true", "hive.llap.daemon.allow.permanent.fns": "false", "hive.llap.daemon.num.executors": "1", "hive.llap.daemon.queue.name": "default", "hive.llap.daemon.rpc.port": "15001", "hive.llap.daemon.service.hosts": "@llap0", "hive.llap.daemon.task.scheduler.enable.preemption": "true", "hive.llap.daemon.vcpus.per.instance": "${hive.llap.daemon.num.executors}", "hive.llap.daemon.yarn.container.mb": "341", "hive.llap.daemon.yarn.shuffle.port": "15551", "hive.llap.execution.mode": "all", "hive.llap.io.enabled": "true", "hive.llap.io.memory.mode": "", "hive.llap.io.memory.size": "0", "hive.llap.io.threadpool.size": "2", "hive.llap.io.use.lrfu": "true", "hive.llap.management.rpc.port": "15004", "hive.llap.object.cache.enabled": "true", "hive.llap.task.scheduler.locality.delay": "-1", "hive.llap.zk.sm.connectionString": "%HOSTGROUP::host_group_1%:2181", "hive.mapjoin.hybridgrace.hashtable": "false", "hive.metastore.event.listeners": "", "hive.metastore.uris": "", "hive.optimize.dynamic.partition.hashjoin": "true", "hive.prewarm.enabled": "false", "hive.server2.enable.doAs": "false", "hive.server2.tez.default.queues": "default", "hive.server2.tez.initialize.default.sessions": "true", "hive.server2.tez.sessions.per.default.queue": "1", "hive.server2.thrift.http.port": "10501", "hive.server2.thrift.port": "10500", "hive.server2.webui.port": "10502", "hive.server2.webui.use.ssl": "false", "hive.server2.zookeeper.namespace": "hiveserver2-hive2", "hive.tez.bucket.pruning": "true", "hive.tez.exec.print.summary": "true", "hive.tez.input.generate.consistent.splits": "true", "hive.vectorized.execution.mapjoin.minmax.enabled": "true", "hive.vectorized.execution.mapjoin.native.enabled": "true", "hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled": "true", "hive.vectorized.execution.reduce.enabled": "true", "llap.shuffle.connection-keep-alive.enable": "true", "llap.shuffle.connection-keep-alive.timeout": "60" }, "hive-site": { "ambari.hive.db.schema.name": "hive", "atlas.hook.hive.maxThreads": "1", "atlas.hook.hive.minThreads": "1", "atlas.rest.address": "http://%HOSTGROUP::host_group_1%:21000", "datanucleus.autoCreateSchema": "false", "datanucleus.cache.level2.type": "none", "datanucleus.fixedDatastore": "true", "hive.auto.convert.join": "true", "hive.auto.convert.join.noconditionaltask": "true", "hive.auto.convert.join.noconditionaltask.size": "286331153", "hive.auto.convert.sortmerge.join": "true", "hive.auto.convert.sortmerge.join.to.mapjoin": "false", "hive.cbo.enable": "true", "hive.cli.print.header": "false", "hive.cluster.delegation.token.store.class": "org.apache.hadoop.hive.thrift.ZooKeeperTokenStore", "hive.cluster.delegation.token.store.zookeeper.connectString": "%HOSTGROUP::host_group_1%:2181", "hive.cluster.delegation.token.store.zookeeper.znode": "/hive/cluster/delegation", "hive.compactor.abortedtxn.threshold": "1000", "hive.compactor.check.interval": "300L", "hive.compactor.delta.num.threshold": "10", "hive.compactor.delta.pct.threshold": "0.1f", "hive.compactor.initiator.on": "false", "hive.compactor.worker.threads": "0", "hive.compactor.worker.timeout": "86400L", "hive.compute.query.using.stats": "true", "hive.conf.restricted.list": "hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role", "hive.convert.join.bucket.mapjoin.tez": "false", "hive.default.fileformat": "TextFile", "hive.default.fileformat.managed": "TextFile", "hive.enforce.bucketing": "false", "hive.enforce.sorting": "true", "hive.enforce.sortmergebucketmapjoin": "true", "hive.exec.compress.intermediate": "false", "hive.exec.compress.output": "false", "hive.exec.dynamic.partition": "true", "hive.exec.dynamic.partition.mode": "strict", "hive.exec.failure.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook", "hive.exec.max.created.files": "100000", "hive.exec.max.dynamic.partitions": "5000", "hive.exec.max.dynamic.partitions.pernode": "2000", "hive.exec.orc.compression.strategy": "SPEED", "hive.exec.orc.default.compress": "ZLIB", "hive.exec.orc.default.stripe.size": "67108864", "hive.exec.orc.encoding.strategy": "SPEED", "hive.exec.parallel": "false", "hive.exec.parallel.thread.number": "8", "hive.exec.post.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.atlas.hive.hook.HiveHook", "hive.exec.pre.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook", "hive.exec.reducers.bytes.per.reducer": "67108864", "hive.exec.reducers.max": "1009", "hive.exec.scratchdir": "/tmp/hive", "hive.exec.submit.local.task.via.child": "true", "hive.exec.submitviachild": "false", "hive.execution.engine": "tez", "hive.fetch.task.aggr": "false", "hive.fetch.task.conversion": "more", "hive.fetch.task.conversion.threshold": "1073741824", "hive.limit.optimize.enable": "true", "hive.limit.pushdown.memory.usage": "0.04", "hive.map.aggr": "true", "hive.map.aggr.hash.force.flush.memory.threshold": "0.9", "hive.map.aggr.hash.min.reduction": "0.5", "hive.map.aggr.hash.percentmemory": "0.5", "hive.mapjoin.bucket.cache.size": "10000", "hive.mapjoin.optimized.hashtable": "true", "hive.mapred.reduce.tasks.speculative.execution": "false", "hive.merge.mapfiles": "true", "hive.merge.mapredfiles": "false", "hive.merge.orcfile.stripe.level": "true", "hive.merge.rcfile.block.level": "true", "hive.merge.size.per.task": "256000000", "hive.merge.smallfiles.avgsize": "16000000", "hive.merge.tezfiles": "false", "hive.metastore.authorization.storage.checks": "false", "hive.metastore.cache.pinobjtypes": "Table,Database,Type,FieldSchema,Order", "hive.metastore.client.connect.retry.delay": "5s", "hive.metastore.client.socket.timeout": "1800s", "hive.metastore.connect.retries": "24", "hive.metastore.execute.setugi": "true", "hive.metastore.failure.retries": "24", "hive.metastore.kerberos.keytab.file": "/etc/security/keytabs/hive.service.keytab", "hive.metastore.kerberos.principal": "hive/_HOST@EXAMPLE.COM", "hive.metastore.pre.event.listeners": "org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener", "hive.metastore.sasl.enabled": "false", "hive.metastore.server.max.threads": "100000", "hive.metastore.uris": "thrift://%HOSTGROUP::host_group_1%:9083", "hive.metastore.warehouse.dir": "/apps/hive/warehouse", "hive.optimize.bucketmapjoin": "true", "hive.optimize.bucketmapjoin.sortedmerge": "false", "hive.optimize.constant.propagation": "true", "hive.optimize.index.filter": "true", "hive.optimize.metadataonly": "true", "hive.optimize.null.scan": "true", "hive.optimize.reducededuplication": "true", "hive.optimize.reducededuplication.min.reducer": "4", "hive.optimize.sort.dynamic.partition": "false", "hive.orc.compute.splits.num.threads": "10", "hive.orc.splits.include.file.footer": "false", "hive.prewarm.enabled": "false", "hive.prewarm.numcontainers": "3", "hive.security.authenticator.manager": "org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator", "hive.security.authorization.enabled": "false", "hive.security.authorization.manager": "org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory", "hive.security.metastore.authenticator.manager": "org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator", "hive.security.metastore.authorization.auth.reads": "true", "hive.security.metastore.authorization.manager": "org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider", "hive.server2.allow.user.substitution": "true", "hive.server2.authentication": "NONE", "hive.server2.authentication.spnego.keytab": "HTTP/_HOST@EXAMPLE.COM", "hive.server2.authentication.spnego.principal": "/etc/security/keytabs/spnego.service.keytab", "hive.server2.enable.doAs": "true", "hive.server2.logging.operation.enabled": "true", "hive.server2.logging.operation.log.location": "/tmp/hive/operation_logs", "hive.server2.max.start.attempts": "5", "hive.server2.support.dynamic.service.discovery": "true", "hive.server2.table.type.mapping": "CLASSIC", "hive.server2.tez.default.queues": "default", "hive.server2.tez.initialize.default.sessions": "false", "hive.server2.tez.sessions.per.default.queue": "1", "hive.server2.thrift.http.path": "cliservice", "hive.server2.thrift.http.port": "10001", "hive.server2.thrift.max.worker.threads": "500", "hive.server2.thrift.port": "10000", "hive.server2.thrift.sasl.qop": "auth", "hive.server2.transport.mode": "binary", "hive.server2.use.SSL": "false", "hive.server2.zookeeper.namespace": "hiveserver2", "hive.smbjoin.cache.rows": "10000", "hive.stats.autogather": "true", "hive.stats.dbclass": "fs", "hive.stats.fetch.column.stats": "true", "hive.stats.fetch.partition.stats": "true", "hive.support.concurrency": "false", "hive.tez.auto.reducer.parallelism": "true", "hive.tez.container.size": "1024", "hive.tez.cpu.vcores": "-1", "hive.tez.dynamic.partition.pruning": "true", "hive.tez.dynamic.partition.pruning.max.data.size": "104857600", "hive.tez.dynamic.partition.pruning.max.event.size": "1048576", "hive.tez.input.format": "org.apache.hadoop.hive.ql.io.HiveInputFormat", "hive.tez.java.opts": "-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps", "hive.tez.log.level": "INFO", "hive.tez.max.partition.factor": "2.0", "hive.tez.min.partition.factor": "0.25", "hive.tez.smb.number.waves": "0.5", "hive.txn.manager": "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager", "hive.txn.max.open.batch": "1000", "hive.txn.timeout": "300", "hive.user.install.directory": "/user/", "hive.vectorized.execution.enabled": "true", "hive.vectorized.execution.reduce.enabled": "false", "hive.vectorized.groupby.checkinterval": "4096", "hive.vectorized.groupby.flush.percent": "0.1", "hive.vectorized.groupby.maxentries": "100000", "hive.zookeeper.client.port": "2181", "hive.zookeeper.namespace": "hive_zookeeper_namespace", "hive.zookeeper.quorum": "%HOSTGROUP::host_group_1%:2181", "javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver", "javax.jdo.option.ConnectionURL": "jdbc:mysql://%HOSTGROUP::host_group_1%/hive?createDatabaseIfNotExist=true", "javax.jdo.option.ConnectionUserName": "hive" }, "hivemetastore-site": { "hive.metastore.metrics.enabled": "true", "hive.service.metrics.file.location": "/var/log/hive/hivemetastore-report.json", "hive.service.metrics.hadoop2.component": "hivemetastore", "hive.service.metrics.reporter": "JSON_FILE, JMX, HADOOP2" }, "hiveserver2-interactive-site": { "hive.async.log.enabled": "false", "hive.metastore.metrics.enabled": "true", "hive.service.metrics.file.location": "/var/log/hive/hiveserver2Interactive-report.json", "hive.service.metrics.hadoop2.component": "hiveserver2", "hive.service.metrics.reporter": "JSON_FILE, JMX, HADOOP2" }, "hiveserver2-site": { "hive.metastore.metrics.enabled": "true", "hive.security.authorization.enabled": "false", "hive.service.metrics.file.location": "/var/log/hive/hiveserver2-report.json", "hive.service.metrics.hadoop2.component": "hiveserver2", "hive.service.metrics.reporter": "JSON_FILE, JMX, HADOOP2" }, "hst-agent-conf": { "agent.loglevel": "INFO", "agent.tmp_dir": "/var/lib/smartsense/hst-agent/data/tmp", "bundle.logs_to_capture": "(.*).log$,(.*).out$", "server.connection_retry_count": "100", "server.connection_retry_interval": "10", "upload.retry_count": "100", "upload.retry_interval": "15" }, "hst-log4j": { "hst-log4j-content": "# Copyright 2014 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # \"License\"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an \"AS IS\" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Define some default values that can be overridden by system properties # Root logger option log4j.rootLogger=INFO,file log4j.appender.file=org.apache.log4j.RollingFileAppender log4j.appender.file.File={{hst_log_dir}}/${log.file.name} log4j.appender.file.MaxFileSize=30MB log4j.appender.file.MaxBackupIndex=10 log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%d{DATE} %5p [%t] %c{1}:%L - %m%n # HST logger log4j.logger.com.hortonworks=INFO com.github.oxo42.stateless4j=WARN log4j.logger.com.sun.jersey=WARN log4j.logger.org.eclipse.jetty.server=INFO", "hst_log_dir": "/var/log/hst" }, "hst-server-conf": { "agent.request.processing.timeout": "7200", "agent.request.syncup.interval": "180", "client.threadpool.size.max": "40", "customer.account.name": "unspecified", "customer.notification.email": "unspecified", "customer.smartsense.id": "unspecified", "gateway.host": "embedded", "gateway.port": "9451", "gateway.registration.port": "9450", "server.port": "9000", "server.storage.dir": "/var/lib/smartsense/hst-server/data", "server.tmp.dir": "/var/lib/smartsense/hst-server/tmp" }, "infra-solr-client-log4j": { "infra_solr_client_log_dir": "/var/log/ambari-infra-solr-client" }, "infra-solr-env": { "infra_solr_datadir": "/opt/ambari_infra_solr/data", "infra_solr_jmx_port": "18886", "infra_solr_kerberos_keytab": "/etc/security/keytabs/infra_solr.service.keytab", "infra_solr_kerberos_name_rules": "DEFAULT", "infra_solr_kerberos_principal": "infra-solr", "infra_solr_keystore_location": "/etc/security/serverKeys/infra.solr.keyStore.jks", "infra_solr_keystore_type": "jks", "infra_solr_log_dir": "/var/log/ambari-infra-solr", "infra_solr_maxmem": "2048", "infra_solr_minmem": "1024", "infra_solr_pid_dir": "/var/run/ambari-infra-solr", "infra_solr_port": "8886", "infra_solr_ssl_enabled": "false", "infra_solr_truststore_location": "/etc/security/serverKeys/infra.solr.trustStore.jks", "infra_solr_truststore_type": "jks", "infra_solr_user": "infra-solr", "infra_solr_web_kerberos_keytab": "/etc/security/keytabs/spnego.service.keytab", "infra_solr_web_kerberos_principal": "HTTP/_HOST@EXAMPLE.COM", "infra_solr_znode": "/infra-solr" }, "kafka-broker": { "auto.create.topics.enable": "true", "auto.leader.rebalance.enable": "true", "compression.type": "producer", "controlled.shutdown.enable": "true", "controlled.shutdown.max.retries": "3", "controlled.shutdown.retry.backoff.ms": "5000", "controller.message.queue.size": "10", "controller.socket.timeout.ms": "30000", "default.replication.factor": "1", "delete.topic.enable": "false", "external.kafka.metrics.exclude.prefix": "kafka.network.RequestMetrics,kafka.server.DelayedOperationPurgatory,kafka.server.BrokerTopicMetrics.BytesRejectedPerSec", "external.kafka.metrics.include.prefix": "kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetCommit.98percentile,kafka.network.RequestMetrics.ResponseQueueTimeMs.request.Offsets.95percentile,kafka.network.RequestMetrics.ResponseSendTimeMs.request.Fetch.95percentile,kafka.network.RequestMetrics.RequestsPerSec.request", "fetch.purgatory.purge.interval.requests": "10000", "kafka.ganglia.metrics.group": "kafka", "kafka.ganglia.metrics.port": "8671", "kafka.ganglia.metrics.reporter.enabled": "true", "kafka.metrics.reporters": "org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter", "kafka.timeline.metrics.host": "{{metric_collector_host}}", "kafka.timeline.metrics.maxRowCacheSize": "10000", "kafka.timeline.metrics.port": "{{metric_collector_port}}", "kafka.timeline.metrics.protocol": "{{metric_collector_protocol}}", "kafka.timeline.metrics.reporter.enabled": "true", "kafka.timeline.metrics.reporter.sendInterval": "5900", "kafka.timeline.metrics.truststore.path": "{{metric_truststore_path}}", "kafka.timeline.metrics.truststore.type": "{{metric_truststore_type}}", "leader.imbalance.check.interval.seconds": "300", "leader.imbalance.per.broker.percentage": "10", "listeners": "PLAINTEXT://localhost:6667", "log.cleanup.interval.mins": "10", "log.dirs": "/kafka-logs", "log.index.interval.bytes": "4096", "log.index.size.max.bytes": "10485760", "log.retention.bytes": "-1", "log.retention.hours": "168", "log.roll.hours": "168", "log.segment.bytes": "1073741824", "message.max.bytes": "1000000", "min.insync.replicas": "1", "num.io.threads": "8", "num.network.threads": "3", "num.partitions": "1", "num.recovery.threads.per.data.dir": "1", "num.replica.fetchers": "1", "offset.metadata.max.bytes": "4096", "offsets.commit.required.acks": "-1", "offsets.commit.timeout.ms": "5000", "offsets.load.buffer.size": "5242880", "offsets.retention.check.interval.ms": "600000", "offsets.retention.minutes": "86400000", "offsets.topic.compression.codec": "0", "offsets.topic.num.partitions": "50", "offsets.topic.replication.factor": "3", "offsets.topic.segment.bytes": "104857600", "port": "6667", "producer.purgatory.purge.interval.requests": "10000", "queued.max.requests": "500", "replica.fetch.max.bytes": "1048576", "replica.fetch.min.bytes": "1", "replica.fetch.wait.max.ms": "500", "replica.high.watermark.checkpoint.interval.ms": "5000", "replica.lag.max.messages": "4000", "replica.lag.time.max.ms": "10000", "replica.socket.receive.buffer.bytes": "65536", "replica.socket.timeout.ms": "30000", "socket.receive.buffer.bytes": "102400", "socket.request.max.bytes": "104857600", "socket.send.buffer.bytes": "102400", "zookeeper.connect": "%HOSTGROUP::host_group_1%:2181", "zookeeper.connection.timeout.ms": "25000", "zookeeper.session.timeout.ms": "30000", "zookeeper.sync.time.ms": "2000" }, "kafka-env": { "is_supported_kafka_ranger": "true", "kafka_log_dir": "/var/log/kafka", "kafka_pid_dir": "/var/run/kafka", "kafka_user": "kafka", "kafka_user_nofile_limit": "128000", "kafka_user_nproc_limit": "65536" }, "knox-env": { "knox_group": "knox", "knox_pid_dir": "/var/run/knox", "knox_user": "knox" }, "livy-conf": { "livy.environment": "production", "livy.impersonation.enabled": "true", "livy.server.csrf_protection.enabled": "true", "livy.server.port": "8998", "livy.server.session.timeout": "3600000" }, "livy-env": { "livy_group": "livy", "livy_log_dir": "/var/log/livy", "livy_pid_dir": "/var/run/livy", "livy_user": "livy", "spark_home": "/usr/hdp/current/spark-client" }, "logfeeder-env": { "logfeeder_kerberos_keytab": "/etc/security/keytabs/logfeeder.service.keytab", "logfeeder_kerberos_principal": "logfeeder", "logfeeder_keystore_location": "/etc/security/serverKeys/logsearch.keyStore.jks", "logfeeder_keystore_type": "jks", "logfeeder_log_dir": "/var/log/ambari-logsearch-logfeeder", "logfeeder_max_mem": "512m", "logfeeder_pid_dir": "/var/run/ambari-logsearch-logfeeder", "logfeeder_solr_audit_logs_enable": "true", "logfeeder_solr_service_logs_enable": "true", "logfeeder_truststore_location": "/etc/security/serverKeys/logsearch.trustStore.jks", "logfeeder_truststore_type": "jks" }, "logfeeder-properties": { "logfeeder.checkpoint.folder": "/etc/ambari-logsearch-logfeeder/conf/checkpoints", "logfeeder.config.files": "{default_config_files}", "logfeeder.log.filter.enable": "true", "logfeeder.metrics.collector.hosts": "{metrics_collector_hosts}", "logfeeder.solr.config.interval": "5" }, "logsearch-admin-json": { "logsearch_admin_username": "ambari_logsearch_admin" }, "logsearch-audit_logs-solrconfig": { "logsearch_audit_logs_max_retention": "7", "logsearch_audit_logs_merge_factor": "5" }, "logsearch-env": { "logsearch_app_max_memory": "1024", "logsearch_debug_enabled": "false", "logsearch_debug_port": "5005", "logsearch_kerberos_keytab": "/etc/security/keytabs/logsearch.service.keytab", "logsearch_kerberos_principal": "logsearch", "logsearch_keystore_location": "/etc/security/serverKeys/logsearch.keyStore.jks", "logsearch_keystore_type": "jks", "logsearch_log_dir": "/var/log/ambari-logsearch-portal", "logsearch_pid_dir": "/var/run/ambari-logsearch-portal", "logsearch_solr_audit_logs_zk_node": "{infra_solr_znode}", "logsearch_solr_audit_logs_zk_quorum": "{zookeeper_quorum}", "logsearch_truststore_location": "/etc/security/serverKeys/logsearch.trustStore.jks", "logsearch_truststore_type": "jks", "logsearch_ui_port": "61888", "logsearch_ui_protocol": "http", "logsearch_user": "logsearch" }, "logsearch-properties": { "logsearch.audit.logs.split.interval.mins": "15", "logsearch.collection.audit.logs.numshards": "2", "logsearch.collection.audit.logs.replication.factor": "1", "logsearch.collection.service.logs.numshards": "2", "logsearch.collection.service.logs.replication.factor": "1", "logsearch.external.auth.enabled": "true", "logsearch.external.auth.host_url": "{ambari_server_auth_host_url}", "logsearch.external.auth.login_url": "/api/v1/users/$USERNAME/privileges?fields=*", "logsearch.logfeeder.include.default.level": "FATAL,ERROR,WARN", "logsearch.service.logs.fields": "logtime,level,event_count,ip,type,seq_num,path,file,line_number,host,log_message,id", "logsearch.service.logs.split.interval.mins": "15", "logsearch.solr.collection.audit.logs": "audit_logs", "logsearch.solr.collection.service.logs": "hadoop_logs", "logsearch.solr.metrics.collector.hosts": "{metrics_collector_hosts}" }, "logsearch-service_logs-solrconfig": { "logsearch_service_logs_max_retention": "7", "logsearch_service_logs_merge_factor": "5" }, "mahout-env": { "mahout_user": "mahout" }, "mapred-env": { "jobhistory_heapsize": "900", "mapred_log_dir_prefix": "/var/log/hadoop-mapreduce", "mapred_pid_dir_prefix": "/var/run/hadoop-mapreduce", "mapred_user": "mapred", "mapred_user_nofile_limit": "32768", "mapred_user_nproc_limit": "65536" }, "mapred-site": { "mapreduce.admin.map.child.java.opts": "-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "mapreduce.admin.reduce.child.java.opts": "-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "mapreduce.admin.user.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "mapreduce.am.max-attempts": "2", "mapreduce.application.classpath": "$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure", "mapreduce.application.framework.path": "/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework", "mapreduce.cluster.administrators": "hadoop", "mapreduce.framework.name": "yarn", "mapreduce.job.counters.max": "130", "mapreduce.job.emit-timeline-data": "false", "mapreduce.job.queuename": "default", "mapreduce.job.reduce.slowstart.completedmaps": "0.05", "mapreduce.jobhistory.address": "%HOSTGROUP::host_group_1%:10020", "mapreduce.jobhistory.bind-host": "0.0.0.0", "mapreduce.jobhistory.done-dir": "/mr-history/done", "mapreduce.jobhistory.http.policy": "HTTP_ONLY", "mapreduce.jobhistory.intermediate-done-dir": "/mr-history/tmp", "mapreduce.jobhistory.recovery.enable": "true", "mapreduce.jobhistory.recovery.store.class": "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService", "mapreduce.jobhistory.recovery.store.leveldb.path": "/hadoop/mapreduce/jhs", "mapreduce.jobhistory.webapp.address": "%HOSTGROUP::host_group_1%:19888", "mapreduce.map.java.opts": "-Xmx1228m", "mapreduce.map.log.level": "INFO", "mapreduce.map.memory.mb": "1536", "mapreduce.map.output.compress": "false", "mapreduce.map.sort.spill.percent": "0.7", "mapreduce.map.speculative": "false", "mapreduce.output.fileoutputformat.compress": "false", "mapreduce.output.fileoutputformat.compress.type": "BLOCK", "mapreduce.reduce.input.buffer.percent": "0.0", "mapreduce.reduce.java.opts": "-Xmx1638m", "mapreduce.reduce.log.level": "INFO", "mapreduce.reduce.memory.mb": "2048", "mapreduce.reduce.shuffle.fetch.retry.enabled": "1", "mapreduce.reduce.shuffle.fetch.retry.interval-ms": "1000", "mapreduce.reduce.shuffle.fetch.retry.timeout-ms": "30000", "mapreduce.reduce.shuffle.input.buffer.percent": "0.7", "mapreduce.reduce.shuffle.merge.percent": "0.66", "mapreduce.reduce.shuffle.parallelcopies": "30", "mapreduce.reduce.speculative": "false", "mapreduce.shuffle.port": "13562", "mapreduce.task.io.sort.factor": "100", "mapreduce.task.io.sort.mb": "859", "mapreduce.task.timeout": "300000", "yarn.app.mapreduce.am.admin-command-opts": "-Dhdp.version=${hdp.version}", "yarn.app.mapreduce.am.command-opts": "-Xmx819m -Dhdp.version=${hdp.version}", "yarn.app.mapreduce.am.log.level": "INFO", "yarn.app.mapreduce.am.resource.mb": "1024", "yarn.app.mapreduce.am.staging-dir": "/user" }, "oozie-env": { "oozie_admin_port": "11001", "oozie_admin_users": "{oozie_user}, oozie-admin,falcon", "oozie_data_dir": "/hadoop/oozie/data", "oozie_database": "New Derby Database", "oozie_heapsize": "2048m", "oozie_log_dir": "/var/log/oozie", "oozie_permsize": "256m", "oozie_pid_dir": "/var/run/oozie", "oozie_tmp_dir": "/var/tmp/oozie", "oozie_user": "oozie", "oozie_user_nofile_limit": "32000", "oozie_user_nproc_limit": "16000" }, "oozie-site": { "oozie.action.retry.interval": "30", "oozie.authentication.simple.anonymous.allowed": "true", "oozie.authentication.type": "simple", "oozie.base.url": "http://%HOSTGROUP::host_group_1%:11000/oozie", "oozie.credentials.credentialclasses": "hcat=org.apache.oozie.action.hadoop.HCatCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials", "oozie.db.schema.name": "oozie", "oozie.service.AuthorizationService.security.enabled": "true", "oozie.service.ELService.ext.functions.coord-action-create": "now=org.apache.oozie.extensions.OozieELExtensions#ph2_now, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_currentWeek, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_lastWeek, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-action-create-inst": "now=org.apache.oozie.extensions.OozieELExtensions#ph2_now_inst, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today_inst, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday_inst, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_currentWeek_inst, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_lastWeek_inst, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth_inst, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth_inst, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear_inst, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear_inst, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-action-start": "now=org.apache.oozie.extensions.OozieELExtensions#ph2_now, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_currentWeek, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_lastWeek, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear, latest=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latest, future=org.apache.oozie.coord.CoordELFunctions#ph3_coord_future, dataIn=org.apache.oozie.extensions.OozieELExtensions#ph3_dataIn, instanceTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime, dateOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateOffset, formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-job-submit-data": "now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo, today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_currentWeek_echo, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_lastWeek_echo, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo, dataIn=org.apache.oozie.extensions.OozieELExtensions#ph1_dataIn_echo, instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_wrap, formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-job-submit-instances": "now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo, today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_currentWeek_echo, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_lastWeek_echo, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo", "oozie.service.ELService.ext.functions.coord-sla-create": "instanceTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_nominalTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-sla-submit": "instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.workflow": "now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo, today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo", "oozie.service.HadoopAccessorService.hadoop.configurations": "*={{hadoop_conf_dir}}", "oozie.service.HadoopAccessorService.kerberos.enabled": "false", "oozie.service.HadoopAccessorService.supported.filesystems": "*", "oozie.service.JPAService.jdbc.driver": "org.apache.derby.jdbc.EmbeddedDriver", "oozie.service.JPAService.jdbc.username": "oozie", "oozie.service.ProxyUserService.proxyuser.falcon.groups": "*", "oozie.service.ProxyUserService.proxyuser.falcon.hosts": "*", "oozie.service.SparkConfigurationService.spark.configurations": "*=spark-conf", "oozie.service.URIHandlerService.uri.handlers": "org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler", "oozie.services.ext": "org.apache.oozie.service.JMSAccessorService,org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService" }, "ranger-hbase-audit": { "xasecure.audit.destination.hdfs": "true", "xasecure.audit.destination.hdfs.batch.filespool.dir": "/var/log/hbase/audit/hdfs/spool", "xasecure.audit.destination.hdfs.dir": "hdfs://NAMENODE_HOSTNAME:8020/ranger/audit", "xasecure.audit.destination.solr": "false", "xasecure.audit.destination.solr.batch.filespool.dir": "/var/log/hbase/audit/solr/spool", "xasecure.audit.destination.solr.urls": "", "xasecure.audit.destination.solr.zookeepers": "NONE", "xasecure.audit.is.enabled": "true", "xasecure.audit.provider.summary.enabled": "true" }, "ranger-hbase-plugin-properties": { "REPOSITORY_CONFIG_USERNAME": "hbase", "common.name.for.certificate": "", "policy_user": "ambari-qa", "ranger-hbase-plugin-enabled": "No" }, "ranger-hbase-policymgr-ssl": { "xasecure.policymgr.clientssl.keystore": "/usr/hdp/current/hbase-client/conf/ranger-plugin-keystore.jks", "xasecure.policymgr.clientssl.keystore.credential.file": "jceks://file{{credential_file}}", "xasecure.policymgr.clientssl.truststore": "/usr/hdp/current/hbase-client/conf/ranger-plugin-truststore.jks", "xasecure.policymgr.clientssl.truststore.credential.file": "jceks://file{{credential_file}}" }, "ranger-hbase-security": { "ranger.plugin.hbase.policy.cache.dir": "/etc/ranger/{{repo_name}}/policycache", "ranger.plugin.hbase.policy.pollIntervalMs": "30000", "ranger.plugin.hbase.policy.rest.ssl.config.file": "/etc/hbase/conf/ranger-policymgr-ssl.xml", "ranger.plugin.hbase.policy.rest.url": "{{policymgr_mgr_url}}", "ranger.plugin.hbase.policy.source.impl": "org.apache.ranger.admin.client.RangerAdminRESTClient", "ranger.plugin.hbase.service.name": "{{repo_name}}", "xasecure.hbase.update.xapolicies.on.grant.revoke": "true" }, "ranger-hdfs-audit": { "xasecure.audit.destination.hdfs": "true", "xasecure.audit.destination.hdfs.batch.filespool.dir": "/var/log/hadoop/hdfs/audit/hdfs/spool", "xasecure.audit.destination.hdfs.dir": "hdfs://NAMENODE_HOSTNAME:8020/ranger/audit", "xasecure.audit.destination.solr": "false", "xasecure.audit.destination.solr.batch.filespool.dir": "/var/log/hadoop/hdfs/audit/solr/spool", "xasecure.audit.destination.solr.urls": "", "xasecure.audit.destination.solr.zookeepers": "NONE", "xasecure.audit.is.enabled": "true", "xasecure.audit.provider.summary.enabled": "false" }, "ranger-hdfs-plugin-properties": { "REPOSITORY_CONFIG_USERNAME": "hadoop", "common.name.for.certificate": "", "hadoop.rpc.protection": "authentication", "policy_user": "ambari-qa", "ranger-hdfs-plugin-enabled": "No" }, "ranger-hdfs-policymgr-ssl": { "xasecure.policymgr.clientssl.keystore": "/usr/hdp/current/hadoop-client/conf/ranger-plugin-keystore.jks", "xasecure.policymgr.clientssl.keystore.credential.file": "jceks://file{{credential_file}}", "xasecure.policymgr.clientssl.truststore": "/usr/hdp/current/hadoop-client/conf/ranger-plugin-truststore.jks", "xasecure.policymgr.clientssl.truststore.credential.file": "jceks://file{{credential_file}}" }, "ranger-hdfs-security": { "ranger.plugin.hdfs.policy.cache.dir": "/etc/ranger/{{repo_name}}/policycache", "ranger.plugin.hdfs.policy.pollIntervalMs": "30000", "ranger.plugin.hdfs.policy.rest.ssl.config.file": "/etc/hadoop/conf/ranger-policymgr-ssl.xml", "ranger.plugin.hdfs.policy.rest.url": "{{policymgr_mgr_url}}", "ranger.plugin.hdfs.policy.source.impl": "org.apache.ranger.admin.client.RangerAdminRESTClient", "ranger.plugin.hdfs.service.name": "{{repo_name}}", "xasecure.add-hadoop-authorization": "true" }, "ranger-hive-audit": { "xasecure.audit.destination.hdfs": "true", "xasecure.audit.destination.hdfs.batch.filespool.dir": "/var/log/hive/audit/hdfs/spool", "xasecure.audit.destination.hdfs.dir": "hdfs://NAMENODE_HOSTNAME:8020/ranger/audit", "xasecure.audit.destination.solr": "false", "xasecure.audit.destination.solr.batch.filespool.dir": "/var/log/hive/audit/solr/spool", "xasecure.audit.destination.solr.urls": "", "xasecure.audit.destination.solr.zookeepers": "NONE", "xasecure.audit.is.enabled": "true", "xasecure.audit.provider.summary.enabled": "false" }, "ranger-hive-plugin-properties": { "REPOSITORY_CONFIG_USERNAME": "hive", "common.name.for.certificate": "", "jdbc.driverClassName": "org.apache.hive.jdbc.HiveDriver", "policy_user": "ambari-qa" }, "ranger-hive-policymgr-ssl": { "xasecure.policymgr.clientssl.keystore": "/usr/hdp/current/hive-server2/conf/ranger-plugin-keystore.jks", "xasecure.policymgr.clientssl.keystore.credential.file": "jceks://file{{credential_file}}", "xasecure.policymgr.clientssl.truststore": "/usr/hdp/current/hive-server2/conf/ranger-plugin-truststore.jks", "xasecure.policymgr.clientssl.truststore.credential.file": "jceks://file{{credential_file}}" }, "ranger-hive-security": { "ranger.plugin.hive.policy.cache.dir": "/etc/ranger/{{repo_name}}/policycache", "ranger.plugin.hive.policy.pollIntervalMs": "30000", "ranger.plugin.hive.policy.rest.ssl.config.file": "/usr/hdp/current/{{ranger_hive_component}}/conf/conf.server/ranger-policymgr-ssl.xml", "ranger.plugin.hive.policy.rest.url": "{{policymgr_mgr_url}}", "ranger.plugin.hive.policy.source.impl": "org.apache.ranger.admin.client.RangerAdminRESTClient", "ranger.plugin.hive.service.name": "{{repo_name}}", "xasecure.hive.update.xapolicies.on.grant.revoke": "true" }, "ranger-kafka-audit": { "xasecure.audit.destination.hdfs": "true", "xasecure.audit.destination.hdfs.batch.filespool.dir": "/var/log/kafka/audit/hdfs/spool", "xasecure.audit.destination.hdfs.dir": "hdfs://NAMENODE_HOSTNAME:8020/ranger/audit", "xasecure.audit.destination.solr": "false", "xasecure.audit.destination.solr.batch.filespool.dir": "/var/log/kafka/audit/solr/spool", "xasecure.audit.destination.solr.urls": "", "xasecure.audit.destination.solr.zookeepers": "NONE", "xasecure.audit.is.enabled": "true", "xasecure.audit.provider.summary.enabled": "true" }, "ranger-kafka-plugin-properties": { "REPOSITORY_CONFIG_USERNAME": "kafka", "common.name.for.certificate": "", "hadoop.rpc.protection": "", "policy_user": "ambari-qa", "ranger-kafka-plugin-enabled": "No", "zookeeper.connect": "localhost:2181" }, "ranger-kafka-policymgr-ssl": { "xasecure.policymgr.clientssl.keystore": "/usr/hdp/current/kafka-broker/config/ranger-plugin-keystore.jks", "xasecure.policymgr.clientssl.keystore.credential.file": "jceks://file/{{credential_file}}", "xasecure.policymgr.clientssl.truststore": "/usr/hdp/current/kafka-broker/config/ranger-plugin-truststore.jks", "xasecure.policymgr.clientssl.truststore.credential.file": "jceks://file/{{credential_file}}" }, "ranger-kafka-security": { "ranger.plugin.kafka.policy.cache.dir": "/etc/ranger/{{repo_name}}/policycache", "ranger.plugin.kafka.policy.pollIntervalMs": "30000", "ranger.plugin.kafka.policy.rest.ssl.config.file": "/etc/kafka/conf/ranger-policymgr-ssl.xml", "ranger.plugin.kafka.policy.rest.url": "{{policymgr_mgr_url}}", "ranger.plugin.kafka.policy.source.impl": "org.apache.ranger.admin.client.RangerAdminRESTClient", "ranger.plugin.kafka.service.name": "{{repo_name}}" }, "ranger-knox-audit": { "xasecure.audit.destination.hdfs": "true", "xasecure.audit.destination.hdfs.batch.filespool.dir": "/var/log/knox/audit/hdfs/spool", "xasecure.audit.destination.hdfs.dir": "hdfs://NAMENODE_HOSTNAME:8020/ranger/audit", "xasecure.audit.destination.solr": "false", "xasecure.audit.destination.solr.batch.filespool.dir": "/var/log/knox/audit/solr/spool", "xasecure.audit.destination.solr.urls": "", "xasecure.audit.destination.solr.zookeepers": "NONE", "xasecure.audit.is.enabled": "true", "xasecure.audit.provider.summary.enabled": "false" }, "ranger-knox-plugin-properties": { "KNOX_HOME": "/usr/hdp/current/knox-server", "REPOSITORY_CONFIG_USERNAME": "admin", "common.name.for.certificate": "", "policy_user": "ambari-qa", "ranger-knox-plugin-enabled": "No" }, "ranger-knox-policymgr-ssl": { "xasecure.policymgr.clientssl.keystore": "/usr/hdp/current/knox-server/conf/ranger-plugin-keystore.jks", "xasecure.policymgr.clientssl.keystore.credential.file": "jceks://file{{credential_file}}", "xasecure.policymgr.clientssl.truststore": "/usr/hdp/current/knox-server/conf/ranger-plugin-truststore.jks", "xasecure.policymgr.clientssl.truststore.credential.file": "jceks://file{{credential_file}}" }, "ranger-knox-security": { "ranger.plugin.knox.policy.cache.dir": "/etc/ranger/{{repo_name}}/policycache", "ranger.plugin.knox.policy.pollIntervalMs": "30000", "ranger.plugin.knox.policy.rest.ssl.config.file": "/usr/hdp/current/knox-server/conf/ranger-policymgr-ssl.xml", "ranger.plugin.knox.policy.rest.url": "{{policymgr_mgr_url}}", "ranger.plugin.knox.policy.source.impl": "org.apache.ranger.admin.client.RangerAdminJersey2RESTClient", "ranger.plugin.knox.service.name": "{{repo_name}}" }, "ranger-yarn-audit": { "xasecure.audit.destination.hdfs": "true", "xasecure.audit.destination.hdfs.batch.filespool.dir": "/var/log/hadoop/yarn/audit/hdfs/spool", "xasecure.audit.destination.hdfs.dir": "hdfs://NAMENODE_HOSTNAME:8020/ranger/audit", "xasecure.audit.destination.solr": "false", "xasecure.audit.destination.solr.batch.filespool.dir": "/var/log/hadoop/yarn/audit/solr/spool", "xasecure.audit.destination.solr.urls": "", "xasecure.audit.destination.solr.zookeepers": "NONE", "xasecure.audit.is.enabled": "true", "xasecure.audit.provider.summary.enabled": "false" }, "ranger-yarn-plugin-properties": { "REPOSITORY_CONFIG_USERNAME": "yarn", "common.name.for.certificate": "", "hadoop.rpc.protection": "", "policy_user": "ambari-qa", "ranger-yarn-plugin-enabled": "No" }, "ranger-yarn-policymgr-ssl": { "xasecure.policymgr.clientssl.keystore": "/usr/hdp/current/hadoop-client/conf/ranger-yarn-plugin-keystore.jks", "xasecure.policymgr.clientssl.keystore.credential.file": "jceks://file{{credential_file}}", "xasecure.policymgr.clientssl.truststore": "/usr/hdp/current/hadoop-client/conf/ranger-yarn-plugin-truststore.jks", "xasecure.policymgr.clientssl.truststore.credential.file": "jceks://file{{credential_file}}" }, "ranger-yarn-security": { "ranger.plugin.yarn.policy.cache.dir": "/etc/ranger/{{repo_name}}/policycache", "ranger.plugin.yarn.policy.pollIntervalMs": "30000", "ranger.plugin.yarn.policy.rest.ssl.config.file": "/etc/hadoop/conf/ranger-policymgr-ssl-yarn.xml", "ranger.plugin.yarn.policy.rest.url": "{{policymgr_mgr_url}}", "ranger.plugin.yarn.policy.source.impl": "org.apache.ranger.admin.client.RangerAdminRESTClient", "ranger.plugin.yarn.service.name": "{{repo_name}}" }, "spark-defaults": { "spark.driver.extraLibraryPath": "{{spark_hadoop_lib_native}}", "spark.eventLog.dir": "hdfs:///spark-history", "spark.eventLog.enabled": "true", "spark.executor.extraLibraryPath": "{{spark_hadoop_lib_native}}", "spark.history.fs.logDirectory": "hdfs:///spark-history", "spark.history.kerberos.keytab": "none", "spark.history.kerberos.principal": "none", "spark.history.provider": "org.apache.spark.deploy.history.FsHistoryProvider", "spark.history.ui.port": "18080", "spark.yarn.containerLauncherMaxThreads": "25", "spark.yarn.driver.memoryOverhead": "384", "spark.yarn.executor.memoryOverhead": "384", "spark.yarn.historyServer.address": "{{spark_history_server_host}}:{{spark_history_ui_port}}", "spark.yarn.preserve.staging.files": "false", "spark.yarn.queue": "default", "spark.yarn.scheduler.heartbeat.interval-ms": "5000", "spark.yarn.submit.file.replication": "3" }, "spark-env": { "hive_kerberos_keytab": "{{hive_kerberos_keytab}}", "hive_kerberos_principal": "{{hive_kerberos_principal}}", "spark_daemon_memory": "1024", "spark_group": "spark", "spark_log_dir": "/var/log/spark", "spark_pid_dir": "/var/run/spark", "spark_thrift_cmd_opts": "", "spark_user": "spark" }, "spark-hive-site-override": { "hive.metastore.client.connect.retry.delay": "5", "hive.metastore.client.socket.timeout": "1800", "hive.server2.enable.doAs": "false", "hive.server2.thrift.port": "10015", "hive.server2.transport.mode": "binary" }, "spark-thrift-fairscheduler": { "fairscheduler_content": " FAIR 1 2 " }, "spark-thrift-sparkconf": { "spark.driver.extraLibraryPath": "{{spark_hadoop_lib_native}}", "spark.dynamicAllocation.enabled": "true", "spark.dynamicAllocation.initialExecutors": "0", "spark.dynamicAllocation.maxExecutors": "10", "spark.dynamicAllocation.minExecutors": "0", "spark.eventLog.dir": "{{spark_history_dir}}", "spark.eventLog.enabled": "true", "spark.executor.extraLibraryPath": "{{spark_hadoop_lib_native}}", "spark.executor.memory": "1g", "spark.hadoop.cacheConf": "false", "spark.history.fs.logDirectory": "{{spark_history_dir}}", "spark.history.provider": "org.apache.spark.deploy.history.FsHistoryProvider", "spark.master": "{{spark_thrift_master}}", "spark.scheduler.allocation.file": "{{spark_conf}}/spark-thrift-fairscheduler.xml", "spark.scheduler.mode": "FAIR", "spark.shuffle.service.enabled": "true", "spark.yarn.am.memory": "512m", "spark.yarn.queue": "default" }, "spark2-defaults": { "spark.driver.extraLibraryPath": "{{spark_hadoop_lib_native}}", "spark.eventLog.dir": "hdfs:///spark2-history/", "spark.eventLog.enabled": "true", "spark.executor.extraLibraryPath": "{{spark_hadoop_lib_native}}", "spark.history.fs.logDirectory": "hdfs:///spark2-history/", "spark.history.kerberos.keytab": "none", "spark.history.kerberos.principal": "none", "spark.history.provider": "org.apache.spark.deploy.history.FsHistoryProvider", "spark.history.ui.port": "18081", "spark.yarn.historyServer.address": "{{spark_history_server_host}}:{{spark_history_ui_port}}", "spark.yarn.queue": "default" }, "spark2-env": { "hive_kerberos_keytab": "{{hive_kerberos_keytab}}", "hive_kerberos_principal": "{{hive_kerberos_principal}}", "spark_daemon_memory": "1024", "spark_group": "spark", "spark_log_dir": "/var/log/spark2", "spark_pid_dir": "/var/run/spark2", "spark_thrift_cmd_opts": "", "spark_user": "spark" }, "spark2-hive-site-override": { "hive.metastore.client.connect.retry.delay": "5", "hive.metastore.client.socket.timeout": "1800", "hive.server2.enable.doAs": "false", "hive.server2.thrift.port": "10016", "hive.server2.transport.mode": "binary" }, "spark2-thrift-fairscheduler": { "fairscheduler_content": " FAIR 1 2 " }, "spark2-thrift-sparkconf": { "spark.driver.extraLibraryPath": "{{spark_hadoop_lib_native}}", "spark.dynamicAllocation.enabled": "true", "spark.dynamicAllocation.initialExecutors": "0", "spark.dynamicAllocation.maxExecutors": "10", "spark.dynamicAllocation.minExecutors": "0", "spark.eventLog.dir": "{{spark_history_dir}}", "spark.eventLog.enabled": "true", "spark.executor.extraLibraryPath": "{{spark_hadoop_lib_native}}", "spark.hadoop.cacheConf": "false", "spark.history.fs.logDirectory": "{{spark_history_dir}}", "spark.history.provider": "org.apache.spark.deploy.history.FsHistoryProvider", "spark.master": "{{spark_thrift_master}}", "spark.scheduler.allocation.file": "{{spark_conf}}/spark-thrift-fairscheduler.xml", "spark.scheduler.mode": "FAIR", "spark.shuffle.service.enabled": "true", "spark.yarn.queue": "default" }, "sqoop-atlas-application.properties": { "atlas.jaas.KafkaClient.option.renewTicket": "true", "atlas.jaas.KafkaClient.option.useTicketCache": "true" }, "sqoop-env": { "jdbc_drivers": "", "sqoop_user": "sqoop" }, "ssl-client": { "ssl.client.keystore.location": "/etc/security/clientKeys/keystore.jks", "ssl.client.keystore.type": "jks", "ssl.client.truststore.location": "/etc/security/clientKeys/all.jks", "ssl.client.truststore.reload.interval": "10000", "ssl.client.truststore.type": "jks" }, "ssl-server": { "ssl.server.keystore.location": "/etc/security/serverKeys/keystore.jks", "ssl.server.keystore.type": "jks", "ssl.server.truststore.location": "/etc/security/serverKeys/all.jks", "ssl.server.truststore.reload.interval": "10000", "ssl.server.truststore.type": "jks" }, "storm-atlas-application.properties": { "atlas.hook.storm.numRetries": "3" }, "storm-env": { "jmxremote_port": "56431", "nimbus_seeds_supported": "true", "storm_log_dir": "/var/log/storm", "storm_logs_supported": "true", "storm_pid_dir": "/var/run/storm", "storm_user": "storm", "storm_user_nofile_limit": "128000", "storm_user_nproc_limit": "65536" }, "storm-site": { "_storm.min.ruid": "null", "_storm.thrift.nonsecure.transport": "org.apache.storm.security.auth.SimpleTransportPlugin", "_storm.thrift.secure.transport": "org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin", "client.jartransformer.class": "org.apache.storm.hack.StormShadeTransformer", "dev.zookeeper.path": "/tmp/dev-storm-zookeeper", "drpc.childopts": "-Xmx768m _JAAS_PLACEHOLDER", "drpc.invocations.port": "3773", "drpc.port": "3772", "drpc.queue.size": "128", "drpc.request.timeout.secs": "600", "drpc.worker.threads": "64", "java.library.path": "/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib", "logviewer.appender.name": "A1", "logviewer.childopts": "-Xmx128m _JAAS_PLACEHOLDER", "logviewer.port": "8000", "metrics.reporter.register": "org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter", "nimbus.cleanup.inbox.freq.secs": "600", "nimbus.file.copy.expiration.secs": "600", "nimbus.inbox.jar.expiration.secs": "3600", "nimbus.monitor.freq.secs": "10", "nimbus.reassign": "true", "nimbus.seeds": "['%HOSTGROUP::host_group_2%']", "nimbus.supervisor.timeout.secs": "60", "nimbus.task.launch.secs": "120", "nimbus.task.timeout.secs": "30", "nimbus.thrift.max_buffer_size": "1048576", "nimbus.thrift.port": "6627", "nimbus.thrift.threads": "196", "nimbus.topology.validator": "org.apache.storm.nimbus.DefaultTopologyValidator", "storm.cluster.mode": "distributed", "storm.local.dir": "/hadoop/storm", "storm.local.mode.zmq": "false", "storm.log.dir": "{{log_dir}}", "storm.messaging.netty.buffer_size": "5242880", "storm.messaging.netty.client_worker_threads": "1", "storm.messaging.netty.max_retries": "30", "storm.messaging.netty.max_wait_ms": "1000", "storm.messaging.netty.min_wait_ms": "100", "storm.messaging.netty.server_worker_threads": "1", "storm.messaging.transport": "org.apache.storm.messaging.netty.Context", "storm.thrift.transport": "{{storm_thrift_transport}}", "storm.topology.submission.notifier.plugin.class": "org.apache.atlas.storm.hook.StormAtlasHook", "storm.zookeeper.connection.timeout": "30000", "storm.zookeeper.port": "2181", "storm.zookeeper.retry.interval": "1000", "storm.zookeeper.retry.intervalceiling.millis": "30000", "storm.zookeeper.retry.times": "5", "storm.zookeeper.root": "/storm", "storm.zookeeper.servers": "['%HOSTGROUP::host_group_1%']", "storm.zookeeper.session.timeout": "30000", "supervisor.heartbeat.frequency.secs": "5", "supervisor.monitor.frequency.secs": "3", "supervisor.slots.ports": "[6700, 6701]", "supervisor.worker.start.timeout.secs": "120", "supervisor.worker.timeout.secs": "30", "task.heartbeat.frequency.secs": "3", "task.refresh.poll.secs": "10", "topology.acker.executors": "null", "topology.builtin.metrics.bucket.size.secs": "60", "topology.debug": "false", "topology.disruptor.wait.strategy": "com.lmax.disruptor.BlockingWaitStrategy", "topology.enable.message.timeouts": "true", "topology.error.throttle.interval.secs": "10", "topology.executor.receive.buffer.size": "1024", "topology.executor.send.buffer.size": "1024", "topology.fall.back.on.java.serialization": "true", "topology.kryo.factory": "org.apache.storm.serialization.DefaultKryoFactory", "topology.max.error.report.per.interval": "5", "topology.max.replication.wait.time.sec": "{{actual_topology_max_replication_wait_time_sec}}", "topology.max.replication.wait.time.sec.default": "60", "topology.max.spout.pending": "1000", "topology.max.task.parallelism": "null", "topology.message.timeout.secs": "30", "topology.min.replication.count": "{{actual_topology_min_replication_count}}", "topology.min.replication.count.default": "1", "topology.optimize": "true", "topology.receiver.buffer.size": "8", "topology.skip.missing.kryo.registrations": "false", "topology.sleep.spout.wait.strategy.time.ms": "1", "topology.spout.wait.strategy": "org.apache.storm.spout.SleepSpoutWaitStrategy", "topology.state.synchronization.timeout.secs": "60", "topology.stats.sample.rate": "0.05", "topology.tick.tuple.freq.secs": "null", "topology.transfer.buffer.size": "1024", "topology.trident.batch.emit.interval.millis": "500", "topology.tuple.serializer": "org.apache.storm.serialization.types.ListDelegateSerializer", "topology.worker.childopts": "null", "topology.worker.shared.thread.pool.size": "4", "topology.workers": "1", "transactional.zookeeper.port": "null", "transactional.zookeeper.root": "/transactional", "transactional.zookeeper.servers": "null", "ui.childopts": "-Xmx768m _JAAS_PLACEHOLDER", "ui.filter": "null", "ui.port": "8744", "worker.heartbeat.frequency.secs": "1", "zmq.hwm": "0", "zmq.linger.millis": "5000", "zmq.threads": "1" }, "tez-env": { "tez_user": "tez" }, "tez-interactive-site": { "tez.am.resource.memory.mb": "1536", "tez.dag.recovery.enabled": "false", "tez.grouping.node.local.only": "true", "tez.lib.uris": "/hdp/apps/${hdp.version}/tez_hive2/tez.tar.gz", "tez.runtime.pipelined-shuffle.enabled": "false", "tez.runtime.pipelined.sorter.lazy-allocate.memory": "true", "tez.runtime.report.partition.stats": "true", "tez.runtime.shuffle.fetch.buffer.percent": "0.6", "tez.runtime.shuffle.fetch.verify-disk-checksum": "false", "tez.runtime.shuffle.memory.limit.percent": "0.25", "tez.session.am.dag.submit.timeout.secs": "3600" }, "tez-site": { "tez.am.am-rm.heartbeat.interval-ms.max": "250", "tez.am.container.idle.release-timeout-max.millis": "20000", "tez.am.container.idle.release-timeout-min.millis": "10000", "tez.am.container.reuse.enabled": "true", "tez.am.container.reuse.locality.delay-allocation-millis": "250", "tez.am.container.reuse.non-local-fallback.enabled": "false", "tez.am.container.reuse.rack-fallback.enabled": "true", "tez.am.launch.cluster-default.cmd-opts": "-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "tez.am.launch.cmd-opts": "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC", "tez.am.launch.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "tez.am.log.level": "INFO", "tez.am.max.app.attempts": "2", "tez.am.maxtaskfailures.per.node": "10", "tez.am.resource.memory.mb": "2048", "tez.am.tez-ui.history-url.template": "__HISTORY_URL_BASE__?viewPath=%2F%23%2Ftez-app%2F__APPLICATION_ID__", "tez.am.view-acls": "*", "tez.cluster.additional.classpath.prefix": "/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure", "tez.counters.max": "10000", "tez.counters.max.groups": "3000", "tez.generate.debug.artifacts": "false", "tez.grouping.max-size": "1073741824", "tez.grouping.min-size": "16777216", "tez.grouping.split-waves": "1.7", "tez.history.logging.service.class": "org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService", "tez.lib.uris": "/hdp/apps/${hdp.version}/tez/tez.tar.gz", "tez.queue.name": "default", "tez.runtime.compress": "true", "tez.runtime.compress.codec": "org.apache.hadoop.io.compress.SnappyCodec", "tez.runtime.convert.user-payload.to.history-text": "false", "tez.runtime.io.sort.mb": "270", "tez.runtime.optimize.local.fetch": "true", "tez.runtime.pipelined.sorter.sort.threads": "2", "tez.runtime.shuffle.fetch.buffer.percent": "0.6", "tez.runtime.shuffle.memory.limit.percent": "0.25", "tez.runtime.sorter.class": "PIPELINED", "tez.runtime.unordered.output.buffer.size-mb": "76", "tez.session.am.dag.submit.timeout.secs": "600", "tez.session.client.timeout.secs": "-1", "tez.shuffle-vertex-manager.max-src-fraction": "0.4", "tez.shuffle-vertex-manager.min-src-fraction": "0.2", "tez.staging-dir": "/tmp/${user.name}/staging", "tez.task.am.heartbeat.counter.interval-ms.max": "4000", "tez.task.generate.counters.per.io": "true", "tez.task.get-task.sleep.interval-ms.max": "200", "tez.task.launch.cluster-default.cmd-opts": "-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "tez.task.launch.cmd-opts": "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC", "tez.task.launch.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "tez.task.max-events-per-heartbeat": "500", "tez.task.resource.memory.mb": "1024", "tez.use.cluster.hadoop-libs": "false" }, "webhcat-site": { "templeton.exec.timeout": "60000", "templeton.hadoop": "/usr/hdp/${hdp.version}/hadoop/bin/hadoop", "templeton.hadoop.conf.dir": "/etc/hadoop/conf", "templeton.hadoop.queue.name": "default", "templeton.hcat": "/usr/hdp/${hdp.version}/hive/bin/hcat", "templeton.hcat.home": "hive.tar.gz/hive/hcatalog", "templeton.hive.archive": "hdfs:///hdp/apps/${hdp.version}/hive/hive.tar.gz", "templeton.hive.extra.files": "/usr/hdp/${hdp.version}/tez/conf/tez-site.xml,/usr/hdp/${hdp.version}/tez,/usr/hdp/${hdp.version}/tez/lib", "templeton.hive.home": "hive.tar.gz/hive", "templeton.hive.path": "hive.tar.gz/hive/bin/hive", "templeton.hive.properties": "hive.metastore.local=false,hive.metastore.uris=thrift://%HOSTGROUP::host_group_1%:9083,hive.metastore.sasl.enabled=false", "templeton.jar": "/usr/hdp/${hdp.version}/hive/share/webhcat/svr/lib/hive-webhcat-*.jar", "templeton.libjars": "/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar", "templeton.override.enabled": "false", "templeton.pig.archive": "hdfs:///hdp/apps/${hdp.version}/pig/pig.tar.gz", "templeton.pig.path": "pig.tar.gz/pig/bin/pig", "templeton.port": "50111", "templeton.python": "${env.PYTHON_CMD}", "templeton.sqoop.archive": "hdfs:///hdp/apps/${hdp.version}/sqoop/sqoop.tar.gz", "templeton.sqoop.home": "sqoop.tar.gz/sqoop", "templeton.sqoop.path": "sqoop.tar.gz/sqoop/bin/sqoop", "templeton.storage.class": "org.apache.hive.hcatalog.templeton.tool.ZooKeeperStorage", "templeton.streaming.jar": "hdfs:///hdp/apps/${hdp.version}/mapreduce/hadoop-streaming.jar", "templeton.zookeeper.hosts": "%HOSTGROUP::host_group_1%:2181", "webhcat.proxyuser.root.groups": "*", "webhcat.proxyuser.root.hosts": "vgt-imaster-0.novalocal" }, "yarn-env": { "apptimelineserver_heapsize": "1024", "is_supported_yarn_ranger": "true", "min_user_id": "1000", "nodemanager_heapsize": "1024", "resourcemanager_heapsize": "1024", "service_check.queue.name": "default", "yarn_cgroups_enabled": "false", "yarn_heapsize": "1024", "yarn_log_dir_prefix": "/var/log/hadoop-yarn", "yarn_pid_dir_prefix": "/var/run/hadoop-yarn", "yarn_user": "yarn", "yarn_user_nofile_limit": "32768", "yarn_user_nproc_limit": "65536" }, "yarn-site": { "hadoop.registry.rm.enabled": "true", "hadoop.registry.zk.quorum": "%HOSTGROUP::host_group_1%:2181", "yarn.acl.enable": "false", "yarn.admin.acl": "yarn", "yarn.application.classpath": "$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*", "yarn.client.nodemanager-connect.max-wait-ms": "60000", "yarn.client.nodemanager-connect.retry-interval-ms": "10000", "yarn.http.policy": "HTTP_ONLY", "yarn.log-aggregation-enable": "true", "yarn.log-aggregation.retain-seconds": "2592000", "yarn.log.server.url": "http://%HOSTGROUP::host_group_1%:19888/jobhistory/logs", "yarn.node-labels.enabled": "false", "yarn.node-labels.fs-store.retry-policy-spec": "2000, 500", "yarn.node-labels.fs-store.root-dir": "/system/yarn/node-labels", "yarn.nodemanager.address": "0.0.0.0:45454", "yarn.nodemanager.admin-env": "MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX", "yarn.nodemanager.aux-services": "mapreduce_shuffle,spark_shuffle,spark2_shuffle", "yarn.nodemanager.aux-services.mapreduce_shuffle.class": "org.apache.hadoop.mapred.ShuffleHandler", "yarn.nodemanager.aux-services.spark2_shuffle.class": "org.apache.spark.network.yarn.YarnShuffleService", "yarn.nodemanager.aux-services.spark2_shuffle.classpath": "{{stack_root}}/${hdp.version}/spark2/aux/*", "yarn.nodemanager.aux-services.spark_shuffle.class": "org.apache.spark.network.yarn.YarnShuffleService", "yarn.nodemanager.aux-services.spark_shuffle.classpath": "{{stack_root}}/${hdp.version}/spark/aux/*", "yarn.nodemanager.bind-host": "0.0.0.0", "yarn.nodemanager.container-executor.class": "org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor", "yarn.nodemanager.container-metrics.unregister-delay-ms": "60000", "yarn.nodemanager.container-monitor.interval-ms": "3000", "yarn.nodemanager.delete.debug-delay-sec": "0", "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage": "90", "yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb": "1000", "yarn.nodemanager.disk-health-checker.min-healthy-disks": "0.25", "yarn.nodemanager.health-checker.interval-ms": "135000", "yarn.nodemanager.health-checker.script.timeout-ms": "60000", "yarn.nodemanager.linux-container-executor.cgroups.hierarchy": "hadoop-yarn", "yarn.nodemanager.linux-container-executor.cgroups.mount": "false", "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage": "false", "yarn.nodemanager.linux-container-executor.group": "hadoop", "yarn.nodemanager.linux-container-executor.resources-handler.class": "org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler", "yarn.nodemanager.local-dirs": "/hadoop/yarn/local", "yarn.nodemanager.log-aggregation.compression-type": "gz", "yarn.nodemanager.log-aggregation.debug-enabled": "false", "yarn.nodemanager.log-aggregation.num-log-files-per-app": "30", "yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds": "3600", "yarn.nodemanager.log-dirs": "/hadoop/yarn/log", "yarn.nodemanager.log.retain-second": "604800", "yarn.nodemanager.recovery.dir": "{{yarn_log_dir_prefix}}/nodemanager/recovery-state", "yarn.nodemanager.recovery.enabled": "true", "yarn.nodemanager.remote-app-log-dir": "/app-logs", "yarn.nodemanager.remote-app-log-dir-suffix": "logs", "yarn.nodemanager.resource.cpu-vcores": "6", "yarn.nodemanager.resource.memory-mb": "9216", "yarn.nodemanager.resource.percentage-physical-cpu-limit": "80", "yarn.nodemanager.vmem-check-enabled": "false", "yarn.nodemanager.vmem-pmem-ratio": "2.1", "yarn.resourcemanager.address": "%HOSTGROUP::host_group_1%:8050", "yarn.resourcemanager.admin.address": "%HOSTGROUP::host_group_1%:8141", "yarn.resourcemanager.am.max-attempts": "2", "yarn.resourcemanager.bind-host": "0.0.0.0", "yarn.resourcemanager.connect.max-wait.ms": "900000", "yarn.resourcemanager.connect.retry-interval.ms": "30000", "yarn.resourcemanager.fs.state-store.retry-policy-spec": "2000, 500", "yarn.resourcemanager.fs.state-store.uri": "", "yarn.resourcemanager.ha.enabled": "false", "yarn.resourcemanager.hostname": "%HOSTGROUP::host_group_1%", "yarn.resourcemanager.nodes.exclude-path": "/etc/hadoop/conf/yarn.exclude", "yarn.resourcemanager.recovery.enabled": "true", "yarn.resourcemanager.resource-tracker.address": "%HOSTGROUP::host_group_1%:8025", "yarn.resourcemanager.scheduler.address": "%HOSTGROUP::host_group_1%:8030", "yarn.resourcemanager.scheduler.class": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler", "yarn.resourcemanager.scheduler.monitor.enable": "false", "yarn.resourcemanager.state-store.max-completed-applications": "${yarn.resourcemanager.max-completed-applications}", "yarn.resourcemanager.store.class": "org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore", "yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size": "10", "yarn.resourcemanager.system-metrics-publisher.enabled": "true", "yarn.resourcemanager.webapp.address": "%HOSTGROUP::host_group_1%:8088", "yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled": "false", "yarn.resourcemanager.webapp.https.address": "%HOSTGROUP::host_group_1%:8090", "yarn.resourcemanager.work-preserving-recovery.enabled": "true", "yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms": "10000", "yarn.resourcemanager.zk-acl": "world:anyone:rwcda", "yarn.resourcemanager.zk-address": "%HOSTGROUP::host_group_1%:2181", "yarn.resourcemanager.zk-num-retries": "1000", "yarn.resourcemanager.zk-retry-interval-ms": "1000", "yarn.resourcemanager.zk-state-store.parent-path": "/rmstore", "yarn.resourcemanager.zk-timeout-ms": "10000", "yarn.scheduler.maximum-allocation-mb": "9216", "yarn.scheduler.maximum-allocation-vcores": "6", "yarn.scheduler.minimum-allocation-mb": "1024", "yarn.scheduler.minimum-allocation-vcores": "1", "yarn.timeline-service.address": "%HOSTGROUP::host_group_1%:10200", "yarn.timeline-service.bind-host": "0.0.0.0", "yarn.timeline-service.client.max-retries": "30", "yarn.timeline-service.client.retry-interval-ms": "1000", "yarn.timeline-service.enabled": "true", "yarn.timeline-service.entity-group-fs-store.active-dir": "/ats/active/", "yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds": "3600", "yarn.timeline-service.entity-group-fs-store.done-dir": "/ats/done/", "yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes": "org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl,org.apache.spark.deploy.history.yarn.plugin.SparkATSPlugin", "yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath": "/usr/hdp/${hdp.version}/spark/hdpLib/*", "yarn.timeline-service.entity-group-fs-store.retain-seconds": "604800", "yarn.timeline-service.entity-group-fs-store.scan-interval-seconds": "60", "yarn.timeline-service.entity-group-fs-store.summary-store": "org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore", "yarn.timeline-service.generic-application-history.store-class": "org.apache.hadoop.yarn.server.applicationhistoryservice.NullApplicationHistoryStore", "yarn.timeline-service.http-authentication.proxyuser.root.groups": "*", "yarn.timeline-service.http-authentication.proxyuser.root.hosts": "vgt-imaster-0.novalocal", "yarn.timeline-service.http-authentication.simple.anonymous.allowed": "true", "yarn.timeline-service.http-authentication.type": "simple", "yarn.timeline-service.leveldb-state-store.path": "/hadoop/yarn/timeline", "yarn.timeline-service.leveldb-timeline-store.path": "/hadoop/yarn/timeline", "yarn.timeline-service.leveldb-timeline-store.read-cache-size": "104857600", "yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size": "10000", "yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size": "10000", "yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms": "300000", "yarn.timeline-service.recovery.enabled": "true", "yarn.timeline-service.state-store-class": "org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore", "yarn.timeline-service.store-class": "org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore", "yarn.timeline-service.ttl-enable": "true", "yarn.timeline-service.ttl-ms": "2678400000", "yarn.timeline-service.version": "1.5", "yarn.timeline-service.webapp.address": "%HOSTGROUP::host_group_1%:8188", "yarn.timeline-service.webapp.https.address": "%HOSTGROUP::host_group_1%:8190" }, "zeppelin-config": { "zeppelin.anonymous.allowed": "true", "zeppelin.interpreter.connect.timeout": "30000", "zeppelin.interpreter.dir": "interpreter", "zeppelin.interpreters": "org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter", "zeppelin.notebook.dir": "notebook", "zeppelin.notebook.homescreen": "", "zeppelin.notebook.homescreen.hide": "false", "zeppelin.notebook.s3.bucket": "zeppelin", "zeppelin.notebook.s3.user": "user", "zeppelin.notebook.storage": "org.apache.zeppelin.notebook.repo.VFSNotebookRepo", "zeppelin.server.addr": "0.0.0.0", "zeppelin.server.allowed.origins": "*", "zeppelin.server.port": "9995", "zeppelin.ssl": "false", "zeppelin.ssl.client.auth": "false", "zeppelin.ssl.keystore.path": "conf/keystore", "zeppelin.ssl.keystore.type": "JKS", "zeppelin.ssl.truststore.path": "conf/truststore", "zeppelin.ssl.truststore.type": "JKS", "zeppelin.websocket.max.text.message.size": "1024000" }, "zeppelin-env": { "log4j_properties_content": "log4j.rootLogger = INFO, dailyfile log4j.appender.stdout = org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout = org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L) - %m%n log4j.appender.dailyfile.DatePattern=.yyyy-MM-dd log4j.appender.dailyfile.Threshold = INFO log4j.appender.dailyfile = org.apache.log4j.DailyRollingFileAppender log4j.appender.dailyfile.File = ${zeppelin.log.file} log4j.appender.dailyfile.layout = org.apache.log4j.PatternLayout log4j.appender.dailyfile.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L) - %m%n", "shiro_ini_content": "[users] # List of users with their password allowed to access Zeppelin. # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections #admin = password1 #user1 = password2, role1, role2 #user2 = password3, role3 #user3 = password4, role2 # Sample LDAP configuration, for user Authentication, currently tested for single Realm [main] #activeDirectoryRealm = org.apache.zeppelin.server.ActiveDirectoryGroupRealm #activeDirectoryRealm.systemUsername = CN=Administrator,CN=Users,DC=HW,DC=EXAMPLE,DC=COM #activeDirectoryRealm.systemPassword = Password1! #activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://user/zeppelin/zeppelin.jceks #activeDirectoryRealm.searchBase = CN=Users,DC=HW,DC=TEST,DC=COM #activeDirectoryRealm.url = ldap://ad-nano.test.example.com:389 #activeDirectoryRealm.groupRolesMap = \"\" #activeDirectoryRealm.authorizationCachingEnabled = true #ldapRealm = org.apache.shiro.realm.ldap.JndiLdapRealm #ldapRealm.userDnTemplate = uid={0},cn=users,cn=accounts,dc=example,dc=com #ldapRealm.contextFactory.url = ldap://ldaphost:389 #ldapRealm.contextFactory.authenticationMechanism = SIMPLE #sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager #securityManager.sessionManager = $sessionManager # 86,400,000 milliseconds = 24 hour #securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login [urls] # anon means the access is anonymous. # authcBasic means Basic Auth Security # To enfore security, comment the line below and uncomment the next one /api/version = anon /** = anon #/** = authc", "zeppelin.executor.instances": "2", "zeppelin.executor.mem": "512m", "zeppelin.server.kerberos.keytab": "", "zeppelin.server.kerberos.principal": "", "zeppelin.spark.jar.dir": "/apps/zeppelin", "zeppelin_env_content": "# Spark master url. eg. spark://master_addr:7077. Leave empty if you want to use local mode export MASTER=yarn-client export SPARK_YARN_JAR={{spark_jar}} # Where log files are stored. PWD by default. export ZEPPELIN_LOG_DIR={{zeppelin_log_dir}} # The pid files are stored. /tmp by default. export ZEPPELIN_PID_DIR={{zeppelin_pid_dir}} export JAVA_HOME={{java64_home}} # Additional jvm options. for example, export ZEPPELIN_JAVA_OPTS=\"-Dspark.executor.memory=8g -Dspark.cores.max=16\" export ZEPPELIN_JAVA_OPTS=\"-Dhdp.version={{full_stack_version}} -Dspark.executor.memory={{executor_mem}} -Dspark.executor.instances={{executor_instances}} -Dspark.yarn.queue={{spark_queue}}\" # Zeppelin jvm mem options Default -Xmx1024m -XX:MaxPermSize=512m # export ZEPPELIN_MEM # zeppelin interpreter process jvm mem options. Defualt = ZEPPELIN_MEM # export ZEPPELIN_INTP_MEM # zeppelin interpreter process jvm options. Default = ZEPPELIN_JAVA_OPTS # export ZEPPELIN_INTP_JAVA_OPTS # Where notebook saved # export ZEPPELIN_NOTEBOOK_DIR # Id of notebook to be displayed in homescreen. ex) 2A94M5J1Z # export ZEPPELIN_NOTEBOOK_HOMESCREEN # hide homescreen notebook from list when this value set to \"true\". default \"false\" # export ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE # Bucket where notebook saved # export ZEPPELIN_NOTEBOOK_S3_BUCKET # User in bucket where notebook saved. For example bucket/user/notebook/2A94M5J1Z/note.json # export ZEPPELIN_NOTEBOOK_S3_USER # A string representing this instance of zeppelin. $USER by default # export ZEPPELIN_IDENT_STRING # The scheduling priority for daemons. Defaults to 0. # export ZEPPELIN_NICENESS #### Spark interpreter configuration #### ## Use provided spark installation ## ## defining SPARK_HOME makes Zeppelin run spark interpreter process using spark-submit ## # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries export SPARK_HOME={{spark_home}} # (optional) extra options to pass to spark submit. eg) \"--driver-memory 512M --executor-memory 1G\". # export SPARK_SUBMIT_OPTIONS ## Use embedded spark binaries ## ## without SPARK_HOME defined, Zeppelin still able to run spark interpreter process using embedded spark binaries. ## however, it is not encouraged when you can define SPARK_HOME ## # Options read in YARN client mode # yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR. export HADOOP_CONF_DIR=/etc/hadoop/conf # Pyspark (supported with Spark 1.2.1 and above) # To configure pyspark, you need to set spark distribution's path to 'spark.home' property in Interpreter setting screen in Zeppelin GUI # path to the python command. must be the same path on the driver(Zeppelin) and all workers. # export PYSPARK_PYTHON export PYTHONPATH=\"${SPARK_HOME}/python:${SPARK_HOME}/python/lib/py4j-0.8.2.1-src.zip\" export SPARK_YARN_USER_ENV=\"PYTHONPATH=${PYTHONPATH}\" ## Spark interpreter options ## ## # Use HiveContext instead of SQLContext if set true. true by default. # export ZEPPELIN_SPARK_USEHIVECONTEXT # Execute multiple SQL concurrently if set true. false by default. # export ZEPPELIN_SPARK_CONCURRENTSQL # Max number of SparkSQL result to display. 1000 by default. # export ZEPPELIN_SPARK_MAXRESULT", "zeppelin_group": "zeppelin", "zeppelin_log_dir": "/var/log/zeppelin", "zeppelin_pid_dir": "/var/run/zeppelin", "zeppelin_user": "zeppelin" }, "zoo.cfg": { "autopurge.purgeInterval": "24", "autopurge.snapRetainCount": "30", "clientPort": "2181", "dataDir": "/hadoop/zookeeper", "initLimit": "10", "syncLimit": "5", "tickTime": "2000" }, "zookeeper-env": { "zk_log_dir": "/var/log/zookeeper", "zk_pid_dir": "/var/run/zookeeper", "zk_user": "zookeeper" } }sahara-8.0.0/sahara/plugins/ambari/resources/images/0000775000175100017510000000000013245515026022437 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/ambari/resources/images/common/0000775000175100017510000000000013245515026023727 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/ambari/resources/images/common/add_jar0000666000175100017510000000143313245514476025251 0ustar zuulzuul00000000000000#! /bin/bash hadoop="2.7.1" HDFS_LIB_DIR=${hdfs_lib_dir:-"/usr/share/hadoop/lib"} JAR_BUILD_DATE="2016-03-17" SWIFT_LIB_URI="http://tarballs.openstack.org/sahara/dist/hadoop-openstack/hadoop-openstack-${hadoop}.jar" HADOOP_SWIFT_JAR_NAME=hadoop-openstack.jar if [ ! -f $HDFS_LIB_DIR/$HADOOP_SWIFT_JAR_NAME ]; then if [ $test_only -eq 0 ]; then if [ -z "${swift_url:-}" ]; then wget -O $HDFS_LIB_DIR/$HADOOP_SWIFT_JAR_NAME $SWIFT_LIB_URI else wget -O $HDFS_LIB_DIR/$HADOOP_SWIFT_JAR_NAME $swift_url fi if [ $? -ne 0 ]; then echo -e "Could not download Swift Hadoop FS implementation.\nAborting" exit 1 fi chmod 0644 $HDFS_LIB_DIR/$HADOOP_SWIFT_JAR_NAME else exit 0 fi fi sahara-8.0.0/sahara/plugins/ambari/resources/images/common/oracle_java0000666000175100017510000000253613245514472026134 0ustar zuulzuul00000000000000#!/bin/bash # NOTE: $(dirname $0) is read-only, use space under $TARGET_ROOT JAVA_LOCATION=${JAVA_TARGET_LOCATION:-"/usr/java"} JAVA_NAME="oracle-jdk" JAVA_HOME=$JAVA_LOCATION/$JAVA_NAME JAVA_DOWNLOAD_URL=${JAVA_DOWNLOAD_URL:-"http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.tar.gz"} if [ ! -d $JAVA_LOCATION ]; then if [ $test_only -eq 0 ]; then echo "Begin: installation of Java" mkdir -p $JAVA_LOCATION if [ -n "$JAVA_DOWNLOAD_URL" ]; then JAVA_FILE=$(basename $JAVA_DOWNLOAD_URL) wget --no-check-certificate --no-cookies -c \ --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \ -O $JAVA_LOCATION/$JAVA_FILE $JAVA_DOWNLOAD_URL elif [ -n "$JAVA_FILE" ]; then install -D -g root -o root -m 0755 $(dirname $0)/$JAVA_FILE $JAVA_LOCATION fi cd $JAVA_LOCATION echo "Decompressing Java archive" echo -e "\n" | tar -zxf $JAVA_FILE echo "Setting up $JAVA_NAME" chown -R root:root $JAVA_LOCATION JAVA_DIR=`ls -1 $JAVA_LOCATION | grep -v tar.gz` ln -s $JAVA_LOCATION/$JAVA_DIR $JAVA_HOME setup-java-home $JAVA_HOME $JAVA_HOME rm $JAVA_FILE echo "End: installation of Java" else exit 0 fi fi sahara-8.0.0/sahara/plugins/ambari/resources/images/image.yaml0000666000175100017510000000532413245514476024423 0ustar zuulzuul00000000000000arguments: ambari_version: description: The version of Ambari to install. Defaults to 2.2.1.0. default: 2.2.1.0 choices: - 2.2.0.0 # HDP 2.3 - 2.2.1.0 # HDP 2.4 java_distro: default: openjdk description: The distribution of Java to install. Defaults to openjdk. choices: - openjdk - oracle-java hdfs_lib_dir: default: /opt description: The path to HDFS lib. Defaults to /opt. required: False swift_url: default: https://tarballs.openstack.org/sahara/dist/hadoop-openstack/master/ description: Location of the swift jar file. required: False validators: - argument_case: argument_name: java_distro cases: openjdk: - package: java-1.7.0-openjdk-devel oracle-java: - script: common/oracle_java - os_case: - centos: - script: centos/disable_selinux - script: centos/setup_java_home: env_vars: [java_distro] - package: wget - script: centos/wget_repo: env_vars: [ambari_version] - package: redhat-lsb - package: - mariadb - mariadb-libs - mariadb-server - mysql-connector-java - package: ntp - package: - ambari-metrics-monitor - ambari-server - ambari-metrics-collector - ambari-metrics-hadoop-sink - package: nmap-ncat - package: fuse-libs - package: snappy-devel - ubuntu: - script: ubuntu/wget_repo: env_vars: [ambari_version] - package: - ambari-metrics-assembly - netcat - package: fuse - package: - mysql-client-5.5 - mysql-server-5.5 - libmysql-java - package: ambari-agent - package: - unzip - zip - curl - tar - rpcbind - rng-tools - iptables-services - os_case: - centos: - script: centos/disable_ambari - script: centos/disable_firewall - script: common/add_jar: env_vars: [hdfs_lib_dir, swift_url] - script: centos/unlimited_security_artifacts: env_vars: [unlimited_security_location] - ubuntu: - script: common/add_jar: env_vars: [hdfs_lib_dir, swift_url] - os_case: - centos: - package: - krb5-server - krb5-libs - krb5-workstation - ubuntu: - package: - krb5-admin-server - libpam-krb5 - krb5-user - ldap-utils sahara-8.0.0/sahara/plugins/ambari/resources/images/centos/0000775000175100017510000000000013245515026023732 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/ambari/resources/images/centos/disable_ambari0000666000175100017510000000020513245514472026576 0ustar zuulzuul00000000000000#!/usr/bin/env bash if [ $test_only -eq 0 ]; then chkconfig ambari-server off chkconfig ambari-agent off else exit 0 fi sahara-8.0.0/sahara/plugins/ambari/resources/images/centos/wget_repo0000666000175100017510000000042113245514472025653 0ustar zuulzuul00000000000000#!/usr/bin/env bash if [ ! -f /etc/yum.repos.d/ambari.repo ]; then if [ $test_only -eq 0 ]; then wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/$ambari_version/ambari.repo -O /etc/yum.repos.d/ambari.repo else exit 0 fi fi sahara-8.0.0/sahara/plugins/ambari/resources/images/centos/disable_firewall0000666000175100017510000000103513245514472027152 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files iptables.service | grep 'enabled' | wc -l) if [ $check -eq 1 ]; then if [ $test_only -eq 0 ]; then if type -p systemctl && [[ "$(systemctl --no-pager list-unit-files firewalld)" =~ 'enabled' ]]; then systemctl disable firewalld fi if type -p service; then service ip6tables save service iptables save chkconfig ip6tables off chkconfig iptables off fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/ambari/resources/images/centos/unlimited_security_artifacts0000666000175100017510000000065113245514476031652 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -d /tmp/UnlimitedPolicy/ ]; then if [ $test_only -eq 0 ]; then mkdir /tmp/UnlimitedPolicy/ wget http://tarballs.openstack.org/sahara/dist/common-artifacts/local_policy.jar -O /tmp/UnlimitedPolicy/local_policy.jar wget http://tarballs.openstack.org/sahara/dist/common-artifacts/US_export_policy.jar -O /tmp/UnlimitedPolicy/US_export_policy.jar else exit 0 fi fi sahara-8.0.0/sahara/plugins/ambari/resources/images/centos/setup_java_home0000666000175100017510000000162113245514472027034 0ustar zuulzuul00000000000000#!/bin/bash JAVA_RC="/etc/profile.d/99-java.sh" JAVA_BIN_RC="/etc/profile.d/98-java-bin.sh" if [ ! -f $JAVA_RC ]; then if [ $test_only -eq 0 ]; then case "$java_distro" in openjdk ) JRE_HOME="/usr/lib/jvm/java-openjdk/jre" JDK_HOME="/usr/lib/jvm/java-openjdk" ;; oracle-java ) JRE_HOME="/usr/java/oracle-jdk" JDK_HOME="/usr/java/oracle-jdk" ;; esac echo "export JAVA_HOME=$JRE_HOME" >> $JAVA_RC chmod +x $JAVA_RC echo "export PATH=$JRE_HOME/bin:\$PATH" >> $JAVA_BIN_RC echo "export PATH=$JDK_HOME/bin:\$PATH" >> $JAVA_BIN_RC chmod +x $JAVA_BIN_RC alternatives --install /usr/bin/java java $JRE_HOME/bin/java 200000 alternatives --install /usr/bin/javac javac $JDK_HOME/bin/javac 200000 else exit 0 fi fi sahara-8.0.0/sahara/plugins/ambari/resources/images/centos/disable_selinux0000666000175100017510000000043513245514472027037 0ustar zuulzuul00000000000000#!/bin/bash check=$(cat /etc/selinux/config | grep 'SELINUX=disabled' | wc -l) if [ $check -eq 0 ]; then if [ $test_only -eq 0 ]; then config=/etc/selinux/config [ -e $config ] && sed -i "s%^\(SELINUX=\s*\).*$%SELINUX=disabled%" $config else exit 0 fi fi sahara-8.0.0/sahara/plugins/ambari/resources/images/ubuntu/0000775000175100017510000000000013245515026023761 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/ambari/resources/images/ubuntu/wget_repo0000666000175100017510000000056213245514472025710 0ustar zuulzuul00000000000000#!/usr/bin/env bash if [ ! -f /etc/apt/sources.list.d/ambari.list ]; then if [ $test_only -eq 0 ]; then wget http://public-repo-1.hortonworks.com/ambari/ubuntu12/2.x/updates/$ambari_version/ambari.list -O /etc/apt/sources.list.d/ambari.list apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD else exit 0 fi fi sahara-8.0.0/sahara/plugins/ambari/resources/configs-2.4.json0000666000175100017510000024343713245514472024041 0ustar zuulzuul00000000000000{ "accumulo-env": { "accumulo_gc_heapsize": "256", "accumulo_instance_name": "hdp-accumulo-instance", "accumulo_log_dir": "/var/log/accumulo", "accumulo_master_heapsize": "1024", "accumulo_monitor_bind_all": "false", "accumulo_monitor_heapsize": "1024", "accumulo_other_heapsize": "1024", "accumulo_pid_dir": "/var/run/accumulo", "accumulo_tserver_heapsize": "1536", "accumulo_user": "accumulo", "instance_secret": "SECRET:accumulo-env:1:instance_secret", "server_content": "#! /usr/bin/env bash export HADOOP_PREFIX={{hadoop_prefix}} export HADOOP_CONF_DIR={{hadoop_conf_dir}} export JAVA_HOME={{java64_home}} export ZOOKEEPER_HOME={{zookeeper_home}} export ACCUMULO_PID_DIR={{pid_dir}} export ACCUMULO_LOG_DIR={{log_dir}} export ACCUMULO_CONF_DIR={{server_conf_dir}} export ACCUMULO_TSERVER_OPTS=\"-Xmx{{accumulo_tserver_heapsize}}m -Xms{{accumulo_tserver_heapsize}}m\" export ACCUMULO_MASTER_OPTS=\"-Xmx{{accumulo_master_heapsize}}m -Xms{{accumulo_master_heapsize}}m\" export ACCUMULO_MONITOR_OPTS=\"-Xmx{{accumulo_monitor_heapsize}}m -Xms{{accumulo_monitor_heapsize}}m\" export ACCUMULO_GC_OPTS=\"-Xmx{{accumulo_gc_heapsize}}m -Xms{{accumulo_gc_heapsize}}m\" export ACCUMULO_GENERAL_OPTS=\"-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -Djava.net.preferIPv4Stack=true ${ACCUMULO_GENERAL_OPTS}\" export ACCUMULO_OTHER_OPTS=\"-Xmx{{accumulo_other_heapsize}}m -Xms{{accumulo_other_heapsize}}m ${ACCUMULO_OTHER_OPTS}\" export ACCUMULO_MONITOR_BIND_ALL={{monitor_bind_str}} # what do when the JVM runs out of heap memory export ACCUMULO_KILL_CMD='kill -9 %p'" }, "accumulo-log4j": { "audit_log_level": "OFF", "debug_log_size": "512M", "debug_num_logs": "10", "info_log_size": "512M", "info_num_logs": "10", "monitor_forwarding_log_level": "WARN" }, "accumulo-site": { "gc.port.client": "50092", "general.classpaths": "$ACCUMULO_HOME/lib/accumulo-server.jar, $ACCUMULO_HOME/lib/accumulo-core.jar, $ACCUMULO_HOME/lib/accumulo-start.jar, $ACCUMULO_HOME/lib/accumulo-fate.jar, $ACCUMULO_HOME/lib/accumulo-proxy.jar, $ACCUMULO_HOME/lib/[^.].*.jar, $ZOOKEEPER_HOME/zookeeper[^.].*.jar, $HADOOP_CONF_DIR, /usr/hdp/current/hadoop-client/[^.].*.jar, /usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar, /usr/hdp/current/hadoop-hdfs-client/[^.].*.jar, /usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar, /usr/hdp/current/hadoop-yarn-client/[^.].*.jar, /usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar, /usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar,", "instance.volumes": "hdfs://%HOSTGROUP::host_group_1%:8020/apps/accumulo/data", "instance.zookeeper.host": "%HOSTGROUP::host_group_1%:2181", "instance.zookeeper.timeout": "30s", "master.port.client": "9999", "monitor.port.client": "50095", "monitor.port.log4j": "4560", "trace.port.client": "12234", "trace.user": "trace", "tserver.cache.data.size": "128M", "tserver.cache.index.size": "256M", "tserver.memory.maps.max": "1G", "tserver.memory.maps.native.enabled": "true", "tserver.port.client": "9997", "tserver.sort.buffer.size": "200M", "tserver.walog.max.size": "1G" }, "ams-env": { "ambari_metrics_user": "ams", "metrics_collector_heapsize": "512", "metrics_collector_log_dir": "/var/log/ambari-metrics-collector", "metrics_collector_pid_dir": "/var/run/ambari-metrics-collector", "metrics_monitor_log_dir": "/var/log/ambari-metrics-monitor", "metrics_monitor_pid_dir": "/var/run/ambari-metrics-monitor" }, "ams-hbase-env": { "hbase_classpath_additional": "", "hbase_log_dir": "/var/log/ambari-metrics-collector", "hbase_master_heapsize": "768", "hbase_master_maxperm_size": "128", "hbase_master_xmn_size": "192", "hbase_pid_dir": "/var/run/ambari-metrics-collector/", "hbase_regionserver_heapsize": "512", "hbase_regionserver_xmn_ratio": "0.2", "max_open_files_limit": "32768", "regionserver_xmn_size": "256" }, "ams-hbase-policy": { "security.admin.protocol.acl": "*", "security.client.protocol.acl": "*", "security.masterregion.protocol.acl": "*" }, "ams-hbase-security-site": { "ams.zookeeper.keytab": "", "ams.zookeeper.principal": "", "hadoop.security.authentication": "", "hbase.coprocessor.master.classes": "", "hbase.coprocessor.region.classes": "", "hbase.master.kerberos.principal": "", "hbase.master.keytab.file": "", "hbase.myclient.keytab": "", "hbase.myclient.principal": "", "hbase.regionserver.kerberos.principal": "", "hbase.regionserver.keytab.file": "", "hbase.security.authentication": "", "hbase.security.authorization": "", "hbase.zookeeper.property.authProvider.1": "", "hbase.zookeeper.property.jaasLoginRenew": "", "hbase.zookeeper.property.kerberos.removeHostFromPrincipal": "", "hbase.zookeeper.property.kerberos.removeRealmFromPrincipal": "" }, "ams-hbase-site": { "dfs.client.read.shortcircuit": "true", "hbase.client.scanner.caching": "10000", "hbase.client.scanner.timeout.period": "900000", "hbase.cluster.distributed": "false", "hbase.hregion.majorcompaction": "0", "hbase.hregion.max.filesize": "4294967296", "hbase.hregion.memstore.block.multiplier": "4", "hbase.hregion.memstore.flush.size": "134217728", "hbase.hstore.blockingStoreFiles": "200", "hbase.hstore.flusher.count": "2", "hbase.local.dir": "${hbase.tmp.dir}/local", "hbase.master.info.bindAddress": "0.0.0.0", "hbase.master.info.port": "61310", "hbase.master.normalizer.class": "org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer", "hbase.master.port": "61300", "hbase.master.wait.on.regionservers.mintostart": "1", "hbase.normalizer.enabled": "true", "hbase.normalizer.period": "600000", "hbase.regionserver.global.memstore.lowerLimit": "0.3", "hbase.regionserver.global.memstore.upperLimit": "0.35", "hbase.regionserver.info.port": "61330", "hbase.regionserver.port": "61320", "hbase.regionserver.thread.compaction.large": "2", "hbase.regionserver.thread.compaction.small": "3", "hbase.replication": "false", "hbase.rootdir": "file:///var/lib/ambari-metrics-collector/hbase", "hbase.snapshot.enabled": "false", "hbase.tmp.dir": "/var/lib/ambari-metrics-collector/hbase-tmp", "hbase.zookeeper.leaderport": "61388", "hbase.zookeeper.peerport": "61288", "hbase.zookeeper.property.clientPort": "{{zookeeper_clientPort}}", "hbase.zookeeper.property.dataDir": "${hbase.tmp.dir}/zookeeper", "hbase.zookeeper.property.tickTime": "6000", "hbase.zookeeper.quorum": "{{zookeeper_quorum_hosts}}", "hfile.block.cache.size": "0.3", "phoenix.coprocessor.maxMetaDataCacheSize": "20480000", "phoenix.coprocessor.maxServerCacheTimeToLiveMs": "60000", "phoenix.groupby.maxCacheSize": "307200000", "phoenix.mutate.batchSize": "10000", "phoenix.query.maxGlobalMemoryPercentage": "15", "phoenix.query.rowKeyOrderSaltedTable": "true", "phoenix.query.spoolThresholdBytes": "12582912", "phoenix.query.timeoutMs": "1200000", "phoenix.sequence.saltBuckets": "2", "phoenix.spool.directory": "${hbase.tmp.dir}/phoenix-spool", "zookeeper.session.timeout": "120000", "zookeeper.session.timeout.localHBaseCluster": "120000", "zookeeper.znode.parent": "/ams-hbase-unsecure" }, "ams-site": { "phoenix.query.maxGlobalMemoryPercentage": "25", "phoenix.spool.directory": "/tmp", "timeline.metrics.aggregator.checkpoint.dir": "/var/lib/ambari-metrics-collector/checkpoint", "timeline.metrics.cluster.aggregate.splitpoints": "", "timeline.metrics.cluster.aggregator.daily.checkpointCutOffMultiplier": "1", "timeline.metrics.cluster.aggregator.daily.disabled": "false", "timeline.metrics.cluster.aggregator.daily.interval": "86400", "timeline.metrics.cluster.aggregator.daily.ttl": "63072000", "timeline.metrics.cluster.aggregator.hourly.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.hourly.disabled": "false", "timeline.metrics.cluster.aggregator.hourly.interval": "3600", "timeline.metrics.cluster.aggregator.hourly.ttl": "31536000", "timeline.metrics.cluster.aggregator.minute.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.minute.disabled": "false", "timeline.metrics.cluster.aggregator.minute.interval": "300", "timeline.metrics.cluster.aggregator.minute.ttl": "7776000", "timeline.metrics.cluster.aggregator.second.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.second.disabled": "false", "timeline.metrics.cluster.aggregator.second.interval": "120", "timeline.metrics.cluster.aggregator.second.timeslice.interval": "30", "timeline.metrics.cluster.aggregator.second.ttl": "2592000", "timeline.metrics.daily.aggregator.minute.interval": "86400", "timeline.metrics.hbase.compression.scheme": "SNAPPY", "timeline.metrics.hbase.data.block.encoding": "FAST_DIFF", "timeline.metrics.hbase.fifo.compaction.enabled": "true", "timeline.metrics.host.aggregate.splitpoints": "", "timeline.metrics.host.aggregator.daily.checkpointCutOffMultiplier": "1", "timeline.metrics.host.aggregator.daily.disabled": "false", "timeline.metrics.host.aggregator.daily.ttl": "31536000", "timeline.metrics.host.aggregator.hourly.checkpointCutOffMultiplier": "2", "timeline.metrics.host.aggregator.hourly.disabled": "false", "timeline.metrics.host.aggregator.hourly.interval": "3600", "timeline.metrics.host.aggregator.hourly.ttl": "2592000", "timeline.metrics.host.aggregator.minute.checkpointCutOffMultiplier": "2", "timeline.metrics.host.aggregator.minute.disabled": "false", "timeline.metrics.host.aggregator.minute.interval": "300", "timeline.metrics.host.aggregator.minute.ttl": "604800", "timeline.metrics.host.aggregator.ttl": "86400", "timeline.metrics.service.checkpointDelay": "60", "timeline.metrics.service.cluster.aggregator.appIds": "datanode,nodemanager,hbase", "timeline.metrics.service.default.result.limit": "15840", "timeline.metrics.service.operation.mode": "embedded", "timeline.metrics.service.resultset.fetchSize": "2000", "timeline.metrics.service.rpc.address": "0.0.0.0:60200", "timeline.metrics.service.use.groupBy.aggregators": "true", "timeline.metrics.service.watcher.delay": "30", "timeline.metrics.service.watcher.initial.delay": "600", "timeline.metrics.service.watcher.timeout": "30", "timeline.metrics.service.webapp.address": "0.0.0.0:6188", "timeline.metrics.sink.collection.period": "60", "timeline.metrics.sink.report.interval": "60" }, "anonymization-rules": { "anonymization-rules-content": "{ \"rules\":[ { \"name\":\"ip_address\", \"path\":null, \"pattern\": \"[ :\\\\/]?[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}[ :\\\\/]?\", \"extract\": \"[ :\\\\/]?([0-9\\\\.]+)[ :\\\\/]?\", \"shared\":true }, { \"name\":\"domain\", \"path\":null, \"pattern\": \"$DOMAIN_RULE$\", \"shared\":true }, { \"name\":\"delete_oozie_jdbc_password\", \"path\":\"oozie-site.xml\", \"property\": \"oozie.service.JPAService.jdbc.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_sqoop_metastore_password\", \"path\":\"sqoop-site.xml\", \"property\": \"sqoop.metastore.client.autoconnect.password\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" }, { \"name\":\"delete_hive_metastore_password\", \"path\":\"hive-site.xml\", \"property\": \"javax.jdo.option.ConnectionPassword\", \"operation\":\"REPLACE\", \"value\":\"Hidden\" } ] }" }, "application-properties": { "atlas.authentication.keytab": "/etc/security/keytabs/atlas.service.keytab", "atlas.authentication.method": "simple", "atlas.authentication.principal": "atlas", "atlas.enableTLS": "false", "atlas.graph.index.search.backend": "elasticsearch", "atlas.graph.index.search.directory": "/var/lib/atlas/data/es", "atlas.graph.index.search.elasticsearch.client-only": "false", "atlas.graph.index.search.elasticsearch.local-mode": "true", "atlas.graph.storage.backend": "berkeleyje", "atlas.graph.storage.directory": "/var/lib/atlas/data/berkeley", "atlas.http.authentication.enabled": "false", "atlas.http.authentication.kerberos.keytab": "/etc/security/keytabs/spnego.service.keytab", "atlas.http.authentication.kerberos.name.rules": "RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// DEFAULT", "atlas.http.authentication.kerberos.principal": "HTTP/_HOST@EXAMPLE.COM", "atlas.http.authentication.type": "simple", "atlas.lineage.hive.process.inputs.name": "inputs", "atlas.lineage.hive.process.outputs.name": "outputs", "atlas.lineage.hive.process.type.name": "Process", "atlas.lineage.hive.table.schema.query.Table": "Table where name='%s'\\, columns", "atlas.lineage.hive.table.schema.query.hive_table": "hive_table where name='%s'\\, columns", "atlas.lineage.hive.table.type.name": "DataSet", "atlas.server.bind.address": "localhost", "atlas.server.http.port": "21000", "atlas.server.https.port": "21443" }, "atlas-env": { "metadata_classpath": "", "metadata_data_dir": "/var/lib/atlas/data", "metadata_expanded_war_dir": "./server/webapp", "metadata_log_dir": "/var/log/atlas", "metadata_opts": "-Xmx1024m -Dlog4j.configuration=atlas-log4j.xml", "metadata_pid_dir": "/var/run/atlas", "metadata_user": "atlas" }, "capacity-scheduler": { "yarn.scheduler.capacity.default.minimum-user-limit-percent": "100", "yarn.scheduler.capacity.maximum-am-resource-percent": "0.2", "yarn.scheduler.capacity.maximum-applications": "10000", "yarn.scheduler.capacity.node-locality-delay": "40", "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator", "yarn.scheduler.capacity.root.accessible-node-labels": "*", "yarn.scheduler.capacity.root.acl_administer_queue": "*", "yarn.scheduler.capacity.root.capacity": "100", "yarn.scheduler.capacity.root.default.acl_administer_jobs": "*", "yarn.scheduler.capacity.root.default.acl_submit_applications": "*", "yarn.scheduler.capacity.root.default.capacity": "100", "yarn.scheduler.capacity.root.default.maximum-capacity": "100", "yarn.scheduler.capacity.root.default.state": "RUNNING", "yarn.scheduler.capacity.root.default.user-limit-factor": "1", "yarn.scheduler.capacity.root.queues": "default" }, "capture-levels": { "capture-levels-content": "[ { \"name\":\"L1\", \"description\":\"Configurations\", \"filter\":\"$type == 'CONF' or ($type == 'REPORTS' and $service is None) or ($type == 'LOG' and $service is None)\" }, { \"name\":\"L2\", \"description\":\"Configurations and Metrics\", \"filter\":\"$type in ['CONF', 'REPORTS'] or ($type == 'REPORTS' and $service is None) or ($type == 'LOG' and $service is None)\" }, { \"name\":\"L3\", \"description\":\"Configurations, Metrics and Logs\", \"filter\":\"$type in ['CONF', 'REPORTS', 'LOG'] or ($type == 'REPORTS' and $service is None) or ($type == 'LOG' and $service is None)\" } ]" }, "cluster-env": { "fetch_nonlocal_groups": "true", "ignore_groupsusers_create": "false", "kerberos_domain": "EXAMPLE.COM", "override_uid": "true", "repo_suse_rhel_template": "[{{repo_id}}] name={{repo_id}} {% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %} path=/ enabled=1 gpgcheck=0", "repo_ubuntu_template": "{{package_type}} {{base_url}} {{components}}", "security_enabled": "false", "smokeuser": "ambari-qa", "smokeuser_keytab": "/etc/security/keytabs/smokeuser.headless.keytab", "user_group": "hadoop" }, "core-site": { "fs.defaultFS": "hdfs://%HOSTGROUP::host_group_1%:8020", "fs.trash.interval": "360", "ha.failover-controller.active-standby-elector.zk.op.retries": "120", "hadoop.http.authentication.simple.anonymous.allowed": "true", "hadoop.proxyuser.falcon.groups": "*", "hadoop.proxyuser.falcon.hosts": "*", "hadoop.proxyuser.hcat.groups": "*", "hadoop.proxyuser.hcat.hosts": "%HOSTGROUP::host_group_1%", "hadoop.proxyuser.hdfs.groups": "*", "hadoop.proxyuser.hdfs.hosts": "*", "hadoop.proxyuser.hive.groups": "*", "hadoop.proxyuser.hive.hosts": "%HOSTGROUP::host_group_1%", "hadoop.proxyuser.oozie.groups": "*", "hadoop.proxyuser.oozie.hosts": "%HOSTGROUP::host_group_1%", "hadoop.security.auth_to_local": "DEFAULT", "hadoop.security.authentication": "simple", "hadoop.security.authorization": "false", "hadoop.security.key.provider.path": "", "io.compression.codecs": "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec", "io.file.buffer.size": "131072", "io.serializations": "org.apache.hadoop.io.serializer.WritableSerialization", "ipc.client.connect.max.retries": "50", "ipc.client.connection.maxidletime": "30000", "ipc.client.idlethreshold": "8000", "ipc.server.tcpnodelay": "true", "mapreduce.jobtracker.webinterface.trusted": "false", "net.topology.script.file.name": "/etc/hadoop/conf/topology_script.py" }, "falcon-env": { "falcon.embeddedmq": "true", "falcon.embeddedmq.data": "/hadoop/falcon/embeddedmq/data", "falcon.emeddedmq.port": "61616", "falcon_local_dir": "/hadoop/falcon", "falcon_log_dir": "/var/log/falcon", "falcon_pid_dir": "/var/run/falcon", "falcon_port": "15000", "falcon_store_uri": "file:///hadoop/falcon/store", "falcon_user": "falcon", "supports_hive_dr": "true" }, "falcon-runtime.properties": { "*.domain": "${falcon.app.type}", "*.log.cleanup.frequency.days.retention": "days(7)", "*.log.cleanup.frequency.hours.retention": "minutes(1)", "*.log.cleanup.frequency.minutes.retention": "hours(6)", "*.log.cleanup.frequency.months.retention": "months(3)" }, "falcon-startup.properties": { "*.ConfigSyncService.impl": "org.apache.falcon.resource.ConfigSyncService", "*.ProcessInstanceManager.impl": "org.apache.falcon.resource.InstanceManager", "*.SchedulableEntityManager.impl": "org.apache.falcon.resource.SchedulableEntityManager", "*.application.services": "org.apache.falcon.security.AuthenticationInitializationService,\\ org.apache.falcon.workflow.WorkflowJobEndNotificationService, \\ org.apache.falcon.service.ProcessSubscriberService,\\ org.apache.falcon.entity.store.ConfigurationStore,\\ org.apache.falcon.rerun.service.RetryService,\\ org.apache.falcon.rerun.service.LateRunService,\\ org.apache.falcon.service.LogCleanupService,\\ org.apache.falcon.metadata.MetadataMappingService", "*.broker.impl.class": "org.apache.activemq.ActiveMQConnectionFactory", "*.broker.ttlInMins": "4320", "*.broker.url": "tcp://%HOSTGROUP::host_group_1%:61616", "*.catalog.service.impl": "org.apache.falcon.catalog.HiveCatalogService", "*.config.store.uri": "file:///hadoop/falcon/store", "*.configstore.listeners": "org.apache.falcon.entity.v0.EntityGraph,\\ org.apache.falcon.entity.ColoClusterRelation,\\ org.apache.falcon.group.FeedGroupMap,\\ org.apache.falcon.service.SharedLibraryHostingService", "*.domain": "${falcon.app.type}", "*.entity.topic": "FALCON.ENTITY.TOPIC", "*.falcon.authentication.type": "simple", "*.falcon.cleanup.service.frequency": "days(1)", "*.falcon.enableTLS": "false", "*.falcon.graph.blueprints.graph": "com.thinkaurelius.titan.core.TitanFactory", "*.falcon.graph.preserve.history": "false", "*.falcon.graph.serialize.path": "/hadoop/falcon/data/lineage", "*.falcon.graph.storage.backend": "berkeleyje", "*.falcon.graph.storage.directory": "/hadoop/falcon/data/lineage/graphdb", "*.falcon.http.authentication.blacklisted.users": "", "*.falcon.http.authentication.cookie.domain": "EXAMPLE.COM", "*.falcon.http.authentication.kerberos.name.rules": "DEFAULT", "*.falcon.http.authentication.signature.secret": "falcon", "*.falcon.http.authentication.simple.anonymous.allowed": "true", "*.falcon.http.authentication.token.validity": "36000", "*.falcon.http.authentication.type": "simple", "*.falcon.security.authorization.admin.groups": "falcon", "*.falcon.security.authorization.admin.users": "falcon,ambari-qa", "*.falcon.security.authorization.enabled": "false", "*.falcon.security.authorization.provider": "org.apache.falcon.security.DefaultAuthorizationProvider", "*.falcon.security.authorization.superusergroup": "falcon", "*.hive.shared.libs": "hive-exec,hive-metastore,hive-common,hive-service,hive-hcatalog-server-extensions,\\ hive-hcatalog-core,hive-jdbc,hive-webhcat-java-client", "*.internal.queue.size": "1000", "*.journal.impl": "org.apache.falcon.transaction.SharedFileSystemJournal", "*.max.retry.failure.count": "1", "*.oozie.feed.workflow.builder": "org.apache.falcon.workflow.OozieFeedWorkflowBuilder", "*.oozie.process.workflow.builder": "org.apache.falcon.workflow.OozieProcessWorkflowBuilder", "*.retry.recorder.path": "${falcon.log.dir}/retry", "*.shared.libs": "activemq-core,ant,geronimo-j2ee-management,jms,json-simple,oozie-client,spring-jms,commons-lang3,commons-el", "*.system.lib.location": "${falcon.home}/server/webapp/${falcon.app.type}/WEB-INF/lib", "*.workflow.engine.impl": "org.apache.falcon.workflow.engine.OozieWorkflowEngine", "prism.application.services": "org.apache.falcon.entity.store.ConfigurationStore", "prism.configstore.listeners": "org.apache.falcon.entity.v0.EntityGraph,\\ org.apache.falcon.entity.ColoClusterRelation,\\ org.apache.falcon.group.FeedGroupMap" }, "flume-env": { "flume_conf_dir": "/etc/flume/conf", "flume_log_dir": "/var/log/flume", "flume_run_dir": "/var/run/flume", "flume_user": "flume" }, "gateway-site": { "gateway.gateway.conf.dir": "deployments", "gateway.hadoop.kerberos.secured": "false", "gateway.path": "gateway", "gateway.port": "8443", "java.security.auth.login.config": "/etc/knox/conf/krb5JAASLogin.conf", "java.security.krb5.conf": "/etc/knox/conf/krb5.conf", "sun.security.krb5.debug": "true" }, "hadoop-env": { "dtnode_heapsize": "1024m", "hadoop_heapsize": "1024", "hadoop_pid_dir_prefix": "/var/run/hadoop", "hadoop_root_logger": "INFO,RFA", "hdfs_log_dir_prefix": "/var/log/hadoop", "hdfs_user": "hdfs", "hdfs_user_nofile_limit": "128000", "hdfs_user_nproc_limit": "65536", "keyserver_host": "", "keyserver_port": "", "namenode_heapsize": "1024m", "namenode_opt_maxnewsize": "128m", "namenode_opt_maxpermsize": "256m", "namenode_opt_newsize": "128m", "namenode_opt_permsize": "128m", "nfsgateway_heapsize": "1024", "proxyuser_group": "users" }, "hadoop-policy": { "security.admin.operations.protocol.acl": "hadoop", "security.client.datanode.protocol.acl": "*", "security.client.protocol.acl": "*", "security.datanode.protocol.acl": "*", "security.inter.datanode.protocol.acl": "*", "security.inter.tracker.protocol.acl": "*", "security.job.client.protocol.acl": "*", "security.job.task.protocol.acl": "*", "security.namenode.protocol.acl": "*", "security.refresh.policy.protocol.acl": "hadoop", "security.refresh.usertogroups.mappings.protocol.acl": "hadoop" }, "hbase-env": { "hbase_java_io_tmpdir": "/tmp", "hbase_log_dir": "/var/log/hbase", "hbase_master_heapsize": "8192", "hbase_pid_dir": "/var/run/hbase", "hbase_regionserver_heapsize": "8192", "hbase_regionserver_xmn_max": "512", "hbase_regionserver_xmn_ratio": "0.2", "hbase_user": "hbase", "hbase_user_nofile_limit": "32000", "hbase_user_nproc_limit": "16000", "phoenix_sql_enabled": "false" }, "hbase-policy": { "security.admin.protocol.acl": "*", "security.client.protocol.acl": "*", "security.masterregion.protocol.acl": "*" }, "hbase-site": { "dfs.domain.socket.path": "/var/lib/hadoop-hdfs/dn_socket", "hbase.bulkload.staging.dir": "/apps/hbase/staging", "hbase.client.keyvalue.maxsize": "1048576", "hbase.client.retries.number": "35", "hbase.client.scanner.caching": "100", "hbase.cluster.distributed": "true", "hbase.coprocessor.master.classes": "", "hbase.coprocessor.region.classes": "org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint", "hbase.defaults.for.version.skip": "true", "hbase.hregion.majorcompaction": "604800000", "hbase.hregion.majorcompaction.jitter": "0.50", "hbase.hregion.max.filesize": "10737418240", "hbase.hregion.memstore.block.multiplier": "4", "hbase.hregion.memstore.flush.size": "134217728", "hbase.hregion.memstore.mslab.enabled": "true", "hbase.hstore.blockingStoreFiles": "10", "hbase.hstore.compaction.max": "10", "hbase.hstore.compactionThreshold": "3", "hbase.local.dir": "${hbase.tmp.dir}/local", "hbase.master.info.bindAddress": "0.0.0.0", "hbase.master.info.port": "16010", "hbase.master.port": "16000", "hbase.regionserver.global.memstore.size": "0.4", "hbase.regionserver.handler.count": "30", "hbase.regionserver.info.port": "16030", "hbase.regionserver.port": "16020", "hbase.regionserver.wal.codec": "org.apache.hadoop.hbase.regionserver.wal.WALCellCodec", "hbase.rootdir": "hdfs://%HOSTGROUP::host_group_1%:8020/apps/hbase/data", "hbase.rpc.protection": "authentication", "hbase.rpc.timeout": "90000", "hbase.security.authentication": "simple", "hbase.security.authorization": "false", "hbase.superuser": "hbase", "hbase.tmp.dir": "/tmp/hbase-${user.name}", "hbase.zookeeper.property.clientPort": "2181", "hbase.zookeeper.quorum": "%HOSTGROUP::host_group_1%", "hbase.zookeeper.useMulti": "true", "hfile.block.cache.size": "0.4", "phoenix.query.timeoutMs": "60000", "zookeeper.session.timeout": "90000", "zookeeper.znode.parent": "/hbase-unsecure" }, "hdfs-site": { "dfs.block.access.token.enable": "true", "dfs.blockreport.initialDelay": "120", "dfs.blocksize": "134217728", "dfs.client.read.shortcircuit": "true", "dfs.client.read.shortcircuit.streams.cache.size": "4096", "dfs.client.retry.policy.enabled": "false", "dfs.cluster.administrators": "hdfs", "dfs.content-summary.limit": "5000", "dfs.datanode.address": "0.0.0.0:50010", "dfs.datanode.balance.bandwidthPerSec": "6250000", "dfs.datanode.data.dir": "/hadoop/hdfs/data", "dfs.datanode.data.dir.perm": "750", "dfs.datanode.du.reserved": "1073741824", "dfs.datanode.failed.volumes.tolerated": "0", "dfs.datanode.http.address": "0.0.0.0:50075", "dfs.datanode.https.address": "0.0.0.0:50475", "dfs.datanode.ipc.address": "0.0.0.0:8010", "dfs.datanode.max.transfer.threads": "16384", "dfs.domain.socket.path": "/var/lib/hadoop-hdfs/dn_socket", "dfs.encrypt.data.transfer.cipher.suites": "AES/CTR/NoPadding", "dfs.encryption.key.provider.uri": "", "dfs.heartbeat.interval": "3", "dfs.hosts.exclude": "/etc/hadoop/conf/dfs.exclude", "dfs.http.policy": "HTTP_ONLY", "dfs.https.port": "50470", "dfs.journalnode.edits.dir": "/hadoop/hdfs/journalnode", "dfs.journalnode.http-address": "0.0.0.0:8480", "dfs.journalnode.https-address": "0.0.0.0:8481", "dfs.namenode.accesstime.precision": "0", "dfs.namenode.audit.log.async": "true", "dfs.namenode.avoid.read.stale.datanode": "true", "dfs.namenode.avoid.write.stale.datanode": "true", "dfs.namenode.checkpoint.dir": "/hadoop/hdfs/namesecondary", "dfs.namenode.checkpoint.edits.dir": "${dfs.namenode.checkpoint.dir}", "dfs.namenode.checkpoint.period": "21600", "dfs.namenode.checkpoint.txns": "1000000", "dfs.namenode.fslock.fair": "false", "dfs.namenode.handler.count": "800", "dfs.namenode.http-address": "%HOSTGROUP::host_group_1%:50070", "dfs.namenode.https-address": "%HOSTGROUP::host_group_1%:50470", "dfs.namenode.name.dir": "/hadoop/hdfs/namenode", "dfs.namenode.name.dir.restore": "true", "dfs.namenode.rpc-address": "%HOSTGROUP::host_group_1%:8020", "dfs.namenode.safemode.threshold-pct": "1", "dfs.namenode.secondary.http-address": "%HOSTGROUP::host_group_1%:50090", "dfs.namenode.stale.datanode.interval": "30000", "dfs.namenode.startup.delay.block.deletion.sec": "3600", "dfs.namenode.write.stale.datanode.ratio": "1.0f", "dfs.permissions.enabled": "true", "dfs.permissions.superusergroup": "hdfs", "dfs.replication": "3", "dfs.replication.max": "50", "dfs.support.append": "true", "dfs.webhdfs.enabled": "true", "fs.permissions.umask-mode": "022", "nfs.exports.allowed.hosts": "* rw", "nfs.file.dump.dir": "/tmp/.hdfs-nfs" }, "hive-env": { "cost_based_optimizer": "On", "hcat_log_dir": "/var/log/webhcat", "hcat_pid_dir": "/var/run/webhcat", "hcat_user": "hcat", "hive.client.heapsize": "1024", "hive.heapsize": "11963", "hive.metastore.heapsize": "3987", "hive_ambari_database": "MySQL", "hive_database": "New MySQL Database", "hive_database_name": "hive", "hive_database_type": "mysql", "hive_exec_orc_storage_strategy": "SPEED", "hive_log_dir": "/var/log/hive", "hive_pid_dir": "/var/run/hive", "hive_security_authorization": "None", "hive_timeline_logging_enabled": "true", "hive_txn_acid": "off", "hive_user": "hive", "hive_user_nofile_limit": "32000", "hive_user_nproc_limit": "16000", "webhcat_user": "hcat" }, "hive-site": { "ambari.hive.db.schema.name": "hive", "atlas.cluster.name": "default", "atlas.hook.hive.maxThreads": "1", "atlas.hook.hive.minThreads": "1", "atlas.rest.address": "", "datanucleus.autoCreateSchema": "false", "datanucleus.cache.level2.type": "none", "datanucleus.fixedDatastore": "true", "hive.auto.convert.join": "true", "hive.auto.convert.join.noconditionaltask": "true", "hive.auto.convert.join.noconditionaltask.size": "715827882", "hive.auto.convert.sortmerge.join": "true", "hive.auto.convert.sortmerge.join.to.mapjoin": "false", "hive.cbo.enable": "true", "hive.cli.print.header": "false", "hive.cluster.delegation.token.store.class": "org.apache.hadoop.hive.thrift.ZooKeeperTokenStore", "hive.cluster.delegation.token.store.zookeeper.connectString": "%HOSTGROUP::host_group_1%:2181", "hive.cluster.delegation.token.store.zookeeper.znode": "/hive/cluster/delegation", "hive.compactor.abortedtxn.threshold": "1000", "hive.compactor.check.interval": "300L", "hive.compactor.delta.num.threshold": "10", "hive.compactor.delta.pct.threshold": "0.1f", "hive.compactor.initiator.on": "false", "hive.compactor.worker.threads": "0", "hive.compactor.worker.timeout": "86400L", "hive.compute.query.using.stats": "true", "hive.conf.restricted.list": "hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role", "hive.convert.join.bucket.mapjoin.tez": "false", "hive.default.fileformat": "TextFile", "hive.default.fileformat.managed": "TextFile", "hive.enforce.bucketing": "false", "hive.enforce.sorting": "true", "hive.enforce.sortmergebucketmapjoin": "true", "hive.exec.compress.intermediate": "false", "hive.exec.compress.output": "false", "hive.exec.dynamic.partition": "true", "hive.exec.dynamic.partition.mode": "strict", "hive.exec.failure.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook", "hive.exec.max.created.files": "100000", "hive.exec.max.dynamic.partitions": "5000", "hive.exec.max.dynamic.partitions.pernode": "2000", "hive.exec.orc.compression.strategy": "SPEED", "hive.exec.orc.default.compress": "ZLIB", "hive.exec.orc.default.stripe.size": "67108864", "hive.exec.orc.encoding.strategy": "SPEED", "hive.exec.parallel": "false", "hive.exec.parallel.thread.number": "8", "hive.exec.post.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.atlas.hive.hook.HiveHook", "hive.exec.pre.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook", "hive.exec.reducers.bytes.per.reducer": "67108864", "hive.exec.reducers.max": "1009", "hive.exec.scratchdir": "/tmp/hive", "hive.exec.submit.local.task.via.child": "true", "hive.exec.submitviachild": "false", "hive.execution.engine": "tez", "hive.fetch.task.aggr": "false", "hive.fetch.task.conversion": "more", "hive.fetch.task.conversion.threshold": "1073741824", "hive.limit.optimize.enable": "true", "hive.limit.pushdown.memory.usage": "0.04", "hive.map.aggr": "true", "hive.map.aggr.hash.force.flush.memory.threshold": "0.9", "hive.map.aggr.hash.min.reduction": "0.5", "hive.map.aggr.hash.percentmemory": "0.5", "hive.mapjoin.bucket.cache.size": "10000", "hive.mapjoin.optimized.hashtable": "true", "hive.mapred.reduce.tasks.speculative.execution": "false", "hive.merge.mapfiles": "true", "hive.merge.mapredfiles": "false", "hive.merge.orcfile.stripe.level": "true", "hive.merge.rcfile.block.level": "true", "hive.merge.size.per.task": "256000000", "hive.merge.smallfiles.avgsize": "16000000", "hive.merge.tezfiles": "false", "hive.metastore.authorization.storage.checks": "false", "hive.metastore.cache.pinobjtypes": "Table,Database,Type,FieldSchema,Order", "hive.metastore.client.connect.retry.delay": "5s", "hive.metastore.client.socket.timeout": "1800s", "hive.metastore.connect.retries": "24", "hive.metastore.execute.setugi": "true", "hive.metastore.failure.retries": "24", "hive.metastore.kerberos.keytab.file": "/etc/security/keytabs/hive.service.keytab", "hive.metastore.kerberos.principal": "hive/_HOST@EXAMPLE.COM", "hive.metastore.pre.event.listeners": "org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener", "hive.metastore.sasl.enabled": "false", "hive.metastore.server.max.threads": "100000", "hive.metastore.uris": "thrift://%HOSTGROUP::host_group_1%:9083", "hive.metastore.warehouse.dir": "/apps/hive/warehouse", "hive.optimize.bucketmapjoin": "true", "hive.optimize.bucketmapjoin.sortedmerge": "false", "hive.optimize.constant.propagation": "true", "hive.optimize.index.filter": "true", "hive.optimize.metadataonly": "true", "hive.optimize.null.scan": "true", "hive.optimize.reducededuplication": "true", "hive.optimize.reducededuplication.min.reducer": "4", "hive.optimize.sort.dynamic.partition": "false", "hive.orc.compute.splits.num.threads": "10", "hive.orc.splits.include.file.footer": "false", "hive.prewarm.enabled": "false", "hive.prewarm.numcontainers": "3", "hive.security.authenticator.manager": "org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator", "hive.security.authorization.enabled": "false", "hive.security.authorization.manager": "org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory", "hive.security.metastore.authenticator.manager": "org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator", "hive.security.metastore.authorization.auth.reads": "true", "hive.security.metastore.authorization.manager": "org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider", "hive.server2.allow.user.substitution": "true", "hive.server2.authentication": "NONE", "hive.server2.authentication.spnego.keytab": "HTTP/_HOST@EXAMPLE.COM", "hive.server2.authentication.spnego.principal": "/etc/security/keytabs/spnego.service.keytab", "hive.server2.enable.doAs": "true", "hive.server2.logging.operation.enabled": "true", "hive.server2.logging.operation.log.location": "/tmp/hive/operation_logs", "hive.server2.support.dynamic.service.discovery": "true", "hive.server2.table.type.mapping": "CLASSIC", "hive.server2.tez.default.queues": "default", "hive.server2.tez.initialize.default.sessions": "false", "hive.server2.tez.sessions.per.default.queue": "1", "hive.server2.thrift.http.path": "cliservice", "hive.server2.thrift.http.port": "10001", "hive.server2.thrift.max.worker.threads": "500", "hive.server2.thrift.port": "10000", "hive.server2.thrift.sasl.qop": "auth", "hive.server2.transport.mode": "binary", "hive.server2.use.SSL": "false", "hive.server2.zookeeper.namespace": "hiveserver2", "hive.smbjoin.cache.rows": "10000", "hive.stats.autogather": "true", "hive.stats.dbclass": "fs", "hive.stats.fetch.column.stats": "true", "hive.stats.fetch.partition.stats": "true", "hive.support.concurrency": "false", "hive.tez.auto.reducer.parallelism": "true", "hive.tez.container.size": "2048", "hive.tez.cpu.vcores": "-1", "hive.tez.dynamic.partition.pruning": "true", "hive.tez.dynamic.partition.pruning.max.data.size": "104857600", "hive.tez.dynamic.partition.pruning.max.event.size": "1048576", "hive.tez.input.format": "org.apache.hadoop.hive.ql.io.HiveInputFormat", "hive.tez.java.opts": "-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps", "hive.tez.log.level": "INFO", "hive.tez.max.partition.factor": "2.0", "hive.tez.min.partition.factor": "0.25", "hive.tez.smb.number.waves": "0.5", "hive.txn.manager": "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager", "hive.txn.max.open.batch": "1000", "hive.txn.timeout": "300", "hive.user.install.directory": "/user/", "hive.vectorized.execution.enabled": "true", "hive.vectorized.execution.reduce.enabled": "false", "hive.vectorized.groupby.checkinterval": "4096", "hive.vectorized.groupby.flush.percent": "0.1", "hive.vectorized.groupby.maxentries": "100000", "hive.zookeeper.client.port": "2181", "hive.zookeeper.namespace": "hive_zookeeper_namespace", "hive.zookeeper.quorum": "%HOSTGROUP::host_group_1%:2181", "javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver", "javax.jdo.option.ConnectionURL": "jdbc:mysql://%HOSTGROUP::host_group_1%/hive?createDatabaseIfNotExist=true", "javax.jdo.option.ConnectionUserName": "hive" }, "hiveserver2-site": { "hive.security.authorization.enabled": "false" }, "hst-agent-conf": { "agent.loglevel": "INFO", "agent.tmp_dir": "/var/lib/smartsense/hst-agent/data/tmp", "bundle.compress_captured_log_locally": "false", "bundle.logs_to_capture": "(.*).log$|(.*).log.1$|(.*).out$", "management.patch.auto.apply.enabled": "true", "server.connection_retry_count": "50", "server.connection_retry_interval": "5", "upload.retry_count": "50", "upload.retry_interval": "15" }, "hst-common-conf": { "agent.capture.level": "L2", "security.anonymization.enabled": "true" }, "hst-log4j": { "hst-log4j-content": "# Copyright 2014 The Apache Software Foundation # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # \"License\"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an \"AS IS\" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Define some default values that can be overridden by system properties # Root logger option log4j.rootLogger=INFO,file log4j.appender.file=org.apache.log4j.RollingFileAppender log4j.appender.file.File=/var/log/hst/${log.file.name} log4j.appender.file.MaxFileSize=80MB log4j.appender.file.MaxBackupIndex=60 log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%d{DATE} %5p [%t] %c{1}:%L - %m%n # HST logger log4j.logger.com.hortonworks=DEBUG com.github.oxo42.stateless4j=WARN log4j.logger.com.sun.jersey=WARN log4j.logger.org.eclipse.jetty.server=INFO" }, "hst-server-conf": { "agent.request.processing.timeout": "1800", "agent.request.syncup.interval": "60", "bundle.monitor.interval": "20", "client.password_less_ssh.enabled": "false", "client.threadpool.size.max": "50", "customer.account.name": "", "customer.notification.email": "", "customer.smartsense.id": "", "gateway.enabled": "false", "gateway.host": "Please Specify", "gateway.port": "9451", "gateway.registration.port": "9450", "server.http.session.inactive_timeout": "1800", "server.min_required_storage": "1", "server.port": "9000", "server.storage.dir": "/var/lib/smartsense/hst-server/data", "server.tmp.dir": "/var/lib/smartsense/hst-server/tmp" }, "kafka-broker": { "auto.create.topics.enable": "true", "auto.leader.rebalance.enable": "true", "compression.type": "producer", "controlled.shutdown.enable": "true", "controlled.shutdown.max.retries": "3", "controlled.shutdown.retry.backoff.ms": "5000", "controller.message.queue.size": "10", "controller.socket.timeout.ms": "30000", "default.replication.factor": "1", "delete.topic.enable": "false", "external.kafka.metrics.exclude.prefix": "kafka.network.RequestMetrics,kafka.server.DelayedOperationPurgatory,kafka.server.BrokerTopicMetrics.BytesRejectedPerSec", "external.kafka.metrics.include.prefix": "kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetCommit.98percentile,kafka.network.RequestMetrics.ResponseQueueTimeMs.request.Offsets.95percentile,kafka.network.RequestMetrics.ResponseSendTimeMs.request.Fetch.95percentile,kafka.network.RequestMetrics.RequestsPerSec.request", "fetch.purgatory.purge.interval.requests": "10000", "kafka.ganglia.metrics.group": "kafka", "kafka.ganglia.metrics.port": "8671", "kafka.ganglia.metrics.reporter.enabled": "true", "kafka.timeline.metrics.host": "{{metric_collector_host}}", "kafka.timeline.metrics.maxRowCacheSize": "10000", "kafka.timeline.metrics.port": "{{metric_collector_port}}", "kafka.timeline.metrics.reporter.enabled": "true", "kafka.timeline.metrics.reporter.sendInterval": "5900", "leader.imbalance.check.interval.seconds": "300", "leader.imbalance.per.broker.percentage": "10", "listeners": "PLAINTEXT://localhost:6667", "log.cleanup.interval.mins": "10", "log.dirs": "/kafka-logs", "log.index.interval.bytes": "4096", "log.index.size.max.bytes": "10485760", "log.retention.bytes": "-1", "log.retention.hours": "168", "log.roll.hours": "168", "log.segment.bytes": "1073741824", "message.max.bytes": "1000000", "min.insync.replicas": "1", "num.io.threads": "8", "num.network.threads": "3", "num.partitions": "1", "num.recovery.threads.per.data.dir": "1", "num.replica.fetchers": "1", "offset.metadata.max.bytes": "4096", "offsets.commit.required.acks": "-1", "offsets.commit.timeout.ms": "5000", "offsets.load.buffer.size": "5242880", "offsets.retention.check.interval.ms": "600000", "offsets.retention.minutes": "86400000", "offsets.topic.compression.codec": "0", "offsets.topic.num.partitions": "50", "offsets.topic.replication.factor": "3", "offsets.topic.segment.bytes": "104857600", "producer.purgatory.purge.interval.requests": "10000", "queued.max.requests": "500", "replica.fetch.max.bytes": "1048576", "replica.fetch.min.bytes": "1", "replica.fetch.wait.max.ms": "500", "replica.high.watermark.checkpoint.interval.ms": "5000", "replica.lag.max.messages": "4000", "replica.lag.time.max.ms": "10000", "replica.socket.receive.buffer.bytes": "65536", "replica.socket.timeout.ms": "30000", "socket.receive.buffer.bytes": "102400", "socket.request.max.bytes": "104857600", "socket.send.buffer.bytes": "102400", "zookeeper.connect": "%HOSTGROUP::host_group_1%:2181", "zookeeper.connection.timeout.ms": "25000", "zookeeper.session.timeout.ms": "30000", "zookeeper.sync.time.ms": "2000" }, "kafka-env": { "is_supported_kafka_ranger": "true", "kafka_log_dir": "/var/log/kafka", "kafka_pid_dir": "/var/run/kafka", "kafka_user": "kafka", "kafka_user_nofile_limit": "32000", "kafka_user_nproc_limit": "16000" }, "knox-env": { "knox_group": "knox", "knox_master_secret": "SECRET:knox-env:1:knox_master_secret", "knox_pid_dir": "/var/run/knox", "knox_user": "knox" }, "mahout-env": { "mahout_user": "mahout" }, "mapred-env": { "jobhistory_heapsize": "900", "mapred_log_dir_prefix": "/var/log/hadoop-mapreduce", "mapred_pid_dir_prefix": "/var/run/hadoop-mapreduce", "mapred_user": "mapred", "mapred_user_nofile_limit": "32768", "mapred_user_nproc_limit": "65536" }, "mapred-site": { "mapreduce.admin.map.child.java.opts": "-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "mapreduce.admin.reduce.child.java.opts": "-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "mapreduce.admin.user.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "mapreduce.am.max-attempts": "2", "mapreduce.application.classpath": "$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure", "mapreduce.application.framework.path": "/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework", "mapreduce.cluster.administrators": "hadoop", "mapreduce.framework.name": "yarn", "mapreduce.job.counters.max": "130", "mapreduce.job.emit-timeline-data": "false", "mapreduce.job.reduce.slowstart.completedmaps": "0.05", "mapreduce.jobhistory.address": "%HOSTGROUP::host_group_1%:10020", "mapreduce.jobhistory.bind-host": "0.0.0.0", "mapreduce.jobhistory.done-dir": "/mr-history/done", "mapreduce.jobhistory.intermediate-done-dir": "/mr-history/tmp", "mapreduce.jobhistory.recovery.enable": "true", "mapreduce.jobhistory.recovery.store.class": "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService", "mapreduce.jobhistory.recovery.store.leveldb.path": "/hadoop/mapreduce/jhs", "mapreduce.jobhistory.webapp.address": "%HOSTGROUP::host_group_1%:19888", "mapreduce.map.java.opts": "-Xmx1638m", "mapreduce.map.log.level": "INFO", "mapreduce.map.memory.mb": "2048", "mapreduce.map.output.compress": "false", "mapreduce.map.sort.spill.percent": "0.7", "mapreduce.map.speculative": "false", "mapreduce.output.fileoutputformat.compress": "false", "mapreduce.output.fileoutputformat.compress.type": "BLOCK", "mapreduce.reduce.input.buffer.percent": "0.0", "mapreduce.reduce.java.opts": "-Xmx3276m", "mapreduce.reduce.log.level": "INFO", "mapreduce.reduce.memory.mb": "4096", "mapreduce.reduce.shuffle.fetch.retry.enabled": "1", "mapreduce.reduce.shuffle.fetch.retry.interval-ms": "1000", "mapreduce.reduce.shuffle.fetch.retry.timeout-ms": "30000", "mapreduce.reduce.shuffle.input.buffer.percent": "0.7", "mapreduce.reduce.shuffle.merge.percent": "0.66", "mapreduce.reduce.shuffle.parallelcopies": "30", "mapreduce.reduce.speculative": "false", "mapreduce.shuffle.port": "13562", "mapreduce.task.io.sort.factor": "100", "mapreduce.task.io.sort.mb": "1146", "mapreduce.task.timeout": "300000", "yarn.app.mapreduce.am.admin-command-opts": "-Dhdp.version=${hdp.version}", "yarn.app.mapreduce.am.command-opts": "-Xmx1638m -Dhdp.version=${hdp.version}", "yarn.app.mapreduce.am.log.level": "INFO", "yarn.app.mapreduce.am.resource.mb": "2048", "yarn.app.mapreduce.am.staging-dir": "/user" }, "oozie-env": { "oozie_admin_port": "11001", "oozie_admin_users": "{oozie_user}, oozie-admin", "oozie_data_dir": "/hadoop/oozie/data", "oozie_database": "New Derby Database", "oozie_derby_database": "Derby", "oozie_heapsize": "2048m", "oozie_hostname": "%HOSTGROUP::host_group_1%", "oozie_log_dir": "/var/log/oozie", "oozie_permsize": "256m", "oozie_pid_dir": "/var/run/oozie", "oozie_user": "oozie" }, "oozie-site": { "oozie.authentication.simple.anonymous.allowed": "true", "oozie.authentication.type": "simple", "oozie.base.url": "http://%HOSTGROUP::host_group_1%:11000/oozie", "oozie.credentials.credentialclasses": "hcat=org.apache.oozie.action.hadoop.HCatCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials", "oozie.db.schema.name": "oozie", "oozie.service.AuthorizationService.security.enabled": "true", "oozie.service.ELService.ext.functions.coord-action-create": "now=org.apache.oozie.extensions.OozieELExtensions#ph2_now, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_currentWeek, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_lastWeek, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-action-create-inst": "now=org.apache.oozie.extensions.OozieELExtensions#ph2_now_inst, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today_inst, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday_inst, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_currentWeek_inst, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_lastWeek_inst, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth_inst, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth_inst, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear_inst, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear_inst, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-action-start": "now=org.apache.oozie.extensions.OozieELExtensions#ph2_now, today=org.apache.oozie.extensions.OozieELExtensions#ph2_today, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_currentWeek, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph2_lastWeek, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear, latest=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latest, future=org.apache.oozie.coord.CoordELFunctions#ph3_coord_future, dataIn=org.apache.oozie.extensions.OozieELExtensions#ph3_dataIn, instanceTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime, dateOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateOffset, formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-job-submit-data": "now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo, today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_currentWeek_echo, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_lastWeek_echo, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo, dataIn=org.apache.oozie.extensions.OozieELExtensions#ph1_dataIn_echo, instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_wrap, formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-job-submit-instances": "now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo, today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo, yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo, currentWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_currentWeek_echo, lastWeek=org.apache.oozie.extensions.OozieELExtensions#ph1_lastWeek_echo, currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo, lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo, currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo, lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo, formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo", "oozie.service.ELService.ext.functions.coord-sla-create": "instanceTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_nominalTime, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.ELService.ext.functions.coord-sla-submit": "instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed, user=org.apache.oozie.coord.CoordELFunctions#coord_user", "oozie.service.HadoopAccessorService.hadoop.configurations": "*={{hadoop_conf_dir}}", "oozie.service.HadoopAccessorService.kerberos.enabled": "false", "oozie.service.HadoopAccessorService.supported.filesystems": "*", "oozie.service.JPAService.jdbc.driver": "org.apache.derby.jdbc.EmbeddedDriver", "oozie.service.JPAService.jdbc.username": "oozie", "oozie.service.ProxyUserService.proxyuser.falcon.groups": "*", "oozie.service.ProxyUserService.proxyuser.falcon.hosts": "*", "oozie.service.URIHandlerService.uri.handlers": "org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler", "oozie.services.ext": "org.apache.oozie.service.JMSAccessorService,org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService" }, "ranger-hdfs-plugin-properties": { "REPOSITORY_CONFIG_USERNAME": "hadoop", "common.name.for.certificate": "", "hadoop.rpc.protection": "", "policy_user": "ambari-qa", "ranger-hdfs-plugin-enabled": "No" }, "spark-defaults": { "spark.eventLog.dir": "hdfs:///spark-history", "spark.eventLog.enabled": "true", "spark.history.fs.logDirectory": "hdfs:///spark-history", "spark.history.kerberos.keytab": "none", "spark.history.kerberos.principal": "none", "spark.history.provider": "org.apache.spark.deploy.history.FsHistoryProvider", "spark.history.ui.port": "18080", "spark.yarn.containerLauncherMaxThreads": "25", "spark.yarn.driver.memoryOverhead": "384", "spark.yarn.executor.memoryOverhead": "384", "spark.yarn.historyServer.address": "{{spark_history_server_host}}:{{spark_history_ui_port}}", "spark.yarn.max.executor.failures": "3", "spark.yarn.preserve.staging.files": "false", "spark.yarn.queue": "default", "spark.yarn.scheduler.heartbeat.interval-ms": "5000", "spark.yarn.submit.file.replication": "3" }, "spark-env": { "spark_group": "spark", "spark_log_dir": "/var/log/spark", "spark_pid_dir": "/var/run/spark", "spark_thrift_cmd_opts": "", "spark_user": "spark" }, "spark-hive-site-override": { "hive.metastore.client.connect.retry.delay": "5", "hive.metastore.client.socket.timeout": "1800", "hive.server2.enable.doAs": "false", "hive.server2.thrift.port": "10015", "hive.server2.transport.mode": "binary" }, "spark-thrift-fairscheduler": { "fairscheduler_content": " FAIR 1 2 " }, "spark-thrift-sparkconf": { "spark.dynamicAllocation.enabled": "true", "spark.dynamicAllocation.initialExecutors": "0", "spark.dynamicAllocation.maxExecutors": "10", "spark.dynamicAllocation.minExecutors": "0", "spark.eventLog.dir": "{{spark_history_dir}}", "spark.eventLog.enabled": "true", "spark.executor.memory": "1g", "spark.history.fs.logDirectory": "{{spark_history_dir}}", "spark.history.provider": "org.apache.spark.deploy.history.FsHistoryProvider", "spark.master": "{{spark_thrift_master}}", "spark.scheduler.allocation.file": "{{spark_conf}}/spark-thrift-fairscheduler.xml", "spark.scheduler.mode": "FAIR", "spark.shuffle.service.enabled": "true", "spark.yarn.am.memory": "512m", "spark.yarn.queue": "default" }, "sqoop-env": { "jdbc_drivers": "", "sqoop_user": "sqoop" }, "ssl-client": { "ssl.client.keystore.location": "/etc/security/clientKeys/keystore.jks", "ssl.client.keystore.type": "jks", "ssl.client.truststore.location": "/etc/security/clientKeys/all.jks", "ssl.client.truststore.reload.interval": "10000", "ssl.client.truststore.type": "jks" }, "ssl-server": { "ssl.server.keystore.location": "/etc/security/serverKeys/keystore.jks", "ssl.server.keystore.type": "jks", "ssl.server.truststore.location": "/etc/security/serverKeys/all.jks", "ssl.server.truststore.reload.interval": "10000", "ssl.server.truststore.type": "jks" }, "storm-env": { "jmxremote_port": "56431", "nimbus_seeds_supported": "true", "storm_log_dir": "/var/log/storm", "storm_logs_supported": "true", "storm_pid_dir": "/var/run/storm", "storm_user": "storm" }, "storm-site": { "_storm.min.ruid": "null", "_storm.thrift.nonsecure.transport": "backtype.storm.security.auth.SimpleTransportPlugin", "_storm.thrift.secure.transport": "backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin", "dev.zookeeper.path": "/tmp/dev-storm-zookeeper", "drpc.childopts": "-Xmx768m _JAAS_PLACEHOLDER", "drpc.invocations.port": "3773", "drpc.port": "3772", "drpc.queue.size": "128", "drpc.request.timeout.secs": "600", "drpc.worker.threads": "64", "drpc_server_host": "[%HOSTGROUP::host_group_1%]", "java.library.path": "/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib", "logviewer.appender.name": "A1", "logviewer.childopts": "-Xmx128m _JAAS_PLACEHOLDER", "logviewer.port": "8000", "nimbus.cleanup.inbox.freq.secs": "600", "nimbus.file.copy.expiration.secs": "600", "nimbus.inbox.jar.expiration.secs": "3600", "nimbus.monitor.freq.secs": "120", "nimbus.reassign": "true", "nimbus.seeds": "['%HOSTGROUP::host_group_1%']", "nimbus.supervisor.timeout.secs": "60", "nimbus.task.launch.secs": "120", "nimbus.task.timeout.secs": "30", "nimbus.thrift.max_buffer_size": "1048576", "nimbus.thrift.port": "6627", "nimbus.topology.validator": "backtype.storm.nimbus.DefaultTopologyValidator", "nimbus_hosts": "[%HOSTGROUP::host_group_1%]", "storm.cluster.mode": "distributed", "storm.local.dir": "/hadoop/storm", "storm.local.mode.zmq": "false", "storm.log.dir": "{{log_dir}}", "storm.messaging.netty.buffer_size": "5242880", "storm.messaging.netty.client_worker_threads": "1", "storm.messaging.netty.max_retries": "30", "storm.messaging.netty.max_wait_ms": "1000", "storm.messaging.netty.min_wait_ms": "100", "storm.messaging.netty.server_worker_threads": "1", "storm.messaging.transport": "backtype.storm.messaging.netty.Context", "storm.thrift.transport": "{{storm_thrift_transport}}", "storm.zookeeper.connection.timeout": "15000", "storm.zookeeper.port": "2181", "storm.zookeeper.retry.interval": "1000", "storm.zookeeper.retry.intervalceiling.millis": "30000", "storm.zookeeper.retry.times": "5", "storm.zookeeper.root": "/storm", "storm.zookeeper.servers": "['%HOSTGROUP::host_group_1%']", "storm.zookeeper.session.timeout": "20000", "storm_ui_server_host": "[%HOSTGROUP::host_group_1%]", "supervisor.heartbeat.frequency.secs": "5", "supervisor.monitor.frequency.secs": "3", "supervisor.slots.ports": "[6700, 6701]", "supervisor.worker.start.timeout.secs": "120", "supervisor.worker.timeout.secs": "30", "supervisor_hosts": "[%HOSTGROUP::host_group_1%]", "task.heartbeat.frequency.secs": "3", "task.refresh.poll.secs": "10", "topology.acker.executors": "null", "topology.builtin.metrics.bucket.size.secs": "60", "topology.debug": "false", "topology.disruptor.wait.strategy": "com.lmax.disruptor.BlockingWaitStrategy", "topology.enable.message.timeouts": "true", "topology.error.throttle.interval.secs": "10", "topology.executor.receive.buffer.size": "1024", "topology.executor.send.buffer.size": "1024", "topology.fall.back.on.java.serialization": "true", "topology.kryo.factory": "backtype.storm.serialization.DefaultKryoFactory", "topology.max.error.report.per.interval": "5", "topology.max.replication.wait.time.sec": "{{actual_topology_max_replication_wait_time_sec}}", "topology.max.replication.wait.time.sec.default": "60", "topology.max.spout.pending": "1000", "topology.max.task.parallelism": "null", "topology.message.timeout.secs": "30", "topology.min.replication.count": "{{actual_topology_min_replication_count}}", "topology.min.replication.count.default": "1", "topology.optimize": "true", "topology.receiver.buffer.size": "8", "topology.skip.missing.kryo.registrations": "false", "topology.sleep.spout.wait.strategy.time.ms": "1", "topology.spout.wait.strategy": "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.state.synchronization.timeout.secs": "60", "topology.stats.sample.rate": "0.05", "topology.tick.tuple.freq.secs": "null", "topology.transfer.buffer.size": "1024", "topology.trident.batch.emit.interval.millis": "500", "topology.tuple.serializer": "backtype.storm.serialization.types.ListDelegateSerializer", "topology.worker.childopts": "null", "topology.worker.shared.thread.pool.size": "4", "topology.workers": "1", "transactional.zookeeper.port": "null", "transactional.zookeeper.root": "/transactional", "transactional.zookeeper.servers": "null", "ui.childopts": "-Xmx768m _JAAS_PLACEHOLDER", "ui.filter": "null", "ui.port": "8744", "worker.heartbeat.frequency.secs": "1", "zmq.hwm": "0", "zmq.linger.millis": "5000", "zmq.threads": "1" }, "tez-env": { "tez_user": "tez" }, "tez-site": { "tez.am.am-rm.heartbeat.interval-ms.max": "250", "tez.am.container.idle.release-timeout-max.millis": "20000", "tez.am.container.idle.release-timeout-min.millis": "10000", "tez.am.container.reuse.enabled": "true", "tez.am.container.reuse.locality.delay-allocation-millis": "250", "tez.am.container.reuse.non-local-fallback.enabled": "false", "tez.am.container.reuse.rack-fallback.enabled": "true", "tez.am.launch.cluster-default.cmd-opts": "-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "tez.am.launch.cmd-opts": "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB", "tez.am.launch.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "tez.am.log.level": "INFO", "tez.am.max.app.attempts": "2", "tez.am.maxtaskfailures.per.node": "10", "tez.am.resource.memory.mb": "4096", "tez.am.tez-ui.history-url.template": "__HISTORY_URL_BASE__?viewPath=%2F%23%2Ftez-app%2F__APPLICATION_ID__", "tez.am.view-acls": "*", "tez.cluster.additional.classpath.prefix": "/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure", "tez.counters.max": "10000", "tez.counters.max.groups": "3000", "tez.generate.debug.artifacts": "false", "tez.grouping.max-size": "1073741824", "tez.grouping.min-size": "16777216", "tez.grouping.split-waves": "1.7", "tez.history.logging.service.class": "org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService", "tez.lib.uris": "/hdp/apps/${hdp.version}/tez/tez.tar.gz", "tez.runtime.compress": "true", "tez.runtime.compress.codec": "org.apache.hadoop.io.compress.SnappyCodec", "tez.runtime.convert.user-payload.to.history-text": "false", "tez.runtime.io.sort.mb": "819", "tez.runtime.optimize.local.fetch": "true", "tez.runtime.pipelined.sorter.sort.threads": "2", "tez.runtime.sorter.class": "PIPELINED", "tez.runtime.unordered.output.buffer.size-mb": "153", "tez.session.am.dag.submit.timeout.secs": "600", "tez.session.client.timeout.secs": "-1", "tez.shuffle-vertex-manager.max-src-fraction": "0.4", "tez.shuffle-vertex-manager.min-src-fraction": "0.2", "tez.staging-dir": "/tmp/${user.name}/staging", "tez.task.am.heartbeat.counter.interval-ms.max": "4000", "tez.task.generate.counters.per.io": "true", "tez.task.get-task.sleep.interval-ms.max": "200", "tez.task.launch.cluster-default.cmd-opts": "-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "tez.task.launch.cmd-opts": "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB", "tez.task.launch.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "tez.task.max-events-per-heartbeat": "500", "tez.task.resource.memory.mb": "2048", "tez.use.cluster.hadoop-libs": "false" }, "webhcat-site": { "templeton.exec.timeout": "60000", "templeton.hadoop": "/usr/hdp/${hdp.version}/hadoop/bin/hadoop", "templeton.hadoop.conf.dir": "/etc/hadoop/conf", "templeton.hcat": "/usr/hdp/${hdp.version}/hive/bin/hcat", "templeton.hcat.home": "hive.tar.gz/hive/hcatalog", "templeton.hive.archive": "hdfs:///hdp/apps/${hdp.version}/hive/hive.tar.gz", "templeton.hive.extra.files": "/usr/hdp/${hdp.version}/tez/conf/tez-site.xml,/usr/hdp/${hdp.version}/tez,/usr/hdp/${hdp.version}/tez/lib", "templeton.hive.home": "hive.tar.gz/hive", "templeton.hive.path": "hive.tar.gz/hive/bin/hive", "templeton.hive.properties": "hive.metastore.local=false,hive.metastore.uris=thrift://%HOSTGROUP::host_group_1%:9083,hive.metastore.sasl.enabled=false,hive.metastore.execute.setugi=true", "templeton.jar": "/usr/hdp/${hdp.version}/hive/share/webhcat/svr/lib/hive-webhcat-*.jar", "templeton.libjars": "/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar", "templeton.override.enabled": "false", "templeton.pig.archive": "hdfs:///hdp/apps/${hdp.version}/pig/pig.tar.gz", "templeton.pig.path": "pig.tar.gz/pig/bin/pig", "templeton.port": "50111", "templeton.python": "${env.PYTHON_CMD}", "templeton.sqoop.archive": "hdfs:///hdp/apps/${hdp.version}/sqoop/sqoop.tar.gz", "templeton.sqoop.home": "sqoop.tar.gz/sqoop", "templeton.sqoop.path": "sqoop.tar.gz/sqoop/bin/sqoop", "templeton.storage.class": "org.apache.hive.hcatalog.templeton.tool.ZooKeeperStorage", "templeton.streaming.jar": "hdfs:///hdp/apps/${hdp.version}/mapreduce/hadoop-streaming.jar", "templeton.zookeeper.hosts": "%HOSTGROUP::host_group_1%:2181" }, "yarn-env": { "apptimelineserver_heapsize": "1024", "is_supported_yarn_ranger": "true", "min_user_id": "1000", "nodemanager_heapsize": "1024", "resourcemanager_heapsize": "1024", "yarn_cgroups_enabled": "false", "yarn_heapsize": "1024", "yarn_log_dir_prefix": "/var/log/hadoop-yarn", "yarn_pid_dir_prefix": "/var/run/hadoop-yarn", "yarn_user": "yarn", "yarn_user_nofile_limit": "32768", "yarn_user_nproc_limit": "65536" }, "yarn-site": { "hadoop.registry.rm.enabled": "true", "hadoop.registry.zk.quorum": "%HOSTGROUP::host_group_1%:2181", "yarn.acl.enable": "false", "yarn.admin.acl": "yarn", "yarn.application.classpath": "$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*", "yarn.client.nodemanager-connect.max-wait-ms": "60000", "yarn.client.nodemanager-connect.retry-interval-ms": "10000", "yarn.http.policy": "HTTP_ONLY", "yarn.log-aggregation-enable": "true", "yarn.log-aggregation.retain-seconds": "2592000", "yarn.log.server.url": "http://%HOSTGROUP::host_group_1%:19888/jobhistory/logs", "yarn.node-labels.enabled": "false", "yarn.node-labels.fs-store.retry-policy-spec": "2000, 500", "yarn.node-labels.fs-store.root-dir": "/system/yarn/node-labels", "yarn.nodemanager.address": "0.0.0.0:45454", "yarn.nodemanager.admin-env": "MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX", "yarn.nodemanager.aux-services": "mapreduce_shuffle,spark_shuffle", "yarn.nodemanager.aux-services.mapreduce_shuffle.class": "org.apache.hadoop.mapred.ShuffleHandler", "yarn.nodemanager.aux-services.spark_shuffle.class": "org.apache.spark.network.yarn.YarnShuffleService", "yarn.nodemanager.bind-host": "0.0.0.0", "yarn.nodemanager.container-executor.class": "org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor", "yarn.nodemanager.container-monitor.interval-ms": "3000", "yarn.nodemanager.delete.debug-delay-sec": "0", "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage": "90", "yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb": "1000", "yarn.nodemanager.disk-health-checker.min-healthy-disks": "0.25", "yarn.nodemanager.health-checker.interval-ms": "135000", "yarn.nodemanager.health-checker.script.timeout-ms": "60000", "yarn.nodemanager.linux-container-executor.cgroups.hierarchy": "hadoop-yarn", "yarn.nodemanager.linux-container-executor.cgroups.mount": "false", "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage": "false", "yarn.nodemanager.linux-container-executor.group": "hadoop", "yarn.nodemanager.linux-container-executor.resources-handler.class": "org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler", "yarn.nodemanager.local-dirs": "/hadoop/yarn/local", "yarn.nodemanager.log-aggregation.compression-type": "gz", "yarn.nodemanager.log-aggregation.debug-enabled": "false", "yarn.nodemanager.log-aggregation.num-log-files-per-app": "30", "yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds": "-1", "yarn.nodemanager.log-dirs": "/hadoop/yarn/log", "yarn.nodemanager.log.retain-second": "604800", "yarn.nodemanager.recovery.dir": "{{yarn_log_dir_prefix}}/nodemanager/recovery-state", "yarn.nodemanager.recovery.enabled": "true", "yarn.nodemanager.remote-app-log-dir": "/app-logs", "yarn.nodemanager.remote-app-log-dir-suffix": "logs", "yarn.nodemanager.resource.cpu-vcores": "25", "yarn.nodemanager.resource.memory-mb": "16384", "yarn.nodemanager.resource.percentage-physical-cpu-limit": "80", "yarn.nodemanager.vmem-check-enabled": "false", "yarn.nodemanager.vmem-pmem-ratio": "2.1", "yarn.resourcemanager.address": "%HOSTGROUP::host_group_1%:8050", "yarn.resourcemanager.admin.address": "%HOSTGROUP::host_group_1%:8141", "yarn.resourcemanager.am.max-attempts": "2", "yarn.resourcemanager.bind-host": "0.0.0.0", "yarn.resourcemanager.connect.max-wait.ms": "900000", "yarn.resourcemanager.connect.retry-interval.ms": "30000", "yarn.resourcemanager.fs.state-store.retry-policy-spec": "2000, 500", "yarn.resourcemanager.fs.state-store.uri": "", "yarn.resourcemanager.ha.enabled": "false", "yarn.resourcemanager.hostname": "%HOSTGROUP::host_group_1%", "yarn.resourcemanager.nodes.exclude-path": "/etc/hadoop/conf/yarn.exclude", "yarn.resourcemanager.recovery.enabled": "true", "yarn.resourcemanager.resource-tracker.address": "%HOSTGROUP::host_group_1%:8025", "yarn.resourcemanager.scheduler.address": "%HOSTGROUP::host_group_1%:8030", "yarn.resourcemanager.scheduler.class": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler", "yarn.resourcemanager.scheduler.monitor.enable": "false", "yarn.resourcemanager.state-store.max-completed-applications": "${yarn.resourcemanager.max-completed-applications}", "yarn.resourcemanager.store.class": "org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore", "yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size": "10", "yarn.resourcemanager.system-metrics-publisher.enabled": "true", "yarn.resourcemanager.webapp.address": "%HOSTGROUP::host_group_1%:8088", "yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled": "false", "yarn.resourcemanager.webapp.https.address": "%HOSTGROUP::host_group_1%:8090", "yarn.resourcemanager.work-preserving-recovery.enabled": "true", "yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms": "10000", "yarn.resourcemanager.zk-acl": "world:anyone:rwcda", "yarn.resourcemanager.zk-address": "%HOSTGROUP::host_group_1%:2181", "yarn.resourcemanager.zk-num-retries": "1000", "yarn.resourcemanager.zk-retry-interval-ms": "1000", "yarn.resourcemanager.zk-state-store.parent-path": "/rmstore", "yarn.resourcemanager.zk-timeout-ms": "10000", "yarn.scheduler.maximum-allocation-mb": "16384", "yarn.scheduler.maximum-allocation-vcores": "25", "yarn.scheduler.minimum-allocation-mb": "2048", "yarn.scheduler.minimum-allocation-vcores": "1", "yarn.timeline-service.address": "%HOSTGROUP::host_group_1%:10200", "yarn.timeline-service.bind-host": "0.0.0.0", "yarn.timeline-service.client.max-retries": "30", "yarn.timeline-service.client.retry-interval-ms": "1000", "yarn.timeline-service.enabled": "true", "yarn.timeline-service.entity-group-fs-store.active-dir": "/ats/active/", "yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds": "3600", "yarn.timeline-service.entity-group-fs-store.done-dir": "/ats/done/", "yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes": "org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl", "yarn.timeline-service.entity-group-fs-store.retain-seconds": "604800", "yarn.timeline-service.entity-group-fs-store.scan-interval-seconds": "60", "yarn.timeline-service.entity-group-fs-store.summary-store": "org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore", "yarn.timeline-service.generic-application-history.store-class": "org.apache.hadoop.yarn.server.applicationhistoryservice.NullApplicationHistoryStore", "yarn.timeline-service.http-authentication.simple.anonymous.allowed": "true", "yarn.timeline-service.http-authentication.type": "simple", "yarn.timeline-service.leveldb-state-store.path": "/hadoop/yarn/timeline", "yarn.timeline-service.leveldb-timeline-store.path": "/hadoop/yarn/timeline", "yarn.timeline-service.leveldb-timeline-store.read-cache-size": "104857600", "yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size": "10000", "yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size": "10000", "yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms": "300000", "yarn.timeline-service.recovery.enabled": "true", "yarn.timeline-service.state-store-class": "org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore", "yarn.timeline-service.store-class": "org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore", "yarn.timeline-service.ttl-enable": "true", "yarn.timeline-service.ttl-ms": "2678400000", "yarn.timeline-service.version": "1.5", "yarn.timeline-service.webapp.address": "%HOSTGROUP::host_group_1%:8188", "yarn.timeline-service.webapp.https.address": "%HOSTGROUP::host_group_1%:8190" }, "zoo.cfg": { "autopurge.purgeInterval": "24", "autopurge.snapRetainCount": "30", "clientPort": "2181", "dataDir": "/hadoop/zookeeper", "initLimit": "10", "syncLimit": "5", "tickTime": "2000" }, "zookeeper-env": { "zk_log_dir": "/var/log/zookeeper", "zk_pid_dir": "/var/run/zookeeper", "zk_user": "zookeeper" } } sahara-8.0.0/sahara/plugins/ambari/resources/configs-2.3.json0000666000175100017510000021752713245514472024041 0ustar zuulzuul00000000000000{ "accumulo-env": { "accumulo_gc_heapsize": "256", "accumulo_instance_name": "hdp-accumulo-instance", "accumulo_log_dir": "/var/log/accumulo", "accumulo_master_heapsize": "1024", "accumulo_monitor_bind_all": "false", "accumulo_monitor_heapsize": "1024", "accumulo_other_heapsize": "1024", "accumulo_pid_dir": "/var/run/accumulo", "accumulo_tserver_heapsize": "1536", "accumulo_user": "accumulo", "instance_secret": "pass", "server_content": "#! /usr/bin/env bash export HADOOP_PREFIX={{hadoop_prefix}} export HADOOP_CONF_DIR={{hadoop_conf_dir}} export JAVA_HOME={{java64_home}} export ZOOKEEPER_HOME={{zookeeper_home}} export ACCUMULO_PID_DIR={{pid_dir}} export ACCUMULO_LOG_DIR={{log_dir}} export ACCUMULO_CONF_DIR={{server_conf_dir}} export ACCUMULO_TSERVER_OPTS=\"-Xmx{{accumulo_tserver_heapsize}}m -Xms{{accumulo_tserver_heapsize}}m\" export ACCUMULO_MASTER_OPTS=\"-Xmx{{accumulo_master_heapsize}}m -Xms{{accumulo_master_heapsize}}m\" export ACCUMULO_MONITOR_OPTS=\"-Xmx{{accumulo_monitor_heapsize}}m -Xms{{accumulo_monitor_heapsize}}m\" export ACCUMULO_GC_OPTS=\"-Xmx{{accumulo_gc_heapsize}}m -Xms{{accumulo_gc_heapsize}}m\" export ACCUMULO_GENERAL_OPTS=\"-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -Djava.net.preferIPv4Stack=true ${ACCUMULO_GENERAL_OPTS}\" export ACCUMULO_OTHER_OPTS=\"-Xmx{{accumulo_other_heapsize}}m -Xms{{accumulo_other_heapsize}}m ${ACCUMULO_OTHER_OPTS}\" export ACCUMULO_MONITOR_BIND_ALL={{monitor_bind_str}} # what do when the JVM runs out of heap memory export ACCUMULO_KILL_CMD='kill -9 %p'" }, "accumulo-log4j": { "audit_log_level": "OFF", "debug_log_size": "512M", "debug_num_logs": "10", "info_log_size": "512M", "info_num_logs": "10", "monitor_forwarding_log_level": "WARN" }, "accumulo-site": { "gc.port.client": "50092", "general.classpaths": "$ACCUMULO_HOME/lib/accumulo-server.jar, $ACCUMULO_HOME/lib/accumulo-core.jar, $ACCUMULO_HOME/lib/accumulo-start.jar, $ACCUMULO_HOME/lib/accumulo-fate.jar, $ACCUMULO_HOME/lib/accumulo-proxy.jar, $ACCUMULO_HOME/lib/[^.].*.jar, $ZOOKEEPER_HOME/zookeeper[^.].*.jar, $HADOOP_CONF_DIR, /usr/hdp/current/hadoop-client/[^.].*.jar, /usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar, /usr/hdp/current/hadoop-hdfs-client/[^.].*.jar, /usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar, /usr/hdp/current/hadoop-yarn-client/[^.].*.jar, /usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar, /usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar,", "instance.volumes": "hdfs://%HOSTGROUP::host_group_1%:8020/apps/accumulo/data", "instance.zookeeper.host": "%HOSTGROUP::host_group_1%:2181", "instance.zookeeper.timeout": "30s", "master.port.client": "9999", "monitor.port.client": "50095", "monitor.port.log4j": "4560", "trace.port.client": "12234", "trace.user": "trace", "tserver.cache.data.size": "128M", "tserver.cache.index.size": "256M", "tserver.memory.maps.max": "1G", "tserver.memory.maps.native.enabled": "true", "tserver.port.client": "9997", "tserver.sort.buffer.size": "200M", "tserver.walog.max.size": "1G" }, "ams-env": { "ambari_metrics_user": "ams", "metrics_collector_heapsize": "512m", "metrics_collector_log_dir": "/var/log/ambari-metrics-collector", "metrics_collector_pid_dir": "/var/run/ambari-metrics-collector", "metrics_monitor_log_dir": "/var/log/ambari-metrics-monitor", "metrics_monitor_pid_dir": "/var/run/ambari-metrics-monitor" }, "ams-hbase-env": { "hbase_log_dir": "/var/log/ambari-metrics-collector", "hbase_master_heapsize": "1024m", "hbase_master_maxperm_size": "128m", "hbase_master_xmn_size": "256m", "hbase_pid_dir": "/var/run/ambari-metrics-collector/", "hbase_regionserver_heapsize": "1024m", "hbase_regionserver_xmn_ratio": "0.2", "max_open_files_limit": "32768", "regionserver_xmn_size": "256m" }, "ams-hbase-policy": { "security.admin.protocol.acl": "*", "security.client.protocol.acl": "*", "security.masterregion.protocol.acl": "*" }, "ams-hbase-security-site": { "ams.zookeeper.keytab": "", "ams.zookeeper.principal": "", "hadoop.security.authentication": "", "hbase.coprocessor.master.classes": "", "hbase.coprocessor.region.classes": "", "hbase.master.kerberos.principal": "", "hbase.master.keytab.file": "", "hbase.myclient.keytab": "", "hbase.myclient.principal": "", "hbase.regionserver.kerberos.principal": "", "hbase.regionserver.keytab.file": "", "hbase.security.authentication": "", "hbase.security.authorization": "", "hbase.zookeeper.property.authProvider.1": "", "hbase.zookeeper.property.jaasLoginRenew": "", "hbase.zookeeper.property.kerberos.removeHostFromPrincipal": "", "hbase.zookeeper.property.kerberos.removeRealmFromPrincipal": "", "zookeeper.znode.parent": "" }, "ams-hbase-site": { "hbase.client.scanner.caching": "10000", "hbase.client.scanner.timeout.period": "900000", "hbase.cluster.distributed": "false", "hbase.hregion.majorcompaction": "0", "hbase.hregion.memstore.block.multiplier": "4", "hbase.hregion.memstore.flush.size": "134217728", "hbase.hstore.blockingStoreFiles": "200", "hbase.hstore.flusher.count": "2", "hbase.local.dir": "${hbase.tmp.dir}/local", "hbase.master.info.bindAddress": "0.0.0.0", "hbase.master.info.port": "61310", "hbase.master.port": "61300", "hbase.master.wait.on.regionservers.mintostart": "1", "hbase.regionserver.global.memstore.lowerLimit": "0.4", "hbase.regionserver.global.memstore.upperLimit": "0.5", "hbase.regionserver.info.port": "61330", "hbase.regionserver.port": "61320", "hbase.regionserver.thread.compaction.large": "2", "hbase.regionserver.thread.compaction.small": "3", "hbase.replication": "false", "hbase.rootdir": "file:///var/lib/ambari-metrics-collector/hbase", "hbase.snapshot.enabled": "false", "hbase.tmp.dir": "/var/lib/ambari-metrics-collector/hbase-tmp", "hbase.zookeeper.leaderport": "61388", "hbase.zookeeper.peerport": "61288", "hbase.zookeeper.property.clientPort": "61181", "hbase.zookeeper.property.dataDir": "${hbase.tmp.dir}/zookeeper", "hbase.zookeeper.quorum": "{{zookeeper_quorum_hosts}}", "hbase_master_xmn_size": "128m", "hfile.block.cache.size": "0.3", "phoenix.groupby.maxCacheSize": "307200000", "phoenix.query.maxGlobalMemoryPercentage": "15", "phoenix.query.spoolThresholdBytes": "12582912", "phoenix.query.timeoutMs": "1200000", "phoenix.sequence.saltBuckets": "2", "phoenix.spool.directory": "${hbase.tmp.dir}/phoenix-spool", "zookeeper.session.timeout": "120000", "zookeeper.session.timeout.localHBaseCluster": "20000" }, "ams-site": { "phoenix.query.maxGlobalMemoryPercentage": "25", "phoenix.spool.directory": "/tmp", "timeline.metrics.aggregator.checkpoint.dir": "/var/lib/ambari-metrics-collector/checkpoint", "timeline.metrics.cluster.aggregator.daily.checkpointCutOffMultiplier": "1", "timeline.metrics.cluster.aggregator.daily.disabled": "false", "timeline.metrics.cluster.aggregator.daily.interval": "86400", "timeline.metrics.cluster.aggregator.daily.ttl": "63072000", "timeline.metrics.cluster.aggregator.hourly.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.hourly.disabled": "false", "timeline.metrics.cluster.aggregator.hourly.interval": "3600", "timeline.metrics.cluster.aggregator.hourly.ttl": "31536000", "timeline.metrics.cluster.aggregator.minute.checkpointCutOffMultiplier": "2", "timeline.metrics.cluster.aggregator.minute.disabled": "false", "timeline.metrics.cluster.aggregator.minute.interval": "120", "timeline.metrics.cluster.aggregator.minute.timeslice.interval": "30", "timeline.metrics.cluster.aggregator.minute.ttl": "2592000", "timeline.metrics.daily.aggregator.minute.interval": "86400", "timeline.metrics.hbase.compression.scheme": "SNAPPY", "timeline.metrics.hbase.data.block.encoding": "FAST_DIFF", "timeline.metrics.host.aggregator.daily.checkpointCutOffMultiplier": "1", "timeline.metrics.host.aggregator.daily.disabled": "false", "timeline.metrics.host.aggregator.daily.ttl": "31536000", "timeline.metrics.host.aggregator.hourly.checkpointCutOffMultiplier": "2", "timeline.metrics.host.aggregator.hourly.disabled": "false", "timeline.metrics.host.aggregator.hourly.interval": "3600", "timeline.metrics.host.aggregator.hourly.ttl": "2592000", "timeline.metrics.host.aggregator.minute.checkpointCutOffMultiplier": "2", "timeline.metrics.host.aggregator.minute.disabled": "false", "timeline.metrics.host.aggregator.minute.interval": "120", "timeline.metrics.host.aggregator.minute.ttl": "604800", "timeline.metrics.host.aggregator.ttl": "86400", "timeline.metrics.service.checkpointDelay": "60", "timeline.metrics.service.cluster.aggregator.appIds": "datanode,nodemanager,hbase", "timeline.metrics.service.default.result.limit": "5760", "timeline.metrics.service.operation.mode": "embedded", "timeline.metrics.service.resultset.fetchSize": "2000", "timeline.metrics.service.rpc.address": "0.0.0.0:60200", "timeline.metrics.service.webapp.address": "0.0.0.0:6188" }, "application-properties": { "atlas.authentication.keytab": "/etc/security/keytabs/atlas.service.keytab", "atlas.authentication.method": "simple", "atlas.authentication.principal": "atlas", "atlas.enableTLS": "false", "atlas.graph.index.search.backend": "elasticsearch", "atlas.graph.index.search.directory": "/var/lib/atlas/data/es", "atlas.graph.index.search.elasticsearch.client-only": "false", "atlas.graph.index.search.elasticsearch.local-mode": "true", "atlas.graph.storage.backend": "berkeleyje", "atlas.graph.storage.directory": "/var/lib/atlas/data/berkeley", "atlas.http.authentication.enabled": "false", "atlas.http.authentication.kerberos.keytab": "/etc/security/keytabs/spnego.service.keytab", "atlas.http.authentication.kerberos.name.rules": "RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// DEFAULT", "atlas.http.authentication.kerberos.principal": "HTTP/_HOST@EXAMPLE.COM", "atlas.http.authentication.type": "simple", "atlas.lineage.hive.process.inputs.name": "inputs", "atlas.lineage.hive.process.outputs.name": "outputs", "atlas.lineage.hive.process.type.name": "Process", "atlas.lineage.hive.table.schema.query.Table": "Table where name='%s'\\, columns", "atlas.lineage.hive.table.schema.query.hive_table": "hive_table where name='%s'\\, columns", "atlas.lineage.hive.table.type.name": "DataSet", "atlas.server.bind.address": "localhost" }, "atlas-env": { "metadata_classpath": "", "metadata_data_dir": "/var/lib/atlas/data", "metadata_expanded_war_dir": "./server/webapp", "metadata_log_dir": "/var/log/atlas", "metadata_opts": "-Xmx1024m", "metadata_pid_dir": "/var/run/atlas", "metadata_port": "21000", "metadata_user": "atlas" }, "capacity-scheduler": { "yarn.scheduler.capacity.default.minimum-user-limit-percent": "100", "yarn.scheduler.capacity.maximum-am-resource-percent": "0.2", "yarn.scheduler.capacity.maximum-applications": "10000", "yarn.scheduler.capacity.node-locality-delay": "40", "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator", "yarn.scheduler.capacity.root.accessible-node-labels": "*", "yarn.scheduler.capacity.root.acl_administer_queue": "*", "yarn.scheduler.capacity.root.capacity": "100", "yarn.scheduler.capacity.root.default.acl_administer_jobs": "*", "yarn.scheduler.capacity.root.default.acl_submit_applications": "*", "yarn.scheduler.capacity.root.default.capacity": "100", "yarn.scheduler.capacity.root.default.maximum-capacity": "100", "yarn.scheduler.capacity.root.default.state": "RUNNING", "yarn.scheduler.capacity.root.default.user-limit-factor": "1", "yarn.scheduler.capacity.root.queues": "default" }, "cluster-env": { "command_retry_enabled": "true", "command_retry_max_time_in_sec": "600", "commands_to_retry": "INSTALL,START", "ignore_groupsusers_create": "false", "kerberos_domain": "EXAMPLE.COM", "repo_suse_rhel_template": "[{{repo_id}}] name={{repo_id}} {% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %} path=/ enabled=1 gpgcheck=0", "repo_ubuntu_template": "{{package_type}} {{base_url}} {{components}}", "security_enabled": "false", "smokeuser": "ambari-qa", "smokeuser_keytab": "/etc/security/keytabs/smokeuser.headless.keytab", "user_group": "hadoop" }, "core-site": { "fs.defaultFS": "hdfs://%HOSTGROUP::host_group_1%:8020", "fs.swift.impl": "org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem", "fs.swift.service.sahara.auth.endpoint.prefix": "/endpoints/AUTH_", "fs.swift.service.sahara.auth.url": "http://172.18.168.2:5000/v2.0/tokens/", "fs.swift.service.sahara.http.port": "8080", "fs.swift.service.sahara.https.port": "443", "fs.swift.service.sahara.public": "true", "fs.swift.service.sahara.tenant": "devs", "fs.trash.interval": "360", "ha.failover-controller.active-standby-elector.zk.op.retries": "120", "hadoop.http.authentication.simple.anonymous.allowed": "true", "hadoop.proxyuser.falcon.groups": "users", "hadoop.proxyuser.falcon.hosts": "*", "hadoop.proxyuser.hcat.groups": "users", "hadoop.proxyuser.hcat.hosts": "%HOSTGROUP::host_group_2%", "hadoop.proxyuser.hdfs.groups": "*", "hadoop.proxyuser.hdfs.hosts": "*", "hadoop.proxyuser.hive.groups": "users", "hadoop.proxyuser.hive.hosts": "%HOSTGROUP::host_group_2%", "hadoop.proxyuser.oozie.groups": "users", "hadoop.proxyuser.oozie.hosts": "*", "hadoop.security.auth_to_local": "DEFAULT", "hadoop.security.authentication": "simple", "hadoop.security.authorization": "false", "hadoop.security.key.provider.path": "", "io.compression.codecs": "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec", "io.file.buffer.size": "131072", "io.serializations": "org.apache.hadoop.io.serializer.WritableSerialization", "ipc.client.connect.max.retries": "50", "ipc.client.connection.maxidletime": "30000", "ipc.client.idlethreshold": "8000", "ipc.server.tcpnodelay": "true", "mapreduce.jobtracker.webinterface.trusted": "false", "net.topology.script.file.name": "/etc/hadoop/conf/topology_script.py", "proxyuser_group": "users" }, "falcon-env": { "falcon.embeddedmq": "true", "falcon.embeddedmq.data": "/hadoop/falcon/embeddedmq/data", "falcon.emeddedmq.port": "61616", "falcon_local_dir": "/hadoop/falcon", "falcon_log_dir": "/var/log/falcon", "falcon_pid_dir": "/var/run/falcon", "falcon_port": "15000", "falcon_store_uri": "file:///hadoop/falcon/store", "falcon_user": "falcon", "supports_hive_dr": "true" }, "falcon-runtime.properties": { "*.domain": "${falcon.app.type}", "*.log.cleanup.frequency.days.retention": "days(7)", "*.log.cleanup.frequency.hours.retention": "minutes(1)", "*.log.cleanup.frequency.minutes.retention": "hours(6)", "*.log.cleanup.frequency.months.retention": "months(3)" }, "falcon-startup.properties": { "*.ConfigSyncService.impl": "org.apache.falcon.resource.ConfigSyncService", "*.ProcessInstanceManager.impl": "org.apache.falcon.resource.InstanceManager", "*.SchedulableEntityManager.impl": "org.apache.falcon.resource.SchedulableEntityManager", "*.application.services": "org.apache.falcon.security.AuthenticationInitializationService,\\ org.apache.falcon.workflow.WorkflowJobEndNotificationService, \\ org.apache.falcon.service.ProcessSubscriberService,\\ org.apache.falcon.entity.store.ConfigurationStore,\\ org.apache.falcon.rerun.service.RetryService,\\ org.apache.falcon.rerun.service.LateRunService,\\ org.apache.falcon.service.LogCleanupService,\\ org.apache.falcon.metadata.MetadataMappingService", "*.broker.impl.class": "org.apache.activemq.ActiveMQConnectionFactory", "*.broker.ttlInMins": "4320", "*.broker.url": "tcp://%HOSTGROUP::host_group_2%:61616", "*.catalog.service.impl": "org.apache.falcon.catalog.HiveCatalogService", "*.config.store.uri": "file:///hadoop/falcon/store", "*.configstore.listeners": "org.apache.falcon.entity.v0.EntityGraph,\\ org.apache.falcon.entity.ColoClusterRelation,\\ org.apache.falcon.group.FeedGroupMap,\\ org.apache.falcon.service.SharedLibraryHostingService", "*.domain": "${falcon.app.type}", "*.entity.topic": "FALCON.ENTITY.TOPIC", "*.falcon.authentication.type": "simple", "*.falcon.cleanup.service.frequency": "days(1)", "*.falcon.enableTLS": "false", "*.falcon.graph.blueprints.graph": "com.thinkaurelius.titan.core.TitanFactory", "*.falcon.graph.preserve.history": "false", "*.falcon.graph.serialize.path": "/hadoop/falcon/data/lineage", "*.falcon.graph.storage.backend": "berkeleyje", "*.falcon.graph.storage.directory": "/hadoop/falcon/data/lineage/graphdb", "*.falcon.http.authentication.blacklisted.users": "", "*.falcon.http.authentication.cookie.domain": "EXAMPLE.COM", "*.falcon.http.authentication.kerberos.name.rules": "DEFAULT", "*.falcon.http.authentication.signature.secret": "falcon", "*.falcon.http.authentication.simple.anonymous.allowed": "true", "*.falcon.http.authentication.token.validity": "36000", "*.falcon.http.authentication.type": "simple", "*.falcon.security.authorization.admin.groups": "falcon", "*.falcon.security.authorization.admin.users": "falcon,ambari-qa", "*.falcon.security.authorization.enabled": "false", "*.falcon.security.authorization.provider": "org.apache.falcon.security.DefaultAuthorizationProvider", "*.falcon.security.authorization.superusergroup": "falcon", "*.hive.shared.libs": "hive-exec,hive-metastore,hive-common,hive-service,hive-hcatalog-server-extensions,\\ hive-hcatalog-core,hive-jdbc,hive-webhcat-java-client", "*.internal.queue.size": "1000", "*.journal.impl": "org.apache.falcon.transaction.SharedFileSystemJournal", "*.max.retry.failure.count": "1", "*.oozie.feed.workflow.builder": "org.apache.falcon.workflow.OozieFeedWorkflowBuilder", "*.oozie.process.workflow.builder": "org.apache.falcon.workflow.OozieProcessWorkflowBuilder", "*.retry.recorder.path": "${falcon.log.dir}/retry", "*.shared.libs": "activemq-core,ant,geronimo-j2ee-management,jms,json-simple,oozie-client,spring-jms,commons-lang3,commons-el", "*.system.lib.location": "${falcon.home}/server/webapp/${falcon.app.type}/WEB-INF/lib", "*.workflow.engine.impl": "org.apache.falcon.workflow.engine.OozieWorkflowEngine", "prism.application.services": "org.apache.falcon.entity.store.ConfigurationStore", "prism.configstore.listeners": "org.apache.falcon.entity.v0.EntityGraph,\\ org.apache.falcon.entity.ColoClusterRelation,\\ org.apache.falcon.group.FeedGroupMap" }, "flume-env": { "flume_conf_dir": "/etc/flume/conf", "flume_log_dir": "/var/log/flume", "flume_run_dir": "/var/run/flume", "flume_user": "flume" }, "gateway-site": { "gateway.gateway.conf.dir": "deployments", "gateway.hadoop.kerberos.secured": "false", "gateway.path": "gateway", "gateway.port": "8443", "java.security.auth.login.config": "/etc/knox/conf/krb5JAASLogin.conf", "java.security.krb5.conf": "/etc/knox/conf/krb5.conf", "sun.security.krb5.debug": "true" }, "hadoop-env": { "dfs.datanode.data.dir.mount.file": "/etc/hadoop/conf/dfs_data_dir_mount.hist", "dtnode_heapsize": "1024m", "hadoop_heapsize": "1024", "hadoop_pid_dir_prefix": "/var/run/hadoop", "hadoop_root_logger": "INFO,RFA", "hdfs_log_dir_prefix": "/var/log/hadoop", "hdfs_user": "hdfs", "keyserver_host": "", "keyserver_port": "", "namenode_heapsize": "1024m", "namenode_opt_maxnewsize": "200m", "namenode_opt_maxpermsize": "256m", "namenode_opt_newsize": "200m", "namenode_opt_permsize": "128m", "nfsgateway_heapsize": "1024", "proxyuser_group": "users" }, "hadoop-policy": { "security.admin.operations.protocol.acl": "hadoop", "security.client.datanode.protocol.acl": "*", "security.client.protocol.acl": "*", "security.datanode.protocol.acl": "*", "security.inter.datanode.protocol.acl": "*", "security.inter.tracker.protocol.acl": "*", "security.job.client.protocol.acl": "*", "security.job.task.protocol.acl": "*", "security.namenode.protocol.acl": "*", "security.refresh.policy.protocol.acl": "hadoop", "security.refresh.usertogroups.mappings.protocol.acl": "hadoop" }, "hbase-env": { "hbase_log_dir": "/var/log/hbase", "hbase_master_heapsize": "1024m", "hbase_pid_dir": "/var/run/hbase", "hbase_regionserver_heapsize": "1024m", "hbase_regionserver_xmn_max": "512", "hbase_regionserver_xmn_ratio": "0.2", "hbase_user": "hbase", "override_hbase_uid": "true", "phoenix_sql_enabled": "false" }, "hbase-policy": { "security.admin.protocol.acl": "*", "security.client.protocol.acl": "*", "security.masterregion.protocol.acl": "*" }, "hbase-site": { "dfs.domain.socket.path": "/var/lib/hadoop-hdfs/dn_socket", "hbase.bulkload.staging.dir": "/apps/hbase/staging", "hbase.client.keyvalue.maxsize": "1048576", "hbase.client.retries.number": "35", "hbase.client.scanner.caching": "100", "hbase.cluster.distributed": "true", "hbase.coprocessor.master.classes": "", "hbase.coprocessor.region.classes": "org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint", "hbase.coprocessor.regionserver.classes": "", "hbase.defaults.for.version.skip": "true", "hbase.hregion.majorcompaction": "604800000", "hbase.hregion.majorcompaction.jitter": "0.50", "hbase.hregion.max.filesize": "10737418240", "hbase.hregion.memstore.block.multiplier": "4", "hbase.hregion.memstore.flush.size": "134217728", "hbase.hregion.memstore.mslab.enabled": "true", "hbase.hstore.blockingStoreFiles": "10", "hbase.hstore.compaction.max": "10", "hbase.hstore.compactionThreshold": "3", "hbase.local.dir": "${hbase.tmp.dir}/local", "hbase.master.info.bindAddress": "0.0.0.0", "hbase.master.info.port": "16010", "hbase.master.port": "16000", "hbase.regionserver.global.memstore.size": "0.4", "hbase.regionserver.handler.count": "30", "hbase.regionserver.info.port": "16030", "hbase.regionserver.port": "16020", "hbase.regionserver.wal.codec": "org.apache.hadoop.hbase.regionserver.wal.WALCellCodec", "hbase.rootdir": "hdfs://%HOSTGROUP::host_group_1%:8020/apps/hbase/data", "hbase.rpc.controllerfactory.class": "", "hbase.rpc.protection": "authentication", "hbase.rpc.timeout": "90000", "hbase.security.authentication": "simple", "hbase.security.authorization": "false", "hbase.superuser": "hbase", "hbase.tmp.dir": "/tmp/hbase-${user.name}", "hbase.zookeeper.property.clientPort": "2181", "hbase.zookeeper.quorum": "%HOSTGROUP::host_group_1%", "hbase.zookeeper.useMulti": "true", "hfile.block.cache.size": "0.40", "phoenix.functions.allowUserDefinedFunctions": "", "phoenix.query.timeoutMs": "60000", "zookeeper.session.timeout": "90000", "zookeeper.znode.parent": "/hbase-unsecure" }, "hdfs-site": { "dfs.block.access.token.enable": "true", "dfs.blockreport.initialDelay": "120", "dfs.blocksize": "134217728", "dfs.client.read.shortcircuit": "true", "dfs.client.read.shortcircuit.streams.cache.size": "4096", "dfs.client.retry.policy.enabled": "false", "dfs.cluster.administrators": "hdfs", "dfs.datanode.address": "0.0.0.0:50010", "dfs.datanode.balance.bandwidthPerSec": "6250000", "dfs.datanode.data.dir": "/hadoop/hdfs/data", "dfs.datanode.data.dir.perm": "750", "dfs.datanode.du.reserved": "1073741824", "dfs.datanode.failed.volumes.tolerated": "0", "dfs.datanode.http.address": "0.0.0.0:50075", "dfs.datanode.https.address": "0.0.0.0:50475", "dfs.datanode.ipc.address": "0.0.0.0:8010", "dfs.datanode.max.transfer.threads": "1024", "dfs.domain.socket.path": "/var/lib/hadoop-hdfs/dn_socket", "dfs.encrypt.data.transfer.cipher.suites": "AES/CTR/NoPadding", "dfs.encryption.key.provider.uri": "", "dfs.heartbeat.interval": "3", "dfs.hosts.exclude": "/etc/hadoop/conf/dfs.exclude", "dfs.http.policy": "HTTP_ONLY", "dfs.https.port": "50470", "dfs.journalnode.edits.dir": "/hadoop/hdfs/journalnode", "dfs.journalnode.http-address": "0.0.0.0:8480", "dfs.journalnode.https-address": "0.0.0.0:8481", "dfs.namenode.accesstime.precision": "0", "dfs.namenode.audit.log.async": "true", "dfs.namenode.avoid.read.stale.datanode": "true", "dfs.namenode.avoid.write.stale.datanode": "true", "dfs.namenode.checkpoint.dir": "/hadoop/hdfs/namesecondary", "dfs.namenode.checkpoint.edits.dir": "${dfs.namenode.checkpoint.dir}", "dfs.namenode.checkpoint.period": "21600", "dfs.namenode.checkpoint.txns": "1000000", "dfs.namenode.fslock.fair": "false", "dfs.namenode.handler.count": "100", "dfs.namenode.http-address": "%HOSTGROUP::host_group_1%:50070", "dfs.namenode.https-address": "%HOSTGROUP::host_group_1%:50470", "dfs.namenode.name.dir": "/hadoop/hdfs/namenode", "dfs.namenode.name.dir.restore": "true", "dfs.namenode.rpc-address": "%HOSTGROUP::host_group_1%:8020", "dfs.namenode.safemode.threshold-pct": "0.999", "dfs.namenode.secondary.http-address": "%HOSTGROUP::host_group_1%:50090", "dfs.namenode.stale.datanode.interval": "30000", "dfs.namenode.startup.delay.block.deletion.sec": "3600", "dfs.namenode.write.stale.datanode.ratio": "1.0f", "dfs.permissions.enabled": "true", "dfs.permissions.superusergroup": "hdfs", "dfs.replication": "3", "dfs.replication.max": "50", "dfs.support.append": "true", "dfs.webhdfs.enabled": "true", "fs.permissions.umask-mode": "022", "nfs.exports.allowed.hosts": "* rw", "nfs.file.dump.dir": "/tmp/.hdfs-nfs" }, "hive-env": { "cost_based_optimizer": "On", "hcat_log_dir": "/var/log/webhcat", "hcat_pid_dir": "/var/run/webhcat", "hcat_user": "hcat", "hive_database": "New MySQL Database", "hive_database_name": "hive", "hive_database_type": "mysql", "hive_exec_orc_storage_strategy": "SPEED", "hive_log_dir": "/var/log/hive", "hive_pid_dir": "/var/run/hive", "hive_security_authorization": "None", "hive_timeline_logging_enabled": "true", "hive_txn_acid": "off", "hive_user": "hive", "webhcat_user": "hcat" }, "hive-site": { "ambari.hive.db.schema.name": "hive", "datanucleus.autoCreateSchema": "false", "datanucleus.cache.level2.type": "none", "hive.auto.convert.join": "true", "hive.auto.convert.join.noconditionaltask": "true", "hive.auto.convert.join.noconditionaltask.size": "59419306", "hive.auto.convert.sortmerge.join": "true", "hive.auto.convert.sortmerge.join.to.mapjoin": "false", "hive.cbo.enable": "true", "hive.cli.print.header": "false", "hive.cluster.delegation.token.store.class": "org.apache.hadoop.hive.thrift.ZooKeeperTokenStore", "hive.cluster.delegation.token.store.zookeeper.connectString": "%HOSTGROUP::host_group_1%:2181", "hive.cluster.delegation.token.store.zookeeper.znode": "/hive/cluster/delegation", "hive.compactor.abortedtxn.threshold": "1000", "hive.compactor.check.interval": "300L", "hive.compactor.delta.num.threshold": "10", "hive.compactor.delta.pct.threshold": "0.1f", "hive.compactor.initiator.on": "false", "hive.compactor.worker.threads": "0", "hive.compactor.worker.timeout": "86400L", "hive.compute.query.using.stats": "true", "hive.conf.restricted.list": "hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role", "hive.convert.join.bucket.mapjoin.tez": "false", "hive.default.fileformat": "TextFile", "hive.default.fileformat.managed": "TextFile", "hive.enforce.bucketing": "false", "hive.enforce.sorting": "true", "hive.enforce.sortmergebucketmapjoin": "true", "hive.exec.compress.intermediate": "false", "hive.exec.compress.output": "false", "hive.exec.dynamic.partition": "true", "hive.exec.dynamic.partition.mode": "strict", "hive.exec.failure.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook", "hive.exec.max.created.files": "100000", "hive.exec.max.dynamic.partitions": "5000", "hive.exec.max.dynamic.partitions.pernode": "2000", "hive.exec.orc.compression.strategy": "SPEED", "hive.exec.orc.default.compress": "ZLIB", "hive.exec.orc.default.stripe.size": "67108864", "hive.exec.orc.encoding.strategy": "SPEED", "hive.exec.parallel": "false", "hive.exec.parallel.thread.number": "8", "hive.exec.post.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook", "hive.exec.pre.hooks": "org.apache.hadoop.hive.ql.hooks.ATSHook", "hive.exec.reducers.bytes.per.reducer": "67108864", "hive.exec.reducers.max": "1009", "hive.exec.scratchdir": "/tmp/hive", "hive.exec.submit.local.task.via.child": "true", "hive.exec.submitviachild": "false", "hive.execution.engine": "tez", "hive.fetch.task.aggr": "false", "hive.fetch.task.conversion": "more", "hive.fetch.task.conversion.threshold": "1073741824", "hive.limit.optimize.enable": "true", "hive.limit.pushdown.memory.usage": "0.04", "hive.map.aggr": "true", "hive.map.aggr.hash.force.flush.memory.threshold": "0.9", "hive.map.aggr.hash.min.reduction": "0.5", "hive.map.aggr.hash.percentmemory": "0.5", "hive.mapjoin.bucket.cache.size": "10000", "hive.mapjoin.optimized.hashtable": "true", "hive.mapred.reduce.tasks.speculative.execution": "false", "hive.merge.mapfiles": "true", "hive.merge.mapredfiles": "false", "hive.merge.orcfile.stripe.level": "true", "hive.merge.rcfile.block.level": "true", "hive.merge.size.per.task": "256000000", "hive.merge.smallfiles.avgsize": "16000000", "hive.merge.tezfiles": "false", "hive.metastore.authorization.storage.checks": "false", "hive.metastore.cache.pinobjtypes": "Table,Database,Type,FieldSchema,Order", "hive.metastore.client.connect.retry.delay": "5s", "hive.metastore.client.socket.timeout": "1800s", "hive.metastore.connect.retries": "24", "hive.metastore.execute.setugi": "true", "hive.metastore.failure.retries": "24", "hive.metastore.kerberos.keytab.file": "/etc/security/keytabs/hive.service.keytab", "hive.metastore.kerberos.principal": "hive/_HOST@EXAMPLE.COM", "hive.metastore.pre.event.listeners": "org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener", "hive.metastore.sasl.enabled": "false", "hive.metastore.server.max.threads": "100000", "hive.metastore.uris": "thrift://%HOSTGROUP::host_group_2%:9083", "hive.metastore.warehouse.dir": "/apps/hive/warehouse", "hive.optimize.bucketmapjoin": "true", "hive.optimize.bucketmapjoin.sortedmerge": "false", "hive.optimize.constant.propagation": "true", "hive.optimize.index.filter": "true", "hive.optimize.metadataonly": "true", "hive.optimize.null.scan": "true", "hive.optimize.reducededuplication": "true", "hive.optimize.reducededuplication.min.reducer": "4", "hive.optimize.sort.dynamic.partition": "false", "hive.orc.compute.splits.num.threads": "10", "hive.orc.splits.include.file.footer": "false", "hive.prewarm.enabled": "false", "hive.prewarm.numcontainers": "3", "hive.security.authenticator.manager": "org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator", "hive.security.authorization.enabled": "false", "hive.security.authorization.manager": "org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory", "hive.security.metastore.authenticator.manager": "org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator", "hive.security.metastore.authorization.auth.reads": "true", "hive.security.metastore.authorization.manager": "org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider", "hive.server2.allow.user.substitution": "true", "hive.server2.authentication": "NONE", "hive.server2.authentication.spnego.keytab": "HTTP/_HOST@EXAMPLE.COM", "hive.server2.authentication.spnego.principal": "/etc/security/keytabs/spnego.service.keytab", "hive.server2.enable.doAs": "true", "hive.server2.logging.operation.enabled": "true", "hive.server2.logging.operation.log.location": "${system:java.io.tmpdir}/${system:user.name}/operation_logs", "hive.server2.support.dynamic.service.discovery": "true", "hive.server2.table.type.mapping": "CLASSIC", "hive.server2.tez.default.queues": "default", "hive.server2.tez.initialize.default.sessions": "false", "hive.server2.tez.sessions.per.default.queue": "1", "hive.server2.thrift.http.path": "cliservice", "hive.server2.thrift.http.port": "10001", "hive.server2.thrift.max.worker.threads": "500", "hive.server2.thrift.port": "10000", "hive.server2.thrift.sasl.qop": "auth", "hive.server2.transport.mode": "binary", "hive.server2.use.SSL": "false", "hive.server2.zookeeper.namespace": "hiveserver2", "hive.smbjoin.cache.rows": "10000", "hive.stats.autogather": "true", "hive.stats.dbclass": "fs", "hive.stats.fetch.column.stats": "true", "hive.stats.fetch.partition.stats": "true", "hive.support.concurrency": "false", "hive.tez.auto.reducer.parallelism": "true", "hive.tez.container.size": "170", "hive.tez.cpu.vcores": "-1", "hive.tez.dynamic.partition.pruning": "true", "hive.tez.dynamic.partition.pruning.max.data.size": "104857600", "hive.tez.dynamic.partition.pruning.max.event.size": "1048576", "hive.tez.input.format": "org.apache.hadoop.hive.ql.io.HiveInputFormat", "hive.tez.java.opts": "-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps", "hive.tez.log.level": "INFO", "hive.tez.max.partition.factor": "2.0", "hive.tez.min.partition.factor": "0.25", "hive.tez.smb.number.waves": "0.5", "hive.txn.manager": "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager", "hive.txn.max.open.batch": "1000", "hive.txn.timeout": "300", "hive.user.install.directory": "/user/", "hive.vectorized.execution.enabled": "true", "hive.vectorized.execution.reduce.enabled": "false", "hive.vectorized.groupby.checkinterval": "4096", "hive.vectorized.groupby.flush.percent": "0.1", "hive.vectorized.groupby.maxentries": "100000", "hive.zookeeper.client.port": "2181", "hive.zookeeper.namespace": "hive_zookeeper_namespace", "hive.zookeeper.quorum": "%HOSTGROUP::host_group_1%:2181", "javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver", "javax.jdo.option.ConnectionURL": "jdbc:mysql://%HOSTGROUP::host_group_2%/hive?createDatabaseIfNotExist=true", "javax.jdo.option.ConnectionUserName": "hive" }, "hiveserver2-site": { "hive.security.authorization.enabled": "false" }, "kafka-broker": { "auto.create.topics.enable": "true", "auto.leader.rebalance.enable": "true", "compression.type": "producer", "controlled.shutdown.enable": "true", "controlled.shutdown.max.retries": "3", "controlled.shutdown.retry.backoff.ms": "5000", "controller.message.queue.size": "10", "controller.socket.timeout.ms": "30000", "default.replication.factor": "1", "delete.topic.enable": "false", "fetch.purgatory.purge.interval.requests": "10000", "kafka.ganglia.metrics.group": "kafka", "kafka.ganglia.metrics.port": "8671", "kafka.ganglia.metrics.reporter.enabled": "true", "kafka.metrics.reporters": "{{kafka_metrics_reporters}}", "kafka.timeline.metrics.host": "{{metric_collector_host}}", "kafka.timeline.metrics.maxRowCacheSize": "10000", "kafka.timeline.metrics.port": "{{metric_collector_port}}", "kafka.timeline.metrics.reporter.enabled": "true", "kafka.timeline.metrics.reporter.sendInterval": "5900", "leader.imbalance.check.interval.seconds": "300", "leader.imbalance.per.broker.percentage": "10", "listeners": "PLAINTEXT://localhost:6667", "log.cleanup.interval.mins": "10", "log.dirs": "/kafka-logs", "log.index.interval.bytes": "4096", "log.index.size.max.bytes": "10485760", "log.retention.bytes": "-1", "log.retention.hours": "168", "log.roll.hours": "168", "log.segment.bytes": "1073741824", "message.max.bytes": "1000000", "min.insync.replicas": "1", "num.io.threads": "8", "num.network.threads": "3", "num.partitions": "1", "num.recovery.threads.per.data.dir": "1", "num.replica.fetchers": "1", "offset.metadata.max.bytes": "4096", "offsets.commit.required.acks": "-1", "offsets.commit.timeout.ms": "5000", "offsets.load.buffer.size": "5242880", "offsets.retention.check.interval.ms": "600000", "offsets.retention.minutes": "86400000", "offsets.topic.compression.codec": "0", "offsets.topic.num.partitions": "50", "offsets.topic.replication.factor": "3", "offsets.topic.segment.bytes": "104857600", "producer.purgatory.purge.interval.requests": "10000", "queued.max.requests": "500", "replica.fetch.max.bytes": "1048576", "replica.fetch.min.bytes": "1", "replica.fetch.wait.max.ms": "500", "replica.high.watermark.checkpoint.interval.ms": "5000", "replica.lag.max.messages": "4000", "replica.lag.time.max.ms": "10000", "replica.socket.receive.buffer.bytes": "65536", "replica.socket.timeout.ms": "30000", "socket.receive.buffer.bytes": "102400", "socket.request.max.bytes": "104857600", "socket.send.buffer.bytes": "102400", "zookeeper.connect": "%HOSTGROUP::host_group_1%:2181", "zookeeper.connection.timeout.ms": "15000", "zookeeper.session.timeout.ms": "30000", "zookeeper.sync.time.ms": "2000" }, "kafka-env": { "is_supported_kafka_ranger": "true", "kafka_log_dir": "/var/log/kafka", "kafka_pid_dir": "/var/run/kafka", "kafka_user": "kafka" }, "knox-env": { "knox_group": "knox", "knox_master_secret": "pass", "knox_pid_dir": "/var/run/knox", "knox_user": "knox" }, "mahout-env": { "mahout_user": "mahout" }, "mapred-env": { "jobhistory_heapsize": "900", "mapred_log_dir_prefix": "/var/log/hadoop-mapreduce", "mapred_pid_dir_prefix": "/var/run/hadoop-mapreduce", "mapred_user": "mapred" }, "mapred-site": { "mapreduce.admin.map.child.java.opts": "-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "mapreduce.admin.reduce.child.java.opts": "-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "mapreduce.admin.user.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "mapreduce.am.max-attempts": "2", "mapreduce.application.classpath": "$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure", "mapreduce.application.framework.path": "/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework", "mapreduce.cluster.administrators": "hadoop", "mapreduce.framework.name": "yarn", "mapreduce.job.counters.max": "130", "mapreduce.job.emit-timeline-data": "false", "mapreduce.job.reduce.slowstart.completedmaps": "0.05", "mapreduce.jobhistory.address": "%HOSTGROUP::host_group_1%:10020", "mapreduce.jobhistory.bind-host": "0.0.0.0", "mapreduce.jobhistory.done-dir": "/mr-history/done", "mapreduce.jobhistory.intermediate-done-dir": "/mr-history/tmp", "mapreduce.jobhistory.recovery.enable": "true", "mapreduce.jobhistory.recovery.store.class": "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService", "mapreduce.jobhistory.recovery.store.leveldb.path": "/hadoop/mapreduce/jhs", "mapreduce.jobhistory.webapp.address": "%HOSTGROUP::host_group_1%:19888", "mapreduce.map.java.opts": "-Xmx410m", "mapreduce.map.log.level": "INFO", "mapreduce.map.memory.mb": "512", "mapreduce.map.output.compress": "false", "mapreduce.map.sort.spill.percent": "0.7", "mapreduce.map.speculative": "false", "mapreduce.output.fileoutputformat.compress": "false", "mapreduce.output.fileoutputformat.compress.type": "BLOCK", "mapreduce.reduce.input.buffer.percent": "0.0", "mapreduce.reduce.java.opts": "-Xmx756m", "mapreduce.reduce.log.level": "INFO", "mapreduce.reduce.memory.mb": "1024", "mapreduce.reduce.shuffle.fetch.retry.enabled": "1", "mapreduce.reduce.shuffle.fetch.retry.interval-ms": "1000", "mapreduce.reduce.shuffle.fetch.retry.timeout-ms": "30000", "mapreduce.reduce.shuffle.input.buffer.percent": "0.7", "mapreduce.reduce.shuffle.merge.percent": "0.66", "mapreduce.reduce.shuffle.parallelcopies": "30", "mapreduce.reduce.speculative": "false", "mapreduce.shuffle.port": "13562", "mapreduce.task.io.sort.factor": "100", "mapreduce.task.io.sort.mb": "358", "mapreduce.task.timeout": "300000", "yarn.app.mapreduce.am.admin-command-opts": "-Dhdp.version=${hdp.version}", "yarn.app.mapreduce.am.command-opts": "-Xmx410m", "yarn.app.mapreduce.am.log.level": "INFO", "yarn.app.mapreduce.am.resource.mb": "512", "yarn.app.mapreduce.am.staging-dir": "/user" }, "oozie-env": { "oozie_admin_port": "11001", "oozie_admin_users": "{oozie_user}, oozie-admin", "oozie_data_dir": "/hadoop/oozie/data", "oozie_database": "New Derby Database", "oozie_derby_database": "Derby", "oozie_heapsize": "2048m", "oozie_hostname": "%HOSTGROUP::host_group_1%", "oozie_log_dir": "/var/log/oozie", "oozie_permsize": "256m", "oozie_pid_dir": "/var/run/oozie", "oozie_user": "oozie" }, "oozie-site": { "oozie.authentication.kerberos.name.rules": "", "oozie.authentication.simple.anonymous.allowed": "true", "oozie.authentication.type": "simple", "oozie.base.url": "http://%HOSTGROUP::host_group_1%:11000/oozie", "oozie.credentials.credentialclasses": "hcat=org.apache.oozie.action.hadoop.HCatCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials", "oozie.db.schema.name": "oozie", "oozie.service.AuthorizationService.security.enabled": "true", "oozie.service.HadoopAccessorService.hadoop.configurations": "*=/etc/hadoop/conf", "oozie.service.HadoopAccessorService.kerberos.enabled": "false", "oozie.service.JPAService.jdbc.driver": "org.apache.derby.jdbc.EmbeddedDriver", "oozie.service.JPAService.jdbc.username": "oozie", "oozie.service.URIHandlerService.uri.handlers": "org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler", "oozie.services.ext": "org.apache.oozie.service.JMSAccessorService,org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService" }, "ranger-hdfs-audit": { "xasecure.audit.credential.provider.file": "jceks://file{{credential_file}}", "xasecure.audit.destination.db": "false", "xasecure.audit.destination.db.batch.filespool.dir": "/var/log/hadoop/hdfs/audit/db/spool", "xasecure.audit.destination.db.jdbc.driver": "{{jdbc_driver}}", "xasecure.audit.destination.db.jdbc.url": "{{audit_jdbc_url}}", "xasecure.audit.destination.db.user": "{{xa_audit_db_user}}", "xasecure.audit.destination.hdfs": "true", "xasecure.audit.destination.hdfs.batch.filespool.dir": "/var/log/hadoop/hdfs/audit/hdfs/spool", "xasecure.audit.destination.hdfs.dir": "hdfs://NAMENODE_HOSTNAME:8020/ranger/audit", "xasecure.audit.destination.solr": "false", "xasecure.audit.destination.solr.batch.filespool.dir": "/var/log/hadoop/hdfs/audit/solr/spool", "xasecure.audit.destination.solr.urls": "{{ranger_audit_solr_urls}}", "xasecure.audit.destination.solr.zookeepers": "none", "xasecure.audit.is.enabled": "true", "xasecure.audit.provider.summary.enabled": "false" }, "ranger-hdfs-plugin-properties": { "REPOSITORY_CONFIG_USERNAME": "hadoop", "common.name.for.certificate": "", "hadoop.rpc.protection": "", "policy_user": "ambari-qa", "ranger-hdfs-plugin-enabled": "No" }, "ranger-hdfs-policymgr-ssl": { "xasecure.policymgr.clientssl.keystore": "/usr/hdp/current/hadoop-client/conf/ranger-plugin-keystore.jks", "xasecure.policymgr.clientssl.keystore.credential.file": "jceks://file{{credential_file}}", "xasecure.policymgr.clientssl.truststore": "/usr/hdp/current/hadoop-client/conf/ranger-plugin-truststore.jks", "xasecure.policymgr.clientssl.truststore.credential.file": "jceks://file{{credential_file}}" }, "ranger-hdfs-security": { "ranger.plugin.hdfs.policy.cache.dir": "/etc/ranger/{{repo_name}}/policycache", "ranger.plugin.hdfs.policy.pollIntervalMs": "30000", "ranger.plugin.hdfs.policy.rest.ssl.config.file": "/etc/hadoop/conf/ranger-policymgr-ssl.xml", "ranger.plugin.hdfs.policy.rest.url": "{{policymgr_mgr_url}}", "ranger.plugin.hdfs.policy.source.impl": "org.apache.ranger.admin.client.RangerAdminRESTClient", "ranger.plugin.hdfs.service.name": "{{repo_name}}", "xasecure.add-hadoop-authorization": "true" }, "ranger-yarn-audit": { "xasecure.audit.credential.provider.file": "jceks://file{{credential_file}}", "xasecure.audit.destination.db": "false", "xasecure.audit.destination.db.batch.filespool.dir": "/var/log/hadoop/yarn/audit/db/spool", "xasecure.audit.destination.db.jdbc.driver": "{{jdbc_driver}}", "xasecure.audit.destination.db.jdbc.url": "{{audit_jdbc_url}}", "xasecure.audit.destination.db.user": "{{xa_audit_db_user}}", "xasecure.audit.destination.hdfs": "true", "xasecure.audit.destination.hdfs.batch.filespool.dir": "/var/log/hadoop/yarn/audit/hdfs/spool", "xasecure.audit.destination.hdfs.dir": "hdfs://NAMENODE_HOSTNAME:8020/ranger/audit", "xasecure.audit.destination.solr": "false", "xasecure.audit.destination.solr.batch.filespool.dir": "/var/log/hadoop/yarn/audit/solr/spool", "xasecure.audit.destination.solr.urls": "{{ranger_audit_solr_urls}}", "xasecure.audit.destination.solr.zookeepers": "none", "xasecure.audit.is.enabled": "true", "xasecure.audit.provider.summary.enabled": "false" }, "ranger-yarn-plugin-properties": { "REPOSITORY_CONFIG_USERNAME": "yarn", "common.name.for.certificate": "", "hadoop.rpc.protection": "", "policy_user": "ambari-qa", "ranger-yarn-plugin-enabled": "No" }, "ranger-yarn-policymgr-ssl": { "xasecure.policymgr.clientssl.keystore": "/usr/hdp/current/hadoop-client/conf/ranger-yarn-plugin-keystore.jks", "xasecure.policymgr.clientssl.keystore.credential.file": "jceks://file{{credential_file}}", "xasecure.policymgr.clientssl.truststore": "/usr/hdp/current/hadoop-client/conf/ranger-yarn-plugin-truststore.jks", "xasecure.policymgr.clientssl.truststore.credential.file": "jceks://file{{credential_file}}" }, "ranger-yarn-security": { "ranger.plugin.yarn.policy.cache.dir": "/etc/ranger/{{repo_name}}/policycache", "ranger.plugin.yarn.policy.pollIntervalMs": "30000", "ranger.plugin.yarn.policy.rest.ssl.config.file": "/etc/hadoop/conf/ranger-policymgr-ssl-yarn.xml", "ranger.plugin.yarn.policy.rest.url": "{{policymgr_mgr_url}}", "ranger.plugin.yarn.policy.source.impl": "org.apache.ranger.admin.client.RangerAdminRESTClient", "ranger.plugin.yarn.service.name": "{{repo_name}}" }, "spark-defaults": { "spark.driver.extraJavaOptions": "-Dhdp.version={{hdp_full_version}}", "spark.history.kerberos.keytab": "none", "spark.history.kerberos.principal": "none", "spark.history.provider": "org.apache.spark.deploy.yarn.history.YarnHistoryProvider", "spark.history.ui.port": "18080", "spark.yarn.am.extraJavaOptions": "-Dhdp.version={{hdp_full_version}}", "spark.yarn.applicationMaster.waitTries": "10", "spark.yarn.containerLauncherMaxThreads": "25", "spark.yarn.driver.memoryOverhead": "384", "spark.yarn.executor.memoryOverhead": "384", "spark.yarn.historyServer.address": "{{spark_history_server_host}}:{{spark_history_ui_port}}", "spark.yarn.max.executor.failures": "3", "spark.yarn.preserve.staging.files": "false", "spark.yarn.queue": "default", "spark.yarn.scheduler.heartbeat.interval-ms": "5000", "spark.yarn.services": "org.apache.spark.deploy.yarn.history.YarnHistoryService", "spark.yarn.submit.file.replication": "3" }, "spark-env": { "spark_group": "spark", "spark_log_dir": "/var/log/spark", "spark_pid_dir": "/var/run/spark", "spark_user": "spark" }, "sqoop-env": { "jdbc_drivers": "", "sqoop_user": "sqoop" }, "ssl-client": { "ssl.client.keystore.location": "/etc/security/clientKeys/keystore.jks", "ssl.client.keystore.type": "jks", "ssl.client.truststore.location": "/etc/security/clientKeys/all.jks", "ssl.client.truststore.reload.interval": "10000", "ssl.client.truststore.type": "jks" }, "ssl-server": { "ssl.server.keystore.location": "/etc/security/serverKeys/keystore.jks", "ssl.server.keystore.type": "jks", "ssl.server.truststore.location": "/etc/security/serverKeys/all.jks", "ssl.server.truststore.reload.interval": "10000", "ssl.server.truststore.type": "jks" }, "storm-env": { "jmxremote_port": "56431", "nimbus_seeds_supported": "true", "storm_log_dir": "/var/log/storm", "storm_pid_dir": "/var/run/storm", "storm_user": "storm" }, "storm-site": { "_storm.min.ruid": "null", "_storm.thrift.nonsecure.transport": "backtype.storm.security.auth.SimpleTransportPlugin", "_storm.thrift.secure.transport": "backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin", "dev.zookeeper.path": "/tmp/dev-storm-zookeeper", "drpc.childopts": "-Xmx768m _JAAS_PLACEHOLDER", "drpc.invocations.port": "3773", "drpc.port": "3772", "drpc.queue.size": "128", "drpc.request.timeout.secs": "600", "drpc.worker.threads": "64", "java.library.path": "/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib", "logviewer.appender.name": "A1", "logviewer.childopts": "-Xmx128m _JAAS_PLACEHOLDER", "logviewer.port": "8000", "metrics.reporter.register": "org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter", "nimbus.cleanup.inbox.freq.secs": "600", "nimbus.file.copy.expiration.secs": "600", "nimbus.inbox.jar.expiration.secs": "3600", "nimbus.monitor.freq.secs": "10", "nimbus.reassign": "true", "nimbus.seeds": "['%HOSTGROUP::host_group_2%']", "nimbus.supervisor.timeout.secs": "60", "nimbus.task.launch.secs": "120", "nimbus.task.timeout.secs": "30", "nimbus.thrift.max_buffer_size": "1048576", "nimbus.thrift.port": "6627", "nimbus.topology.validator": "backtype.storm.nimbus.DefaultTopologyValidator", "storm.cluster.mode": "distributed", "storm.local.dir": "/hadoop/storm", "storm.local.mode.zmq": "false", "storm.log.dir": "{{log_dir}}", "storm.messaging.netty.buffer_size": "5242880", "storm.messaging.netty.client_worker_threads": "1", "storm.messaging.netty.max_retries": "30", "storm.messaging.netty.max_wait_ms": "1000", "storm.messaging.netty.min_wait_ms": "100", "storm.messaging.netty.server_worker_threads": "1", "storm.messaging.transport": "backtype.storm.messaging.netty.Context", "storm.thrift.transport": "{{storm_thrift_transport}}", "storm.zookeeper.connection.timeout": "15000", "storm.zookeeper.port": "2181", "storm.zookeeper.retry.interval": "1000", "storm.zookeeper.retry.intervalceiling.millis": "30000", "storm.zookeeper.retry.times": "5", "storm.zookeeper.root": "/storm", "storm.zookeeper.servers": "['%HOSTGROUP::host_group_1%']", "storm.zookeeper.session.timeout": "20000", "supervisor.heartbeat.frequency.secs": "5", "supervisor.monitor.frequency.secs": "3", "supervisor.slots.ports": "[6700, 6701]", "supervisor.worker.start.timeout.secs": "120", "supervisor.worker.timeout.secs": "30", "task.heartbeat.frequency.secs": "3", "task.refresh.poll.secs": "10", "topology.acker.executors": "null", "topology.builtin.metrics.bucket.size.secs": "60", "topology.debug": "false", "topology.disruptor.wait.strategy": "com.lmax.disruptor.BlockingWaitStrategy", "topology.enable.message.timeouts": "true", "topology.error.throttle.interval.secs": "10", "topology.executor.receive.buffer.size": "1024", "topology.executor.send.buffer.size": "1024", "topology.fall.back.on.java.serialization": "true", "topology.kryo.factory": "backtype.storm.serialization.DefaultKryoFactory", "topology.max.error.report.per.interval": "5", "topology.max.replication.wait.time.sec": "{{actual_topology_max_replication_wait_time_sec}}", "topology.max.replication.wait.time.sec.default": "60", "topology.max.spout.pending": "null", "topology.max.task.parallelism": "null", "topology.message.timeout.secs": "30", "topology.metrics.consumer.register": "[{'class': 'org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink', 'parallelism.hint': 1}]", "topology.min.replication.count": "{{actual_topology_min_replication_count}}", "topology.min.replication.count.default": "1", "topology.optimize": "true", "topology.receiver.buffer.size": "8", "topology.skip.missing.kryo.registrations": "false", "topology.sleep.spout.wait.strategy.time.ms": "1", "topology.spout.wait.strategy": "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.state.synchronization.timeout.secs": "60", "topology.stats.sample.rate": "0.05", "topology.tick.tuple.freq.secs": "null", "topology.transfer.buffer.size": "1024", "topology.trident.batch.emit.interval.millis": "500", "topology.tuple.serializer": "backtype.storm.serialization.types.ListDelegateSerializer", "topology.worker.childopts": "null", "topology.worker.shared.thread.pool.size": "4", "topology.workers": "1", "transactional.zookeeper.port": "null", "transactional.zookeeper.root": "/transactional", "transactional.zookeeper.servers": "null", "ui.childopts": "-Xmx768m _JAAS_PLACEHOLDER", "ui.filter": "null", "ui.port": "8744", "worker.heartbeat.frequency.secs": "1", "zmq.hwm": "0", "zmq.linger.millis": "5000", "zmq.threads": "1" }, "tez-env": { "tez_user": "tez" }, "tez-site": { "tez.am.am-rm.heartbeat.interval-ms.max": "250", "tez.am.container.idle.release-timeout-max.millis": "20000", "tez.am.container.idle.release-timeout-min.millis": "10000", "tez.am.container.reuse.enabled": "true", "tez.am.container.reuse.locality.delay-allocation-millis": "250", "tez.am.container.reuse.non-local-fallback.enabled": "false", "tez.am.container.reuse.rack-fallback.enabled": "true", "tez.am.launch.cluster-default.cmd-opts": "-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "tez.am.launch.cmd-opts": "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC", "tez.am.launch.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "tez.am.log.level": "INFO", "tez.am.max.app.attempts": "2", "tez.am.maxtaskfailures.per.node": "10", "tez.am.resource.memory.mb": "1536", "tez.am.tez-ui.history-url.template": "__HISTORY_URL_BASE__?viewPath=%2F%23%2Ftez-app%2F__APPLICATION_ID__", "tez.am.view-acls": "*", "tez.cluster.additional.classpath.prefix": "/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure", "tez.counters.max": "2000", "tez.counters.max.groups": "1000", "tez.generate.debug.artifacts": "false", "tez.grouping.max-size": "1073741824", "tez.grouping.min-size": "16777216", "tez.grouping.split-waves": "1.7", "tez.history.logging.service.class": "org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService", "tez.lib.uris": "/hdp/apps/${hdp.version}/tez/tez.tar.gz", "tez.runtime.compress": "true", "tez.runtime.compress.codec": "org.apache.hadoop.io.compress.SnappyCodec", "tez.runtime.convert.user-payload.to.history-text": "false", "tez.runtime.io.sort.mb": "272", "tez.runtime.optimize.local.fetch": "true", "tez.runtime.pipelined.sorter.sort.threads": "2", "tez.runtime.sorter.class": "PIPELINED", "tez.runtime.unordered.output.buffer.size-mb": "100", "tez.session.am.dag.submit.timeout.secs": "300", "tez.session.client.timeout.secs": "-1", "tez.shuffle-vertex-manager.max-src-fraction": "0.4", "tez.shuffle-vertex-manager.min-src-fraction": "0.2", "tez.staging-dir": "/tmp/${user.name}/staging", "tez.task.am.heartbeat.counter.interval-ms.max": "4000", "tez.task.generate.counters.per.io": "true", "tez.task.get-task.sleep.interval-ms.max": "200", "tez.task.launch.cluster-default.cmd-opts": "-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}", "tez.task.launch.cmd-opts": "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC", "tez.task.launch.env": "LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64", "tez.task.max-events-per-heartbeat": "500", "tez.task.resource.memory.mb": "1536", "tez.use.cluster.hadoop-libs": "false" }, "webhcat-site": { "templeton.exec.timeout": "60000", "templeton.hadoop": "/usr/hdp/${hdp.version}/hadoop/bin/hadoop", "templeton.hadoop.conf.dir": "/etc/hadoop/conf", "templeton.hcat": "/usr/hdp/${hdp.version}/hive/bin/hcat", "templeton.hcat.home": "hive.tar.gz/hive/hcatalog", "templeton.hive.archive": "hdfs:///hdp/apps/${hdp.version}/hive/hive.tar.gz", "templeton.hive.extra.files": "/usr/hdp/${hdp.version}/tez/conf/tez-site.xml,/usr/hdp/${hdp.version}/tez,/usr/hdp/${hdp.version}/tez/lib", "templeton.hive.home": "hive.tar.gz/hive", "templeton.hive.path": "hive.tar.gz/hive/bin/hive", "templeton.hive.properties": "hive.metastore.local=false,hive.metastore.uris=thrift://%HOSTGROUP::host_group_2%:9083,hive.metastore.sasl.enabled=false,hive.metastore.execute.setugi=true", "templeton.jar": "/usr/hdp/${hdp.version}/hive/share/webhcat/svr/lib/hive-webhcat-*.jar", "templeton.libjars": "/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar", "templeton.override.enabled": "false", "templeton.pig.archive": "hdfs:///hdp/apps/${hdp.version}/pig/pig.tar.gz", "templeton.pig.path": "pig.tar.gz/pig/bin/pig", "templeton.port": "50111", "templeton.python": "${env.PYTHON_CMD}", "templeton.sqoop.archive": "hdfs:///hdp/apps/${hdp.version}/sqoop/sqoop.tar.gz", "templeton.sqoop.home": "sqoop.tar.gz/sqoop", "templeton.sqoop.path": "sqoop.tar.gz/sqoop/bin/sqoop", "templeton.storage.class": "org.apache.hive.hcatalog.templeton.tool.ZooKeeperStorage", "templeton.streaming.jar": "hdfs:///hdp/apps/${hdp.version}/mapreduce/hadoop-streaming.jar", "templeton.zookeeper.hosts": "%HOSTGROUP::host_group_1%:2181" }, "yarn-env": { "apptimelineserver_heapsize": "1024", "is_supported_yarn_ranger": "true", "min_user_id": "1000", "nodemanager_heapsize": "1024", "resourcemanager_heapsize": "1024", "yarn_cgroups_enabled": "false", "yarn_heapsize": "1024", "yarn_log_dir_prefix": "/var/log/hadoop-yarn", "yarn_pid_dir_prefix": "/var/run/hadoop-yarn", "yarn_user": "yarn" }, "yarn-site": { "hadoop.registry.rm.enabled": "true", "hadoop.registry.zk.quorum": "%HOSTGROUP::host_group_1%:2181", "yarn.acl.enable": "false", "yarn.admin.acl": "yarn", "yarn.application.classpath": "$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*", "yarn.client.nodemanager-connect.max-wait-ms": "60000", "yarn.client.nodemanager-connect.retry-interval-ms": "10000", "yarn.http.policy": "HTTP_ONLY", "yarn.log-aggregation-enable": "true", "yarn.log-aggregation.retain-seconds": "2592000", "yarn.log.server.url": "http://%HOSTGROUP::host_group_1%:19888/jobhistory/logs", "yarn.node-labels.enabled": "false", "yarn.node-labels.fs-store.retry-policy-spec": "2000, 500", "yarn.node-labels.fs-store.root-dir": "/system/yarn/node-labels", "yarn.nodemanager.address": "0.0.0.0:45454", "yarn.nodemanager.admin-env": "MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX", "yarn.nodemanager.aux-services": "mapreduce_shuffle", "yarn.nodemanager.aux-services.mapreduce_shuffle.class": "org.apache.hadoop.mapred.ShuffleHandler", "yarn.nodemanager.bind-host": "0.0.0.0", "yarn.nodemanager.container-executor.class": "org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor", "yarn.nodemanager.container-monitor.interval-ms": "3000", "yarn.nodemanager.delete.debug-delay-sec": "0", "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage": "90", "yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb": "1000", "yarn.nodemanager.disk-health-checker.min-healthy-disks": "0.25", "yarn.nodemanager.health-checker.interval-ms": "135000", "yarn.nodemanager.health-checker.script.timeout-ms": "60000", "yarn.nodemanager.linux-container-executor.cgroups.hierarchy": "hadoop-yarn", "yarn.nodemanager.linux-container-executor.cgroups.mount": "false", "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage": "false", "yarn.nodemanager.linux-container-executor.group": "hadoop", "yarn.nodemanager.linux-container-executor.resources-handler.class": "org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler", "yarn.nodemanager.local-dirs": "/hadoop/yarn/local", "yarn.nodemanager.log-aggregation.compression-type": "gz", "yarn.nodemanager.log-aggregation.debug-enabled": "false", "yarn.nodemanager.log-aggregation.num-log-files-per-app": "30", "yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds": "-1", "yarn.nodemanager.log-dirs": "/hadoop/yarn/log", "yarn.nodemanager.log.retain-second": "604800", "yarn.nodemanager.recovery.dir": "{{yarn_log_dir_prefix}}/nodemanager/recovery-state", "yarn.nodemanager.recovery.enabled": "true", "yarn.nodemanager.remote-app-log-dir": "/app-logs", "yarn.nodemanager.remote-app-log-dir-suffix": "logs", "yarn.nodemanager.resource.cpu-vcores": "8", "yarn.nodemanager.resource.memory-mb": "5120", "yarn.nodemanager.resource.percentage-physical-cpu-limit": "80", "yarn.nodemanager.vmem-check-enabled": "false", "yarn.nodemanager.vmem-pmem-ratio": "2.1", "yarn.resourcemanager.address": "%HOSTGROUP::host_group_1%:8050", "yarn.resourcemanager.admin.address": "%HOSTGROUP::host_group_1%:8141", "yarn.resourcemanager.am.max-attempts": "2", "yarn.resourcemanager.bind-host": "0.0.0.0", "yarn.resourcemanager.connect.max-wait.ms": "900000", "yarn.resourcemanager.connect.retry-interval.ms": "30000", "yarn.resourcemanager.fs.state-store.retry-policy-spec": "2000, 500", "yarn.resourcemanager.fs.state-store.uri": "", "yarn.resourcemanager.ha.enabled": "false", "yarn.resourcemanager.hostname": "%HOSTGROUP::host_group_1%", "yarn.resourcemanager.nodes.exclude-path": "/etc/hadoop/conf/yarn.exclude", "yarn.resourcemanager.recovery.enabled": "true", "yarn.resourcemanager.resource-tracker.address": "%HOSTGROUP::host_group_1%:8025", "yarn.resourcemanager.scheduler.address": "%HOSTGROUP::host_group_1%:8030", "yarn.resourcemanager.scheduler.class": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler", "yarn.resourcemanager.scheduler.monitor.enable": "false", "yarn.resourcemanager.state-store.max-completed-applications": "${yarn.resourcemanager.max-completed-applications}", "yarn.resourcemanager.store.class": "org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore", "yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size": "10", "yarn.resourcemanager.system-metrics-publisher.enabled": "true", "yarn.resourcemanager.webapp.address": "%HOSTGROUP::host_group_1%:8088", "yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled": "false", "yarn.resourcemanager.webapp.https.address": "%HOSTGROUP::host_group_1%:8090", "yarn.resourcemanager.work-preserving-recovery.enabled": "true", "yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms": "10000", "yarn.resourcemanager.zk-acl": "world:anyone:rwcda", "yarn.resourcemanager.zk-address": "%HOSTGROUP::host_group_1%:2181", "yarn.resourcemanager.zk-num-retries": "1000", "yarn.resourcemanager.zk-retry-interval-ms": "1000", "yarn.resourcemanager.zk-state-store.parent-path": "/rmstore", "yarn.resourcemanager.zk-timeout-ms": "10000", "yarn.scheduler.maximum-allocation-mb": "5120", "yarn.scheduler.maximum-allocation-vcores": "8", "yarn.scheduler.minimum-allocation-mb": "512", "yarn.scheduler.minimum-allocation-vcores": "1", "yarn.timeline-service.address": "%HOSTGROUP::host_group_1%:10200", "yarn.timeline-service.bind-host": "0.0.0.0", "yarn.timeline-service.client.max-retries": "30", "yarn.timeline-service.client.retry-interval-ms": "1000", "yarn.timeline-service.enabled": "true", "yarn.timeline-service.generic-application-history.store-class": "org.apache.hadoop.yarn.server.applicationhistoryservice.NullApplicationHistoryStore", "yarn.timeline-service.http-authentication.simple.anonymous.allowed": "true", "yarn.timeline-service.http-authentication.type": "simple", "yarn.timeline-service.leveldb-state-store.path": "/hadoop/yarn/timeline", "yarn.timeline-service.leveldb-timeline-store.path": "/hadoop/yarn/timeline", "yarn.timeline-service.leveldb-timeline-store.read-cache-size": "104857600", "yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size": "10000", "yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size": "10000", "yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms": "300000", "yarn.timeline-service.recovery.enabled": "true", "yarn.timeline-service.state-store-class": "org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore", "yarn.timeline-service.store-class": "org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore", "yarn.timeline-service.ttl-enable": "true", "yarn.timeline-service.ttl-ms": "2678400000", "yarn.timeline-service.webapp.address": "%HOSTGROUP::host_group_1%:8188", "yarn.timeline-service.webapp.https.address": "%HOSTGROUP::host_group_1%:8190" }, "zoo.cfg": { "autopurge.purgeInterval": "24", "autopurge.snapRetainCount": "30", "clientPort": "2181", "dataDir": "/hadoop/zookeeper", "initLimit": "10", "syncLimit": "5", "tickTime": "2000" }, "zookeeper-env": { "zk_log_dir": "/var/log/zookeeper", "zk_pid_dir": "/var/run/zookeeper", "zk_user": "zookeeper" } }sahara-8.0.0/sahara/plugins/default_templates/0000775000175100017510000000000013245515027021430 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/ambari/0000775000175100017510000000000013245515026022662 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_5/0000775000175100017510000000000013245515027023436 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_5/master-edp.json0000666000175100017510000000061713245514472026403 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.5", "node_processes": [ "Hive Metastore", "HiveServer", "Oozie" ], "name": "hdp-25-default-master-edp", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_5/worker.json0000666000175100017510000000064613245514472025655 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.5", "node_processes": [ "DataNode", "NodeManager" ], "name": "hdp-25-default-worker", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "volume_size": 2, "volumes_per_node": 2, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_5/cluster.json0000666000175100017510000000123713245514472026022 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.5", "node_groups": [ { "name": "worker", "count": 3, "node_group_template_id": "{hdp-25-default-worker}" }, { "name": "master-edp", "count": 1, "node_group_template_id": "{hdp-25-default-master-edp}" }, { "name": "master", "count": 1, "node_group_template_id": "{hdp-25-default-master}" } ], "name": "hdp-25-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_5/master.json0000666000175100017510000000107113245514472025630 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.5", "node_processes": [ "Ambari", "MapReduce History Server", "Spark History Server", "NameNode", "ResourceManager", "SecondaryNameNode", "YARN Timeline Server", "ZooKeeper", "Kafka Broker" ], "name": "hdp-25-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_3/0000775000175100017510000000000013245515027023434 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_3/master-edp.json0000666000175100017510000000061713245514472026401 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.3", "node_processes": [ "Hive Metastore", "HiveServer", "Oozie" ], "name": "hdp-23-default-master-edp", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_3/worker.json0000666000175100017510000000064613245514472025653 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.3", "node_processes": [ "DataNode", "NodeManager" ], "name": "hdp-23-default-worker", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "volume_size": 2, "volumes_per_node": 2, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_3/cluster.json0000666000175100017510000000123413245514472026015 0ustar zuulzuul00000000000000{ "plugin_name": "hdp", "hadoop_version": "2.3", "node_groups": [ { "name": "worker", "count": 3, "node_group_template_id": "{hdp-23-default-worker}" }, { "name": "master-edp", "count": 1, "node_group_template_id": "{hdp-23-default-master-edp}" }, { "name": "master", "count": 1, "node_group_template_id": "{hdp-23-default-master}" } ], "name": "hdp-23-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_3/master.json0000666000175100017510000000107013245514472025625 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.3", "node_processes": [ "Ambari", "MapReduce History Server", "Spark History Server", "NameNode", "ResourceManager", "SecondaryNameNode", "YARN Timeline Server", "ZooKeeper", "Kafka Broker" ], "name": "hdp-23-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_4/0000775000175100017510000000000013245515027023435 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_4/master-edp.json0000666000175100017510000000061713245514472026402 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.4", "node_processes": [ "Hive Metastore", "HiveServer", "Oozie" ], "name": "hdp-24-default-master-edp", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_4/worker.json0000666000175100017510000000064613245514472025654 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.4", "node_processes": [ "DataNode", "NodeManager" ], "name": "hdp-24-default-worker", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "volume_size": 2, "volumes_per_node": 2, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_4/cluster.json0000666000175100017510000000123713245514472026021 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.4", "node_groups": [ { "name": "worker", "count": 3, "node_group_template_id": "{hdp-24-default-worker}" }, { "name": "master-edp", "count": 1, "node_group_template_id": "{hdp-24-default-master-edp}" }, { "name": "master", "count": 1, "node_group_template_id": "{hdp-24-default-master}" } ], "name": "hdp-24-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/ambari/v2_4/master.json0000666000175100017510000000107113245514472025627 0ustar zuulzuul00000000000000{ "plugin_name": "ambari", "hadoop_version": "2.4", "node_processes": [ "Ambari", "MapReduce History Server", "Spark History Server", "NameNode", "ResourceManager", "SecondaryNameNode", "YARN Timeline Server", "ZooKeeper", "Kafka Broker" ], "name": "hdp-24-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/storm/0000775000175100017510000000000013245515026022573 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/storm/v1_1_0/0000775000175100017510000000000013245515027023561 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/storm/v1_1_0/slave.json0000666000175100017510000000054313245514472025575 0ustar zuulzuul00000000000000{ "plugin_name": "storm", "hadoop_version": "1.1.0", "node_processes": [ "supervisor" ], "name": "storm-110-default-slave", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/storm/v1_1_0/cluster.json0000666000175100017510000000102713245514472026142 0ustar zuulzuul00000000000000{ "plugin_name": "storm", "hadoop_version": "1.1.0", "node_groups": [ { "name": "slave", "count": 3, "node_group_template_id": "{storm-110-default-slave}" }, { "name": "master-zk", "count": 1, "node_group_template_id": "{storm-110-default-master}" } ], "name": "storm-110-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/storm/v1_1_0/master.json0000666000175100017510000000056413245514472025761 0ustar zuulzuul00000000000000{ "plugin_name": "storm", "hadoop_version": "1.1.0", "node_processes": [ "nimbus", "zookeeper" ], "name": "storm-110-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/storm/v1_0_1/0000775000175100017510000000000013245515027023561 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/storm/v1_0_1/slave.json0000666000175100017510000000054313245514472025575 0ustar zuulzuul00000000000000{ "plugin_name": "storm", "hadoop_version": "1.0.1", "node_processes": [ "supervisor" ], "name": "storm-101-default-slave", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/storm/v1_0_1/cluster.json0000666000175100017510000000102713245514472026142 0ustar zuulzuul00000000000000{ "plugin_name": "storm", "hadoop_version": "1.0.1", "node_groups": [ { "name": "slave", "count": 3, "node_group_template_id": "{storm-101-default-slave}" }, { "name": "master-zk", "count": 1, "node_group_template_id": "{storm-101-default-master}" } ], "name": "storm-101-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/storm/v1_0_1/master.json0000666000175100017510000000056413245514472025761 0ustar zuulzuul00000000000000{ "plugin_name": "storm", "hadoop_version": "1.0.1", "node_processes": [ "nimbus", "zookeeper" ], "name": "storm-101-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/0000775000175100017510000000000013245515026022366 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/mapr/v5_1_0_mrv2/0000775000175100017510000000000013245515027024326 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/mapr/v5_1_0_mrv2/worker.json0000666000175100017510000000057713245514472026550 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.1.0.mrv2", "node_processes": [ "FileServer", "NodeManager" ], "name": "mapr-510mr2-default-worker", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/v5_1_0_mrv2/cluster.json0000666000175100017510000000104013245514472026702 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.1.0.mrv2", "node_groups": [ { "name": "master", "count": 1, "node_group_template_id": "{mapr-510mr2-default-master}" }, { "name": "worker", "count": 3, "node_group_template_id": "{mapr-510mr2-default-worker}" } ], "name": "mapr-510mr2-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/v5_1_0_mrv2/master.json0000666000175100017510000000124513245514472026523 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.1.0.mrv2", "node_processes": [ "Metrics", "Webserver", "ZooKeeper", "HTTPFS", "Oozie", "FileServer", "CLDB", "Flume", "Hue", "NodeManager", "HistoryServer", "ResourceManager", "HiveServer2", "HiveMetastore", "Sqoop2-Client", "Sqoop2-Server" ], "name": "mapr-510mr2-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/5_0_0_mrv2/0000775000175100017510000000000013245515027024137 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/mapr/5_0_0_mrv2/worker.json0000666000175100017510000000057713245514472026361 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.0.0.mrv2", "node_processes": [ "FileServer", "NodeManager" ], "name": "mapr-500mr2-default-worker", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/5_0_0_mrv2/cluster.json0000666000175100017510000000103713245514472026521 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.0.0.mrv2", "node_groups": [ { "name": "master", "count": 1, "node_group_template_id": "{mapr-500mr2-default-master}" }, { "name": "worker", "count": 3, "node_group_template_id": "{mapr-500mr2-default-worker}" } ], "name": "mapr-500mr2-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/5_0_0_mrv2/master.json0000666000175100017510000000074713245514472026342 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.0.0.mrv2", "node_processes": [ "ZooKeeper", "Webserver", "CLDB", "FileServer", "ResourceManager", "HistoryServer", "Oozie" ], "name": "mapr-500mr2-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/v5_2_0_mrv2/0000775000175100017510000000000013245515027024327 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/mapr/v5_2_0_mrv2/worker.json0000666000175100017510000000057713245514472026551 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.2.0.mrv2", "node_processes": [ "FileServer", "NodeManager" ], "name": "mapr-520mr2-default-worker", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/v5_2_0_mrv2/cluster.json0000666000175100017510000000104013245514472026703 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.2.0.mrv2", "node_groups": [ { "name": "master", "count": 1, "node_group_template_id": "{mapr-520mr2-default-master}" }, { "name": "worker", "count": 3, "node_group_template_id": "{mapr-520mr2-default-worker}" } ], "name": "mapr-520mr2-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/mapr/v5_2_0_mrv2/master.json0000666000175100017510000000124513245514472026524 0ustar zuulzuul00000000000000{ "plugin_name": "mapr", "hadoop_version": "5.2.0.mrv2", "node_processes": [ "Metrics", "Webserver", "ZooKeeper", "HTTPFS", "Oozie", "FileServer", "CLDB", "Flume", "Hue", "NodeManager", "HistoryServer", "ResourceManager", "HiveServer2", "HiveMetastore", "Sqoop2-Client", "Sqoop2-Server" ], "name": "mapr-520mr2-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/0000775000175100017510000000000013245515026022547 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/spark/v1_6_0/0000775000175100017510000000000013245515027023542 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/spark/v1_6_0/slave.json0000666000175100017510000000056013245514472025555 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "1.6.0", "node_processes": [ "datanode", "slave" ], "name": "spark-160-default-slave", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/v1_6_0/cluster.json0000666000175100017510000000102313245514472026117 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "1.6.0", "node_groups": [ { "name": "slave", "count": 3, "node_group_template_id": "{spark-160-default-slave}" }, { "name": "master", "count": 1, "node_group_template_id": "{spark-160-default-master}" } ], "name": "spark-160-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/v1_6_0/master.json0000666000175100017510000000056213245514472025740 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "1.6.0", "node_processes": [ "namenode", "master" ], "name": "spark-160-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/v2_1_0/0000775000175100017510000000000013245515027023536 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/spark/v2_1_0/slave.json0000666000175100017510000000056013245514472025551 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "2.1.0", "node_processes": [ "datanode", "slave" ], "name": "spark-210-default-slave", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/v2_1_0/cluster.json0000666000175100017510000000102313245514472026113 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "2.1.0", "node_groups": [ { "name": "slave", "count": 3, "node_group_template_id": "{spark-210-default-slave}" }, { "name": "master", "count": 1, "node_group_template_id": "{spark-210-default-master}" } ], "name": "spark-210-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/v2_1_0/master.json0000666000175100017510000000056213245514472025734 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "2.1.0", "node_processes": [ "namenode", "master" ], "name": "spark-210-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/v1_3_1/0000775000175100017510000000000013245515027023540 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/spark/v1_3_1/slave.json0000666000175100017510000000056013245514472025553 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "1.3.1", "node_processes": [ "datanode", "slave" ], "name": "spark-131-default-slave", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/v1_3_1/cluster.json0000666000175100017510000000102313245514472026115 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "1.3.1", "node_groups": [ { "name": "slave", "count": 3, "node_group_template_id": "{spark-131-default-slave}" }, { "name": "master", "count": 1, "node_group_template_id": "{spark-131-default-master}" } ], "name": "spark-131-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/spark/v1_3_1/master.json0000666000175100017510000000056213245514472025736 0ustar zuulzuul00000000000000{ "plugin_name": "spark", "hadoop_version": "1.3.1", "node_processes": [ "namenode", "master" ], "name": "spark-131-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/0000775000175100017510000000000013245515026022165 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_7_0/0000775000175100017510000000000013245515027023165 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_7_0/master-core.json0000666000175100017510000000073013245514472026306 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.7.0", "node_processes": [ "HDFS_NAMENODE", "YARN_RESOURCEMANAGER", "SENTRY_SERVER", "YARN_NODEMANAGER", "ZOOKEEPER_SERVER" ], "name": "cdh-570-default-master-core", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_7_0/manager.json0000666000175100017510000000056513245514472025505 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.7.0", "node_processes": [ "CLOUDERA_MANAGER", "KMS" ], "name": "cdh-570-default-manager", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_7_0/master-additional.json0000666000175100017510000000103413245514472027464 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.7.0", "node_processes": [ "OOZIE_SERVER", "YARN_JOBHISTORY", "YARN_NODEMANAGER", "HDFS_SECONDARYNAMENODE", "HIVE_METASTORE", "HIVE_SERVER2", "SPARK_YARN_HISTORY_SERVER" ], "name": "cdh-570-default-master-additional", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_7_0/worker-nm-dn.json0000666000175100017510000000067413245514472026414 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.7.0", "node_processes": [ "HDFS_DATANODE", "YARN_NODEMANAGER" ], "name": "cdh-570-default-nodemanager-datanode", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "volume_size": 2, "volumes_per_node": 2, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_7_0/cluster.json0000666000175100017510000000155213245514472025551 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.7.0", "node_groups": [ { "name": "worker-nodemanager-datanode", "count": 3, "node_group_template_id": "{cdh-570-default-nodemanager-datanode}" }, { "name": "manager", "count": 1, "node_group_template_id": "{cdh-570-default-manager}" }, { "name": "master-core", "count": 1, "node_group_template_id": "{cdh-570-default-master-core}" }, { "name": "master-additional", "count": 1, "node_group_template_id": "{cdh-570-default-master-additional}" } ], "name": "cdh-570-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_9_0/0000775000175100017510000000000013245515027023167 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_9_0/master-core.json0000666000175100017510000000073013245514472026310 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.9.0", "node_processes": [ "HDFS_NAMENODE", "YARN_RESOURCEMANAGER", "SENTRY_SERVER", "YARN_NODEMANAGER", "ZOOKEEPER_SERVER" ], "name": "cdh-590-default-master-core", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_9_0/manager.json0000666000175100017510000000056513245514472025507 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.9.0", "node_processes": [ "CLOUDERA_MANAGER", "KMS" ], "name": "cdh-590-default-manager", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_9_0/master-additional.json0000666000175100017510000000103413245514472027466 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.9.0", "node_processes": [ "OOZIE_SERVER", "YARN_JOBHISTORY", "YARN_NODEMANAGER", "HDFS_SECONDARYNAMENODE", "HIVE_METASTORE", "HIVE_SERVER2", "SPARK_YARN_HISTORY_SERVER" ], "name": "cdh-590-default-master-additional", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_9_0/worker-nm-dn.json0000666000175100017510000000067413245514472026416 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.9.0", "node_processes": [ "HDFS_DATANODE", "YARN_NODEMANAGER" ], "name": "cdh-590-default-nodemanager-datanode", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "volume_size": 2, "volumes_per_node": 2, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_9_0/cluster.json0000666000175100017510000000155213245514472025553 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.9.0", "node_groups": [ { "name": "worker-nodemanager-datanode", "count": 3, "node_group_template_id": "{cdh-590-default-nodemanager-datanode}" }, { "name": "manager", "count": 1, "node_group_template_id": "{cdh-590-default-manager}" }, { "name": "master-core", "count": 1, "node_group_template_id": "{cdh-590-default-master-core}" }, { "name": "master-additional", "count": 1, "node_group_template_id": "{cdh-590-default-master-additional}" } ], "name": "cdh-590-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_5_0/0000775000175100017510000000000013245515027023163 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_5_0/master-core.json0000666000175100017510000000073013245514472026304 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.5.0", "node_processes": [ "HDFS_NAMENODE", "YARN_RESOURCEMANAGER", "SENTRY_SERVER", "YARN_NODEMANAGER", "ZOOKEEPER_SERVER" ], "name": "cdh-550-default-master-core", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_5_0/manager.json0000666000175100017510000000056513245514472025503 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.5.0", "node_processes": [ "CLOUDERA_MANAGER", "KMS" ], "name": "cdh-550-default-manager", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_5_0/master-additional.json0000666000175100017510000000103413245514472027462 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.5.0", "node_processes": [ "OOZIE_SERVER", "YARN_JOBHISTORY", "YARN_NODEMANAGER", "HDFS_SECONDARYNAMENODE", "HIVE_METASTORE", "HIVE_SERVER2", "SPARK_YARN_HISTORY_SERVER" ], "name": "cdh-550-default-master-additional", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_5_0/worker-nm-dn.json0000666000175100017510000000067413245514472026412 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.5.0", "node_processes": [ "HDFS_DATANODE", "YARN_NODEMANAGER" ], "name": "cdh-550-default-nodemanager-datanode", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "volume_size": 2, "volumes_per_node": 2, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/cdh/v5_5_0/cluster.json0000666000175100017510000000155213245514472025547 0ustar zuulzuul00000000000000{ "plugin_name": "cdh", "hadoop_version": "5.5.0", "node_groups": [ { "name": "worker-nodemanager-datanode", "count": 3, "node_group_template_id": "{cdh-550-default-nodemanager-datanode}" }, { "name": "manager", "count": 1, "node_group_template_id": "{cdh-550-default-manager}" }, { "name": "master-core", "count": 1, "node_group_template_id": "{cdh-550-default-master-core}" }, { "name": "master-additional", "count": 1, "node_group_template_id": "{cdh-550-default-master-additional}" } ], "name": "cdh-550-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/vanilla/0000775000175100017510000000000013245515026023055 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/vanilla/v2_7_1/0000775000175100017510000000000013245515027024053 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/default_templates/vanilla/v2_7_1/worker.json0000666000175100017510000000065413245514472026271 0ustar zuulzuul00000000000000{ "plugin_name": "vanilla", "hadoop_version": "2.7.1", "node_processes": [ "datanode", "nodemanager" ], "name": "vanilla-271-default-worker", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "volume_size": 2, "volumes_per_node": 2, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/vanilla/v2_7_1/cluster.json0000666000175100017510000000103513245514472026433 0ustar zuulzuul00000000000000{ "plugin_name": "vanilla", "hadoop_version": "2.7.1", "node_groups": [ { "name": "worker", "count": 3, "node_group_template_id": "{vanilla-271-default-worker}" }, { "name": "master", "count": 1, "node_group_template_id": "{vanilla-271-default-master}" } ], "name": "vanilla-271-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {}, "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/vanilla/v2_7_1/master.json0000666000175100017510000000074013245514472026247 0ustar zuulzuul00000000000000{ "plugin_name": "vanilla", "hadoop_version": "2.7.1", "node_processes": [ "namenode", "resourcemanager", "historyserver", "oozie", "hive server", "spark history server" ], "name": "vanilla-271-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", "security_groups": "{security_groups}", "is_protected": true } sahara-8.0.0/sahara/plugins/default_templates/template.conf0000666000175100017510000000326413245514472024124 0ustar zuulzuul00000000000000[DEFAULT] # Set the flavor_id to 2 which is m1.small in the # default flavor set flavor_id = 2 auto_security_group = true [cdh-550-default-manager] # For the CDH plugin, version 550, set the flavor_id # of the manager node to 4 which is m1.large in the # default flavor set flavor_id = 4 [cdh-550-default-master-core] # For the CDH plugin, version 550, set the flavor_id # of the master-core node to 3 which is m1.medium in the # default flavor set flavor_id = 3 [cdh-550-default-master-additional] # For the CDH plugin, version 550, set the flavor_id # of the master-additional node to 3 which is m1.medium in the # default flavor set flavor_id = 3 [cdh-570-default-manager] # For the CDH plugin, version 570, set the flavor_id # of the manager node to 4 which is m1.large in the # default flavor set flavor_id = 4 [cdh-570-default-master-core] # For the CDH plugin, version 570, set the flavor_id # of the master-core node to 3 which is m1.medium in the # default flavor set flavor_id = 3 [cdh-570-default-master-additional] # For the CDH plugin, version 570, set the flavor_id # of the master-additional node to 3 which is m1.medium in the # default flavor set flavor_id = 3 [cdh-590-default-manager] # For the CDH plugin, version 590, set the flavor_id # of the manager node to 4 which is m1.large in the # default flavor set flavor_id = 4 [cdh-590-default-master-core] # For the CDH plugin, version 590, set the flavor_id # of the master-core node to 3 which is m1.medium in the # default flavor set flavor_id = 3 [cdh-590-default-master-additional] # For the CDH plugin, version 590, set the flavor_id # of the master-additional node to 3 which is m1.medium in the # default flavor set flavor_id = 3 sahara-8.0.0/sahara/plugins/storm/0000775000175100017510000000000013245515027017072 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/storm/plugin.py0000666000175100017510000003544213245514472020757 0ustar zuulzuul00000000000000# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from oslo_log import log as logging import six import yaml from sahara import conductor from sahara import context from sahara.i18n import _ from sahara.plugins import exceptions as ex from sahara.plugins import provisioning as p from sahara.plugins.storm import config_helper as c_helper from sahara.plugins.storm import edp_engine from sahara.plugins.storm import run_scripts as run from sahara.plugins import utils from sahara.utils import cluster_progress_ops as cpo from sahara.utils import general as ug from sahara.utils import remote conductor = conductor.API LOG = logging.getLogger(__name__) class StormProvider(p.ProvisioningPluginBase): def __init__(self): self.processes = { "Zookeeper": ["zookeeper"], "Storm": ["nimbus", "supervisor"] } def get_title(self): return "Apache Storm" def get_description(self): return ( _("This plugin provides an ability to launch Storm " "cluster without any management consoles.")) def get_labels(self): default = {'enabled': {'status': True}, 'stable': {'status': True}} deprecated = {'enabled': {'status': True}, 'deprecated': {'status': True}} result = {'plugin_labels': copy.deepcopy(default)} result['version_labels'] = { '1.1.0': copy.deepcopy(default), '1.0.1': copy.deepcopy(default), '0.9.2': copy.deepcopy(deprecated), } return result def get_versions(self): return ['0.9.2', '1.0.1', '1.1.0'] def get_configs(self, storm_version): return c_helper.get_plugin_configs() def get_node_processes(self, storm_version): return self.processes def validate(self, cluster): # validate Storm Master Node and Storm Slaves sm_count = sum([ng.count for ng in utils.get_node_groups(cluster, "nimbus")]) if sm_count < 1: raise ex.RequiredServiceMissingException("Storm nimbus") if sm_count >= 2: raise ex.InvalidComponentCountException("Storm nimbus", "1", sm_count) sl_count = sum([ng.count for ng in utils.get_node_groups(cluster, "supervisor")]) if sl_count < 1: raise ex.InvalidComponentCountException("Storm supervisor", _("1 or more"), sl_count) def update_infra(self, cluster): pass def configure_cluster(self, cluster): self._setup_instances(cluster) def start_cluster(self, cluster): sm_instance = utils.get_instance(cluster, "nimbus") sl_instances = utils.get_instances(cluster, "supervisor") zk_instances = utils.get_instances(cluster, "zookeeper") # start zookeeper processes self._start_zookeeper_processes(zk_instances) # start storm master if sm_instance: self._start_storm_master(sm_instance) # start storm slaves self._start_slave_processes(sl_instances) LOG.info("Cluster {cluster} has been started successfully".format( cluster=cluster.name)) self._set_cluster_info(cluster) def get_edp_engine(self, cluster, job_type): if job_type in edp_engine.EdpStormEngine.get_supported_job_types(): return edp_engine.EdpStormEngine(cluster) if job_type in edp_engine.EdpPyleusEngine.get_supported_job_types(): return edp_engine.EdpPyleusEngine(cluster) return None def get_edp_job_types(self, versions=None): res = {} for vers in self.get_versions(): if not versions or vers in versions: storm_engine = edp_engine.EdpStormEngine pyleus_engine = edp_engine.EdpPyleusEngine res[vers] = (storm_engine.get_supported_job_types() + pyleus_engine.get_supported_job_types()) return res def get_edp_config_hints(self, job_type, version): if edp_engine.EdpStormEngine.edp_supported(version): return edp_engine.EdpStormEngine.get_possible_job_config(job_type) if edp_engine.EdpPyleusEngine.edp_supported(version): return edp_engine.EdpPyleusEngine.get_possible_job_config(job_type) return {} def get_open_ports(self, node_group): ports_map = { 'nimbus': [8080] } ports = [] for process in node_group.node_processes: if process in ports_map: ports.extend(ports_map[process]) return ports def _extract_configs_to_extra(self, cluster): st_master = utils.get_instance(cluster, "nimbus") zk_servers = utils.get_instances(cluster, "zookeeper") extra = dict() config_instances = '' if st_master is not None: if zk_servers is not None: zknames = [] for zk in zk_servers: zknames.append(zk.hostname()) config_instances = c_helper.generate_storm_config( st_master.hostname(), zknames, cluster.hadoop_version) config = self._convert_dict_to_yaml(config_instances) supervisor_conf = c_helper.generate_slave_supervisor_conf() nimbus_ui_conf = c_helper.generate_master_supervisor_conf() zk_conf = c_helper.generate_zookeeper_conf() pyleus_conf = c_helper.generate_pyleus_config() for ng in cluster.node_groups: extra[ng.id] = { 'st_instances': config, 'slave_sv_conf': supervisor_conf, 'master_sv_conf': nimbus_ui_conf, 'zk_conf': zk_conf, 'pyleus_conf': pyleus_conf } return extra @cpo.event_wrapper( True, step=utils.start_process_event_message("StormMaster")) def _start_storm_master(self, sm_instance): with remote.get_remote(sm_instance) as r: run.start_storm_nimbus_and_ui(r) LOG.info("Storm master at {host} has been started".format( host=sm_instance.hostname())) def _start_slave_processes(self, sl_instances): if len(sl_instances) == 0: return cpo.add_provisioning_step( sl_instances[0].cluster_id, utils.start_process_event_message("Slave"), len(sl_instances)) with context.ThreadGroup() as tg: for i in sl_instances: tg.spawn('storm-start-sl-%s' % i.instance_name, self._start_slaves, i) @cpo.event_wrapper(True) def _start_slaves(self, instance): with instance.remote() as r: run.start_storm_supervisor(r) def _start_zookeeper_processes(self, zk_instances): if len(zk_instances) == 0: return cpo.add_provisioning_step( zk_instances[0].cluster_id, utils.start_process_event_message("Zookeeper"), len(zk_instances)) with context.ThreadGroup() as tg: for i in zk_instances: tg.spawn('storm-start-zk-%s' % i.instance_name, self._start_zookeeper, i) @cpo.event_wrapper(True) def _start_zookeeper(self, instance): with instance.remote() as r: run.start_zookeeper(r) def _setup_instances(self, cluster, instances=None): extra = self._extract_configs_to_extra(cluster) if instances is None: instances = utils.get_instances(cluster) self._push_configs_to_nodes(cluster, extra, instances) def _push_configs_to_nodes(self, cluster, extra, new_instances): all_instances = utils.get_instances(cluster) cpo.add_provisioning_step( cluster.id, _("Push configs to nodes"), len(all_instances)) with context.ThreadGroup() as tg: for instance in all_instances: if instance in new_instances: tg.spawn('storm-configure-%s' % instance.instance_name, self._push_configs_to_new_node, cluster, extra, instance) else: tg.spawn('storm-reconfigure-%s' % instance.instance_name, self._push_configs_to_existing_node, cluster, extra, instance) def _convert_dict_to_yaml(self, dict_to_convert): new_dict = dict_to_convert.copy() for key in dict_to_convert: if isinstance(dict_to_convert[key], six.string_types): new_dict[key] = "\"" + dict_to_convert[key] + "\"" stream = yaml.dump(new_dict, default_flow_style=False) stream = stream.replace("\'", "") return stream @cpo.event_wrapper(True) def _push_configs_to_new_node(self, cluster, extra, instance): ng_extra = extra[instance.node_group.id] files_supervisor = { '/etc/supervisor/supervisord.conf': ng_extra['slave_sv_conf'] } files_storm = { '/usr/local/storm/conf/storm.yaml': ng_extra['st_instances'] } files_zk = { '/opt/zookeeper/zookeeper/conf/zoo.cfg': ng_extra['zk_conf'] } files_supervisor_master = { '/etc/supervisor/supervisord.conf': ng_extra['master_sv_conf'] } file_pyleus_conf = { '/home/ubuntu/.pyleus.conf': ng_extra['pyleus_conf'] } with remote.get_remote(instance) as r: node_processes = instance.node_group.node_processes r.write_files_to(files_storm, run_as_root=True) if 'zookeeper' in node_processes: self._push_zk_configs(r, files_zk) if 'nimbus' in node_processes: self._push_supervisor_configs(r, files_supervisor_master) self._push_supervisor_configs(r, file_pyleus_conf) if 'supervisor' in node_processes: self._push_supervisor_configs(r, files_supervisor) @cpo.event_wrapper(True) def _push_configs_to_existing_node(self, cluster, extra, instance): node_processes = instance.node_group.node_processes need_storm_update = ('nimbus' in node_processes or 'supervisor' in node_processes) need_zookeeper_update = 'zookeeper' in node_processes ng_extra = extra[instance.node_group.id] r = remote.get_remote(instance) if need_storm_update: storm_path = '/usr/local/storm/conf/storm.yaml' files_storm = {storm_path: ng_extra['st_instances']} r.write_files_to(files_storm) if need_zookeeper_update: zk_path = '/opt/zookeeper/zookeeper/conf/zoo.cfg' files_zookeeper = {zk_path: ng_extra['zk_conf']} self._push_zk_configs(r, files_zookeeper) def _set_cluster_info(self, cluster): st_master = utils.get_instance(cluster, "nimbus") info = {} if st_master: port = "8080" info['Strom'] = { 'Web UI': 'http://%s:%s' % ( st_master.get_ip_or_dns_name(), port) } ctx = context.ctx() conductor.cluster_update(ctx, cluster, {'info': info}) def _push_zk_configs(self, r, files): r.write_files_to(files, run_as_root=True) def _push_supervisor_configs(self, r, files): r.append_to_files(files, run_as_root=True) # Scaling def _get_running_topologies_names(self, cluster): master = utils.get_instance(cluster, "nimbus") cmd = ( "%(storm)s -c nimbus.host=%(host)s " "list | grep ACTIVE | awk '{print $1}'") % ( { "storm": "/usr/local/storm/bin/storm", "host": master.hostname() }) with remote.get_remote(master) as r: ret, stdout = r.execute_command(cmd) names = stdout.split('\n') topology_names = names[0:len(names)-1] return topology_names @cpo.event_wrapper(True, step=_("Rebalance Topology"), param=('cluster', 1)) def rebalance_topology(self, cluster): topology_names = self._get_running_topologies_names(cluster) master = utils.get_instance(cluster, "nimbus") for topology_name in topology_names: cmd = ( '%(rebalance)s -c nimbus.host=%(host)s %(topology_name)s') % ( { "rebalance": "/usr/local/storm/bin/storm rebalance", "host": master.hostname(), "topology_name": topology_name }) with remote.get_remote(master) as r: ret, stdout = r.execute_command(cmd) def validate_scaling(self, cluster, existing, additional): self._validate_existing_ng_scaling(cluster, existing) self._validate_additional_ng_scaling(cluster, additional) def scale_cluster(self, cluster, instances): self._setup_instances(cluster, instances) # start storm slaves self._start_slave_processes(instances) self.rebalance_topology(cluster) LOG.info("Storm scaling has been started.") def _get_scalable_processes(self): return ["supervisor"] def _validate_additional_ng_scaling(self, cluster, additional): scalable_processes = self._get_scalable_processes() for ng_id in additional: ng = ug.get_by_id(cluster.node_groups, ng_id) if not set(ng.node_processes).issubset(scalable_processes): raise ex.NodeGroupCannotBeScaled( ng.name, _("Storm plugin cannot scale nodegroup" " with processes: %s") % ' '.join(ng.node_processes)) def _validate_existing_ng_scaling(self, cluster, existing): scalable_processes = self._get_scalable_processes() for ng in cluster.node_groups: if ng.id in existing: if not set(ng.node_processes).issubset(scalable_processes): raise ex.NodeGroupCannotBeScaled( ng.name, _("Storm plugin cannot scale nodegroup" " with processes: %s") % ' '.join(ng.node_processes)) sahara-8.0.0/sahara/plugins/storm/config_helper.py0000666000175100017510000001174413245514472022264 0ustar zuulzuul00000000000000# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from sahara import conductor as c from sahara import exceptions as ex from sahara.i18n import _ conductor = c.API CONF = cfg.CONF def get_config_value(service, name, cluster=None): if cluster: for ng in cluster.node_groups: if (ng.configuration().get(service) and ng.configuration()[service].get(name)): return ng.configuration()[service][name] raise ex.ConfigurationError( _("Unable to get parameter '%(param_name)s' from " "service %(service)s") % {'param_name': name, 'service': service}) def get_plugin_configs(): return {} def generate_storm_config(master_hostname, zk_hostnames, version): if version in ['1.0.1', '1.1.0']: host_cfg = 'nimbus.seeds' master_value = [master_hostname.encode('ascii', 'ignore')] else: host_cfg = 'nimbus.host' master_value = master_hostname.encode('ascii', 'ignore') cfg = { host_cfg: master_value, "worker.childopts": "-Xmx768m -Djava.net.preferIPv4Stack=true", "nimbus.childopts": "-Xmx1024m -Djava.net.preferIPv4Stack=true", "supervisor.childopts": "-Djava.net.preferIPv4Stack=true", "storm.zookeeper.servers": [i.encode('ascii', 'ignore') for i in zk_hostnames], "ui.childopts": "-Xmx768m -Djava.net.preferIPv4Stack=true", "storm.local.dir": "/app/storm" } # Since pyleus is built using previous versions os Storm we need this # option to allow the cluster to be compatible with pyleus topologies as # well as with topologies built using older versions of Storm if version in ['1.0.1', '1.1.0']: cfg['client.jartransformer.class'] = ( "org.apache.storm.hack.StormShadeTransformer") return cfg def generate_pyleus_config(): separator = "\n" conf = ("[storm]", "storm_cmd_path: /usr/local/storm/bin/storm") return separator.join(conf) def generate_slave_supervisor_conf(): separator = "\n" conf = ("[program:storm-supervisor]", 'command=bash -exec "cd /usr/local/storm && bin/storm supervisor"', "user=storm", "autostart=true", "autorestart=true", "startsecs=10", "startretries=999", "log_stdout=true", "log_stderr=true", "logfile=/var/log/storm/supervisor.out", "logfile_maxbytes=20MB", "logfile_backups=10") return separator.join(conf) def generate_master_supervisor_conf(): separator = "\n" seq_n = ("[program:storm-nimbus]", "command=/usr/local/storm/bin/storm nimbus", "user=storm", "autostart=true", "autorestart=true", "startsecs=10", "startretries=999", "log_stdout=true", "log_stderr=true", "logfile=/var/log/storm/supervisor.out", "logfile_maxbytes=20MB", "logfile_backups=10") seq_u = ("[program:storm-ui]", "command=/usr/local/storm/bin/storm ui", "user=storm", "autostart=true", "autorestart=true", "startsecs=10", "startretries=999", "log_stdout=true", "log_stderr=true", "logfile=/var/log/storm/ui.out", "logfile_maxbytes=20MB", "logfile_backups=10") conf_n = separator.join(seq_n) conf_u = separator.join(seq_u) conf = (conf_n, conf_u) return separator.join(conf) def generate_zookeeper_conf(): separator = "\n" conf = ("tickTime=2000", "dataDir=/var/zookeeper", "clientPort=2181") return separator.join(conf) def generate_storm_setup_script(env_configs): separator = "\n" script_lines = ["#!/bin/bash -x"] script_lines.append("echo -n > /usr/local/storm/conf/storm.yaml") for line in env_configs: script_lines.append('echo "%s" >> /usr/local/storm/conf/storm.yaml' % line) return separator.join(script_lines) def extract_name_values(configs): return {cfg['name']: cfg['value'] for cfg in configs} def _set_config(cfg, gen_cfg, name=None): if name in gen_cfg: cfg.update(gen_cfg[name]['conf']) if name is None: for name in gen_cfg: cfg.update(gen_cfg[name]['conf']) return cfg sahara-8.0.0/sahara/plugins/storm/run_scripts.py0000666000175100017510000000310513245514472022023 0ustar zuulzuul00000000000000# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. def start_zookeeper(remote): remote.execute_command("sudo %s %s" % ( "/opt/zookeeper/zookeeper/bin/zkServer.sh", "start")) def start_storm_supervisor(node): _create_supervisor_log_file(node) _stop_supervisor_deamon(node) _start_supervisor_deamon(node) def start_storm_nimbus_and_ui(node): _create_supervisor_log_file(node) _stop_supervisor_deamon(node) _start_supervisor_deamon(node) def stop_storm_nimbus_and_ui(node): _stop_supervisor_deamon(node) def stop_storm_supervisor(node): _stop_supervisor_deamon(node) def _start_supervisor_deamon(node): node.execute_command("sudo service supervisor start") def _stop_supervisor_deamon(node): node.execute_command("sudo service supervisor stop") def _create_supervisor_log_file(node): node.execute_command("sudo mkdir -p /var/log/storm") node.execute_command("sudo chmod -R 777 /var/log/storm") node.execute_command("sudo chown -R storm:storm /var/log/storm") sahara-8.0.0/sahara/plugins/storm/__init__.py0000666000175100017510000000000013245514472021176 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/storm/edp_engine.py0000666000175100017510000000351513245514472021552 0ustar zuulzuul00000000000000# Copyright (c) 2015 Telles Nobrega. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.edp.storm import engine as edp_engine class EdpStormEngine(edp_engine.StormJobEngine): edp_base_version = "0.9.2" @staticmethod def edp_supported(version): return version >= EdpStormEngine.edp_base_version def validate_job_execution(self, cluster, job, data): if not self.edp_supported(cluster.hadoop_version): raise ex.InvalidDataException( _('Storm {base} required to run {type} jobs').format( base=EdpStormEngine.edp_base_version, type=job.type)) super(EdpStormEngine, self).validate_job_execution(cluster, job, data) class EdpPyleusEngine(edp_engine.StormPyleusJobEngine): edp_base_version = "0.9.2" @staticmethod def edp_supported(version): return version >= EdpPyleusEngine.edp_base_version def validate_job_execution(self, cluster, job, data): if not self.edp_supported(cluster.hadoop_version): raise ex.InvalidDataException( _('Storm {base} required to run {type} jobs').format( base=EdpPyleusEngine.edp_base_version, type=job.type)) super(EdpPyleusEngine, self).validate_job_execution(cluster, job, data) sahara-8.0.0/sahara/plugins/recommendations_utils.py0000666000175100017510000003573613245514472022732 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc from oslo_log import log as logging import six from sahara import conductor as cond from sahara import context from sahara.utils.openstack import nova conductor = cond.API LOG = logging.getLogger(__name__) @six.add_metaclass(abc.ABCMeta) class AutoConfigsProvider(object): def __init__(self, mapper, plugin_configs, cluster, scaling): """This meta class provides general recommendation utils for cluster configuration. :param mapper: dictionary, that describes which cluster configs and node_configs to configure. It should maps to following dicts: node_configs to configure and cluster_configs to configure. This dicts should contains abstract names of configs as keys and tuple (correct_applicable_target, correct_name) as values. Such representation allows to use same AutoConfigsProvider for plugins with almost same configs and configuring principles. :param plugin_configs: all plugins_configs for specified plugin :param cluster: cluster which is required to configure :param scaling: indicates that current cluster operation is scaling """ self.plugin_configs = plugin_configs self.cluster = cluster self.node_configs_to_update = mapper.get('node_configs', {}) self.cluster_configs_to_update = mapper.get('cluster_configs', {}) self.scaling = scaling @abc.abstractmethod def _get_recommended_node_configs(self, node_group): """Method calculates and returns recommended configs for node_group. It's not required to update node_configs of node_group using the conductor api in this method, because it will be done in the method apply_node_configs. :param node_group: NodeGroup Sahara resource. :return: dictionary with calculated recommended configs for node_group. """ pass @abc.abstractmethod def _get_recommended_cluster_configs(self): """Method calculates and returns recommended configs for cluster. It's not required to update cluster_configs of cluster using the conductor api in this method, because it will be done in the method apply_cluster_configs. :return: dictionary with calculated recommended configs for cluster. """ pass def _can_be_recommended(self, configs_list, node_group=None): """Method calculates and returns True, when it's possible to automatically configure provided list of configs configs_list. Otherwise, method should return False. :param configs_list: list of configs which we want to configure :param node_group: optional argument, which should be provided if some config can be used in node_configs of some node_group :return: True if all configs can be configured and False otherwise """ # cluster configs is Frozen Dict, so let's call to_dict() cl_configs = self.cluster.cluster_configs.to_dict() for ncfg in configs_list: section, name = self._get_correct_section_and_name(ncfg) if section in cl_configs and name in cl_configs[section]: return False if not node_group: return True cl_configs = node_group.node_configs.to_dict() for ncfg in configs_list: section, name = self._get_correct_section_and_name(ncfg) if section in cl_configs and name in cl_configs[section]: return False return True def _get_correct_section_and_name(self, config_name): """Calculates and returns correct applicable target and name from abstract name of config. :param config_name: abstract name of config. :return: correct applicable target and name for config_name """ section, name = None, None if config_name in self.cluster_configs_to_update: section = self.cluster_configs_to_update[config_name][0] name = self.cluster_configs_to_update[config_name][1] elif config_name in self.node_configs_to_update: section = self.node_configs_to_update[config_name][0] name = self.node_configs_to_update[config_name][1] return section, name def _get_default_config_value(self, config_name): """Calculates and return default value of config from abstract name of config. :param config_name: abstract name of config. :return: default config value for config_name. """ section, name = self._get_correct_section_and_name(config_name) for config in self.plugin_configs: if config.applicable_target == section and config.name == name: return config.default_value def _merge_configs(self, current_configs, proposed_configs): """Correctly merges old configs and new extra configs""" result = {} for (section, configs) in six.iteritems(proposed_configs): cfg_values = {} if section in current_configs: cfg_values = (current_configs[section] if current_configs[section] else {}) cfg_values.update(configs) result.update({section: cfg_values}) for (section, configs) in six.iteritems(current_configs): if section not in result: result.update({section: configs}) return result def _get_cluster_extra(self): cluster = self.cluster return cluster.extra.to_dict() if cluster.extra else {} def finalize_autoconfiguration(self): if not self.cluster.use_autoconfig: return cluster_extra = self._get_cluster_extra() cluster_extra['auto-configured'] = True conductor.cluster_update( context.ctx(), self.cluster, {'extra': cluster_extra}) def apply_node_configs(self, node_group): """Method applies configs for node_group using conductor api, which were calculated with recommend_node_configs method. :param node_group: NodeGroup Sahara resource. :return: None. """ if not node_group.use_autoconfig or not self.cluster.use_autoconfig: return to_update = self.node_configs_to_update recommended_node_configs = self._get_recommended_node_configs( node_group) if not recommended_node_configs: # Nothing to configure return current_dict = node_group.node_configs.to_dict() configuration = {} for ncfg in six.iterkeys(to_update): if ncfg not in recommended_node_configs: continue n_section = to_update[ncfg][0] n_name = to_update[ncfg][1] proposed_config_value = recommended_node_configs[ncfg] if n_section not in configuration: configuration.update({n_section: {}}) configuration[n_section].update({n_name: proposed_config_value}) current_dict = self._merge_configs(current_dict, configuration) conductor.node_group_update(context.ctx(), node_group, {'node_configs': current_dict}) def apply_cluster_configs(self): """Method applies configs for cluster using conductor api, which were calculated with recommend_cluster_configs method. :return: None. """ cluster = self.cluster if not cluster.use_autoconfig: return to_update = self.cluster_configs_to_update recommended_cluster_configs = self._get_recommended_cluster_configs() if not recommended_cluster_configs: # Nothing to configure return current_dict = cluster.cluster_configs.to_dict() configuration = {} for ncfg in six.iterkeys(to_update): if ncfg not in recommended_cluster_configs: continue n_section = to_update[ncfg][0] n_name = to_update[ncfg][1] proposed_config_value = recommended_cluster_configs[ncfg] if n_section not in configuration: configuration.update({n_section: {}}) configuration[n_section].update({n_name: proposed_config_value}) current_dict = self._merge_configs(current_dict, configuration) conductor.cluster_update(context.ctx(), cluster, {'cluster_configs': current_dict}) def apply_recommended_configs(self): """Method applies recommended configs for cluster and for all node_groups using conductor api. :return: None. """ if self.scaling: # Validate cluster is not an old created cluster cluster_extra = self._get_cluster_extra() if 'auto-configured' not in cluster_extra: # Don't configure return for ng in self.cluster.node_groups: self.apply_node_configs(ng) self.apply_cluster_configs() configs = list(self.cluster_configs_to_update.keys()) configs.extend(list(self.node_configs_to_update.keys())) LOG.debug("Following configs were auto-configured: {configs}".format( configs=configs)) self.finalize_autoconfiguration() class HadoopAutoConfigsProvider(AutoConfigsProvider): def __init__(self, mapper, plugin_configs, cluster, scaling, hbase=False): super(HadoopAutoConfigsProvider, self).__init__( mapper, plugin_configs, cluster, scaling) self.requested_flavors = {} self.is_hbase_enabled = hbase def _get_java_opts(self, value): return "-Xmx%dm" % int(value) def _transform_mb_to_gb(self, mb): return mb / 1024. def _transform_gb_to_mb(self, gb): return gb * 1024. def _get_min_size_of_container(self, ram): if ram <= 4: return 256 if ram <= 8: return 512 if ram <= 24: return 1024 return 2048 def _get_os_ram_recommendation(self, ram): upper_bounds = [4, 8, 16, 24, 48, 64, 72, 96, 128, 256] reserve_for_os = [1, 2, 2, 4, 6, 8, 8, 12, 24, 32] for (upper, reserve) in zip(upper_bounds, reserve_for_os): if ram <= upper: return reserve return 64 def _get_hbase_ram_recommendations(self, ram): if not self.is_hbase_enabled: return 0 upper_bounds = [4, 8, 16, 24, 48, 64, 72, 96, 128, 256] reserve_for_hbase = [1, 1, 2, 4, 8, 8, 8, 16, 24, 32] for (upper, reserve) in zip(upper_bounds, reserve_for_hbase): if ram <= upper: return reserve return 64 def _get_node_group_data(self, node_group): if node_group.flavor_id not in self.requested_flavors: flavor = nova.get_flavor(id=node_group.flavor_id) self.requested_flavors[node_group.flavor_id] = flavor else: flavor = self.requested_flavors[node_group.flavor_id] cpu = flavor.vcpus ram = flavor.ram data = {} # config recommendations was taken from Ambari code os = self._get_os_ram_recommendation(self._transform_mb_to_gb(ram)) hbase = self._get_hbase_ram_recommendations( self._transform_mb_to_gb(ram)) reserved_ram = self._transform_gb_to_mb(os + hbase) min_container_size = self._get_min_size_of_container( self._transform_mb_to_gb(ram)) # we use large amount of containers to allow users to run # at least 4 jobs at same time on clusters based on small flavors data["containers"] = int(max( 8, min(2 * cpu, ram / min_container_size))) data["ramPerContainer"] = (ram - reserved_ram) / data["containers"] data["ramPerContainer"] = max(data["ramPerContainer"], min_container_size) data["ramPerContainer"] = min(2048, int(data["ramPerContainer"])) data["ramPerContainer"] = int(data["ramPerContainer"]) data["mapMemory"] = int(data["ramPerContainer"]) data["reduceMemory"] = int(2 * data["ramPerContainer"]) data["amMemory"] = int(min(data["mapMemory"], data["reduceMemory"])) return data def _get_recommended_node_configs(self, node_group): """Calculates recommended MapReduce and YARN configs for specified node_group. :param node_group: NodeGroup Sahara resource :return: dictionary with recommended MapReduce and YARN configs """ configs_to_update = list(self.node_configs_to_update.keys()) if not self._can_be_recommended(configs_to_update, node_group): return {} data = self._get_node_group_data(node_group) r = {} r['yarn.nodemanager.resource.memory-mb'] = (data['containers'] * data['ramPerContainer']) r['yarn.scheduler.minimum-allocation-mb'] = data['ramPerContainer'] r['yarn.scheduler.maximum-allocation-mb'] = (data['containers'] * data['ramPerContainer']) r['yarn.nodemanager.vmem-check-enabled'] = "false" r['yarn.app.mapreduce.am.resource.mb'] = data['amMemory'] r['yarn.app.mapreduce.am.command-opts'] = self._get_java_opts( 0.8 * data['amMemory']) r['mapreduce.map.memory.mb'] = data['mapMemory'] r['mapreduce.reduce.memory.mb'] = data['reduceMemory'] r['mapreduce.map.java.opts'] = self._get_java_opts( 0.8 * data['mapMemory']) r['mapreduce.reduce.java.opts'] = self._get_java_opts( 0.8 * data['reduceMemory']) r['mapreduce.task.io.sort.mb'] = int(min( 0.4 * data['mapMemory'], 1024)) return r def get_datanode_name(self): return "datanode" def _get_recommended_cluster_configs(self): """Method recommends dfs_replication for cluster. :return: recommended value of dfs_replication. """ if not self._can_be_recommended(['dfs.replication']): return {} datanode_count = 0 datanode_proc_name = self.get_datanode_name() for ng in self.cluster.node_groups: if datanode_proc_name in ng.node_processes: datanode_count += ng.count replica = 'dfs.replication' recommended_value = self._get_default_config_value(replica) if recommended_value: return {replica: min(recommended_value, datanode_count)} else: return {} sahara-8.0.0/sahara/plugins/opts.py0000666000175100017510000000164413245514472017277 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # File contains plugins opts to avoid cyclic imports issue from oslo_config import cfg opts = [ cfg.ListOpt('plugins', default=['vanilla', 'spark', 'cdh', 'ambari', 'storm', 'mapr'], help='List of plugins to be loaded. Sahara preserves the ' 'order of the list when returning it.'), ] CONF = cfg.CONF CONF.register_opts(opts) sahara-8.0.0/sahara/plugins/exceptions.py0000666000175100017510000001532513245514472020474 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import sahara.exceptions as e from sahara.i18n import _ class NodeGroupCannotBeScaled(e.SaharaException): def __init__(self, ng_name, reason): self.message = _("Chosen node group %(ng_name)s cannot be scaled : " "%(reason)s") % {"ng_name": ng_name, "reason": reason} self.code = "NODE_GROUP_CANNOT_BE_SCALED" super(NodeGroupCannotBeScaled, self).__init__() class DecommissionError(e.SaharaException): code = "DECOMMISSION_ERROR" message = _("Failed to decommission cluster") def __init__(self, message=None): if message: self.message = message super(DecommissionError, self).__init__() class ClusterCannotBeScaled(e.SaharaException): def __init__(self, cluster_name, reason): self.message = _("Cluster %(cluster_name)s cannot be scaled : " "%(reason)s") % {"cluster_name": cluster_name, "reason": reason} self.code = "CLUSTER_CANNOT_BE_SCALED" super(ClusterCannotBeScaled, self).__init__() class RequiredServiceMissingException(e.SaharaException): """Exception indicating that a required service has not been deployed.""" def __init__(self, service_name, required_by=None): self.message = (_('Cluster is missing a service: %s') % service_name) if required_by: self.message = (_('%(message)s, required by service: ' '%(required_by)s') % {'message': self.message, 'required_by': required_by}) self.code = 'MISSING_SERVICE' super(RequiredServiceMissingException, self).__init__() class InvalidComponentCountException(e.SaharaException): """Exception indicating invalid number of components in a cluster. A message indicating a number of components cluster should contain and an invalid number of components are being deployed in a cluster. """ def __init__(self, component, expected_count, count, description=None): message = _("Hadoop cluster should contain %(expected_count)s " "%(component)s component(s)." " Actual %(component)s count is %(count)s") if description: message = ("%(message)s. %(description)s" % {'message': message, 'description': description}) self.message = message % {"expected_count": expected_count, "component": component, "count": count} self.code = "INVALID_COMPONENT_COUNT" super(InvalidComponentCountException, self).__init__() class InvalidClusterTopology(e.SaharaException): """Exception indicating another problems in a cluster topology, which is different from InvalidComponentCountException and RequiredServiceMissingException. """ code = "INVALID_TOPOLOGY" message = _("Cluster has invalid topology: {description}") def __init__(self, description): self.message = self.message.format(description=description) super(InvalidClusterTopology, self).__init__() class HadoopProvisionError(e.SaharaException): """Exception indicating that cluster provisioning failed. A message indicating the reason for failure must be provided. """ base_message = _("Failed to Provision Hadoop Cluster: %s") def __init__(self, message): self.code = "HADOOP_PROVISION_FAILED" self.message = self.base_message % message super(HadoopProvisionError, self).__init__() class NameNodeHAConfigurationError(e.SaharaException): """Exception indicating that hdp or cdh HDFS HA failed. A message indicating the reason for failure must be provided. """ base_message = _("NameNode High Availability: %s") def __init__(self, message): self.code = "NAMENODE_HIGHAVAILABILITY_CONFIGURATION_FAILED" self.message = self.base_message % message super(NameNodeHAConfigurationError, self).__init__() class ResourceManagerHAConfigurationError(e.SaharaException): """Exception indicating that cdh YARN HA failed. A message indicating the reason for failure must be provided. """ base_message = _("ResourceManager High Availability: %s") def __init__(self, message): self.code = "RESOURCEMANAGER_HIGHAVAILABILITY_CONFIGURATION_FAILED" self.message = self.base_message % message super(ResourceManagerHAConfigurationError, self).__init__() class ImageValidationSpecificationError(e.SaharaException): """Exception indicating that an image validation spec is in error.""" base_message = _("Image validation spec is in error: %s") def __init__(self, message, *args): self.code = "IMAGE_SPECIFICATION_ERROR" self.message = self.base_message % message super(ImageValidationSpecificationError, self).__init__() class ImageValidationError(e.SaharaException): """Exception indicating that an image has failed validation.""" base_message = _("Image has failed validation: %s") def __init__(self, message): self.code = "IMAGE_VALIDATION_FAILED" self.message = self.base_message % message super(ImageValidationError, self).__init__() class AllValidationsFailedError(ImageValidationError): """Exception indicating that all validations in an any block failed.""" sub_message = _("All validations have failed: %s") def __init__(self, exceptions): data = ";".join(ex.message for ex in exceptions) message = self.sub_message % data super(AllValidationsFailedError, self).__init__(message) class InvalidVolumeSizeException(e.SaharaException): """Exception indicating invalid configuration of components in a cluster. """ def __init__(self, volume_size, reserved): message = _("Volume size: %(volume_size)s GB should be greater than " "value of \"dfs_datanode_du_reserved\": %(reserved)s GB") self.message = message % {"volume_size": volume_size, "reserved": reserved} self.code = "INVALID_CONFIGURATION" super(InvalidVolumeSizeException, self).__init__() sahara-8.0.0/sahara/plugins/base.py0000666000175100017510000001124513245514472017222 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc from oslo_config import cfg from oslo_log import log as logging import six from stevedore import enabled from sahara import conductor as cond from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import labels from sahara.utils import resources conductor = cond.API LOG = logging.getLogger(__name__) CONF = cfg.CONF def required(fun): return abc.abstractmethod(fun) def required_with_default(fun): return fun def optional(fun): fun.__not_implemented__ = True return fun @six.add_metaclass(abc.ABCMeta) class PluginInterface(resources.BaseResource): __resource_name__ = 'plugin' name = 'plugin_interface' @required def get_title(self): """Plugin title For example: "Vanilla Provisioning" """ pass @required_with_default def get_description(self): """Optional description of the plugin This information is targeted to be displayed in UI. """ pass def to_dict(self): return { 'name': self.name, 'title': self.get_title(), 'description': self.get_description(), } class PluginManager(object): def __init__(self): self.plugins = {} self.default_label_schema = {} self._load_cluster_plugins() self.label_handler = labels.LabelHandler(self.plugins) def _load_cluster_plugins(self): config_plugins = CONF.plugins extension_manager = enabled.EnabledExtensionManager( check_func=lambda ext: ext.name in config_plugins, namespace='sahara.cluster.plugins', invoke_on_load=True ) for ext in extension_manager.extensions: if ext.name in self.plugins: raise ex.ConfigurationError( _("Plugin with name '%s' already exists.") % ext.name) ext.obj.name = ext.name self.plugins[ext.name] = ext.obj LOG.info("Plugin {plugin_name} loaded {entry_point}".format( plugin_name=ext.name, entry_point=ext.entry_point_target)) if len(self.plugins) < len(config_plugins): loaded_plugins = set(six.iterkeys(self.plugins)) requested_plugins = set(config_plugins) raise ex.ConfigurationError( _("Plugins couldn't be loaded: %s") % ", ".join(requested_plugins - loaded_plugins)) def get_plugins(self, serialized=False): if serialized: return [self.serialize_plugin(name) for name in CONF.plugins] return [self.get_plugin(name) for name in CONF.plugins] def get_plugin(self, plugin_name): return self.plugins.get(plugin_name) def is_plugin_implements(self, plugin_name, fun_name): plugin = self.get_plugin(plugin_name) fun = getattr(plugin, fun_name) if not (fun and callable(fun)): return False return not hasattr(fun, '__not_implemented__') def serialize_plugin(self, plugin_name, version=None): plugin = self.get_plugin(plugin_name) if plugin: res = plugin.as_resource() res._info.update(self.label_handler.get_label_full_details( plugin_name)) if version: if version in plugin.get_versions(): res._info.update(plugin.get_version_details(version)) else: return None return res def update_plugin(self, plugin_name, values): self.label_handler.update_plugin(plugin_name, values) return self.serialize_plugin(plugin_name) def validate_plugin_update(self, plugin_name, values): return self.label_handler.validate_plugin_update(plugin_name, values) def get_plugin_update_validation_jsonschema(self): return self.label_handler.get_plugin_update_validation_jsonschema() def validate_plugin_labels(self, plugin, version): self.label_handler.validate_plugin_labels(plugin, version) PLUGINS = None def setup_plugins(): global PLUGINS PLUGINS = PluginManager() sahara-8.0.0/sahara/plugins/mapr/0000775000175100017510000000000013245515027016665 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/versions/0000775000175100017510000000000013245515027020535 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/versions/v5_1_0_mrv2/0000775000175100017510000000000013245515027022474 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/versions/v5_1_0_mrv2/__init__.py0000666000175100017510000000000013245514472024600 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/versions/v5_1_0_mrv2/version_handler.py0000666000175100017510000000527313245514472026244 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.base.base_version_handler as bvh from sahara.plugins.mapr.services.drill import drill from sahara.plugins.mapr.services.flume import flume from sahara.plugins.mapr.services.hbase import hbase from sahara.plugins.mapr.services.hive import hive from sahara.plugins.mapr.services.httpfs import httpfs from sahara.plugins.mapr.services.hue import hue from sahara.plugins.mapr.services.impala import impala from sahara.plugins.mapr.services.mahout import mahout from sahara.plugins.mapr.services.management import management as mng from sahara.plugins.mapr.services.maprfs import maprfs from sahara.plugins.mapr.services.oozie import oozie from sahara.plugins.mapr.services.pig import pig from sahara.plugins.mapr.services.spark import spark from sahara.plugins.mapr.services.sqoop import sqoop2 from sahara.plugins.mapr.services.swift import swift from sahara.plugins.mapr.services.yarn import yarn import sahara.plugins.mapr.versions.v5_1_0_mrv2.context as c version = "5.1.0.mrv2" class VersionHandler(bvh.BaseVersionHandler): def __init__(self): super(VersionHandler, self).__init__() self._version = version self._required_services = [ yarn.YARNv270(), maprfs.MapRFS(), mng.Management(), oozie.Oozie(), ] self._services = [ hive.HiveV013(), hive.HiveV12(), impala.ImpalaV220(), pig.PigV014(), pig.PigV015(), flume.FlumeV16(), flume.FlumeV15(), sqoop2.Sqoop2(), mahout.MahoutV010(), oozie.OozieV410(), oozie.OozieV420(), hue.HueV381(), hue.HueV390(), hbase.HBaseV09812(), drill.DrillV11(), drill.DrillV12(), drill.DrillV14(), yarn.YARNv270(), maprfs.MapRFS(), mng.Management(), httpfs.HttpFS(), swift.Swift(), spark.SparkOnYarn() ] def get_context(self, cluster, added=None, removed=None): return c.Context(cluster, self, added, removed) sahara-8.0.0/sahara/plugins/mapr/versions/v5_1_0_mrv2/context.py0000666000175100017510000000433713245514472024546 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.base.base_cluster_context as bc import sahara.plugins.mapr.services.yarn.yarn as yarn class Context(bc.BaseClusterContext): def __init__(self, cluster, version_handler, added=None, removed=None): super(Context, self).__init__(cluster, version_handler, added, removed) self._hadoop_version = yarn.YARNv270().version self._hadoop_lib = None self._hadoop_conf = None self._cluster_mode = yarn.YARNv270.cluster_mode self._node_aware = True self._resource_manager_uri = "maprfs:///" self._mapr_version = "5.1.0" self._ubuntu_ecosystem_repo = ( "http://package.mapr.com/releases/ecosystem-5.x/ubuntu binary/") self._centos_ecosystem_repo = ( "http://package.mapr.com/releases/ecosystem-5.x/redhat") @property def hadoop_lib(self): if not self._hadoop_lib: self._hadoop_lib = "%s/share/hadoop/common" % self.hadoop_home return self._hadoop_lib @property def hadoop_conf(self): if not self._hadoop_conf: self._hadoop_conf = "%s/etc/hadoop" % self.hadoop_home return self._hadoop_conf @property def resource_manager_uri(self): return self._resource_manager_uri @property def configure_sh(self): if not self._configure_sh: configure_sh_template = "%(base)s -HS %(history_server)s" args = { "base": super(Context, self).configure_sh, "history_server": self.get_historyserver_ip(), } self._configure_sh = configure_sh_template % args return self._configure_sh sahara-8.0.0/sahara/plugins/mapr/versions/__init__.py0000666000175100017510000000000013245514472022641 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/versions/version_handler_factory.py0000666000175100017510000000334313245514472026030 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import os def _load_versions(): d_name = os.path.dirname(__file__) m_template = 'sahara.plugins.mapr.versions.%s.version_handler' def predicate(v_dir): if v_dir != "__pycache__": v_dir = os.path.join(d_name, v_dir) return os.path.isdir(v_dir) and os.listdir(v_dir) return False def mapper(v_dir): return m_template % v_dir v_dirs = filter(predicate, os.listdir(d_name)) m_names = map(mapper, v_dirs) versions = {} for m_name in m_names: m = __import__(m_name, fromlist=['sahara']) versions[m.version] = getattr(m, 'VersionHandler')() return versions class VersionHandlerFactory(object): instance = None versions = None @staticmethod def get(): if not VersionHandlerFactory.instance: VersionHandlerFactory.versions = _load_versions() VersionHandlerFactory.instance = VersionHandlerFactory() return VersionHandlerFactory.instance def get_versions(self): return list(VersionHandlerFactory.versions.keys()) def get_handler(self, version): return VersionHandlerFactory.versions[version] sahara-8.0.0/sahara/plugins/mapr/versions/v5_2_0_mrv2/0000775000175100017510000000000013245515027022475 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/versions/v5_2_0_mrv2/__init__.py0000666000175100017510000000000013245514472024601 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/versions/v5_2_0_mrv2/version_handler.py0000666000175100017510000000537013245514472026243 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.base.base_version_handler as bvh from sahara.plugins.mapr.services.drill import drill from sahara.plugins.mapr.services.flume import flume from sahara.plugins.mapr.services.hbase import hbase from sahara.plugins.mapr.services.hive import hive from sahara.plugins.mapr.services.httpfs import httpfs from sahara.plugins.mapr.services.hue import hue from sahara.plugins.mapr.services.impala import impala from sahara.plugins.mapr.services.kafka import kafka from sahara.plugins.mapr.services.mahout import mahout from sahara.plugins.mapr.services.management import management as mng from sahara.plugins.mapr.services.maprfs import maprfs from sahara.plugins.mapr.services.oozie import oozie from sahara.plugins.mapr.services.pig import pig from sahara.plugins.mapr.services.sentry import sentry from sahara.plugins.mapr.services.spark import spark from sahara.plugins.mapr.services.sqoop import sqoop2 from sahara.plugins.mapr.services.swift import swift from sahara.plugins.mapr.services.yarn import yarn import sahara.plugins.mapr.versions.v5_2_0_mrv2.context as c version = "5.2.0.mrv2" class VersionHandler(bvh.BaseVersionHandler): def __init__(self): super(VersionHandler, self).__init__() self._version = version self._required_services = [ yarn.YARNv270(), maprfs.MapRFS(), mng.Management(), oozie.Oozie(), ] self._services = [ hive.HiveV12(), pig.PigV016(), impala.ImpalaV250(), flume.FlumeV16(), sqoop2.Sqoop2(), mahout.MahoutV012(), oozie.OozieV420(), hue.HueV310(), hue.HueLivyV310(), hbase.HBaseV111(), drill.DrillV19(), yarn.YARNv270(), maprfs.MapRFS(), mng.Management(), httpfs.HttpFS(), swift.Swift(), sentry.SentryV16(), spark.SparkOnYarnV201(), kafka.Kafka(), kafka.KafkaConnect(), kafka.KafkaRest(), ] def get_context(self, cluster, added=None, removed=None): return c.Context(cluster, self, added, removed) sahara-8.0.0/sahara/plugins/mapr/versions/v5_2_0_mrv2/context.py0000666000175100017510000000436513245514472024550 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.base.base_cluster_context as bc import sahara.plugins.mapr.services.yarn.yarn as yarn class Context(bc.BaseClusterContext): def __init__(self, cluster, version_handler, added=None, removed=None): super(Context, self).__init__(cluster, version_handler, added, removed) self._hadoop_version = yarn.YARNv270().version self._hadoop_lib = None self._hadoop_conf = None self._cluster_mode = yarn.YARNv270.cluster_mode self._node_aware = True self._resource_manager_uri = "maprfs:///" self._mapr_version = "5.2.0" self._ubuntu_ecosystem_repo = ( "http://package.mapr.com/releases/MEP/MEP-2.0.0/ubuntu/" " binary trusty") self._centos_ecosystem_repo = ( "http://package.mapr.com/releases/MEP/MEP-2.0.0/redhat") @property def hadoop_lib(self): if not self._hadoop_lib: self._hadoop_lib = "%s/share/hadoop/common" % self.hadoop_home return self._hadoop_lib @property def hadoop_conf(self): if not self._hadoop_conf: self._hadoop_conf = "%s/etc/hadoop" % self.hadoop_home return self._hadoop_conf @property def resource_manager_uri(self): return self._resource_manager_uri @property def configure_sh(self): if not self._configure_sh: configure_sh_template = "%(base)s -HS %(history_server)s" args = { "base": super(Context, self).configure_sh, "history_server": self.get_historyserver_ip(), } self._configure_sh = configure_sh_template % args return self._configure_sh sahara-8.0.0/sahara/plugins/mapr/abstract/0000775000175100017510000000000013245515027020470 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/abstract/cluster_validator.py0000666000175100017510000000154313245514472024600 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import abc import six @six.add_metaclass(abc.ABCMeta) class AbstractValidator(object): @abc.abstractmethod def validate(self, cluster_context): pass @abc.abstractmethod def validate_scaling(self, cluster_context, existing, additional): pass sahara-8.0.0/sahara/plugins/mapr/abstract/node_manager.py0000666000175100017510000000203213245514472023463 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import abc import six @six.add_metaclass(abc.ABCMeta) class AbstractNodeManager(object): @abc.abstractmethod def start(self, cluster_context, instances=None): pass @abc.abstractmethod def stop(self, cluster_context, instances=None): pass @abc.abstractmethod def move_nodes(self, cluster_context, instances): pass @abc.abstractmethod def remove_nodes(self, cluster_context, instances): pass sahara-8.0.0/sahara/plugins/mapr/abstract/health_checker.py0000666000175100017510000000141313245514472023777 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import abc import six @six.add_metaclass(abc.ABCMeta) class AbstractHealthChecker(object): @abc.abstractmethod def get_checks(self, cluster_context, instances=None): pass sahara-8.0.0/sahara/plugins/mapr/abstract/configurer.py0000666000175100017510000000154513245514472023217 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import abc import six @six.add_metaclass(abc.ABCMeta) class AbstractConfigurer(object): @abc.abstractmethod def configure(self, cluster_context, instances=None): pass @abc.abstractmethod def update(self, cluster_context, instances=None): pass sahara-8.0.0/sahara/plugins/mapr/abstract/__init__.py0000666000175100017510000000000013245514472022574 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/abstract/version_handler.py0000666000175100017510000000357313245514472024241 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import abc import six @six.add_metaclass(abc.ABCMeta) class AbstractVersionHandler(object): @abc.abstractmethod def get_node_processes(self): return @abc.abstractmethod def get_configs(self): return @abc.abstractmethod def configure_cluster(self, cluster): pass @abc.abstractmethod def start_cluster(self, cluster): pass @abc.abstractmethod def validate(self, cluster): pass @abc.abstractmethod def validate_scaling(self, cluster, existing, additional): pass @abc.abstractmethod def scale_cluster(self, cluster, instances): pass @abc.abstractmethod def decommission_nodes(self, cluster, instances): pass @abc.abstractmethod def get_edp_engine(self, cluster, job_type): return @abc.abstractmethod def get_edp_job_types(self): return [] @abc.abstractmethod def get_edp_config_hints(self, job_type): return {} @abc.abstractmethod def get_context(self, cluster, added=None, removed=None): return @abc.abstractmethod def get_services(self): return @abc.abstractmethod def get_required_services(self): return @abc.abstractmethod def get_open_ports(self, node_group): return sahara-8.0.0/sahara/plugins/mapr/abstract/cluster_context.py0000666000175100017510000000632313245514472024300 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import abc import six @six.add_metaclass(abc.ABCMeta) class AbstractClusterContext(object): @abc.abstractproperty def mapr_home(self): return @abc.abstractproperty def configure_sh_path(self): return @abc.abstractproperty def configure_sh(self): return @abc.abstractproperty def hadoop_version(self): return @abc.abstractproperty def hadoop_home(self): return @abc.abstractproperty def hadoop_lib(self): return @abc.abstractproperty def hadoop_conf(self): return @abc.abstractproperty def cluster(self): return @abc.abstractproperty def name_node_uri(self): return @abc.abstractproperty def resource_manager_uri(self): return @abc.abstractproperty def oozie_server_uri(self): return @abc.abstractproperty def oozie_server(self): return @abc.abstractproperty def oozie_http(self): return @abc.abstractproperty def cluster_mode(self): return @abc.abstractproperty def is_node_aware(self): return @abc.abstractproperty def some_instance(self): return @abc.abstractproperty def distro(self): return @abc.abstractproperty def mapr_db(self): return @abc.abstractmethod def filter_instances(self, instances, node_process=None, service=None): return @abc.abstractmethod def removed_instances(self, node_process=None, service=None): return @abc.abstractmethod def added_instances(self, node_process=None, service=None): return @abc.abstractmethod def changed_instances(self, node_process=None, service=None): return @abc.abstractmethod def existing_instances(self, node_process=None, service=None): return @abc.abstractproperty def should_be_restarted(self): return @abc.abstractproperty def mapr_repos(self): return @abc.abstractproperty def is_prebuilt(self): return @abc.abstractproperty def local_repo(self): return @abc.abstractproperty def required_services(self): return @abc.abstractproperty def all_services(self): return @abc.abstractproperty def mapr_version(self): return @abc.abstractproperty def ubuntu_base_repo(self): return @abc.abstractproperty def ubuntu_ecosystem_repo(self): return @abc.abstractproperty def centos_base_repo(self): return @abc.abstractproperty def centos_ecosystem_repo(self): return sahara-8.0.0/sahara/plugins/mapr/plugin.py0000666000175100017510000001050613245514472020544 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from sahara.i18n import _ import sahara.plugins.mapr.versions.version_handler_factory as vhf import sahara.plugins.provisioning as p class MapRPlugin(p.ProvisioningPluginBase): title = 'MapR Hadoop Distribution' description = _('The MapR Distribution provides a full Hadoop stack that' ' includes the MapR File System (MapR-FS), MapReduce,' ' a complete Hadoop ecosystem, and the MapR Control System' ' user interface') def _get_handler(self, hadoop_version): return vhf.VersionHandlerFactory.get().get_handler(hadoop_version) def get_title(self): return MapRPlugin.title def get_description(self): return MapRPlugin.description def get_labels(self): return { 'plugin_labels': {'enabled': {'status': True}}, 'version_labels': { '5.2.0.mrv2': {'enabled': {'status': True}}, '5.1.0.mrv2': {'enabled': {'status': False}, 'deprecated': {'status': True}}, } } def get_versions(self): return vhf.VersionHandlerFactory.get().get_versions() def get_node_processes(self, hadoop_version): return self._get_handler(hadoop_version).get_node_processes() def get_configs(self, hadoop_version): return self._get_handler(hadoop_version).get_configs() def configure_cluster(self, cluster): self._get_handler(cluster.hadoop_version).configure_cluster(cluster) def start_cluster(self, cluster): self._get_handler(cluster.hadoop_version).start_cluster(cluster) def validate(self, cluster): self._get_handler(cluster.hadoop_version).validate(cluster) def validate_scaling(self, cluster, existing, additional): v_handler = self._get_handler(cluster.hadoop_version) v_handler.validate_scaling(cluster, existing, additional) def scale_cluster(self, cluster, instances): v_handler = self._get_handler(cluster.hadoop_version) v_handler.scale_cluster(cluster, instances) def decommission_nodes(self, cluster, instances): v_handler = self._get_handler(cluster.hadoop_version) v_handler.decommission_nodes(cluster, instances) def get_edp_engine(self, cluster, job_type): v_handler = self._get_handler(cluster.hadoop_version) return v_handler.get_edp_engine(cluster, job_type) def get_edp_job_types(self, versions=None): res = {} for vers in self.get_versions(): if not versions or vers in versions: vh = self._get_handler(vers) res[vers] = vh.get_edp_job_types() return res def get_edp_config_hints(self, job_type, version): v_handler = self._get_handler(version) return v_handler.get_edp_config_hints(job_type) def get_open_ports(self, node_group): v_handler = self._get_handler(node_group.cluster.hadoop_version) return v_handler.get_open_ports(node_group) def get_health_checks(self, cluster): v_handler = self._get_handler(cluster.hadoop_version) return v_handler.get_cluster_checks(cluster) def get_image_arguments(self, hadoop_version): return self._get_handler(hadoop_version).get_image_arguments() def pack_image(self, hadoop_version, remote, test_only=False, image_arguments=None): version = self._get_handler(hadoop_version) version.pack_image(hadoop_version, remote, test_only=test_only, image_arguments=image_arguments) def validate_images(self, cluster, test_only=False, image_arguments=None): self._get_handler(cluster.hadoop_version).validate_images( cluster, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/mapr/base/0000775000175100017510000000000013245515027017577 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/base/base_node_manager.py0000666000175100017510000002124013245514472023566 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import random from oslo_log import log as logging from oslo_serialization import jsonutils as json from oslo_utils import timeutils from sahara import context from sahara.i18n import _ import sahara.plugins.exceptions as ex import sahara.plugins.mapr.abstract.node_manager as s import sahara.plugins.mapr.services.management.management as mng import sahara.plugins.mapr.services.maprfs.maprfs as mfs import sahara.plugins.mapr.util.event_log as el from sahara.utils import cluster_progress_ops as cpo LOG = logging.getLogger(__name__) GET_SERVER_ID_CMD = ('maprcli node list -json -filter [ip==%s] -columns id' ' | grep id | grep -o \'[0-9]*\'') NODE_LIST_CMD = 'maprcli node list -json' MOVE_NODE_CMD = 'maprcli node move -serverids %s -topology /decommissioned' REMOVE_NODE_CMD = ('maprcli node remove -filter [ip==%(ip)s] -nodes %(nodes)s' ' -zkconnect %(zookeepers)s') WAIT_NODE_ALARM_NO_HEARTBEAT = 360 WARDEN_SERVICE = 'warden' START = 'start' STOP = 'stop' DELAY = 5 DEFAULT_RETRY_COUNT = 10 class BaseNodeManager(s.AbstractNodeManager): def move_nodes(self, cluster_context, instances): LOG.debug("Moving the nodes to /decommissioned topology") cldb_instances = self._get_cldb_instances(cluster_context, instances) with random.choice(cldb_instances).remote() as cldb_remote: for instance in instances: with instance.remote() as r: command = GET_SERVER_ID_CMD % instance.internal_ip ec, out = r.execute_command(command, run_as_root=True) command = MOVE_NODE_CMD % out.strip() cldb_remote.execute_command(command, run_as_root=True) LOG.info("Nodes successfully moved") def remove_nodes(self, cluster_context, instances): LOG.debug("Removing nodes from cluster") cldb_instances = self._get_cldb_instances(cluster_context, instances) with random.choice(cldb_instances).remote() as cldb_remote: for instance in instances: args = { 'ip': instance.internal_ip, 'nodes': instance.fqdn(), 'zookeepers': cluster_context.get_zookeeper_nodes_ip_with_port(), } command = REMOVE_NODE_CMD % args cldb_remote.execute_command(command, run_as_root=True) LOG.info("Nodes successfully removed") def start(self, cluster_context, instances=None): instances = instances or cluster_context.get_instances() zookeepers = cluster_context.filter_instances(instances, mng.ZOOKEEPER) cldbs = cluster_context.filter_instances(instances, mfs.CLDB) others = filter( lambda i: not cluster_context.check_for_process(i, mfs.CLDB), instances) cpo.add_provisioning_step(cluster_context.cluster.id, _("Start ZooKeepers nodes"), len(zookeepers)) self._start_zk_nodes(zookeepers) cpo.add_provisioning_step(cluster_context.cluster.id, _("Start CLDB nodes"), len(cldbs)) self._start_cldb_nodes(cldbs) if others: cpo.add_provisioning_step(cluster_context.cluster.id, _("Start non-CLDB nodes"), len(list(others))) self._start_non_cldb_nodes(others) self._await_cldb(cluster_context, instances) def stop(self, cluster_context, instances=None): instances = instances or cluster_context.get_instances() zookeepers = cluster_context.filter_instances(instances, mng.ZOOKEEPER) cpo.add_provisioning_step(cluster_context.cluster.id, _("Stop ZooKeepers nodes"), len(zookeepers)) self._stop_zk_nodes(zookeepers) cpo.add_provisioning_step(cluster_context.cluster.id, _("Stop Warden nodes"), len(instances)) self._stop_warden_on_nodes(instances) def _await_cldb(self, cluster_context, instances=None, timeout=600): instances = instances or cluster_context.get_instances() cldb_node = cluster_context.get_instance(mfs.CLDB) start_time = timeutils.utcnow() retry_count = 0 with cldb_node.remote() as r: LOG.debug("Waiting {count} seconds for CLDB initialization".format( count=timeout)) while timeutils.delta_seconds(start_time, timeutils.utcnow()) < timeout: ec, out = r.execute_command(NODE_LIST_CMD, raise_when_error=False) resp = json.loads(out) status = resp['status'] if str(status).lower() == 'ok': ips = [n['ip'] for n in resp['data']] retry_count += 1 for i in instances: if (i.internal_ip not in ips and (retry_count > DEFAULT_RETRY_COUNT)): msg = _("Node failed to connect to CLDB: %s" ) % i.internal_ip raise ex.HadoopProvisionError(msg) break else: context.sleep(DELAY) else: raise ex.HadoopProvisionError(_("CLDB failed to start")) def _start_nodes(self, instances, sys_service): with context.ThreadGroup() as tg: for instance in instances: tg.spawn('start-%s-%s' % (sys_service, instance.id), self._start_service, instance, sys_service) def _stop_nodes(self, instances, sys_service): with context.ThreadGroup() as tg: for instance in instances: tg.spawn('stop-%s-%s' % (sys_service, instance.id), self._stop_service, instance, sys_service) def _start_zk_nodes(self, instances): LOG.debug('Starting ZooKeeper nodes') self._start_nodes(instances, mng.ZOOKEEPER.ui_name) LOG.info('ZooKeeper nodes successfully started') def _start_cldb_nodes(self, instances): LOG.debug('Starting CLDB nodes') self._start_nodes(instances, WARDEN_SERVICE) LOG.info('CLDB nodes successfully started') def _start_non_cldb_nodes(self, instances): LOG.debug('Starting non-control nodes') self._start_nodes(instances, WARDEN_SERVICE) LOG.info('Non-control nodes successfully started') def _stop_zk_nodes(self, instances): self._stop_nodes(instances, mng.ZOOKEEPER.ui_name) def _stop_warden_on_nodes(self, instances): self._stop_nodes(instances, WARDEN_SERVICE) @staticmethod def _do_service_action(instance, service, action): with instance.remote() as r: cmd = "service mapr-%(service)s %(action)s" args = {'service': service.lower(), 'action': action} cmd = cmd % args LOG.debug( 'Executing "{command}" on node={ip}'.format( command=cmd, ip=instance.internal_ip)) r.execute_command(cmd, run_as_root=True) @el.provision_event(instance_reference=1) def _start_service(self, instance, service): return self._do_service_action(instance, service, START) @el.provision_event(instance_reference=1) def _stop_service(self, instance, service): return self._do_service_action(instance, service, STOP) def _get_cldb_instances(self, cluster_context, instances): current = self._get_current_cluster_instances(cluster_context, instances) return cluster_context.filter_instances(current, mfs.CLDB) @staticmethod def await_no_heartbeat(): delay = WAIT_NODE_ALARM_NO_HEARTBEAT LOG.debug('Waiting for "NO_HEARBEAT" alarm') context.sleep(delay) def _get_current_cluster_instances(self, cluster_context, instances): all_instances = cluster_context.get_instances() return [x for x in all_instances if x not in instances] sahara-8.0.0/sahara/plugins/mapr/base/base_cluster_context.py0000666000175100017510000003755613245514472024415 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import collections from oslo_config import cfg import sahara.exceptions as e from sahara.i18n import _ import sahara.plugins.mapr.abstract.cluster_context as cc import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.distro as distro import sahara.plugins.mapr.services.management.management as mng import sahara.plugins.mapr.services.maprfs.maprfs as mfs import sahara.plugins.mapr.services.oozie.oozie as oozie from sahara.plugins.mapr.services.swift import swift import sahara.plugins.mapr.services.yarn.yarn as yarn import sahara.plugins.mapr.util.general as g import sahara.plugins.mapr.util.service_utils as su import sahara.plugins.utils as u from sahara.topology import topology_helper as th import sahara.utils.configs as sahara_configs CONF = cfg.CONF CONF.import_opt("enable_data_locality", "sahara.topology.topology_helper") class BaseClusterContext(cc.AbstractClusterContext): ubuntu_base = 'http://package.mapr.com/releases/v%s/ubuntu/ mapr optional' centos_base = 'http://package.mapr.com/releases/v%s/redhat/' def __init__(self, cluster, version_handler, added=None, removed=None): self._cluster = cluster self._distro = None self._distro_version = None self._all_services = version_handler.get_services() self._required_services = version_handler.get_required_services() self._cluster_services = None self._mapr_home = '/opt/mapr' self._name_node_uri = 'maprfs:///' self._cluster_mode = None self._node_aware = None self._oozie_server_uri = None self._oozie_server = None self._oozie_http = None self._some_instance = None self._configure_sh_path = None self._configure_sh = None self._mapr_db = None self._hadoop_home = None self._hadoop_version = None self._added_instances = added or [] self._removed_instances = removed or [] self._changed_instances = ( self._added_instances + self._removed_instances) self._existing_instances = [i for i in self.get_instances() if i not in self._changed_instances] self._restart = collections.defaultdict(list) self._ubuntu_base_repo = None self._ubuntu_ecosystem_repo = None self._centos_base_repo = None self._centos_ecosystem_repo = None self._repos = {} self._is_prebuilt = None self._local_repo = '/opt/mapr-repository' self._mapr_version = None @property def cluster(self): return self._cluster @property def cluster_services(self): if not self._cluster_services: self._cluster_services = self.get_cluster_services() return self._cluster_services @property def required_services(self): return self._required_services @property def all_services(self): return self._all_services @property def mapr_home(self): return self._mapr_home @property def hadoop_version(self): return self._hadoop_version @property def hadoop_home(self): if not self._hadoop_home: f = '%(mapr_home)s/hadoop/hadoop-%(hadoop_version)s' args = { 'mapr_home': self.mapr_home, 'hadoop_version': self.hadoop_version, } self._hadoop_home = f % args return self._hadoop_home @property def name_node_uri(self): return self._name_node_uri @property def oozie_server_uri(self): if not self._oozie_server_uri: oozie_http = self.oozie_http url = 'http://%s/oozie' % oozie_http if oozie_http else None self._oozie_server_uri = url return self._oozie_server_uri @property def oozie_server(self): if not self._oozie_server: self._oozie_server = self.get_instance(oozie.OOZIE) return self._oozie_server @property def oozie_http(self): if not self._oozie_http: oozie_server = self.oozie_server ip = oozie_server.management_ip if oozie_server else None self._oozie_http = '%s:11000' % ip if ip else None return self._oozie_http @property def cluster_mode(self): return self._cluster_mode @property def is_node_aware(self): return self._node_aware and CONF.enable_data_locality @property def some_instance(self): if not self._some_instance: self._some_instance = self.cluster.node_groups[0].instances[0] return self._some_instance @property def distro_version(self): if not self._distro_version: self._distro_version = distro.get_version(self.some_instance) return self._distro_version @property def distro(self): if not self._distro: self._distro = distro.get(self.some_instance) return self._distro @property def mapr_db(self): if self._mapr_db is None: mapr_db = mfs.MapRFS.ENABLE_MAPR_DB_CONFIG mapr_db = self._get_cluster_config_value(mapr_db) self._mapr_db = '-noDB' if not mapr_db else '' return self._mapr_db @property def configure_sh_path(self): if not self._configure_sh_path: self._configure_sh_path = '%s/server/configure.sh' % self.mapr_home return self._configure_sh_path @property def configure_sh(self): if not self._configure_sh: f = ('%(script_path)s' ' -N %(cluster_name)s' ' -C %(cldbs)s' ' -Z %(zookeepers)s' ' -no-autostart -f %(m7)s') args = { 'script_path': self.configure_sh_path, 'cluster_name': self.cluster.name, 'cldbs': self.get_cldb_nodes_ip(), 'zookeepers': self.get_zookeeper_nodes_ip(), 'm7': self.mapr_db } self._configure_sh = f % args return self._configure_sh def _get_cluster_config_value(self, config): cluster_configs = self.cluster.cluster_configs service = config.applicable_target name = config.name if service in cluster_configs and name in cluster_configs[service]: return cluster_configs[service][name] else: return config.default_value def get_node_processes(self): node_processes = [] for ng in self.cluster.node_groups: for np in ng.node_processes: if np not in node_processes: node_processes.append(self.get_node_process_by_name(np)) return node_processes def get_node_process_by_name(self, name): for service in self.cluster_services: for node_process in service.node_processes: if node_process.ui_name == name: return node_process def get_instances(self, node_process=None): if node_process is not None: node_process = su.get_node_process_name(node_process) return u.get_instances(self.cluster, node_process) def get_instance(self, node_process): node_process_name = su.get_node_process_name(node_process) instances = u.get_instances(self.cluster, node_process_name) return instances[0] if instances else None def get_instances_ip(self, node_process): return [i.internal_ip for i in self.get_instances(node_process)] def get_instance_ip(self, node_process): i = self.get_instance(node_process) return i.internal_ip if i else None def get_zookeeper_nodes_ip_with_port(self, separator=','): return separator.join(['%s:%s' % (ip, mng.ZK_CLIENT_PORT) for ip in self.get_instances_ip(mng.ZOOKEEPER)]) def check_for_process(self, instance, process): return su.has_node_process(instance, process) def get_services_configs_dict(self, services=None): if not services: services = self.cluster_services result = dict() for service in services: result.update(service.get_configs_dict()) return result def get_chosen_service_version(self, service_name): service_configs = self.cluster.cluster_configs.get(service_name, None) if not service_configs: return None return service_configs.get('%s Version' % service_name, None) def get_cluster_services(self, node_group=None): node_processes = None if node_group: node_processes = node_group.node_processes else: node_processes = [np for ng in self.cluster.node_groups for np in ng.node_processes] node_processes = g.unique_list(node_processes) services = g.unique_list(node_processes, self.get_service) return services + [swift.Swift()] def get_service(self, node_process): ui_name = self.get_service_name_by_node_process(node_process) if ui_name is None: raise e.InvalidDataException( _('Service not found in services list')) version = self.get_chosen_service_version(ui_name) service = self._find_service_instance(ui_name, version) if service is None: raise e.InvalidDataException(_('Can not map service')) return service def _find_service_instance(self, ui_name, version): # if version is None, the latest service version is returned for service in self.all_services[::-1]: if service.ui_name == ui_name: if version is not None and service.version != version: continue return service def get_service_name_by_node_process(self, node_process): node_process_name = su.get_node_process_name(node_process) for service in self.all_services: service_node_processes = [np.ui_name for np in service.node_processes] if node_process_name in service_node_processes: return service.ui_name def get_instances_count(self, node_process=None): if node_process is not None: node_process = su.get_node_process_name(node_process) return u.get_instances_count(self.cluster, node_process) def get_node_groups(self, node_process=None): if node_process is not None: node_process = su.get_node_process_name(node_process) return u.get_node_groups(self.cluster, node_process) def get_cldb_nodes_ip(self, separator=','): return separator.join(self.get_instances_ip(mfs.CLDB)) def get_zookeeper_nodes_ip(self, separator=','): return separator.join( self.get_instances_ip(mng.ZOOKEEPER)) def get_resourcemanager_ip(self): return self.get_instance_ip(yarn.RESOURCE_MANAGER) def get_historyserver_ip(self): return self.get_instance_ip(yarn.HISTORY_SERVER) def has_control_nodes(self, instances): for inst in instances: zookeepers = self.check_for_process(inst, mng.ZOOKEEPER) cldbs = self.check_for_process(inst, mfs.CLDB) if zookeepers or cldbs: return True return False def is_present(self, service): is_service_subclass = lambda s: isinstance(s, service.__class__) return any(is_service_subclass(s) for s in self.cluster_services) def filter_instances(self, instances, node_process=None, service=None): if node_process: return su.filter_by_node_process(instances, node_process) if service: return su.filter_by_service(instances, service) return list(instances) def removed_instances(self, node_process=None, service=None): instances = self._removed_instances return self.filter_instances(instances, node_process, service) def added_instances(self, node_process=None, service=None): instances = self._added_instances return self.filter_instances(instances, node_process, service) def changed_instances(self, node_process=None, service=None): instances = self._changed_instances return self.filter_instances(instances, node_process, service) def existing_instances(self, node_process=None, service=None): instances = self._existing_instances return self.filter_instances(instances, node_process, service) @property def should_be_restarted(self): return self._restart @property def mapr_repos(self): if not self._repos: self._repos = { "ubuntu_mapr_base_repo": self.ubuntu_base_repo, "ubuntu_mapr_ecosystem_repo": self.ubuntu_ecosystem_repo, "centos_mapr_base_repo": self.centos_base_repo, "centos_mapr_ecosystem_repo": self.centos_ecosystem_repo, } return self._repos @property def local_repo(self): return self._local_repo @property def is_prebuilt(self): if self._is_prebuilt is None: self._is_prebuilt = g.is_directory( self.some_instance, self.local_repo) return self._is_prebuilt @property def mapr_version(self): return self._mapr_version @property def ubuntu_base_repo(self): default_value = self._ubuntu_base_repo or self.ubuntu_base % self\ .mapr_version return self.cluster.cluster_configs.get( 'general', {}).get('Ubuntu base repo', default_value) @property def ubuntu_ecosystem_repo(self): default_value = self._ubuntu_ecosystem_repo return self.cluster.cluster_configs.get( 'general', {}).get('Ubuntu ecosystem repo', default_value) @property def centos_base_repo(self): default_value = self._centos_base_repo or self.centos_base % self\ .mapr_version return self.cluster.cluster_configs.get( 'general', {}).get('CentOS base repo', default_value) @property def centos_ecosystem_repo(self): default_value = self._centos_ecosystem_repo return self.cluster.cluster_configs.get( 'general', {}).get('CentOS ecosystem repo', default_value) def get_configuration(self, node_group): services = self.get_cluster_services(node_group) user_configs = node_group.configuration() default_configs = self.get_services_configs_dict(services) return sahara_configs.merge_configs(default_configs, user_configs) def get_config_files(self, node_group): services = self.get_cluster_services(node_group) configuration = self.get_configuration(node_group) instance = node_group.instances[0] config_files = [] for service in services: service_conf_files = service.get_config_files( cluster_context=self, configs=configuration[service.ui_name], instance=instance, ) for conf_file in service_conf_files: file_atr = bcf.FileAttr(conf_file.remote_path, conf_file.render(), conf_file.mode, conf_file.owner) config_files.append(file_atr) return config_files @property def topology_map(self): return th.generate_topology_map(self.cluster, self.is_node_aware) sahara-8.0.0/sahara/plugins/mapr/base/base_health_checker.py0000666000175100017510000001101213245514472024074 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import functools from sahara.i18n import _ import sahara.plugins.mapr.abstract.health_checker as hc from sahara.plugins.mapr.domain import node_process as np from sahara.plugins.mapr.services.management import management from sahara.plugins.mapr.services.spark import spark from sahara.service.health import health_check_base class BaseHealthChecker(hc.AbstractHealthChecker): def _is_avaliable(self, process): return process.open_ports and process not in spark.SparkOnYarn().\ node_processes def get_checks(self, cluster_context, instances=None): checks = [ functools.partial(ZookeeperCheck, cluster_context=cluster_context)] for node_process in cluster_context.get_node_processes(): if self._is_avaliable( node_process) and node_process.ui_name != 'ZooKeeper': checks.append(functools.partial (MapRNodeProcessCheck, cluster_context=cluster_context, process=node_process)) return checks class ZookeeperCheck(health_check_base.BasicHealthCheck): def __init__(self, cluster, cluster_context): super(ZookeeperCheck, self).__init__(cluster) self.cluster_context = cluster_context def get_health_check_name(self): return 'MapR ZooKeeper check' def is_available(self): return self.cluster_context.cluster.plugin_name == 'mapr' def _is_zookeeper_running(self, instance): cmd = 'service mapr-zookeeper status' with instance.remote() as r: __, out = r.execute_command(cmd, run_as_root=True) return 'zookeeper running as process' in out \ or 'active (running)' in out def check_health(self): instances = self.cluster_context.get_instances( node_process=management.ZOOKEEPER) active_count = 0 for instance in instances: if self._is_zookeeper_running(instance): active_count += 1 if active_count == 0: raise health_check_base.RedHealthError(_( "ZooKeeper is not in running state")) if active_count < len(instances): raise health_check_base.YellowHealthError(_( "Some ZooKeeper processes are not in running state")) return _("ZooKeeper is in running state") class MapRNodeProcessCheck(health_check_base.BasicHealthCheck): IMPORTANT_PROCESSES = [ 'CLDB', 'FileServer', 'NodeManager', 'ResourceManager' ] def __init__(self, cluster, cluster_context, process): super(MapRNodeProcessCheck, self).__init__(cluster) self.process = process self.cluster_context = cluster_context def get_health_check_name(self): return 'MapR %s check' % self.process.ui_name def is_available(self): return self.cluster_context.cluster.plugin_name == 'mapr' def check_health(self): instances = self.cluster_context.get_instances( node_process=self.process) active_count = 0 for instance in instances: status = self.process.status(instance) if status == np.Status.RUNNING: active_count += 1 if active_count == 0: if self.process.ui_name in self.IMPORTANT_PROCESSES: raise health_check_base.RedHealthError(_( "%s is not in running state") % self.process.ui_name) else: raise health_check_base.YellowHealthError(_( "%s is not in running state") % self.process.ui_name) if active_count < len(instances): if self.process.ui_name in self.IMPORTANT_PROCESSES: raise health_check_base.YellowHealthError(_( "Some %s processes are not in running state") % self.process.ui_name) return _("%s is in running state") % self.process.ui_name sahara-8.0.0/sahara/plugins/mapr/base/__init__.py0000666000175100017510000000000013245514472021703 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/base/base_cluster_validator.py0000666000175100017510000000274313245514472024704 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.abstract.cluster_validator as v import sahara.plugins.mapr.util.validation_utils as vu import sahara.plugins.mapr.versions.version_handler_factory as vhf class BaseValidator(v.AbstractValidator): def validate(self, cluster_context): for service in cluster_context.required_services: vu.assert_present(service, cluster_context) for service in cluster_context.cluster_services: for rule in service.validation_rules: rule(cluster_context) def validate_scaling(self, cluster_context, existing, additional): cluster = cluster_context.cluster version = cluster.hadoop_version handler = vhf.VersionHandlerFactory.get().get_handler(version) cluster = vu.create_fake_cluster(cluster, existing, additional) cluster_context = handler.get_context(cluster) self.validate(cluster_context) sahara-8.0.0/sahara/plugins/mapr/base/base_edp_engine.py0000666000175100017510000000731713245514472023255 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import os from sahara import context import sahara.plugins.mapr.util.maprfs_helper as mfs import sahara.plugins.mapr.versions.version_handler_factory as vhf from sahara.service.edp.job_binaries import manager as jb_manager import sahara.service.edp.oozie.engine as e from sahara.utils import edp class MapROozieJobEngine(e.OozieJobEngine): def __init__(self, cluster): super(MapROozieJobEngine, self).__init__(cluster) self.cluster_context = self._get_cluster_context(self.cluster) hdfs_user = 'mapr' def get_hdfs_user(self): return MapROozieJobEngine.hdfs_user def create_hdfs_dir(self, remote, dir_name): mfs.create_maprfs4_dir(remote, dir_name, self.get_hdfs_user()) def _upload_workflow_file(self, where, job_dir, wf_xml, hdfs_user): f_name = 'workflow.xml' with where.remote() as r: mfs.put_file_to_maprfs(r, wf_xml, f_name, job_dir, hdfs_user) return os.path.join(job_dir, f_name) def _upload_job_files_to_hdfs(self, where, job_dir, job, configs, proxy_configs=None): mains = job.mains or [] libs = job.libs or [] builtin_libs = edp.get_builtin_binaries(job, configs) uploaded_paths = [] hdfs_user = self.get_hdfs_user() lib_dir = job_dir + '/lib' with where.remote() as r: for m in mains: path = jb_manager.JOB_BINARIES. \ get_job_binary_by_url(m.url). \ copy_binary_to_cluster(m, proxy_configs=proxy_configs, remote=r, context=context.ctx()) target = os.path.join(job_dir, m.name) mfs.copy_from_local(r, path, target, hdfs_user) uploaded_paths.append(target) if len(libs) > 0: self.create_hdfs_dir(r, lib_dir) for l in libs: path = jb_manager.JOB_BINARIES. \ get_job_binary_by_url(l.url). \ copy_binary_to_cluster(l, proxy_configs=proxy_configs, remote=r, context=context.ctx()) target = os.path.join(lib_dir, l.name) mfs.copy_from_local(r, path, target, hdfs_user) uploaded_paths.append(target) for lib in builtin_libs: mfs.put_file_to_maprfs(r, lib['raw'], lib['name'], lib_dir, hdfs_user) uploaded_paths.append(lib_dir + '/' + lib['name']) return uploaded_paths def get_name_node_uri(self, cluster): return self.cluster_context.name_node_uri def get_oozie_server_uri(self, cluster): return self.cluster_context.oozie_server_uri def get_oozie_server(self, cluster): return self.cluster_context.oozie_server def get_resource_manager_uri(self, cluster): return self.cluster_context.resource_manager_uri def _get_cluster_context(self, cluster): h_version = cluster.hadoop_version v_handler = vhf.VersionHandlerFactory.get().get_handler(h_version) return v_handler.get_context(cluster) sahara-8.0.0/sahara/plugins/mapr/base/base_cluster_configurer.py0000666000175100017510000004235213245514472025062 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import abc from oslo_log import log as logging import six from sahara import conductor from sahara import context from sahara.i18n import _ import sahara.plugins.mapr.abstract.configurer as ac from sahara.plugins.mapr.domain import distro as d from sahara.plugins.mapr.domain import service as srvc import sahara.plugins.mapr.services.management.management as mng import sahara.plugins.mapr.services.mapreduce.mapreduce as mr from sahara.plugins.mapr.services.maprfs import maprfs from sahara.plugins.mapr.services.mysql import mysql import sahara.plugins.mapr.services.yarn.yarn as yarn from sahara.plugins.mapr.util import event_log as el import sahara.plugins.mapr.util.general as util import sahara.plugins.mapr.util.password_utils as pu import sahara.utils.files as files LOG = logging.getLogger(__name__) conductor = conductor.API _JAVA_HOME = '/usr/java/jdk1.7.0_51' _CONFIGURE_SH_TIMEOUT = 600 _SET_MODE_CMD = 'maprcli cluster mapreduce set -mode ' _TOPO_SCRIPT = 'plugins/mapr/resources/topology.sh' INSTALL_JAVA_SCRIPT = 'plugins/mapr/resources/install_java.sh' INSTALL_SCALA_SCRIPT = 'plugins/mapr/resources/install_scala.sh' INSTALL_MYSQL_CLIENT = 'plugins/mapr/resources/install_mysql_client.sh' ADD_MAPR_REPO_SCRIPT = 'plugins/mapr/resources/add_mapr_repo.sh' ADD_SECURITY_REPO_SCRIPT = 'plugins/mapr/resources/add_security_repos.sh' SERVICE_INSTALL_PRIORITY = [ mng.Management(), yarn.YARNv251(), yarn.YARNv241(), yarn.YARNv270(), mr.MapReduce(), maprfs.MapRFS(), ] @six.add_metaclass(abc.ABCMeta) class BaseConfigurer(ac.AbstractConfigurer): def configure(self, cluster_context, instances=None): instances = instances or cluster_context.get_instances() self._configure_ssh_connection(cluster_context, instances) self._install_mapr_repo(cluster_context, instances) if not cluster_context.is_prebuilt: self._prepare_bare_image(cluster_context, instances) self._install_services(cluster_context, instances) if cluster_context.is_node_aware: self._configure_topology(cluster_context, instances) self._configure_database(cluster_context, instances) self._configure_services(cluster_context, instances) self._configure_sh_cluster(cluster_context, instances) self._set_cluster_mode(cluster_context, instances) self._post_configure_services(cluster_context, instances) self._write_config_files(cluster_context, instances) self._configure_environment(cluster_context, instances) self._update_cluster_info(cluster_context) def update(self, cluster_context, instances=None): LOG.debug('Configuring existing instances') instances = instances or cluster_context.get_instances() existing = cluster_context.existing_instances() if cluster_context.is_node_aware: self._configure_topology(cluster_context, existing) if cluster_context.has_control_nodes(instances): self._configure_sh_cluster(cluster_context, existing) self._post_configure_sh(cluster_context, existing) self._write_config_files(cluster_context, existing) self._update_services(cluster_context, existing) self._restart_services(cluster_context) self._update_cluster_info(cluster_context) LOG.info('Existing instances successfully configured') def _configure_services(self, cluster_context, instances): for service in cluster_context.cluster_services: service.configure(cluster_context, instances) def _install_services(self, cluster_context, instances): for service in self._service_install_sequence(cluster_context): service.install(cluster_context, instances) def _service_install_sequence(self, cluster_context): def key(service): if service in SERVICE_INSTALL_PRIORITY: return SERVICE_INSTALL_PRIORITY.index(service) return -service._priority return sorted(cluster_context.cluster_services, key=key, reverse=True) def _prepare_bare_image(self, cluster_context, instances): LOG.debug('Preparing bare image') if d.UBUNTU == cluster_context.distro: self._install_security_repos(cluster_context, instances) self._install_java(cluster_context, instances) self._install_scala(cluster_context, instances) self._install_mysql_client(cluster_context, instances) LOG.debug('Bare images successfully prepared') @el.provision_step(_("Install security repos")) def _install_security_repos(self, cluster_context, instances): LOG.debug("Installing security repos") @el.provision_event() def install_security_repos(instance): return util.run_script(instance, ADD_SECURITY_REPO_SCRIPT, "root") util.execute_on_instances(instances, install_security_repos) @el.provision_step(_("Install MySQL client")) def _install_mysql_client(self, cluster_context, instances): LOG.debug("Installing MySQL client") distro_name = cluster_context.distro.name @el.provision_event() def install_mysql_client(instance): return util.run_script(instance, INSTALL_MYSQL_CLIENT, "root", distro_name) util.execute_on_instances(instances, install_mysql_client) @el.provision_step(_("Install Scala")) def _install_scala(self, cluster_context, instances): LOG.debug("Installing Scala") distro_name = cluster_context.distro.name @el.provision_event() def install_scala(instance): return util.run_script(instance, INSTALL_SCALA_SCRIPT, "root", distro_name) util.execute_on_instances(instances, install_scala) @el.provision_step(_("Install Java")) def _install_java(self, cluster_context, instances): LOG.debug("Installing Java") distro_name = cluster_context.distro.name @el.provision_event() def install_java(instance): return util.run_script(instance, INSTALL_JAVA_SCRIPT, "root", distro_name) util.execute_on_instances(instances, install_java) @el.provision_step(_("Configure cluster topology")) def _configure_topology(self, cluster_context, instances): LOG.debug("Configuring cluster topology") topology_map = cluster_context.topology_map topology_map = ("%s %s" % item for item in six.iteritems(topology_map)) topology_map = "\n".join(topology_map) + "\n" data_path = "%s/topology.data" % cluster_context.mapr_home script = files.get_file_text(_TOPO_SCRIPT) script_path = '%s/topology.sh' % cluster_context.mapr_home @el.provision_event() def write_topology_data(instance): util.write_file(instance, data_path, topology_map, owner="root") util.write_file(instance, script_path, script, mode="+x", owner="root") util.execute_on_instances(instances, write_topology_data) LOG.info('Cluster topology successfully configured') @el.provision_step(_("Write config files to instances")) def _write_config_files(self, cluster_context, instances): LOG.debug('Writing config files') @el.provision_event() def write_config_files(instance, config_files): for file in config_files: util.write_file(instance, file.path, file.data, mode=file.mode, owner="mapr") node_groups = util.unique_list(instances, lambda i: i.node_group) for node_group in node_groups: config_files = cluster_context.get_config_files(node_group) ng_instances = [i for i in node_group.instances if i in instances] util.execute_on_instances(ng_instances, write_config_files, config_files=config_files) LOG.debug("Config files are successfully written") def _configure_environment(self, cluster_context, instances): self.configure_general_environment(cluster_context, instances) self._post_install_services(cluster_context, instances) def _configure_database(self, cluster_context, instances): mysql_instance = mysql.MySQL.get_db_instance(cluster_context) @el.provision_event(instance=mysql_instance, name=_("Configure database")) def decorated(): distro_name = cluster_context.distro.name distro_version = cluster_context.distro_version mysql.MySQL.install_mysql(mysql_instance, distro_name, distro_version) mysql.MySQL.start_mysql_server(cluster_context) mysql.MySQL.create_databases(cluster_context, instances) decorated() def _post_install_services(self, cluster_context, instances): LOG.debug('Executing service post install hooks') for s in cluster_context.cluster_services: service_instances = cluster_context.filter_instances(instances, service=s) if service_instances: s.post_install(cluster_context, instances) LOG.info('Post install hooks execution successfully executed') def _update_cluster_info(self, cluster_context): LOG.debug('Updating UI information.') info = {'Admin user credentials': {'Username': 'mapr', 'Password': pu.get_mapr_password (cluster_context.cluster)}} for service in cluster_context.cluster_services: for title, node_process, ui_info in ( service.get_ui_info(cluster_context)): removed = cluster_context.removed_instances(node_process) instances = cluster_context.get_instances(node_process) instances = [i for i in instances if i not in removed] if len(instances) == 1: display_name_template = "%(title)s" else: display_name_template = "%(title)s %(index)s" for index, instance in enumerate(instances, start=1): args = {"title": title, "index": index} display_name = display_name_template % args data = ui_info.copy() data[srvc.SERVICE_UI] = (data[srvc.SERVICE_UI] % instance.get_ip_or_dns_name()) info.update({display_name: data}) ctx = context.ctx() conductor.cluster_update(ctx, cluster_context.cluster, {'info': info}) def configure_general_environment(self, cluster_context, instances=None): LOG.debug('Executing post configure hooks') mapr_user_pass = pu.get_mapr_password(cluster_context.cluster) if not instances: instances = cluster_context.get_instances() def set_user_password(instance): LOG.debug('Setting password for user "mapr"') if self.mapr_user_exists(instance): with instance.remote() as r: r.execute_command( 'echo "%s:%s"|chpasswd' % ('mapr', mapr_user_pass), run_as_root=True) else: LOG.warning('User "mapr" does not exists') def create_home_mapr(instance): target_path = '/home/mapr' LOG.debug("Creating home directory for user 'mapr'") args = {'path': target_path, 'user': 'mapr', 'group': 'mapr'} cmd = ('mkdir -p %(path)s && chown %(user)s:%(group)s %(path)s' % args) if self.mapr_user_exists(instance): with instance.remote() as r: r.execute_command(cmd, run_as_root=True) else: LOG.warning('User "mapr" does not exists') util.execute_on_instances(instances, set_user_password) util.execute_on_instances(instances, create_home_mapr) @el.provision_step(_("Execute configure.sh")) def _configure_sh_cluster(self, cluster_context, instances): LOG.debug('Executing configure.sh') if not instances: instances = cluster_context.get_instances() script = cluster_context.configure_sh db_specs = dict(mysql.MySQL.METRICS_SPECS._asdict()) db_specs.update({ 'host': mysql.MySQL.get_db_instance(cluster_context).internal_ip, 'port': mysql.MySQL.MYSQL_SERVER_PORT, }) with context.ThreadGroup() as tg: for instance in instances: tg.spawn('configure-sh-%s' % instance.id, self._configure_sh_instance, cluster_context, instance, script, db_specs) LOG.debug('Executing configure.sh successfully completed') @el.provision_event(instance_reference=2) def _configure_sh_instance(self, cluster_context, instance, command, specs): if not self.mapr_user_exists(instance): command += ' --create-user' if cluster_context.check_for_process(instance, mng.METRICS): command += (' -d %(host)s:%(port)s -du %(user)s -dp %(password)s ' '-ds %(db_name)s') % specs with instance.remote() as r: r.execute_command('sudo -i ' + command, timeout=_CONFIGURE_SH_TIMEOUT) @el.provision_step(_("Configure SSH connection")) def _configure_ssh_connection(self, cluster_context, instances): @el.provision_event() def configure_ssh(instance): echo_param = 'echo "KeepAlive yes" >> ~/.ssh/config' echo_timeout = 'echo "ServerAliveInterval 60" >> ~/.ssh/config' with instance.remote() as r: r.execute_command(echo_param) r.execute_command(echo_timeout) util.execute_on_instances(instances, configure_ssh) def mapr_user_exists(self, instance): with instance.remote() as r: ec, __ = r.execute_command( "id -u %s" % 'mapr', run_as_root=True, raise_when_error=False) return ec == 0 def post_start(self, cluster_context, instances=None): instances = instances or cluster_context.get_instances() LOG.debug('Executing service post start hooks') for service in cluster_context.cluster_services: updated = cluster_context.filter_instances(instances, service=service) service.post_start(cluster_context, updated) LOG.info('Post start hooks successfully executed') @el.provision_step(_("Set cluster mode")) def _set_cluster_mode(self, cluster_context, instances): cluster_mode = cluster_context.cluster_mode if not cluster_mode: return command = "maprcli cluster mapreduce set -mode %s" % cluster_mode @el.provision_event() def set_cluster_mode(instance): return util.execute_command([instance], command, run_as='mapr') util.execute_on_instances(instances, set_cluster_mode) @el.provision_step(_("Install MapR repositories")) def _install_mapr_repo(self, cluster_context, instances): distro_name = cluster_context.distro.name @el.provision_event() def install_mapr_repos(instance): return util.run_script(instance, ADD_MAPR_REPO_SCRIPT, "root", distro_name, **cluster_context.mapr_repos) util.execute_on_instances(instances, install_mapr_repos) def _update_services(self, cluster_context, instances): for service in cluster_context.cluster_services: updated = cluster_context.filter_instances(instances, service=service) service.update(cluster_context, updated) def _restart_services(self, cluster_context): restart = cluster_context.should_be_restarted for service, instances in six.iteritems(restart): service.restart(util.unique_list(instances)) def _post_configure_sh(self, cluster_context, instances): LOG.debug('Executing post configure.sh hooks') for service in cluster_context.cluster_services: service.post_configure_sh(cluster_context, instances) LOG.info('Post configure.sh hooks successfully executed') def _post_configure_services(self, cluster_context, instances): for service in cluster_context.cluster_services: service.post_configure(cluster_context, instances) sahara-8.0.0/sahara/plugins/mapr/base/base_version_handler.py0000666000175100017510000001647313245514472024345 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import collections as c from sahara.i18n import _ import sahara.plugins.mapr.abstract.version_handler as vh import sahara.plugins.mapr.base.base_cluster_configurer as base_conf import sahara.plugins.mapr.base.base_cluster_validator as bv import sahara.plugins.mapr.base.base_edp_engine as edp import sahara.plugins.mapr.base.base_health_checker as health import sahara.plugins.mapr.base.base_node_manager as bs from sahara.plugins.mapr import images import sahara.plugins.mapr.util.general as util import sahara.plugins.provisioning as p import sahara.plugins.utils as u class BaseVersionHandler(vh.AbstractVersionHandler): def __init__(self): self._validator = bv.BaseValidator() self._configurer = base_conf.BaseConfigurer() self._health_checker = health.BaseHealthChecker() self._node_manager = bs.BaseNodeManager() self._version = None self._required_services = [] self._services = [] self._node_processes = {} self._configs = [] self.images = images def get_edp_engine(self, cluster, job_type): if job_type in edp.MapROozieJobEngine.get_supported_job_types(): return edp.MapROozieJobEngine(cluster) return None def get_edp_job_types(self): return edp.MapROozieJobEngine.get_supported_job_types() def get_edp_config_hints(self, job_type): return edp.MapROozieJobEngine.get_possible_job_config(job_type) def get_services(self): return self._services def get_required_services(self): return self._required_services def get_node_processes(self): if not self._node_processes: self._node_processes = { s.ui_name: [np.ui_name for np in s.node_processes] for s in self.get_services() if s.node_processes} return self._node_processes def get_configs(self): if not self._configs: configs = [c for s in self.get_services() for c in s.get_configs()] configs += self._get_version_configs() configs += self._get_repo_configs() self._configs = util.unique_list(configs) return self._configs def _get_repo_configs(self): ubuntu_base = p.Config( name="Ubuntu base repo", applicable_target="general", scope='cluster', priority=1, default_value="", description=_( 'Specifies Ubuntu MapR core repository.') ) centos_base = p.Config( name="CentOS base repo", applicable_target="general", scope='cluster', priority=1, default_value="", description=_( 'Specifies CentOS MapR core repository.') ) ubuntu_eco = p.Config( name="Ubuntu ecosystem repo", applicable_target="general", scope='cluster', priority=1, default_value="", description=_( 'Specifies Ubuntu MapR ecosystem repository.') ) centos_eco = p.Config( name="CentOS ecosystem repo", applicable_target="general", scope='cluster', priority=1, default_value="", description=_( 'Specifies CentOS MapR ecosystem repository.') ) return [ubuntu_base, centos_base, ubuntu_eco, centos_eco] def _get_version_configs(self): services = self.get_services() service_version_dict = c.defaultdict(list) for service in services: service_version_dict[service.ui_name].append(service.version) result = [] for service in services: versions = service_version_dict[service.ui_name] if len(versions) > 1: result.append(service.get_version_config(versions)) return result def get_configs_dict(self): configs = dict() for service in self.get_services(): configs.update(service.get_configs_dict()) return configs def configure_cluster(self, cluster): instances = u.get_instances(cluster) cluster_context = self.get_context(cluster, added=instances) self._configurer.configure(cluster_context) def start_cluster(self, cluster): instances = u.get_instances(cluster) cluster_context = self.get_context(cluster, added=instances) self._node_manager.start(cluster_context) self._configurer.post_start(cluster_context) def validate(self, cluster): cluster_context = self.get_context(cluster) self._validator.validate(cluster_context) def validate_scaling(self, cluster, existing, additional): cluster_context = self.get_context(cluster) self._validator.validate_scaling(cluster_context, existing, additional) def scale_cluster(self, cluster, instances): cluster_context = self.get_context(cluster, added=instances) cluster_context._cluster_services = None self._configurer.configure(cluster_context, instances) self._configurer.update(cluster_context, instances) self._node_manager.start(cluster_context, instances) def decommission_nodes(self, cluster, instances): cluster_context = self.get_context(cluster, removed=instances) cluster_context._cluster_services = None self._node_manager.move_nodes(cluster_context, instances) self._node_manager.stop(cluster_context, instances) self._node_manager.await_no_heartbeat() self._node_manager.remove_nodes(cluster_context, instances) self._configurer.update(cluster_context, instances) def get_open_ports(self, node_group): result = [] for service in self.get_services(): for node_process in service.node_processes: if node_process.ui_name in node_group.node_processes: result += node_process.open_ports return util.unique_list(result) def get_cluster_checks(self, cluster): cluster_context = self.get_context(cluster) return self._health_checker.get_checks(cluster_context) def get_image_arguments(self): if hasattr(self, 'images'): return self.images.get_image_arguments() else: return NotImplemented def pack_image(self, hadoop_version, remote, test_only=False, image_arguments=None): if hasattr(self, 'images'): self.images.pack_image( remote, test_only=test_only, image_arguments=image_arguments) def validate_images(self, cluster, test_only=False, image_arguments=None): if hasattr(self, 'images'): self.images.validate_images( cluster, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/mapr/images.py0000666000175100017510000000275713245514472020524 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import images from sahara.plugins import utils as plugin_utils _validator = images.SaharaImageValidator.from_yaml( 'plugins/mapr/resources/images/image.yaml', resource_roots=['plugins/mapr/resources/images']) def get_image_arguments(): return _validator.get_argument_list() def pack_image(remote, test_only=False, image_arguments=None): _validator.validate(remote, test_only=test_only, image_arguments=image_arguments) def validate_images(cluster, test_only=False, image_arguments=None): image_arguments = get_image_arguments() if not test_only: instances = plugin_utils.get_instances(cluster) else: instances = plugin_utils.get_instances(cluster)[0] for instance in instances: with instance.remote() as r: _validator.validate(r, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/mapr/__init__.py0000666000175100017510000000000013245514472020771 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/0000775000175100017510000000000013245515027020510 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hive/0000775000175100017510000000000013245515027021443 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hive/hive.py0000666000175100017510000001636513245514472022770 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslo_log import log as logging from sahara.i18n import _ import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.services.sentry.sentry as sentry import sahara.plugins.mapr.util.event_log as el import sahara.plugins.mapr.util.validation_utils as vu import sahara.utils.files as files LOG = logging.getLogger(__name__) HIVE_METASTORE = np.NodeProcess( name='hivemeta', ui_name='HiveMetastore', package='mapr-hivemetastore', open_ports=[9083] ) HIVE_SERVER_2 = np.NodeProcess( name='hs2', ui_name='HiveServer2', package='mapr-hiveserver2', open_ports=[10000] ) class Hive(s.Service): def __init__(self): super(Hive, self).__init__() self._name = 'hive' self._ui_name = 'Hive' self._node_processes = [HIVE_METASTORE, HIVE_SERVER_2] self._validation_rules = [ vu.exactly(1, HIVE_METASTORE), vu.exactly(1, HIVE_SERVER_2), ] # hive-site.xml def get_config_files(self, cluster_context, configs, instance=None): hive_default = 'plugins/mapr/services/hive/resources/hive-default.xml' hive_site = bcf.HadoopXML("hive-site.xml") hive_site.remote_path = self.conf_dir(cluster_context) if instance: hive_site.fetch(instance) hive_site.parse(files.get_file_text(hive_default)) hive_site.add_properties(self._get_hive_site_props(cluster_context)) sentry_host = cluster_context.get_instance(sentry.SENTRY) if sentry_host: sentry_mode = cluster_context._get_cluster_config_value( sentry.Sentry().SENTRY_STORAGE_MODE) ui_name = sentry.Sentry().ui_name sentry_version = cluster_context.get_chosen_service_version( ui_name) sentry_service = cluster_context. \ _find_service_instance(ui_name, sentry_version) if sentry_service.supports(self, sentry_mode): sentry_default = 'plugins/mapr/services/hive/resources/' \ 'sentry-default.xml' sentry_db = \ 'plugins/mapr/services/hive/resources/sentry-db.xml' hive_site.parse(files.get_file_text(sentry_default)) hive_site.add_property('hive.sentry.conf.url', 'file://%s/sentry-site.xml' % sentry_service.conf_dir( cluster_context)) if sentry_mode == sentry.DB_STORAGE_SENTRY_MODE: hive_site.parse(files.get_file_text(sentry_db)) return [hive_site] def _get_hive_site_props(self, cluster_context): # Import here to resolve circular dependency from sahara.plugins.mapr.services.mysql import mysql zookeepers = cluster_context.get_zookeeper_nodes_ip() metastore_specs = mysql.MySQL.METASTORE_SPECS return { 'javax.jdo.option.ConnectionDriverName': mysql.MySQL.DRIVER_CLASS, 'javax.jdo.option.ConnectionURL': self._get_jdbc_uri( cluster_context), 'javax.jdo.option.ConnectionUserName': metastore_specs.user, 'javax.jdo.option.ConnectionPassword': metastore_specs.password, 'hive.metastore.uris': self._get_metastore_uri(cluster_context), 'hive.zookeeper.quorum': zookeepers, 'hbase.zookeeper.quorum': zookeepers, } def _get_jdbc_uri(self, cluster_context): # Import here to resolve circular dependency from sahara.plugins.mapr.services.mysql import mysql jdbc_uri = ('jdbc:mysql://%(db_host)s:%(db_port)s/%(db_name)s?' 'createDatabaseIfNotExist=true') jdbc_args = { 'db_host': mysql.MySQL.get_db_instance( cluster_context).internal_ip, 'db_port': mysql.MySQL.MYSQL_SERVER_PORT, 'db_name': mysql.MySQL.METASTORE_SPECS.db_name, } return jdbc_uri % jdbc_args def _get_metastore_uri(self, cluster_context): return 'thrift://%s:9083' % cluster_context.get_instance_ip( HIVE_METASTORE) def post_start(self, cluster_context, instances): # Import here to resolve circular dependency import sahara.plugins.mapr.services.maprfs.maprfs as mfs create_path = lambda p: 'sudo -u mapr hadoop fs -mkdir %s' % p check_path = 'sudo -u mapr hadoop fs -ls %s' cmd = "%(check)s || ( %(parent)s && %(target)s )" args = { 'check': check_path % '/user/hive/warehouse/', 'parent': create_path('/user/hive/'), 'target': create_path('/user/hive/warehouse/') } cldb_node = cluster_context.get_instance(mfs.CLDB) with cldb_node.remote() as r: LOG.debug("Creating Hive warehouse dir") r.execute_command(cmd % args, raise_when_error=False) self._create_sentry_role(cluster_context) def _create_sentry_role(self, cluster_context): @el.provision_event(name=_("Create Sentry role for Hive")) def _create_role(instance): cmd = 'sudo -u mapr hive -e "create role admin_role;' \ 'grant all on server HS2 to role admin_role;' \ 'grant role admin_role to group mapr;"' with instance.remote() as r: LOG.debug("Creating hive role for sentry") r.execute_command(cmd, raise_when_error=False) hive_host = cluster_context.get_instance(HIVE_METASTORE) sentry_host = cluster_context.get_instance(sentry.SENTRY) if sentry_host: sentry_mode = cluster_context._get_cluster_config_value( sentry.Sentry().SENTRY_STORAGE_MODE) ui_name = sentry.Sentry().ui_name sentry_version = cluster_context.get_chosen_service_version( ui_name) sentry_service = cluster_context. \ _find_service_instance(ui_name, sentry_version) if sentry_service.supports(self, sentry_mode): _create_role(hive_host) class HiveV013(Hive): def __init__(self): super(HiveV013, self).__init__() self._version = '0.13' self._dependencies = [('mapr-hive', self.version)] class HiveV10(Hive): def __init__(self): super(HiveV10, self).__init__() self._version = "1.0" self._dependencies = [("mapr-hive", self.version)] class HiveV12(Hive): def __init__(self): super(HiveV12, self).__init__() self._version = "1.2" self._dependencies = [("mapr-hive", self.version)] sahara-8.0.0/sahara/plugins/mapr/services/hive/__init__.py0000666000175100017510000000000013245514472023547 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hive/resources/0000775000175100017510000000000013245515027023455 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hive/resources/hive-default.xml0000666000175100017510000000320113245514472026555 0ustar zuulzuul00000000000000 javax.jdo.option.ConnectionURL jdbc:mysql://localhost:3306/metastore ?createDatabaseIfNotExist=true JDBC connect string for a JDBC metastore datanucleus.schema.autoCreateAll true creates necessary schema on a startup if one doesn't exist. javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver Driver class name for a JDBC metastore javax.jdo.option.ConnectionUserName root username to use against metastore database javax.jdo.option.ConnectionPassword password to use against metastore database hive.metastore.uris thrift://localhost:9083 hive.zookeeper.client.port 5181 hbase.zookeeper.property.clientPort 5181 hive.metastore.execute.setugi true sahara-8.0.0/sahara/plugins/mapr/services/hive/resources/sentry-db.xml0000666000175100017510000000162413245514472026116 0ustar zuulzuul00000000000000 hive.support.concurrency Enable Hive's Table Lock Manager Service true hive.security.authorization.task.factory org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl hive.metastore.rawstore.impl org.apache.sentry.binding.metastore.AuthorizingObjectStore hive.metastore.pre.event.listeners org.apache.sentry.binding.metastore.MetastoreAuthzBinding list of comma separated listeners for metastore events. sahara-8.0.0/sahara/plugins/mapr/services/hive/resources/sentry-default.xml0000666000175100017510000000237513245514472027161 0ustar zuulzuul00000000000000 hive.server2.session.hook org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook hive.metastore.rawstore.impl org.apache.sentry.binding.metastore.AuthorizingObjectStore hive.server2.enable.doAs false Set this property to enable impersonation in Hive Server2 hive.metastore.execute.setugi true hive.internal.ss.authz.settings.applied.marker true hive.sentry.subject.name mapr hive.metastore.filter.hook org.apache.sentry.binding.metastore.SentryMetaStoreFilterHook hive.internal.ss.authz.settings.applied.marker true sahara-8.0.0/sahara/plugins/mapr/services/pig/0000775000175100017510000000000013245515027021267 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/pig/pig.py0000666000175100017510000000277113245514472022434 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu PIG = np.NodeProcess( name='pig', ui_name='Pig', package='mapr-pig' ) class Pig(s.Service): def __init__(self): super(Pig, self).__init__() self._name = 'pig' self._ui_name = 'Pig' self._node_processes = [PIG] self._validation_rules = [vu.at_least(1, PIG)] class PigV013(Pig): def __init__(self): super(PigV013, self).__init__() self._version = '0.13' class PigV014(Pig): def __init__(self): super(PigV014, self).__init__() self._version = '0.14' class PigV015(Pig): def __init__(self): super(PigV015, self).__init__() self._version = '0.15' class PigV016(Pig): def __init__(self): super(PigV016, self).__init__() self._version = '0.16' sahara-8.0.0/sahara/plugins/mapr/services/pig/__init__.py0000666000175100017510000000000013245514472023373 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/yarn/0000775000175100017510000000000013245515027021461 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/yarn/yarn.py0000666000175100017510000001132613245514472023014 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu from sahara.swift import swift_helper from sahara.topology import topology_helper as topo RESOURCE_MANAGER = np.NodeProcess( name='resourcemanager', ui_name='ResourceManager', package='mapr-resourcemanager', open_ports=[8033, 8032, 8031, 8030, 8088] ) NODE_MANAGER = np.NodeProcess( name='nodemanager', ui_name='NodeManager', package='mapr-nodemanager', open_ports=[8041, 8040, 8042, 8044] ) HISTORY_SERVER = np.NodeProcess( name='historyserver', ui_name='HistoryServer', package='mapr-historyserver', open_ports=[10020, 19888, 19890] ) class YARN(s.Service): cluster_mode = 'yarn' def __init__(self): super(YARN, self).__init__() self._name = 'hadoop' self._ui_name = 'YARN' self._node_processes = [RESOURCE_MANAGER, NODE_MANAGER, HISTORY_SERVER] self._ui_info = [ ('NodeManager', NODE_MANAGER, {s.SERVICE_UI: 'http://%s:8042'}), ('ResourceManager', RESOURCE_MANAGER, {s.SERVICE_UI: 'http://%s:8088'}), ('HistoryServer', RESOURCE_MANAGER, {s.SERVICE_UI: 'http://%s:19888'}), ] self._cluster_defaults = ['yarn-cluster.json'] self._node_defaults = ['yarn-node.json'] def get_config_files(self, cluster_context, configs, instance=None): # yarn-site.xml yarn_site = bcf.HadoopXML("yarn-site.xml") yarn_site.remote_path = self.conf_dir(cluster_context) if instance: yarn_site.fetch(instance) yarn_site.add_properties(self._get_yarn_site_props(cluster_context)) yarn_site.load_properties(configs) # core-site.xml core_site = bcf.HadoopXML("core-site.xml") core_site.remote_path = self.conf_dir(cluster_context) if instance: core_site.fetch(instance) core_site.add_properties(self._get_core_site_props(cluster_context)) return [yarn_site, core_site] def _get_core_site_props(self, cluster_context): result = { 'hadoop.proxyuser.mapr.groups': '*', 'hadoop.proxyuser.mapr.hosts': '*', } if cluster_context.is_node_aware: result.update(self._get_core_site_node_aware_props()) for conf in swift_helper.get_swift_configs(): result[conf['name']] = conf['value'] return result def _get_core_site_node_aware_props(self): result = topo.vm_awareness_core_config() result = {c['name']: c['value'] for c in result} result.update({ 'net.topology.script.file.name': '/opt/mapr/topology.sh', 'net.topology.script.number.args': '75', }) return result def _get_yarn_site_props(self, cluster_context): return { 'hadoop.proxyuser.mapr.groups': '*', 'hadoop.proxyuser.mapr.hosts': '*', } def conf_dir(self, cluster_context): return '%s/etc/hadoop' % self.home_dir(cluster_context) def get_file_path(self, file_name): template = 'plugins/mapr/services/yarn/resources/%s' return template % file_name class YARNv241(YARN): def __init__(self): super(YARNv241, self).__init__() self._version = '2.4.1' self._validation_rules = [ vu.exactly(1, RESOURCE_MANAGER), vu.at_least(1, NODE_MANAGER), vu.exactly(1, HISTORY_SERVER), ] class YARNv251(YARN): def __init__(self): super(YARNv251, self).__init__() self._version = '2.5.1' self._validation_rules = [ vu.at_least(1, RESOURCE_MANAGER), vu.at_least(1, NODE_MANAGER), vu.exactly(1, HISTORY_SERVER), ] class YARNv270(YARN): def __init__(self): super(YARNv270, self).__init__() self._version = "2.7.0" self._validation_rules = [ vu.at_least(1, RESOURCE_MANAGER), vu.at_least(1, NODE_MANAGER), vu.exactly(1, HISTORY_SERVER), ] sahara-8.0.0/sahara/plugins/mapr/services/yarn/__init__.py0000666000175100017510000000000013245514472023565 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/yarn/resources/0000775000175100017510000000000013245515027023473 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/yarn/resources/yarn-cluster.json0000666000175100017510000000116313245514472027024 0ustar zuulzuul00000000000000[ { "value": 1024, "name": "yarn.scheduler.minimum-allocation-mb", "priority": 1, "config_type": "int", "description": "The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this won't take effect, and the specified value will get allocated at minimum." }, { "value": 8192, "name": "yarn.scheduler.maximum-allocation-mb", "priority": 1, "config_type": "int", "description": "The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this won't take effect, and will get capped to this value." } ]sahara-8.0.0/sahara/plugins/mapr/services/yarn/resources/yarn-node.json0000666000175100017510000000115513245514472026271 0ustar zuulzuul00000000000000[ { "value": 8192, "name": "yarn.nodemanager.resource.memory-mb", "priority": 1, "config_type": "int", "description": "Amount of physical memory, in MB, that can be allocated for containers." }, { "value": 4, "name": "yarn.nodemanager.resource.cpu-vcores", "priority": 1, "config_type": "int", "description": "Number of CPU cores that can be allocated for containers." }, { "value": 1, "name": "yarn.nodemanager.resource.io-spindles", "priority": 1, "config_type": "int", "description": "Number of spindles that can be allocated for containers." } ]sahara-8.0.0/sahara/plugins/mapr/services/sentry/0000775000175100017510000000000013245515027022034 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/sentry/sentry.py0000666000175100017510000001515113245514472023742 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from sahara.i18n import _ import sahara.plugins.mapr.domain.configuration_file as cf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.services.mysql.mysql as mysql import sahara.plugins.mapr.util.event_log as el import sahara.plugins.mapr.util.maprfs_helper as mfs import sahara.plugins.provisioning as p import sahara.utils.files as files SENTRY = np.NodeProcess( name='sentry', ui_name='Sentry', package='mapr-sentry' ) SENTRY_MODE_CONFIG_NAME = 'Sentry storage mode' FILE_STORAGE_SENTRY_MODE = 'File-base storage' DB_STORAGE_SENTRY_MODE = 'DB-based storage' class Sentry(s.Service): SENTRY_STORAGE_MODE = p.Config( name=SENTRY_MODE_CONFIG_NAME, applicable_target='Sentry', scope='cluster', config_type="dropdown", config_values=[(v, v) for v in (FILE_STORAGE_SENTRY_MODE, DB_STORAGE_SENTRY_MODE)], priority=1, description=_( 'Specifies Sentry storage mode.') ) GLOBAL_POLICY_FILE = '/user/mapr/sentry/global-policy.ini' def __init__(self): super(Sentry, self).__init__() self._name = 'sentry' self._ui_name = 'Sentry' self._node_processes = [SENTRY] self._priority = 2 def get_configs(self): return [Sentry.SENTRY_STORAGE_MODE] def get_config_files(self, cluster_context, configs, instance=None): sentry_default = \ 'plugins/mapr/services/sentry/resources/sentry-default.xml' global_policy_template = \ 'plugins/mapr/services/sentry/resources/global-policy.ini' sentry_site = cf.HadoopXML('sentry-site.xml') sentry_site.remote_path = self.conf_dir(cluster_context) if instance: sentry_site.fetch(instance) sentry_site.load_properties(configs) sentry_mode = configs[self.SENTRY_STORAGE_MODE.name] sentry_site.parse(files.get_file_text(sentry_default)) sentry_site.add_properties( self._get_sentry_site_props(cluster_context, sentry_mode)) global_policy = cf.TemplateFile('global-policy.ini') global_policy.remote_path = self.conf_dir(cluster_context) global_policy.parse(files.get_file_text(global_policy_template)) return [sentry_site, global_policy] def _get_jdbc_uri(self, cluster_context): jdbc_uri = ('jdbc:mysql://%(db_host)s:%(db_port)s/%(db_name)s?' 'createDatabaseIfNotExist=true') jdbc_args = { 'db_host': mysql.MySQL.get_db_instance( cluster_context).internal_ip, 'db_port': mysql.MySQL.MYSQL_SERVER_PORT, 'db_name': mysql.MySQL.SENTRY_SPECS.db_name, } return jdbc_uri % jdbc_args def _get_sentry_site_props(self, cluster_context, setry_mode): sentry_specs = mysql.MySQL.SENTRY_SPECS if setry_mode == FILE_STORAGE_SENTRY_MODE: return { 'sentry.hive.provider.backend': 'org.apache.sentry.provider' '.file.SimpleFileProviderBackend', 'sentry.hive.provider.resource': 'maprfs:///' + self.GLOBAL_POLICY_FILE, } if setry_mode == DB_STORAGE_SENTRY_MODE: return { 'sentry.hive.provider.backend': 'org.apache.sentry.provider.db.SimpleDBProviderBackend', 'sentry.store.jdbc.url': self._get_jdbc_uri(cluster_context), 'sentry.store.jdbc.driver': mysql.MySQL.DRIVER_CLASS, 'sentry.store.jdbc.user': sentry_specs.user, 'sentry.store.jdbc.password': sentry_specs.password, } def _init_db_schema(self, cluster_context): instance = cluster_context.get_instance(SENTRY) cmd = '%(home)s/bin/sentry --command schema-tool' \ ' --conffile %(home)s/conf/sentry-site.xml' \ ' --dbType mysql --initSchema' % { 'home': self.home_dir(cluster_context)} @el.provision_event(name=_("Init Sentry DB schema"), instance=instance) def decorated(): with instance.remote() as r: r.execute_command(cmd, run_as_root=True) decorated() def post_start(self, cluster_context, instances): sentry_host = cluster_context.get_instance(SENTRY) source = self.conf_dir(cluster_context) + '/global-policy.ini' with sentry_host.remote() as r: mfs.mkdir(r, '/user/mapr/sentry', run_as='mapr') mfs.chmod(r, '/user/mapr/sentry', 777, run_as='mapr') mfs.copy_from_local(r, source, self.GLOBAL_POLICY_FILE, hdfs_user='mapr') def _copy_warden_conf(self, cluster_context): sentry_host = cluster_context.get_instance(SENTRY) cmd = 'sudo -u mapr cp %s/conf.d/warden.sentry.conf' \ ' /opt/mapr/conf/conf.d/' % self.home_dir(cluster_context) with sentry_host.remote() as r: r.execute_command(cmd) def post_install(self, cluster_context, instances): self._set_service_dir_owner(cluster_context, instances) if cluster_context._get_cluster_config_value( self.SENTRY_STORAGE_MODE) == DB_STORAGE_SENTRY_MODE: self._init_db_schema(cluster_context) self._copy_warden_conf(cluster_context) def supports(self, service, mode): "return True is Sentry supports integration" service = service.name + '-' + service.version return self.SENTRY_SUPPORT_MATRIX[mode][service] class SentryV16(Sentry): SENTRY_SUPPORT_MATRIX = { DB_STORAGE_SENTRY_MODE: { 'hive-1.2': True, 'hive-2.0': True, 'impala-2.2.0': True, 'impala-2.5.0': True }, FILE_STORAGE_SENTRY_MODE: { 'hive-1.2': True, 'hive-2.0': True, 'impala-2.2.0': True, 'impala-2.5.0': True }, } def __init__(self): super(SentryV16, self).__init__() self._version = '1.6.0' sahara-8.0.0/sahara/plugins/mapr/services/sentry/__init__.py0000666000175100017510000000000013245514472024140 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/sentry/resources/0000775000175100017510000000000013245515027024046 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/sentry/resources/sentry-default.xml0000666000175100017510000000016213245514472027542 0ustar zuulzuul00000000000000 sahara-8.0.0/sahara/plugins/mapr/services/sentry/resources/global-policy.ini0000666000175100017510000000022213245514472027305 0ustar zuulzuul00000000000000[groups] mapr = admin_role testuser = test_role [roles] admin_role = server=HS2 test_role = server=HS2->db=test_db1->table=test_table->action=all sahara-8.0.0/sahara/plugins/mapr/services/hbase/0000775000175100017510000000000013245515027021572 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hbase/hbase.py0000666000175100017510000000721713245514472023242 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu HBASE_MASTER = np.NodeProcess( name='hbmaster', ui_name='HBase-Master', package='mapr-hbase-master', open_ports=[60000, 60010] ) HBASE_REGION_SERVER = np.NodeProcess( name='hbregionserver', ui_name='HBase-RegionServer', package='mapr-hbase-regionserver', open_ports=[60020] ) HBASE_THRIFT = np.NodeProcess( name='hbasethrift', ui_name='HBase-Thrift', package='mapr-hbasethrift', open_ports=[9090] ) HBASE_REST = np.NodeProcess( name="hbaserestgateway", ui_name="HBase REST", package="mapr-hbase-rest", open_ports=[8080, 8085], ) class HBase(s.Service): def __init__(self): super(HBase, self).__init__() self._name = 'hbase' self._ui_name = 'HBase' self._node_processes = [ HBASE_MASTER, HBASE_REGION_SERVER, HBASE_THRIFT, ] self._cluster_defaults = ['hbase-default.json'] self._validation_rules = [ vu.at_least(1, HBASE_MASTER), vu.at_least(1, HBASE_REGION_SERVER), ] self._ui_info = [ ("HBase Master", HBASE_MASTER, {s.SERVICE_UI: "http://%s:60010"}), ] def get_config_files(self, cluster_context, configs, instance=None): hbase_site = bcf.HadoopXML("hbase-site.xml") hbase_site.remote_path = self.conf_dir(cluster_context) if instance: hbase_site.fetch(instance) hbase_site.load_properties(configs) return [hbase_site] class HBaseV094(HBase): def __init__(self): super(HBaseV094, self).__init__() self._version = '0.94.24' self._dependencies = [('mapr-hbase', self.version)] class HBaseV0987(HBase): def __init__(self): super(HBaseV0987, self).__init__() self._version = '0.98.7' self._dependencies = [('mapr-hbase', self.version)] class HBaseV0989(HBase): def __init__(self): super(HBaseV0989, self).__init__() self._version = '0.98.9' self._dependencies = [('mapr-hbase', self.version)] self._node_processes.append(HBASE_REST) self._ui_info.append( ("HBase REST", HBASE_REST, {s.SERVICE_UI: "http://%s:8085"}), ) class HBaseV09812(HBase): def __init__(self): super(HBaseV09812, self).__init__() self._version = "0.98.12" self._dependencies = [("mapr-hbase", self.version)] self._node_processes.append(HBASE_REST) self._ui_info.append( ("HBase REST", HBASE_REST, {s.SERVICE_UI: "http://%s:8085"}), ) class HBaseV111(HBase): def __init__(self): super(HBaseV111, self).__init__() self._version = "1.1.1" self._dependencies = [("mapr-hbase", self.version)] self._node_processes.append(HBASE_REST) self._ui_info.append( ("HBase REST", HBASE_REST, {s.SERVICE_UI: "http://%s:8085"}), ) sahara-8.0.0/sahara/plugins/mapr/services/hbase/__init__.py0000666000175100017510000000000013245514472023676 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hbase/resources/0000775000175100017510000000000013245515027023604 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hbase/resources/hbase-default.json0000666000175100017510000000113213245514472027205 0ustar zuulzuul00000000000000[ { "value": "maprfs:///hbase", "name": "hbase.rootdir", "description": "The directory shared by RegionServers." }, { "value": true, "name": "dfs.support.append", "config_type": "bool" }, { "name": "hbase.fsutil.maprfs.impl", "value": "org.apache.hadoop.hbase.util.FSMapRUtils" }, { "value": 30, "name": "hbase.regionserver.handler.count", "config_type": "int" }, { "value": 64, "name": "fs.mapr.threads", "config_type": "int" }, { "value": false, "name": "hbase.table.sanity.checks", "config_type": "bool" } ]sahara-8.0.0/sahara/plugins/mapr/services/sqoop/0000775000175100017510000000000013245515027021651 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/sqoop/sqoop2.py0000666000175100017510000000267213245514472023462 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu SQOOP_2_SERVER = np.NodeProcess( name='sqoop2', ui_name='Sqoop2-Server', package='mapr-sqoop2-server', open_ports=[12000] ) SQOOP_2_CLIENT = np.NodeProcess( name='sqoop-client', ui_name='Sqoop2-Client', package='mapr-sqoop2-client' ) class Sqoop2(s.Service): def __init__(self): super(Sqoop2, self).__init__() self._name = 'sqoop' self._ui_name = 'Sqoop2' self._version = '2.0.0' self._node_processes = [SQOOP_2_CLIENT, SQOOP_2_SERVER] self._validation_rules = [ vu.exactly(1, SQOOP_2_SERVER), ] def post_install(self, cluster_context, instances): self._set_service_dir_owner(cluster_context, instances) sahara-8.0.0/sahara/plugins/mapr/services/sqoop/__init__.py0000666000175100017510000000000013245514472023755 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/impala/0000775000175100017510000000000013245515027021753 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/impala/__init__.py0000666000175100017510000000000013245514472024057 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/impala/impala.py0000666000175100017510000001573513245514472023610 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.services.hbase.hbase as hbase import sahara.plugins.mapr.services.hive.hive as hive import sahara.plugins.mapr.services.sentry.sentry as sentry import sahara.plugins.mapr.util.general as g import sahara.plugins.mapr.util.validation_utils as vu import sahara.utils.files as files IMPALA_SERVER = np.NodeProcess( name='impalaserver', ui_name='Impala-Server', package='mapr-impala-server', open_ports=[21000, 21050, 25000] ) IMPALA_STATE_STORE = np.NodeProcess( name='impalastore', ui_name='Impala-Statestore', package='mapr-impala-statestore', open_ports=[25010] ) IMPALA_CATALOG = np.NodeProcess( name='impalacatalog', ui_name='Impala-Catalog', package='mapr-impala-catalog', open_ports=[25020] ) class Impala(s.Service): def __init__(self): super(Impala, self).__init__() self._name = 'impala' self._ui_name = 'Impala' self._node_processes = [ IMPALA_CATALOG, IMPALA_SERVER, IMPALA_STATE_STORE, ] def _get_impala_env_props(self, cluster_context): return {} def post_start(self, cluster_context, instances): self._copy_hive_site(cluster_context, instances) def _copy_hive_site(self, cluster_context, instances): hive_site_path = self._hive(cluster_context).conf_dir( cluster_context) + "/hive-site.xml" path = self.conf_dir(cluster_context) + "/hive-site.xml" hive_instance = cluster_context.get_instance(hive.HIVE_METASTORE) impalas = cluster_context.filter_instances(instances, service=self) for instance in impalas: g.copy_file(hive_site_path, hive_instance, path, instance, run_as='root', owner='mapr') # hive service instance def _hive(self, cluster_context): hive_version = cluster_context.get_chosen_service_version('Hive') return cluster_context._find_service_instance('Hive', hive_version) def get_config_files(self, cluster_context, configs, instance=None): defaults = 'plugins/mapr/services/impala/resources/impala-env.sh.j2' impala_env = bcf.TemplateFile("env.sh") impala_env.remote_path = self.conf_dir(cluster_context) if instance: impala_env.fetch(instance) impala_env.parse(files.get_file_text(defaults)) impala_env.add_properties(self._get_impala_env_props(cluster_context)) sentry_host = cluster_context.get_instance(sentry.SENTRY) if sentry_host: sentry_mode = cluster_context._get_cluster_config_value( sentry.Sentry().SENTRY_STORAGE_MODE) ui_name = sentry.Sentry().ui_name sentry_version = cluster_context.get_chosen_service_version( ui_name) sentry_service = cluster_context. \ _find_service_instance(ui_name, sentry_version) if sentry_service.supports(self, sentry_mode): impala_env.add_properties({ 'sentry_home': sentry_service.home_dir(cluster_context), 'sentry_db': sentry_mode == sentry.DB_STORAGE_SENTRY_MODE, 'sentry_policy_file': 'maprfs://' + sentry_service.GLOBAL_POLICY_FILE, }) return [impala_env] def post_install(self, cluster_context, instances): self._set_service_dir_owner(cluster_context, instances) class ImpalaV141(Impala): def __init__(self): super(ImpalaV141, self).__init__() self._version = '1.4.1' self._dependencies = [ ('mapr-hive', hive.HiveV013().version), ('mapr-impala', self.version), ] self._validation_rules = [ vu.depends_on(hive.HiveV013(), self), vu.exactly(1, IMPALA_STATE_STORE), vu.exactly(1, IMPALA_CATALOG), vu.at_least(1, IMPALA_SERVER), ] def _get_impala_env_props(self, cluster_context): return { 'impala_version': self.version, 'statestore_host': cluster_context.get_instance_ip( IMPALA_STATE_STORE), 'catalog_host': cluster_context.get_instance_ip(IMPALA_CATALOG), } class ImpalaV220(Impala): def __init__(self): super(ImpalaV220, self).__init__() self._version = '2.2.0' self._dependencies = [ ('mapr-hive', hive.HiveV12().version), ('mapr-impala', self.version), ] self._validation_rules = [ vu.depends_on(hive.HiveV12(), self), vu.exactly(1, IMPALA_STATE_STORE), vu.exactly(1, IMPALA_CATALOG), vu.at_least(1, IMPALA_SERVER), vu.required_os('centos', self) ] def _get_impala_env_props(self, cluster_context): return { 'impala_version': self.version, 'statestore_host': cluster_context.get_instance_ip( IMPALA_STATE_STORE), 'catalog_host': cluster_context.get_instance_ip(IMPALA_CATALOG), } def _get_packages(self, cluster_context, node_processes): result = [] result += self.dependencies result += [(np.package, self.version) for np in node_processes] # gets the latest version hbase_version = cluster_context.get_chosen_service_version('HBase') result += [('mapr-hbase', hbase_version)] return result class ImpalaV250(Impala): def __init__(self): super(ImpalaV250, self).__init__() self._version = '2.5.0' self._dependencies = [ ('mapr-hive', hive.HiveV12().version), ('mapr-impala', self.version), ('mapr-hbase', hbase.HBaseV111().version) ] self._validation_rules = [ vu.depends_on(hive.HiveV12(), self), vu.exactly(1, IMPALA_STATE_STORE), vu.exactly(1, IMPALA_CATALOG), vu.at_least(1, IMPALA_SERVER), vu.required_os('centos', self) ] def _get_impala_env_props(self, cluster_context): return { 'impala_version': self.version, 'statestore_host': cluster_context.get_instance_ip( IMPALA_STATE_STORE), 'catalog_host': cluster_context.get_instance_ip(IMPALA_CATALOG), } sahara-8.0.0/sahara/plugins/mapr/services/impala/resources/0000775000175100017510000000000013245515027023765 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/impala/resources/impala-env.sh.j20000666000175100017510000000461113245514472026673 0ustar zuulzuul00000000000000# these are configured automatically when impala is installed export IMPALA_HOME=/opt/mapr/impala/impala-{{ impala_version }} export MAPR_HOME=/opt/mapr export IMPALA_VERSION={{ impala_version }} # Get the generic mapr environment variables [ -f ${MAPR_HOME}/conf/env.sh ] && . ${MAPR_HOME}/conf/env.sh # This MUST point to the node running statestore IMPALA_STATE_STORE_HOST={{ statestore_host | default("localhost", True) }} IMPALA_STATE_STORE_PORT={{ statestore_port | default("24000", True) }} CATALOG_SERVICE_HOST={{ catalog_host | default("localhost", True) }} # By default, we use the Hive configuration. # Uncomment this If hive is not configured, or we wish to override it. #HIVE_METASTORE_URI=thrift://localhost:9083 # These impact the impala server and can be optionally changed IMPALA_BACKEND_PORT=22000 IMPALA_LOG_DIR=${IMPALA_HOME}/logs IMPALA_SERVER_ARGS=" \ -log_dir=${IMPALA_LOG_DIR} \ -state_store_port=${IMPALA_STATE_STORE_PORT} \ -use_statestore \ -state_store_host=${IMPALA_STATE_STORE_HOST} \ -catalog_service_host=${CATALOG_SERVICE_HOST} \ -be_port=${IMPALA_BACKEND_PORT} {%- if sentry_db %} -server_name=HS2 \ -sentry_config={{ sentry_home }}/conf/sentry-site.xml {%- endif %} {%- if sentry_policy_file %} -server_name=HS2 \ -authorization_policy_file={{ sentry_policy_file }} {%- endif %}" # These impact the state store daemon and can be optionally changed IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} \ -state_store_port=${IMPALA_STATE_STORE_PORT} \ -catalog_service_host=${CATALOG_SERVICE_HOST} {%- if sentry_db %} -sentry_config={{ sentry_home }}/conf/sentry-site.xml {%- endif %}" IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} \ -state_store_port=${IMPALA_STATE_STORE_PORT} \ -use_statestore \ -state_store_host=${IMPALA_STATE_STORE_HOST} {%- if sentry_db %} -sentry_config={{ sentry_home }}/conf/sentry-site.xml {%- endif %}" # for troubleshooting ENABLE_CORE_DUMPS=false # Impala figures these out at runtime, but they can be overridden here. # (Normally, they should be commented out.) # HIVE_HOME=${MAPR_HOME}/hive/hive-* # HBASE_HOME=${MAPR_HOME}/hbase/hbase-* # HADOOP_HOME=${MAPR_HOME}/hadoop/hodoop-* # No longer used ... # LIBHDFS_OPTS=-Djava.library.path=/usr/lib/impala/lib # MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar sahara-8.0.0/sahara/plugins/mapr/services/swift/0000775000175100017510000000000013245515027021644 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/swift/swift.py0000666000175100017510000000411213245514472023355 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslo_log import log as logging import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.services.maprfs.maprfs as maprfs import sahara.plugins.mapr.util.event_log as el import sahara.utils.files as f LOG = logging.getLogger(__name__) class Swift(s.Service): HADOOP_SWIFT_JAR = ('plugins/mapr/services/swift/' 'resources/hadoop-swift-latest.jar') def __init__(self): super(Swift, self).__init__() self._name = 'swift' self._ui_name = 'Swift' self._cluster_defaults = ['swift-default.json'] def install(self, cluster_context, instances): # swift does not require any package pass def configure(self, cluster_context, instances=None): instances = instances or cluster_context.get_instances() file_servers = cluster_context.filter_instances(instances, maprfs.FILE_SERVER) self._install_swift_jar(cluster_context, file_servers) @el.provision_step("Install Swift service") def _install_swift_jar(self, cluster_context, instances): LOG.debug('Installing Swift jar') jar = f.get_file_text(Swift.HADOOP_SWIFT_JAR) path = '%s/swift.jar' % cluster_context.hadoop_lib @el.provision_event() def install_on_instance(inst): with inst.remote() as r: r.write_file_to(path, jar, run_as_root=True) for instance in instances: install_on_instance(instance) sahara-8.0.0/sahara/plugins/mapr/services/swift/__init__.py0000666000175100017510000000000013245514472023750 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/swift/resources/0000775000175100017510000000000013245515027023656 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/swift/resources/swift-default.json0000666000175100017510000000224413245514472027336 0ustar zuulzuul00000000000000[ { "name": "fs.swift.impl", "config_type": "string", "value": "org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem" }, { "name": "fs.swift.connect.timeout", "config_type": "int", "value": 15000 }, { "name": "fs.swift.socket.timeout", "config_type": "int", "value": 60000 }, { "name": "fs.swift.connect.retry.count", "config_type": "int", "value": 3 }, { "name": "fs.swift.connect.throttle.delay", "config_type": "int", "value": 0 }, { "name": "fs.swift.blocksize", "config_type": "int", "value": 32768 }, { "name": "fs.swift.partsize", "config_type": "int", "value": 4718592 }, { "name": "fs.swift.requestsize", "config_type": "int", "value": 64 }, { "name": "fs.swift.service.sahara.public", "config_type": "bool", "value": true }, { "name": "fs.swift.service.sahara.http.port", "config_type": "int", "value": 8080 }, { "name": "fs.swift.service.sahara.https.port", "config_type": "int", "value": 443 }, { "name": "fs.swift.service.sahara.auth.endpoint.prefix", "value": "/endpoints/AUTH_" } ]sahara-8.0.0/sahara/plugins/mapr/services/swift/resources/hadoop-swift-latest.jar0000666000175100017510000034516313245514472030273 0ustar zuulzuul00000000000000PK lL[D META-INF/PK kL[D Ÿ6Éf}META-INF/MANIFEST.MFóMÌËLK-.Ñ K-*ÎÌϳR0Ô3àår,JÎÈ,K-Bä¤V”+À$x¹œ‹RKRSt*­ “3R|ËRóx¹œJ3sJÀâ‰y)E©å‘]¯”lñæzñ&@;x¹PK jL[Dorg/PK jL[D org/apache/PK jL[Dorg/apache/hadoop/PK jL[Dorg/apache/hadoop/fs/PK kL[Dorg/apache/hadoop/fs/swift/PK kL[D org/apache/hadoop/fs/swift/auth/PK jL[D)org/apache/hadoop/fs/swift/auth/entities/PK kL[D&org/apache/hadoop/fs/swift/exceptions/PK kL[D org/apache/hadoop/fs/swift/http/PK kL[D#org/apache/hadoop/fs/swift/snative/PK kL[D org/apache/hadoop/fs/swift/util/PK kL[DÑ«#|‚Aorg/apache/hadoop/fs/swift/auth/ApiKeyAuthenticationRequest.class¥TmOA~¶-m)§…Š âË!(¥¼¬B4‘*I%c+0D>.×¥],wõnÿÒD ÑÄà2ÎÞ5„” ~¸g§3Ï<3Ó»_¿¿ÿ°ˆ¥,Lõc÷Íñ ƒéR˜Ê¢hìL%cgÍ1—Á| C¢­ªòpÕ—uéj%ZÃRÍó\´…Ó”¼)êž×æ»>«]ÍE¨›¼ÒTfH?S®Ò+ ¯Šµ=q xK¸ ¾©}å6Ê—`œÙbH­zuɯ)W®‡û;Ò/vZä)ÔA_!}Œ ©L)dcœ²MؼZSô¹ÒˆÐFxÃÕ!x´ŽP´LVíËŸC| ÒFiM’rtò#Ïh%a×ɲÀP`25²C,Ÿ9C$?{íB*M"Àß+$èNRM–4[ÇBIö˜S“FÈ¿‰‰°ŒÕæü ís[<l®ödœÄ­ððó` ”Í|Zg—/èÌz 0ÖÊj#Å.|»'H¤äuO;½A" Ûtfç ¡Älƒ| &¬|Gtï}éX:C‘Iešˆ§­ í'tí’ì6‘l¢¿ ƒÉµKä…Kä+aÝ)ú RÑq÷§é¸‡Ìq?`ðPK jL[D­Ì‡×”;org/apache/hadoop/fs/swift/auth/AuthenticationRequest.classR]OA=Ó–¶,[©…‚*-J[ŒŽ!>¨$Jâ‹ &@H|œn‡vH™­»³ú`üOšhI4ñø£Œw¶›‚e}ñåÜ;wÏ=÷ÌýõûûOÐpà`u³X+à†ƒ,V‹¸iã- ëjÔ Ø`pŒÔB›=q**íñ^ðÐ=~`¥{OòÏ”Vf‡!Ûl1ävý.QçÛJ˽è´#ƒCÑÄ;'G"Pöœs¦¯B†'m?èq1^_ò¾èúþ‡<ü Ž ‘éóRå £|½/ßE244½Ô“æð‚ÅÅf+Íd)ü›Vm^fY÷EãOtó? <ùJYŸ+©óï[ %\q1—ayZôe¤]0ÔS>žo÷yÃÅmܱ°é⪠Ûÿµ†ò¹‹7éQ)/†C©» ÷Ò.~©”ئŭ5wÿý5ú‰ú©è!í(ËQNû œ§ÓcŠŒâÌÖØJ2(::‘7p•2wLB ±Ø"ª‰Ÿ|EæóD [©ÍKXNšwˆmå·î~Cfzüêy+,Y‰‚ÍæbUfŸ!ÑzM;§öÙ·gÈUfF¤K¦(”#ÊŸF(N›Ü¾`²61y=f­üPK kL[DY¿cs6Borg/apache/hadoop/fs/swift/auth/AuthenticationRequestWrapper.class¥‘ÏJÃ@Æ¿mcbc­õ¿WA¡­âB=ˆµDðTTô¼k»¥&1Ùêk)(‚À‡gÓEÄC½ÌÌß~óÛÙ·÷—Wu¬¸°°P@‹–,3Xb¨{ ;í êr ¯'yO\Aȯbß©+Í„ïS¾VžÐ*ðOäÍPÆzÁn*_éC¾R='¿ƒàR2Ì´•/†×‰Î€:síÀƒs)sN›–¡5Þô‹H„¡Œ¢Q“ß ;]©÷“-ìVªcïÁ‰?]ÜÓ`yòP™®þE¾Õ·¢“ELÀfhþg eãÇÂïòãN_z«ô×ý=CÎL Ê¢šR,Ði›2£gNwö÷ŸŸ¿ Ï2„•4$€SxJa•B) kIXg:•FÔ„ ¦+'âLð¦pêü]õDÚf“Á„–í3eË2¦4Ý:ƒ)?«cT“W”¦œô¾ª;ÂtÚ’ÁnàõVÅm×¹h »!yCÔ\·Å5ןԱá¢c\:F%5ïRlncÉXyðË£€À¸q?J‡ÁóË#wl[j}@8,ØRŽ2Û ¢ù )»5¼ÖdE9òmç´*Û¢Ú”ä–k‹æ‘h+zîÆLCiÿ%ßÁ@la”ë|ºå:Z"ýX]š7ý–Ìæ aMÓƒIsùá’žÅRûîM÷ öooèðŠ]ËêaÆ™|´åλ÷¹Õsðu0ýJRˆ"‰×?ô¦hï7ê%u]Ê&ò’ø“¸1 ûùYLé~•—#¨,Bb2ûn§mË=EÏ‡Þ ¹nÁ4ÌX‚´³`&-˜‚,þŸ®6Ñø½Î&ÜÅïO?O1ˆî"Ä ,o?Ž?$ööÈkß¡2Œ³øÄqe¸Æ‹?€}õÒæ0&¼ÃEÈa´ü¸×qepn¿AäKü8<·ºàmÌŽPvqé;DþQg¼Ó%Ä,{r~V·íRÞ܆;ø.($ÂC„øæ,àÙ"Zèy?((J‚¢‚Ö»~ ßð{}:ï·à>îÀÃÏbA©/B={„^†x z¶‰˜­ $Ò q !BâA!¯B…±-!BâA!eÄì^ „†ˆ—=Ì“¿PK kL[D‡g¤žQÒ;org/apache/hadoop/fs/swift/auth/AuthenticationWrapper.classRMKA~f]µÝ,íË¢N]B¢È$è$v×QGlgÙëoA‡~@?*zw[ÊK^Þ/žygæóëýÀ!¶\ØXwÁFå<6rÂód1œ´u8à"ÞPò¡èið~Ä£GÕ7\LÌ·(Hß(O¥ýkÚdƒ4Δ¯L“!S©vìsÝ“ Ŷòååä¾+Ã[ÑÓdµ­=1îˆPÅ}:´ÍP‘ýñ?íïB2$wg M+]â´R 'š 5*óêÄ7àÞèIèÉ /¸=óàõ‘xä±P@9†£¹ög(ÅB|,ü¿êޤg°KïkÓ{3X±4UªÉ‰¢C§Ì(gko`ÏTXp)æ’á)~”—¡eÿ_`=ý"ïÍ$—°’’›„¶btmÿÖÔÚM¦Ä©' åTªWÙľPÂYûPK jL[D¸â¿f:org/apache/hadoop/fs/swift/auth/entities/AccessToken.classSëNAþv{/Ejå" ^ \W.â¨A ¦Q#•DþM·C;Pf›Ý­˜ßI-‰&>€e<³»”…ˆ?öœ™3çòïœýõûûO³x–F¦R¸‚i% %$0“F SIÌ*=§Ä|•^Hâ‘Òx’ÀS þ¡!lîhÈwÙ{fÔ™¬›®-duQƒ.*â.—LºfŠ–]5Xƒ™5nÔXŲÆŽc8bÇ5XÓ­\ºÂÜ1J^eˆ/ )Ü‚†H~lKCtÕªp ]E!ùËæ~™Û%V®sUÞ2Y}‹ÙBÝcÔ­ ‚¶ðïuWL“;NÉÚã’Š§«Ü];j°;?ÖbÚ9áÓ“?ë¢`Ç(ÑQs|¢{)`eŽò^ž—”sœa>é TÒµ|ˆÔĦմM¾.kÙ$L«v4ô¿iR‚}¾%AÌ®Hi¹Ì–¤žç½â&Í¥ÆšŽ±ËÌ=Ç’ó}¸ñ‚nUiÙüµm5¸­`P³½•{Ò:ÚG-ƒôfÅÕ rJ\C·†¾vFŸ7E½ÂíÓ@?ŠÊòh‹XRb™šò÷nYh:&‡‚eUŽ#$èÿ³T÷Ñ«ò.7iqÖhpI“ Û€3¦ ¢a ¿zÑó©t~± Òa˜~à.¢2]ÑH']1éi"“tšb™dÝ Ò騸!´/žÛu’qÏ8~’ß7pSÍ<üúç¶àBhð-Ü‚ ä­+ïñ‰oÐK§=ëŬ{z}¯ ƒ:©¾Tá¡P ‘v ÅP Ãá@"í@6)¦t\ä¶Ùr÷ÀDÛ0Š)ŸDMVùËH|¼)ü@ìÝ!â¹D IÅ©ìh ©\Ú7D[èÈeü³~ô˜ýD¢…ÎövNtPêëõdc¤;èmœ¾‰mLþPK jL[D{ÒUº6org/apache/hadoop/fs/swift/auth/entities/Catalog.classTmOAžë«ÀakyQ|E( ,"HÕð%‰ ÔÄlËÒn)»Íݞĥ ÆÄþ”qvïZÚ»k‚|¸ÙÝÙyæyffÛ?ý€ex<Y %mh³¬ÍÃ,¬dá‘Lì·$ʵ _nÐ/”xŠ7I™»j¯wxMPå9Ì‚ÐõZY:5B[´Zg¤N÷¥l‘—¸Çü@ê©:aBqÅ™K6’ÕæÌu(?7¹8DâéHfsnRQ#o+ Võ)APG¡ëvG9\ÔôúÚ»Ì\•,Hçöл.÷Ñ›+sÁ¶¼£ svi¥irÈ*mîQ‡ësàL©:G5Ëg¯k*Ú”šß®1µqÚÊBq.ÚÌçy[h»=l#Å0™.~¼·FìOPçËpøùdhŽBwÙíqÎÄÔ;Ђƒžª‹kò,’o™'1Úfì}Y·0VŒÞ·Sìš—£cýÝàŽôœ*ÛäºWv0âE ·`â‡8b{ÜåØËçBHE—…¯˜.VñÅÕ©ç’­ºRêÇ0òO¯jB:lÛ‘-æè~¢Èan|ïÅ¡Ç"‰¿[ˆKP°á Ø`ksr6ä!gÁÒÿ¾MüY‡ÛSøE¢$4 `ÍxJÀ~v×y?¤6û¼YõjC;‚'‚«…kzþ'XßMØ(ÚŒqNÂZÛ€q¸¬+ƒ+0Q½àH| ŸÄ€}MWÑw ®I¶qMhÔüÂHh I“eÐxŸ"ö™É4îG™ôίûF‡E{†:•ßÄÝ­X©É°Ô;R'Ñ7/5–ú±[}¤Ú©6ަ-õ6îîÀtd$' KÝÉ]˜ À¥n‰©Óú?"æS‰ú•hâÙX!é°J¬b¼tXGL£¼’€9ƒ™‡\õXïáwÿ,þPK jL[D¼ß±\lã 7org/apache/hadoop/fs/swift/auth/entities/Endpoint.class•“¹oA‡ß:qŸ9ƒûÆ ÇB¸¯@,ˆrIP1±'ögÖÚ]CHPÑDE"q!hù£ï­'±3Ìþ½y×Ì·óÆ¿ÿ|ûËC .Ç WHFI®’Œ‘\#¹N2Nrƒä&ÉD+L¶Â- "¢`AWn‰=dv™É¢=ã»B/XÐÆ ËBÎMç,H×ò’ûöÜôæBúÜ•¬¤c•êBYäƒu|c=lA‹Ë‹Â‘¸—Ï%“þž{È]ƒ´N¬¯å¢S÷rÂó±÷¢Â± );8oAó˜Sà´ç„ä·«Ë Üe eNìNž•ç™+ÈWÁf¿$< Nä·h³ Ë—¸]bǩ؋ží=‹¾Íª~ÉæÒ¾àž}]*~~\´ÈÒžì é^¢^-Éþ›%Ò¶n\]Çú& —ç5tf7çi‹44ÞpÚÓI¬˜ªßzÒÛ䦳8‡”·Ùa~Z&æÕׄ>»1)â¬{tâ|}ttbƒ›nÈÃL{z ^QpCE-Ÿqªnž š`j} Gév,˜®â¨–ù¼ðŽxTJÇg>öâ˜OcÎã)±ªg/±üÏ‘6«Õp{½‰¢t\>å:îÒÀq)Äæäé<’Møw²’°¶&! íIè€Î$t‘t“ôd(ÑKÒGÒO2íÿO£þ|î,,ñ¼»ñŸ™BŽ8DˆWÂl—²ÝÊö(›Qu½Êö)Û¯ì@`3`ѧ¡nCÏFk¡}k5(ÛŽÚwÁÔd­v¢ƒh{þi^ƒÈŠÖ|ר¼ö©æ¬ŽPõСO©¢÷±‡;ôÖªÔ´¢û ƒ÷ÃH“R4‚„¬ ¤I‘Øã„€Ð@èàA#H³R5‚ ™AšuÇØó$¤K2‚Du§FÃf¨ò {ž‡€t+#Fä…䨤Ey‰=¯B@zˆm|¬­:Èk#È1ócmÕAÞ`ÏÛŒz¬Ç m:È;#Ȱ¤My=B@zÈ #HLùh9i‰é +سÒ§@NAâ:Èšä´$®ƒ|Áž¯! ý äŒ$¡ƒ|7‚œ5ƒ$tŸØó+d ‰À¹ ç<\@›ÀÈEü]º#PK jL[DéöŽõI5org/apache/hadoop/fs/swift/auth/entities/Tenant.class•’ÍnÓ@Çg'n'-¥-…ò I ]8 ¨Ô*DCËÆÙ&›&»‘ס/Å$*$<…˜Ý¤9΃gvÇóŸùÍØ¿~ÿø `Ëîä!ec*ÆTÙôà®÷8¢C`¹ÖgŸ0Ù¥ûq$d÷1B‡ë0£X(IÀã’µ“I‹€+ÙÈ=RÄÛ2åJ£/T£‹5!y}1eã¸G¹ŒE,¸¦ d1¢y]×-ÁJ¹’Æîé“„Õòì{ÚúÕÉDH3ù¨ùrË-“UÂF/ÿ]CI'YÌØEAVO¼¿¯ÆQÈw„±0!Þ2.¼ã$CÞZ`—çRª˜™*¸…‡v !î¯ÇÆšöYx¨•¤l’ÃéÞv»REüm¤F<2ûÀ9‹ÂÆÞËC©Žd¿9  ‹!@‚|sZ€<úŸ›&°tº»7í>c¸†?‡}\pLQ<9¦ƒõØÎú¢½ç´Kx£è úlõ;¯6í Úœ nÀ2Ú`’gaÅ «°6#þΗ„øQªøœŸŠ·1Û1ÙÕÍcpN[û6ú5Ïl…µIÖ´‚9í øáàb Hæsd'd.Í€¬C& ò5õ9 [•ÀåÔ¸É4RA6Ò7â&A> æãº‘+© Ù$H;äj:H6 "PÓŸ²`Aüæ:Ü@_ÀÈM|nµàöPK jL[Dص$Fâ’3org/apache/hadoop/fs/swift/auth/entities/User.class•“[OA€Ïnï –‹â]D( Œ"^°„„1 ¢¾¡ í@;ÛìÎJŒoú‡4јøàðGÏ̶¥îN£<ìÙ™3çòsf~ýþñ¡œ…4<È@ –”x¨Ä#%+ñ$Ë)xjÍ-(VŽé;JTÔÈŽt¹¨•-ˆ Úd$\§Á< /yƒT¸'Ñ"³Ãk‚JßE³åÐñJÅqk„¶hµÎH:N‹yÄ;åG’P_ÖɶŠ[^Å89b¿ÁÅ &šŒD:ƒ{yp̪R;¥}¹br… .W-ˆ•fö|Ý9DíP… ¶é7˜ûŠ4˜*Ó©ÒÆu¹Ú·•qY瘕ü“— É%gÙÅĨ1¹Í)͘ڗð‚ãÑRôTQ¦Ð}Sã§¼Î*ºí ÝÅNÔÞ†—#Êó´9íu£—ÂÁÒØßýyßêôh%lþÿiUÜÁNYOê0yÐ yNGI Ž*i“îv¯HÎëÝewß­²g\—QúÅ‚ñm§Üd{ÜãXùšޤ’;S/éš«x·êÔ÷È1­žxŽ 4°aäî6jÂqÙ–ë´˜«î –0ÀµnWœçTÄðmZy†‘0€K0®j„Ëp%âüìÏ!ç5£óU¸Öv^Ek[YÏÎ}û,uVkŸ£Ï†Ž0Xµ#¨•ª[%¾n‰…A¶Œ 7Ì ±0ÈkôyÓ$×¹‰c±# ñ0Ⱦ$˜ÎênÃd;ÈV/P\Åz€ŽÐ·Ö(¸wºY”f¨{¦p5mDM„Q›}QK¨›1£&¨}ý>¨…j/iuWsÆñ&èŒã½ko2<Þèó©bQ׆yí³€Á빇ßý·°øPK jL[Dó~(jŽM?org/apache/hadoop/fs/swift/auth/KeystoneApiKeyCredentials.class“[OAÇÿÓ-Ýv] T.ЋнpYAAC$&DÔ‡‡íÐ.ÔÝfw+Á/ä«&RM|5ñC©g¦Š¥ø`šž9sæ\~çÌÎÏ__¿XºÌdЇRäu $1“FQ®%)fÓ˜30¹±¥x¤cQÇC†;Ž×â”!·{Äßs»Î½ª]Ž׫®“C(œ@DÊ!õÜõÜhƒ!_¸êzÕRÜcHnûÁ0¸ëzâuóÝÞòƒºÅ|‡×÷xàÊ}lLF57dXßõƒªÍÜ© »Æ+¾ß°C;nœm²Uj{â¯@¢©ÀG!fí(@ ¢/3³ã™3gή¿}ÿüÀ&d`ÂJãÖ´¹“D!ƒ8¬îj/…ûÚµ)%QN¢ÂçCÙtÏÅûm_t…«$ õ–ç÷l>äN_Ø}Þõ¼¡}ØÁ;y l>R}›:å¹¢9”¿7×¥+Uƒáe©uÈßr{ÀÝžÝV¾t{çA.ï1Ä·½®`ȶ¤+vGGá¿æer-Ïáƒ=îK}ž$ãª/i‡ÿ2´¥Ã+ñf$¥QÂå®Òà§ö`Xé Õü“‚OJåsi¸œÞ(SÅ”ò¢5‰èœå2moä;â™Ô2.Ï‘§¦›LdQ5q—L¬#OOjêéHºÂgXƒañÀŠ$¶Š&6Pc0Š–Žl7q‹aë?.Žîûo]ºZëáp%=wÚ+iq2¥]~D«'øp(Ü.ÃÆœ·|Z¹Éºõ™úCá¨3ê±F­I¿2) å¤(Nq—É^¡Ó>åc䳕°Êú1ŒJõ±”2#›£ D¶Œ4*QÅUÊXQq £%ä TGzAñ2®OÆØäõ·…Ê'¦à‰0Y ͨ`H/7&Í ª6tµ¦fü¤– ³›Ô³",EUSJ‰ˆLw>ÁÚ ‰…/ˆïŸ`!—#I¤bär©(6ÆH‘™¥ùèš…)ÍÛaÕêPK kL[D›•%2,Corg/apache/hadoop/fs/swift/auth/PasswordAuthenticationRequest.class¥SÛnÓ@=›¤¹C -i¹» mœ´%¤"P•x¨BAUêã&Þ&®R;Øú[ ÑT‰à£³¶EIZPx™™Ï9sfÖûë÷÷ŸÁÒ ÃÈá –•YÉ ¤!#‹‡Ê¯f±¦|Y3ƒJU†ù‚SÏ·w|a W:¼0l5<¿mñou„Õá¶çõ¬£À N#iñ¾ìXï'au†ôsÇuä6îÙ8柹ÕånÛÚ—¾ã¶ë3qVR;ž-ò Ç{ý“¦ð?òf—2…†×âÝî;ê'S²ãÐ/ÿ¹Ý+:¨v-.Ïý >õE iM —»rŸ„­&æa(¶…œ"šá‰Y™qƒÅàʧæÌûËJ/’̰@¦ ¢í{}¿%Þ8j+—.¦¦à:òX×QĢޫ¸Æ°8Núºïtmá3,°ƒF´U£¬c› ‰²¡¢šŽ»¸Çðâ¿nþÜ¿á/À%Mµ©4ïõ„k3lNù'7Y«×<-yI=–é}êôhi|µ?ŠRçqì :R>I>_=«®Ÿ!QÝ8Cò+¥(-X%»†ÊDabž2FÃnaDwD¤*R/áVÜÆ"¯¾ÍU¿!ùeHž“µP bB†Û¸ƒ·©:¡ª•¾iZ˜Ý"Ìã¡U %¥C!ª1]xÌõ6”~ ux޹Bz€ ‰J+d£89@nm\æ³™¥¡ÌûaÕƒ?PK jL[DäǨ_—{9org/apache/hadoop/fs/swift/auth/PasswordCredentials.class“mOÓPÇÿwë…É£ ØE1D%Ñ LIxyÙÊVíÒv’ø…|«‰ŒDßšø¡Ôsº2È(¾0Ëι÷ôÜÿùÓÞ_¿¿ý°ˆµ0Ý‹~äØäURˆc:‰"û6³IÌ¥0ƒ7 lªXT±$lz¦kËcS »u$?H£.íªQò]Ë®®Ñó†ô¼Ç­$žY¶å¯ (ùž@|éЩô–e›ÛÍãÓ}+êŽS–õ=éZ¼ƒq¿fyË[Ž[5dC–k¦Q“Çi‡žáX‡¾!›~ÍØ ëm¸fÅ´}KÖ=ÂÈå¯Â]0V_Õôw;ÌCùBTW}Þå¤áñs©wñð™‹]ÒwÚ™ý%_–ß¿‘ Uš?ÍV UršnÙ|eq÷c]ÍsM YÜÐFFà 2£Ý$/›V½bº4~~SzNÃ#,³Y×O,¿¦Ÿ¿"Ý9Ôë¦]¥ŠÇž`•ŸjÐ1!°ô£È\í™eŸHd£aÚ4¸¹¨Á] …ÐØ'óÿzœé~D¥ÚýßÜ& l^/€)úþèRÄèG VƒW!xÐd‡h·D^ï)žA| ҆ɦȺEŒÐJk'acäY`<اL…|šŠ3§ˆgO¡\HeIXà‹†^º¤T“%õö±P’WÌ)‚“Æh}a£ÃùÊçŽx"®F2NâVxx=e3ŸÒÝås:ó"Pigu¾ ëy r'$Ö ²Mgv®I‡ zäc0!`å;âûgèÉ&ZPŠ\&×B2ÛÛ(?¡*Ÿâe ©úZИ\¹D^ºD¾Ö¢ÿb5wÿpšŠ{ˆÑ†9îìþPK jL[D2ÖÇ" ,+org/apache/hadoop/fs/swift/auth/Roles.class’ÛnÓ@†g8NlÇ-¥-PŽ-”&)ÂWH ‚TÄADmEªR‰ Ø8Ûd«Ô޼Ny-@H\ð<bÆ1¤l77ÿììÎ?ûÍÚ?}ÿàž U¸SÖIî’l4Hš´ØdPŽù‰`°Ð>æ§<ò¸v²TÆýG ,Ùcàõ„ŠR9Êd3¨)‘žÊH¼Â“j&bg´¬<–±Ì¶”Ílºô°é\[Æbg|Òé>ïók’ˆx*)/6ËÙ@*í$í‡|Ä£¼—$£ðH…ê£<ÊB>Îá›d(r9}‘í䨋¦ ÜQ –çÏ‰ÐÆn«I 0vvÐ@i>Vt¦ÃûêŸÔÃÓý¿Ïᩳ™ÛIÆi$žKÖ͇¸OP>ÌÁ¼5p}ðH|’:I.ƒõÿzóÓ w»Ç"Ê`¿pÿ,j+‹nÈ£_Äzƒ¨ÃåÇ@fFé´à#+M1 'í8D«z=A0¬9ןPK kL[DpO‹EËâDorg/apache/hadoop/fs/swift/exceptions/SwiftBadRequestException.class¥’KoÓ@…ÏÄ!iMh!„G¡å!X´)e$„ØUˆ D¤À¢)ÝÂԙƃlñŒ ? VH,øü(ĽN¶*/fΜ¹¾Ÿï‘¿~ûüÀ=Üq WyYžÃJך¸.Ðxh2ã7^­Þ¨%•åÐ&÷Žs2íåËí~o`‹±T¹Šb-#›¦6s2ö>£3/Ÿ‘|®}lG½µ]ú–iÅÉô‹2ÝÓÅŽÚKÈil¤’]U>Oͺxô+#V#ks¹ï¤{gö½Ôï#{ÃØ!Õh[¿-µóO~Üôš©vN+Сæm® ŵAY…ß§¤Òj îßO,ðàˆDû'EÊA…Î+_ºI\¢O4tÇx§³ºvÔ(áЖE¤Ÿ®Y9.’»üf 4yéÐk?‹6ÿ/o­hÐÏTb¨‰ËÉÓ³.¸‰:ý¨üü©´ÎÑé5¹ì‡Ýõ;KõOéTÃ<{´·Hß& ÜŸTâ4ZU§g°@}X-’ªUê,΃Uç‰Ê´ÎaÚrE«ýIë’^?‘v§4æ3.Ó.áò”¶A;×íàà Ө,Y!Z“ë)B`©ªºòPK jL[D°Üb„NGorg/apache/hadoop/fs/swift/exceptions/SwiftConfigurationException.class¥‘ÍJ1…Oú3£µZ­¶.¬;mÅ€¸«¤(ƒ›–îÓi:i'e2c}-W>€%&i-¢E³¸É=¹÷˹äíýåÀö Èaׄ=UçJD"iTN¼öÈèˆEí$±ˆ‚æi ×–NPòDÄïÓqŸÇ]Öi¥ìIŸz,&_ˆ¹$Š íÉ8 lÂüÓ ¤œÐ¡¢j*† åO>Ÿ$BFŠvŒÐ–ÑPiÌŒvóyÙ$pÇ\)Ø·~x#¨¯püE醱œWvŒ¼ÏR¥I••…ŽLcŸß 3Eí_禿ˆ<¶®ÿ=*ÁåË6#«¿Ò¬ ˆq¤£«³–Î3zwêȳ½_Ó±`Õ]yˆu}ªÎ«´¾a)ŠØÔ ÃÚZ°îôY½»õÆÙ ™ï°#ÝT³°Ú¼l s0s*aÛZܱÝåPK kL[D|¾„]½iJorg/apache/hadoop/fs/swift/exceptions/SwiftConnectionClosedException.class­R]OA=Cën]ÑòQP$)Ý„à‹­Å—jL–4áqº;´c–™Íîý[¾@‚‰?€E¼³-ÑXbBâÃÜ;÷Ü{Ïý˜¹º¾ü `[5,y¸e=T𤆧V¯X±êbÍEÀà~lGÑÛm†zç ?ãaÊU?ŒL.Uÿ€a¦¥Ua¸2]žƒóZ*iÞ0TšÛ]†jK'„Îv¤Ÿ†§=‘ñ^*,™ŽyÚå¹´ö¬š,;:ï‡<ãñ@„žh…'EX|•'&ßb‘IUÃÈT_‰Ø­T"ißø©¹…æd˶+'¼ÐŠÁ‹ô0Å¡´Õ7ÿÍ÷ÂR1ìþŒÊœ ù™ŒE0àEÐBq™ëcó ¿›x7”i"rë~ÀðüŒ¯ëØð±‰g ïÿÇžöï@óGšÃ³L¨„¸mÍÐxhz–šÑ#ˆa¾9x@Öè_ Ëð0MÚ'kŠ´=´4’³„¼,mÀÝ©³ L}/ƒ’ôH+p°ŠGtóGa¨c®¤¥gS|¦¿nc—v~ r|jýÞ9œ]:çpoá\G%çâ(oÌio e«Œ<6§ñ PK kL[DЭŒlØDorg/apache/hadoop/fs/swift/exceptions/SwiftConnectionException.class¥‘ÍNÂ@…ÏðWEEA!Áˆ+,Æ&êCLˆ&&Ä „ýPÒñµ\‘¸ð|(ãR‰A\»8÷Îí9_oÛ÷×7(&GNK^ËC†Äµt¤_gˆVN; ±†ê †LS:âa:î ·Í»#šd›Êæ£w¥>‡Ã˜?”ÃMS¹‹O¸=Ö÷”šX}Ïòf²ï[âÙ_*dzZzÐPŽ#l=¸ýºScÈUšü‰[#î ¬–ïJgPÓûcáy|,ðÃÀ`®‰}›´‡®šéUVÜæSH¹µ†dKM][ÜIýj¥ß–=×á0´¤µì0Ôÿ÷®þXÆPFŒ~¦¾¢`z'Ò :]ReTãæì…š6I“T…ŠÔ©… [Ô!¤C@œÚ›0«sDV %Ê„üÂt·Lðpú,!ëž–‹R5ÌêÙÑUX™¬'ìxa[ÂŒ¦»]dÉͰ¤÷?PK jL[Dö  f€ :org/apache/hadoop/fs/swift/exceptions/SwiftException.class‘ËJÃP†ÿÓ[´Þ/õ^QA©Q< âFq# BÑEK÷§ilŽÔœ’¤êk¹*¸ð|(q&¢µÝ˜ÀÌœáŸoþ“¼¼¾8D19,qXæ°baÕš@îLû::H—vk™ Óp¦ÊÚwo:u7¨ªz‹:³eã¨VMšÏI3y:8)› )U[9ž+=Õ0¦-ïB>é»HºÏŽÛŽ´ñCYáÆå×ùT P*ß«G%[ÊoÊJh¿yÊ.¬7 U3^ûG `ûÑ©zybƒ1+ë¨NH¤Â@…ÀRièh¾b:ã^i¾ëÜo÷<4 #&9Ìp˜8þÏ× ± mäõíw›ÈÒßâ'ÁÛ(ŽÒ鈲 œµ»/T¤§˜§ I¼Ž1ªÆ{"Ê”0™ÎIÉÚœ½×EªŸÀk·bÂBO•¸šÂt¼œ.œ°®‘¦°ì½ý.Òý°mÚ‰a=Ù7ÌJ`\ÍbŽÔŒd1ÓOÝ%‹ö‹Ìb‹…xfáPK kL[D¼ÌúkGorg/apache/hadoop/fs/swift/exceptions/SwiftInternalStateException.class¥‘_KÂPƟ㟭̲,Í¢À À4:ÝBH0º™xœÇíÄÜd›Ù×êJè¢ЇŠÎ™&aD<{Ï{ž÷·çlï¯o®p˜A%E%û:J:´ቨAP¨ì‰Q—y65£@xvý¼Cjú=N3„ÇFƒ.Ú¬ëÊNÞð-ævX ÔzÖLEŽ š†Ø” ™åpê°žïi?¤áXô#ÊŸ->Œ„ï…ÔT–ñÀc®±ˆß}mÖ ôCfÇïú‘ º$ñ·NÛ ü±J#m±Q(I…¥‚ReåhÆôGÅï…:`ù—È—Š…]INIžàöߟ‚àúŒùN’¿Z]IKêš\57 UkY%°.5wä̱¬âÔ… dcІMlI†båf¬–d'åS¯Ö.&H,ÂÊÒzÃÊSÛ¦Ï`ªÚÆŽt+l~YÄä"õTF<[Q±TÄÝxfïPK kL[DÛRŸ†« Iorg/apache/hadoop/fs/swift/exceptions/SwiftInvalidResponseException.class­UkSU~N.»!l¹„R¤EÀKØk›J5”`AÔJ¸ØZm7É’,Mvw² Úñ‹ãÏðøÅuFÒQgüþ¿ø½£¾çl’H±Î80ç¼÷÷yÏyöä÷¿~ù Àu8Q\ÁJ.c•/òe-‚,ßïôâ#äø².c#‚MGñ ¶¢èE>JÛ\Ú‰`—+{|Á]nºÁg|¿ÏÃ?à ®<ñP†Îu=Ý;t—í’ÁÀ² =¶cÔuÏ´-†Xî@?Òµªn•µ¼W7­òtÞ.CâeéHÜîòqÞ?Zà—NVªVÙ«ÐQ%²hª=Hª=HJÀ‰Ã iåMR—·ÉÆ5²Ž#ð7µ ˸.#%þoÈ„„ɘ{ÆÍ)*H/AsrvÞ ¬þ„Àí#—„qIàSü€&*†÷h–³ÉÁ'§’W»&ÏÓðg“C§“ïtM^èÚY:¼Ñ59ÓNži&cÑÓ©{©Ávêb;õO:o~ _ÿŠÞ»ƒø†·o@yørIµ¾ú÷’±±‘>èë1_§Sòõ¡–jà"éÃ-=ÐÀ¥=¿èÈwJÆ^iy$!40úD@ãÓè£UÇ Äê"¡D$2è†÷é”ËXF9zró8À}z”÷QEþŽàˆ1/ù£4Ç¢è÷q›½æ¯Ïp$è}‡ôÁ?PK kL[D±Q–Iorg/apache/hadoop/fs/swift/exceptions/SwiftJsonMarshallingException.class­‘ÏN1ƧüÙUDP€eœ$F‰‰‰{˜v¾Îüæ›ìÛûË+\À^2°c® e*œ+‹¤I |â=°GF#´“(ÓLK8¢'b~?õ¹ê²~„JÉ“>‹zL “/ÄL M íIP6f~ÈiÈRŽéPS=Äò'Ÿ!cM;F¸Õ2¾cJ‡,Špjûó¹AÀq­Y`§ýpG ¶Âó¥*95¾ì"YŸM4’Ê++ä:r¢|~-ÌÇ¿:;7„òlß×0欺•°Ž·Ê¼ õ Kq ›È0¬Â‚uƒ3ÒxºµúÙ Rßa‡ØTµ°ê¼l s0s+–µ¸m»KPK kL[Dçw’íïForg/apache/hadoop/fs/swift/exceptions/SwiftNotDirectoryException.class¥SÛnÓ@œÓ81&…д”Ò¤…¶8I[#ÄUåòU•‚"ñ¸u¶Î¢Ô¶l‡Ë_‰ÀG!ήMA4„xðzö\fÎÎÚß|ùàö˸mÁº‰†M ØTèŽZ]ÁÝ l™Ø6±ÃPŒx:dX놱ïòˆ{Cáù #÷4q)ùˆ¡ôX2}ÂÐpæÔµúD÷4†Å® ÄÑøìDÄ/ùɈ"µnèñQŸÇRíó`1Ê„áp6iòVž¦®xç‰(•a¸=8 Óg2^ÆïŸÿÊÑŒ÷çÖ}ÍßpwÄßí¥± |=¬y&’„ûzº ”öEz¬ÝYwZsý±zá8öÄ ©Õ¼|Ì}¥ÂUÔVþ=ËÑ@Ä*}Ï„c£…¶vUdϦ¾« ÿkÃàø£Íp”k«—:ÁPNÃì$ KäÙ SK<ŠD0`Øsf\Ê…Pî5ÖgÕ÷± õMt‡ÊBá*i½F»Úè]nwjlŠÂGÂ\§ÕÒñ&LúEj„–³:ÜÀ ‘âcš­ž³ b3TWû+ŒWS;,LPÚ¥gsªd>£ò[¤ªË·°Šm4°£…62Šs¡f.¤Ð2V¨“á&ud’®XhBåÃ9qI[šÐÎ rB†[ºjí'PK kL[DÝúø”P–Iorg/apache/hadoop/fs/swift/exceptions/SwiftOperationFailedException.class­‘ÏN1ƧüÙUDPæ Óy‰»Ÿ×-ö”+Å|óÚwœ ž¿(ƒ ‘síË ’÷ØL!©º±‚@¡/g‰Ç{BÏÑøÕÙ¥&!–%—Àõ(ë6¨C?T¯ í £Yó î–Ó\y6÷[ F=ÆÊØÆSmY…úŽ¡XP„]dhViźÅ7²¸ÛNób™ï°:65 ìtY¶†Ù+˜>•aÏXÜ7Ý•PK kL[DÚݤcN‡Dorg/apache/hadoop/fs/swift/exceptions/SwiftPathExistsException.class¥‘ÏN1ƧüÙUDP<¨o Æ&Ɔ˜LL6ƽ,e·fÙ’m^ˉÀ‡2N £x0ö0í|ùõ›ôíýå.`/Ø1a×…² Εˆ…n(Ÿxì‰ÑˆÅmëDÄAã´K s#ûœ@Ñ1¿{<é°^„JÉ“>‹º,&_ˆ EàÚ“I@Ùˆù!§!ëK9¢EÕD 4åSŸ´±¢m#<0¶¦BiÕú¼ip‡\)؇~#P[a÷‹Ò 91–ì YŸ’Ê++äÚrœøüV˜3unšóÇ„æÿ†$pùÀ² Ž!?hV ˆ±ƒÑŬ‰y w§VŸy¶÷ksVÝÇÊXÇSe^…ú†¥8‡MdVaÁºÃ7Ò¸»µúÙ RßaGØTµ°ê¼l s0s*–µ¸m»KPK kL[D(œi¿Jorg/apache/hadoop/fs/swift/exceptions/SwiftThrottledRequestException.class­RKK1œ´«Õµ¶¾Å£èÁw@¼U¼¨ÅBõЪWM·±Ùݬ»Ùêßò$xðø£Ä/ëEH曄™L’ǧû›˜uQÄT Ó%Ì0 n«H™†³¥æ¥è ˆ¨ÇÛ&QQ¯öIÃOZZS'=.báù’{: u”rß˜Ø ”Œ ? x(¯»µåSgWw%Cµ©"y”…™‹N@ÌDS{"8‰²õ+é_¥ õ¾èjó‹”§×êÂpyãÉØ(k۶ıŸhcÙmÉ«L¦fÿm½ÆP ešŠ^n÷%ðŽe"ì^†b–(†Êç¬tUaž…aå÷¹ܶÎOÖ• µðó17¬cöþ#7ÃîdQ_ŠDÒ˜8ù®‚yèÏØÆ¨Óñh¤êœÐÍîÊêÚúœsvKU%ËÑ ,`‹"´õ²“j7Wr1‚2éX4J¨£ ª¹“‹1Œ“½V®8ù PK kL[D§ˆyñLorg/apache/hadoop/fs/swift/exceptions/SwiftUnsupportedFeatureException.class­QMKÄ0}Yk«uuuAOzð øÄÛÊ^Äâ¡x©î=ÛfÛH·)Iªþ-O‚€?JœÖ‹L`2óæÍ›y}{~p‚aˆ ¬Ø`ðÏT¥Ü˜as?¾w‚—¢Êy〉òÑÁ„Á;×™dĪ’WÍ|*͵˜–„ cŠr"ŒjëOÐs…² —±69µH É ‘i]ó™åö^Í—©¬Ò•åI ÜT¶©kmœÌ")\cäÅcÄÌ¥µ"ïþ0È&º1©ŒT»}ï7½ãV¡‹ ÑÿXd8ýƒÐ÷vÑ£h£KŽ(úT ïÑë==RÖC@1ìÐmâí`‰²­–©ƒ.[A¿ÓZífÖÞPK kL[DcApD 0org/apache/hadoop/fs/swift/http/CopyMethod.classOOÂ0ÆŸnˆ(ŠÀUn‚ÿšxÅp!xšh‚!ñXFÙj¶vÙŠ†o¥'~?”±+DMô`oû>}úëÓ¾¼¾8G«‚êeì»h¸h8B Ý'hv¼{öÀhÄd@Ç:2èu'¥šq‚š'$-â)OoÙ42JÝS>‹&,y¿K:Á‰§Ò€²„ù!§!›)•ÐyF³G1×4Ô:¡•,¯¸Õ¬GP\¤"þ @à\XlÐN÷/Ce¬©Ï/E~}í{–{«Ø€“¿áúæŽàèÿ¡z?̾Šc%3kò#Á¥¦±µet(µÐË¡ô#•™@«Óh£hþ9<ƒ©®éú¦/˜Ù9:~y¶ûeS+V=0Î66ͪµr}ËRT±m;¨­Y§¶ŠõÂÓȱҡ…TWÛkÁ®uí}PK kL[DOC ¢ 4org/apache/hadoop/fs/swift/http/ExceptionDiags.class•ViWW~.I˜*à‚ n$l±ÚjE¡†IÀ½t C 3éd".Ý÷}³_üýjÛsÔ¶§=öK?ô7u{î$†Ûs’;÷¾÷¹ïòÜ÷}gþøû§_Æ7´à‚‚‹Ôà‚.ùqY>¯(¸êÇK ¦ýxYæÇ5i?füÐåþ¬™öbN†×¥¦yY l…©À ‡WØìD^SGhÁ~,nÀMÜRp;€;xUÀKœPc–‰h9-=§GÒÖ‚eæ#Y+“1ÌL$feŽ 4'&¦ÇCÓƒC©þÑXr:94$л®ÝÐ"YÀ¤cOlýÏ;šéLiÙ‚îÇkÁÉøÙxâ||z$‘LùñºÀ†‘þÁDb|úüèÙQ?Þ¨í5LÃé£O¡ð”€wÀšÑ6Æ S®évJ»–Õ¥I+­e§4Ûë’ÐëÌyC•Ìi3–•‹Ìæ#ùEcÖ‰Ì9N.2t3­çÃ2 -“—Î.ÚZ®,­i-‰aEFå³ÇÃU¥tnFÏ;uVN·µ¢•:}Éâ–5ŽÕjé"À{ÓÖgÉËB>C“Ž–žÓrnä¼tnäuRp°Šãáj×ãÓMǾEjeäç gnLÏçµ Uô…ªº²¦ÞÕ‘¦ËØSq‚®gõ´qsÂ.HÂ)Çí¢Rs¶µ(’{ôÄ—Îj·o l®À dµ¼¼±­ËàVîI[ßlogUƒ}TéI¥8†W[ókÊ­8ÆZ—42¦æl:p¤7­ÊL_ˆªÐèšõ÷¦³¥Ô$­‚Ö‡ NÓò\í‘ǃèAD`ÛJU§ FvF·ƒxo‘A5ˆ·ñŽ@‹‹3u—“|T$úæ’ˆ‹ }¶×gv—ñ“æ¼i-š#V¾ò̆ 1¯¹ŒNZéyÝI ºU¨Ä×/Û¨<·&(ÑSÖ õË6xÙê¬F6fTËTË5Ue ï2ÐãAœB8ª[¶º`Ùº:£;<–WYQU••Dy£§ØzdŸ(5ˆ ÞÃû,‡—/°i%ËA|€—É×®“Bñ1>áUÉ— >ÅgO¶–'Fûq@ mÒ”@Õ±TY“KªÖ¬ê0ÁUöè¨j8꜖WMK ý «é¥ôVðy_àKžÿ×|Ä׸+°}-™æ¡Bñêíëó);W.§›Ì§îÿÔ“JÉ+KÁ±Š"¾pBU›×2Eö×טÑÒu6…V8 °Ä"‹8ty,üÔ†¶{]+ÇÔGÝ·bš¥Ý¾ÌF•0J"ò.;ÀÆòè UmGá5úhëú/uvëEÍfÉuT¡´º!¾–¬¯tX“áòåRKZ]#GBUØ|ÊçÚøÁÒÂ%/vÈ ál¢!ÎkænGź“뮊u7|œ³Ur}!¾s!Ïp¬u…ûø=‹Á³|Öá9-'ˆ®¥ìîϨ¹øžÎð6ú8tðÿµqõvýåþ¨÷[tpU'W¹ÚÅÕ¹ ÊUWõ÷ 46D½%}ÝRßF]°‰ú¢¾ïïh+í¶HK›9´x¡±h2êëjñ=BÓ}ºåqC‰¢Á·ã8?úN¿>²t’œÂ1ôsuCl“1 bÃ00 /²5ÅÜðûŠ!Û¸³ç]>åìê­¡æ;ÔÝN{;ù ÙË™—Z:h«´õÙ ß_+8É_}}àOð=yÙ¡K÷ÐG=Rgs)¸Ææ%ﯸ–ñŠki¦÷Ò¯€lôDÉki¢UeS<¿e¬Ëû ¶^ô4a[ò¶Ç»¹l¹èéäbÇc쌒<Âv=ÆnþîÇ¿Gëö”ˆoT‹ntÿˆ6:îÓ®Þf´^׫V(Ò¥èÏ%fåeî^aÂ\E Óe›‰"Ã"êq#<„I6Ê™ÇÕ ù„;;árìÃ9ÞÄYÆ£T%ª¨c¬Œ»©)Q ¢$×[ þ¡I‚q—ðs &tILÁÉròïu)$yMØûû~@«¬QAõ~ŽSîÉóÿPK kL[DŒfÄŒ%5org/apache/hadoop/fs/swift/http/HttpBodyContent.class¥’ÙJ1†ÿtÕ:Z­ûn`[Á ·ŠˆQ) */Ói줴3eš*>”  ^ø>”x’J]ðB‘a’sNòýg!/¯OÏÖ1›B㽈aÂ,“IL%1ÍÐï¾–¾.J¿ª=vÀЧüf[ŸèPŠÃf1«\4…ëIî‰J4ùE‹·®Ô…æžÖM¾GËþr¦´w,ëR´äCbSùJo1lgÿ#t+1Ä AE2¤‹Ê—‡íFY†§¢\§H¦¸¢^¡2þ{0¦=ÕbXûUÚ r]茂ЬJ]ø:˜h6G£ ƒýÏÓÙÊæþ7ŸÔIÐ]¹«LÉ#ßJY­‰Ká … $ô —ÿ±!êÇèðºð«ü¨\“®¶ý”0OO!Nï#‚ŒÑ'+cRЕ†>ZòÎÉÑžÎ?‚åçɯ< zgÑ~‹™c#·@ø"°D?0G aô Ö2òÌZ&a„ì!¢;i¸õxþ‘›®x—­ Ó¹ð.È0Œ‘.̺pôö¼ò#¢-¨I@TúP|[ÛÚ—¶Ö¶?ôŸ°µßsg3bHýôã‡{ïÜ{ιçû=»ñ•ÿ<ý€ ñ÷(ÚðE·Ì@ ¾$×eøŠ ·V᫸MÇíîˆÂÀAÙ<¤ãkîŒÂÄÁ*Ü…¯Ëæ7t|SÇ·¢ˆáÛ:¾Å\±ù]™ï–áîRò°ïÉ|Ÿ¨}ßÀý~ ×< ߊÁ‡dø¡¸õ#?6ðÃQÔâaQ{DöjàQ™èxÌÀã~fàhÇð„ì>ià)¹nÄÀ¨1Ç œ+ž6pÒÀ3²|ÖÀsN‰ØÏ üBÇ/£øž×ñ‚†PjÛ¬”›ïKÚƒvO¿“ìqÜ\!™uûú2¹¾dÊí[KÁR>£aVj}ƒÌ9ÅäŽÎvnW8Å~7­!1•‰þbq°'›qrÅäf.;”,µŒ¼“uì‚C=m·3mí6*”dËÈä¶óŽ= ažw_ÆM¶çKEo›æôPº˜/õ3.…ížë}Ù¬MŸ7îëq匲&u n®-ëª c1Ú#B±.“Ë›5¬Ÿ‰ð££ª»FC¸ÍM;f§29gki ÛÉ_mwg¹Ôí±³×ØùŒ|—75ç €Û‚N‡‹ý™‚†uÁûûí´ë&{ ÉÂP¦·¨üP¸Ù™)öwzäÒÊLEM‡=¨ndàuœÖñ"“œ)¬!êßÈ«"=B$z9B lüÄMÉt#Yéq­¡ÂÙ§ã%ªl§%—î$OywHCóY-N¢¡î,ä„2.›á:y[¶x]äÙê;ýÌšAÝLÖ£š ÚI*½ãÁ|ßìÆm›‚vµvֱǯmmo­›H®m¥¾ÛÛË1ëätÑ›ÉÙÙÌ´l]‚†¹ñº©rËXÇdñ²+ºÝ-å{œMñiéÙC·J̘Ø©†RŽ@sNÚÄ'Ñib®0±í&6`£Ž—M¼‚W5œ3EHÔ‚$€XzÍÄ6\¥aù„\{6ëôÙÙ–|©Ëjçn-e³V­WÛµÖ ·¹vò“ÇÄëxÃD¶Ž_zf|4,ßmÝ_tZòy{¥øòk &ÕZÊdÓNžùÖç0ˆ…A&¥Óê¦÷·<Åx•ÉY~Ý»y«ÑÄ›ø ¿5ñ9ÜlbØt•Èñ:zM\‰-äË{ÄBÎQ}Ã*ºƒÚdY&~‡·L¤°ÅÄïA¦ÿ CпÐf¥Kâ¨UÎè&ѵ9n˜¦F¿Â’Ûe£Í¿ÝëG¾õrŒrr±‹Af“Ÿ¾DÞË¥“Ä{ÕÙT ÌîÌOË¿ÑòÊ1¯ÓPã-¬F›q•n«ˆ”v«ã&þ„?‹å;†ìn°˜ð –pû6Ë€¢ã)/æª3P« CQ^µÊʹEKQŸ¶óR›ÙýVc¹bîZ¥A‰Þ„ ïà/g°;ÔOˆer…f6î€F^ÞÒ?{ÑXCnþzÑâÕ£ã‘J_.8þJ~¬àCs¹†¦ÿ¿»êx×Äßð^°&é|ð¡ßõ9£’¯–†éu°ó¦¡ .SØàt—ú6æD!­z;ò¼)š­4øg)3žÄëÎòêFØE·ÒdGâSÿÐV¹´×N’ßÖ½‡•0­|$-ˆ&½Ae½Ô4)d»ÛÍ'žž‰’£¡i^*f²^‘îàŠ/bõ L=fŒÓÉš`—,øOoYoŸÎóðÏMb°ÒÞ=öùÓ{“-ýo?Á ªà_¼§-Ë×¼þßtXÆŸ˜mü]l`¡¼%\Åäuá\! çVéžjN©¹BÚ¾’ã[¢¾ù$q®‚Æöu5¿wZ=Âü(&Æ %b#%êGN4Œ ’x•' k¸³OÂèŠÍCtamU /bæI˜]±™c˜u*Ñp³5Œ zõ['ë =cŒáîðkn׿ſbAã(ÎEMãž:FOšéýÌA×ЋK0“ãzÔ‚(QÎ&ƹD¹šRe3±m&š-Ä–"ºÝÄV ®ÔJP+Å_ã»ÐEkDÄÓkµ¦4µ®*p× òÒñ)ŸÖñh\~À“ÀV…Z~¶:¥Ã®E7Ã44‡ÿøàxâR^'Æg$b‹´1,ÞyL±-x¢œA!ƒ_â£é –=›0·\ž¿rDji.“C‰§°ä0n•p„&QÏå8¡ç‘PJ…Éil)×õüGr +DNc1§e‡aÊ× ,×à->¦Õ§•i+Â͉ð–ȰlÉ‘¦ðÄWMøTè}“§î&úunV³±VTs´¡IJÉzô‘ñ~Æ(C™ëIãÑíåX !Fa- áÎwp%ÌX*f‡ü˜¢NÅÖ!Zºª³*T®ŸoÌìБ™ ¯B~Lp%ÑxŒ†$¡×$ê$ÃHn Pv~™¦ØÊ@2*¾˜Ð‘ÆŽ†£êR¡OÊóøþi¦ãNîÞF'ngBdBbÂÞå'_ê‘-'ßÈÂö€¬!®2W2¥êùœŸH;ˉ´È‹¼ Í‹Ryµ]\†Æ?uÌ'~–ºçnÎ÷°%¤×¢òÝ×)~*fUùLÕò¯”¯»‘$=W²ôë¼4HIùÕZï1%ÕKµRW0š>/1Åóýàîƒ,݇”óiu&jT!Š?+þTB“R:£’öŽ{¤Í§eAµ[< ­ó|jX7ŒË:& `i š«ècƒç£—ô6;±dƒ¸¼è>vÉáDlµªÌE’È•älÕÜì£if³¦WðäQž=ɳ#Dù–àq\†£X˽fþ5¿O°¡<Å64ÂÔ%Òã~ÄPÇï<ïYϲߥ"NDe*Ùx÷² 4ÅÆJiAsT“ѤûßG-Ç$lã{¶Q=‰­R™­]Ü«äÉúÅKŽã‚ñ(òk^ÆŽ¦ÈmË&ÑVñx«‰xĉЦˆþ‹(CCÇ>î5•šˆ0:/<™Òyaá4ʨ×S™¯œÄé³ôòäcÌÇ«”zžú±Ÿf£‰<¾J—Ùº_a ¿Æ\xß`“ySqº™µ²ˆ}Z8Pj©âTä¿Æà¶M­ö‘Ý µÚÏUˆwîQŒ‡ã+é"Ãÿf¨Ñ„U‡¿QóÈ~–6'Àw˜í)]î,cåzéJÄ.!ÑUD/Vh½4ÐŽ×x¬ÀeF<k ˆ­õÄêO`]DŽոˆ‰ã¸Pøm–Ó;ÜÿOÞeÛyüÓ/µ‹ jÓPt»¥F«[ùv­à ÿà*ƒtÉä¸7—¼iöÖ ûúØ'¼õ2®éàQÿI«Têÿ ô›dùRŸ÷¹Z®Nøžƒ–ãh}•Ò¹´€…U¿ ¬ø/PK kL[Djì–çI 8org/apache/hadoop/fs/swift/http/RestClientBindings.class¥WûWÿNv73»^ K®Ê’]Kk )¥ !”Ø<Ðð²…Éîd3°ÙÙÎÌñQm­mµ­öe ´µÕZµÚ‡ ªh«øÖ¢õÕú~üàÀ§zÎÝÙÙÍîñãç“ܹ÷œsÏûq÷ÜÛ'NX‰†Äm2>BnSðQþÞÎË2>¦àNá.Ü‚Œ{|‚Ÿ ¡÷òî¾îǧøøi>>À˃¼<݇yy$HËgð(Sp9Rp˜ÏË8ÂB{”¿ŸUð9O(x’ï}^ÁS žVð_TðŒ‚/)xVÁ—™ô+ ¾ªà9_SðuÏ+xAÁ‹ ^Rð /óýqÇL„p'B8‰oòîëvZÁ¦xEÆ·Bø6ÎJðõôß !ÚcZé„–Ó’#z"iŽŽšY;‘1Ói#›Nô˜ék%LïÚÕÝ7ØÑÓ½~W_Go—„pÏmŸ–ÈhD2àXDITÓ:馣eA-“×%Ô¯1²†³–äÄ–Jðwš)‚Îè1²z_~tH·¶hC™™I-3¨YŸ] ß1l W•+7¢¥L3—¶ö~cØIŒ8N.±Y·ÎŒ¡guF6EšØ¤Ê¼¡¼‘I 0U·Ð)©o²ôa〄ËcÕº/¯eŽlëÖ>#Iº4Ù}ZÆHõi£tÚZ“ÁEÔÔ$õœc°[…Fä¦a#·4†u‘$Q1m'+„ÌÑ8––dZG#‡YÙ’AÿÝäêGKîíÕr®·BòÙbWÁ¬î$nÚÜ]Û:_Þ"^Ó'SR,\¹ÛYªŒïP8†È¹z*X×0;IÖ%&™XT ï™Ä&ËÌé–cè¦À°M\ˆ;_¢Ð\;¢gGI&‹Þ¡d˹ñ äHyanm¡2¾+ãUªAa–’4scëL3#!»4éÕά-¨šnWÌ,oÔÇH¶PTlëSú0 ÅvýQtlì».v©¬kd$ÉòíÌÍœ#˜ûÙX’ké·æ K§púö1\Y“̸•0óVRß`pR5V—Vœå¨Ø Zc¥Ìu\s‹YÃv\¤~Ü-¡¸Š×ð=ßÇ9 ÿw©HXß]¨È¨¸-ÖMtÙR{Y[é8𷍑›ÖhÔK–¸WØ3K&ôíÑ“%…Šà‡lâ(¤øñ›1 â§øeŠŸãœŠ›°‰!¿Pñ:Ϋø%~¥âf†õóÒ‰õ2ÞPñkœ§öÁî´ÇlG²#~#aaÔ°£y›œÆj•LŒRÄl"ú-~G= Vð%D<ïtmìîìj垬â÷x“‚â!;ûû¶tt÷umn-´ì`\Ë;#ñ¼•)P¦âȯ‘@Ϧr¦‘uâÅ¢Z8ùB>Ï“S ­«Ä¼“ãZÎÙ8£ÄÉ…ã9Ͷ÷›VªünÆ®}‹8z–FK9ƒ"DŽ[zZ¤E W„„â<$â9Ó"°‡.6ˆƒížfO")Båx.?”1’å £ì·ñ Í0Ž`«¶_³ôò@LÆPóQñüQÆŸ¸þ¬â/œ0åʈõ¶È‰Q-›ÒÓ«ÈS¤%ˆƒ¼„•ÿûl”ñ7Ç?&å~¡|%¨å½]Â’Ké‡Ô_´\NçáÐzI“Õí<î³(yN¬æªçÂÕ(«k°ÞÙSY½µ']¤–Z܉É%úþá)(ÈA;?d»Îuw×~&¤ug£G‘؆Ý5šsÆÄCh‡:¢Q-/(—Ü9¢YÔ›uz­\Ë„‹.þ0£ñmØëõ¡|º+ËÏ€ÊLé”R)¾^áâ=òQÍ!·÷ú«jøéâ!wYM!€L ‘÷xþêZ–}\khÕ¬Ø#ŽÄvÔ~ؤytúé8*aéÅ=¸^:Te”htIxôÊIÉPšü¶«õ>Æô¬Ò‹:ÌçÎO»&ta$Ü   TÚÓФõ½IÐW¢o ù¤É´Ö àrôЪЋ>úyª¸—×5ãæœDÝöcð…ýãĈþÇQÿ|¯kÊxÍÁûH% ÄÃÌåu=ý¼a\ä$äíaÅ Áí¾æ „Ž¡¡’YG³¶fAžƒDÅÌ’ðÑhl«ã˜¶fÁQ(͘~¦Ù·`3˜O° ÃOk?ÉÞ„Òj&éìç®»¬e‚b+‰ÛVPáfJØN{?Ád!z‡+z¥ëP…ü0s³*UßV¦º‚x?}Ë9ÝìqtôqˆªŒXàß=ŽpoË8f†ÚÂè9‡\dKɸEÄHѪ“qÃdÄæÁ@ {ÑJ‰Dx†Æ\Cç‘ñ¤:aò4øØä h”q‹¿Bß),TZnýËwawSÝ¿è©.eN½-¯"Ħ’…-ÂBòj-‘6“sÛ/¢ñ8æI8„m‹ÉØTHÆùa y )PHËãxGý]H‹Úê›êÃïl¢¤n¥Ý»Zy³Â½..„nö2µd2r©‡¼¼¹ÌCƪ‘Ë=d³¯ ¹ÂC¶0Á8ZAö?5iÜ#MT#ßí!¯¨F®ôWV#¯òï©F^í!¯!äªÉÈÕ²‘Mõ¥l¼…2 ¸É”‰wRÝÞM=êÊÀ{± ÷QúÜO™÷%ÁƒØƒ‡`ãa:=‚çð(Îá1z)”Â8,%ð¸Ôƒ#’£Ò]xBz OJ¯ài‘\ iM$Ðl‘NSp6Ï—ž¥jØ@:Eé&g¸ˤ#TH³ÝRiª’z‘õÃPÖsÆ¿úþþqÈ2 {<˜øÛ ¿€eh+è0] œ®¼‘„ø ç6Ä1Ò±žû[óŠ¥èµWZZ# ¬á,-uê颀_¦ú¡&€c k …ñ;SŒÞ±yub—nàÝ­T›~â ’Ó-rCˆßt®;Ý.¤r6¯h)J÷•I?A·OR‹9UÖRT! b·Ï•®b¿×R|H$=õÝöq–˜°ÚíÍT×µQÅ’@Ϊµmˆÿ.k ¼†E<Š~=%›[,4#δ´6‘{ÚKÊúßë´ž¿Ýgã ,¤ßA«ð&›·Ê\ÕÝsU»çªvÏUív•ŒÕø€È˜mbòÔ]Àl å’²P*ø 7d †4ïf£cë^B#+)•õÆfZ?$búáÿPK jL[DmÀªŽ%z<org/apache/hadoop/fs/swift/http/SwiftProtocolConstants.class—ù{gÇgã$Þ8NÝ8i7 q ¤ ‰†P’6-]Ëk[¬U¥•c—c»–ÖöÆÊ®ºZÅq9ZŽ–Ê]î³ÜB!)-”rC¹ïû¾Ëùððð}÷ÉüDüDÚÏÌ;óÎñ¾³öCÿ¹ï":Dÿ º«ŸÞ3@ëè.™Þ+¾ß7@杻§n¢Mô¡~º»Ÿ>,ÑДªŒ«C©éSÆquN¢áâIó´™kšÎb®ê{¶³x•D[ò®ÓöMÇŸ1›K¦À²z¢0¡Sº^6ÊZE—H*ôí½C¢ SU5ÐA.Ý/Ñ`´YE)Mª2}›E’qµªJŠ^ÐJ2•h{$/*UÝ˜ÖÆ u\¦ItQ¤Ék%]-éFQ-MêS2Ý“Ūh›K´/ (Ѫ ­2­èFYÑuµ‚]?ßUµ2SÈ«F^Ñ•¢6if2“hW¯*_ÔjãÆD¡¨Ve:/ÑîÞÚØuj^7ªºVA÷J´u6–M+¥Â2–é“íœ V %-Èkµ´÷It1׎ÍéjÕ¨UE)PNyÖ(©'?ŸEÏúTEÓõ¢:n>zDýßð/äš&63*êõ5TÃ‡Ž ýÝð2Qóü4jV*N¦\Q' ³2= üDSBˆh”¡¦U­»Ï¤vZþ¸ª§š‘Á¸:¡Ôн:ìþð(JÔU¯ÌÅi6qçƒóQTædús×­ƒ;¤ʱ"öªn@á?/ÑŽ$‚n%ln¥d¯²RÑ A^¡áÐÓnÃî}4%vŽ Ù~Q¢Kºm¹^Äz-R"ú‰û7©ŠÔ¿”\žT*Ñæq-i‰L_Æu‚ä\ÈôÜ-! nq­R”é«hX"QKãe­€·ó!‰„×DÛ~-²›–”ilòõHRVªÕZîѶɅ—é›]‹ª‘ì[‘ëŠ:\æoGkÊmvΘÒÄiýN—,´û.J)dhN0 å„".Î÷"J¹€ñ$Ó÷#[„šÈ~­)ׯЅ¼L?Ä1é=ÇZY­èXû#~’£{•j,ѶP.UüeŠ5ý©˜á™ jÊL†˜öT”—)ž(ãJ3å/§a¦LõË$¤Lû+‰Fzg0Sÿ:ñv‰©~“¨Âb2Õo“Dµ7ºCúD{B}wÙ’ß'q¥Ç©ÿУî ûÈ9ȶ‚ù!î}xÁ‚µ2ý çS×Êfêx'üg%¸íID2ýŽ-ÙÔw²Šẖiuåmï6, ¢íX¥Î©yËÓÍù¦%Þ¡nÝlΘž-8®÷—ì¶DGŠ®·˜3[f}ÉÊ-™ ×måÚ¹öнàç–|¿•«ŠÇ²çúnÝmÆ/Þ6ÞÂò±z3 ` êv¼º5a Ï»þ·EV¼ÈéMôf$ÖûRëØÍ†åÁX B B \æ›¶åø£ƒô0ýµŸþ6H§ Ò?ÅG™%ºâÿËÓ-C›?iÕ}ٌ̙Òñ—2º»l9m¨@i‰¡gµ}Û1}Û…tKÑlû™i·a/ØV}…O!fŠ–³è/…¿¡‚Ø|~Õ·ÚWïodöcù† 4”­Þt; Q5Ä2èdÚ¾ëYâýœ CÊL›Ž½€ÝÅË9#üšh°kónÇñÅ›™©ÆÄf™Z[„&ÏfJÖJ`./´³ÑÎ[ãÇlÛòNÛu+ î:|g}û”åv°|GºÜ­/[L³{‘gùÞ*(kïZŸKhƒß´² «i®âp&+æ›ðݶo¶xx-ÓóCÙöDæY7uP(@š^FY´Äv[²ÑˆÍˆÁ†»MÆj$Ù”5ÑÝlÇkb´„Ï–Óh¹¶ƒ½#Q­3…¿MÙ¼;æ)K<·Ìv{ÅõPÔ¬8VÙ–ëaÑæÚõ#¼Åàl†C÷ÌjvÉmû)…«†²H68LsÅýîÏš-{ÙZα©A¶Õ™oÚuD›äßcÚ›Þ\Ó*¦9ïé–­É}(ÑÇ5H½¤µàý‰k’ZƉ¤­eÕÚÖ%Œ«6ÄšV/•ÄæÄõI³ï-hºoC¯4Ü~g ‡-Îå‚Ùiú™@«Í‘޳ì¸+l[.^á™õå\t¡6š­Š)QæÀÚ¿X®EóSôÒµ³«ãÛÍ܌嵑MÁYpQŒEËL")nõNÅLöÝð™öá謁F&i„6Ó ~7ÜZG€‡è„·‚‡oogë/_Ìxx'ãü\Âxx7ã=àG0Þ e¼|)ãG‚Åx?øÑŒ>Àø ø2Æ?ŽqœeùåÀg|9ø[ÿðaÆO_ÁøIà#Œ‚¯dþ®cú«Á×0~2øZ¶^1}<Îô*x‚éE?§˜¾¾Žéƒ‹L? .1½ÆKÏ#t=“WÀUÆ:¸Æx|‚ñ,xŽñ à§0~*øiŒŸ6ß6σëŒ`‹ñx‘ñØf|¼Ì¸ >ÅØ»Œ[à›{à6cÜa|¼Âø x•ñÍàg0~&øYŒŸ ¾…ñ­àç0~.øyŒŸ¾ñíà0~!øEŒ_ ~ ã—‚_ÆøåàW0¾üJƯ¿šñkÀ¯eü:Üú׳su'ø Œßˆÿ’øíŸo$‡o ß.;OÒÙ`É[ñ¹1ás0\@o§wà[¦w&Æ—cµø7r?­›;O}ÃëÏц{i#>ÏQÿ=$Ÿ þ¾N>ˆÏw›¼û¿PK kL[D´Iµÿ®7org/apache/hadoop/fs/swift/http/SwiftRestClient$1.class­T[OAþ¦´”–-”K行½ËÝKKK‘’ †"Ã‹Ø ð”ÕØG¹¦Î£éõ}KܶUƒžñrEåBí½éZŽØn;r©aÓÀmÜ1pg äð¡Yœ60‡:ƒ3 †k“XÔÝ^_6¥/x÷‘-;»Â<Êç¶»h(éžûØy=„IZÉÀøÒÀg¸À°6ÞÿøN‡c{f}g8$ ëï€ÃYŠÔ© ¾Ë¦äRÃ{$#áÑX¶< ‘¡S8ÙÝ»5<`¸ûêÎW™®Ž„ïF˜o Ù#-1ˆ§!ܶì0$ ‹Û 7ÇÇõö­n+:·Ô9çWÊÔ.?Ÿà5iFÊeŒWX%vÅq>fè{"™Ë©áˆÓü‡œnÚ™¢ÿÎÑ›nž¤H Eül±tV¼|ˆDqéÉ";BêWÚHà#¢iRG:ó$J&¨ñ)Ô “F“´XƒôÄç_aêñ+¤C¢t­ô!óËÞBˆ7Ù ð”ô9.Òþ%’SÄóDéøØD2ÔÏ6+Iþý¿@uµ™.ÃHWŽg‡àÙð…Ü-/ À¯Wú™bé/œz ã͈@3CÐ Q<qnªRz hv´ ‚½Z^ò2*áÞ2¾Â'$§ÒL#úÍÐêRJÇVôø/PK kL[DtùJ 7org/apache/hadoop/fs/swift/http/SwiftRestClient$2.class¥UßsÛDþÎ’£X•;M](P uˆí$U’¦š4$NÒ&àÄ!.™É˜E¾Æ*Šd$¹ Ïå™<òH™¡…a: o0LÿfxæÇždÜLH[ 4£ÛÕj÷Û½½Ý½_ÿüáGSØNáÞË»*4\SÑY±Ì©¸Žy3xOÁ‚J ‹b)+XR°Ì0x×°óÿ´Íý`• îù #õŠëíéFË0›\7Ýý}×ñõf´LÛâN Gš³ }AÓòó “G-šFÃu[úm_÷ïY·ƒÐR¯ v‹Ü,…ÂxÎr¬`ž¡^ˆmÝc„ÅmyÉmp†Šåðöþ.÷n»6I+®iØÛ†g‰ïŽP;bƒ¶æ8Ü[² ßç$¹7Æüíñìm۽ǵÀÚ¾†ÐäB±¾Ææg˜Âªm W /Ø×>šnÃ×oò`=dg‹õ2¥2’;‡žì†<î·H—ÝÆg‹~-ð¸±O¡Wîw Ýrõ5§Õ"1©k6¥-°\gÙ †„ðŸ¦Í™Ÿ¬­0‹ V\bPWLÞª¾‚ IŸíÃôØå¶‚› ã/²\%¶kBÁPm–z·a˜ã!Êm8{zu÷7EU§jÖžC'í‰ ën`ÓsMîû®Ç°»þ2W/Ï“/µæ¶=“ß°Då3»,‚£º_qLÛõ-g/ÂP°ªa ïk@FC§5dQÑ°Ž UlÒéiø[j¸Å Ç<* E”4¼Š×¨c· Cùÿg‡áÌ ¥{TZíV&ÃâsònGnל€{Ža‹^æG0ô˜A3di0D5Sé´ÃÇσíÀ²#œÈnÓš=ÏÁ:MüfbOÚüdØHi ÷éc E_çHºuâð8W(>c|ÈDiP% õ²€P SZ~z‚:õóøAÏÝZ6|çÚˆ]»“¸H7«F÷q"“}¥&#I½Y Ò­r†¸kH@"š)>+=D¢Ä¾ƒô°ÅP¨OWPâ7œ%^<¡äð2QçñJg¾Ä¿\ßcuGÊâ—ꎜe÷«;É,žTCûýZ®ƒÆDŸ’–@ûš¨,PG`¾8Ä…ˆ»ˆ¡Ã“C¨Ä%F’=ÌzVÆãþ±ñGèûÊøW™ð+…~sH’ß¿0,I‘’“ú0%õ‡q”"_8w¯SšY&ð±øu¢B?Uý ý:ùÈ"ÐT4…q\>ZÈ,Râõg€ªÇO®GPZ'BË)\ é4®†ÿfð6^"î<¥æ¢G#餬âƒèóoPK kL[D3ÀnT7Ù 7org/apache/hadoop/fs/swift/http/SwiftRestClient$3.class¥VmWU~n6²nJ ÐZµª©MBéB¡µŠ%J”7 Ecê˲¹­ËnܽiéOð»?ÀŸPϱà±Û”ǹ»i )BVsNîÎÎÞyæ™Ù™¹û×ß¿ýà*x§qS.“*0¥"[ L'ð±ŠnÜîÁŒ¼¤~VJwä2§¢ óRwWÅ ,$PT â¹|šÀbK ý ;ãñïÜ ܨrÏg¸TYt½mݨf릻³ã:¾^¢nÚw„îœdè5ËÏŒ2Œ¶¨U×­ë[¾î?²¶D`©—¤¸Fnf iÀ¾ÿç÷г¾‡Þõ]$ûšHÒ¹‰M(»HýNo'‘89)HH?A–~†*ý8;¶œ )ƒ äôíÀŠ©¸HBœ*­YäÚÅC–=Ïäui–Ö¡Ò£ŽˆQ\L©VD äDHå~¥%öú÷Ó"‹ÒS(ÒÞ¶©6ÛT‹í ’cé[~¹>CO±?™ýé&ŽÎwšlƒ&¡aü¨šž£3˜øÐÁλ¥õJ`9‰kÁó:Þ ¾ÝÀ»TX â‘¨¼Â¿!Ò^Ë(‡¯ÿPK kL[D"j²Ö 7org/apache/hadoop/fs/swift/http/SwiftRestClient$5.class­VKsEþF’½ÑzmËb  =쬓ð°ãÄ–"cc%Ìi¼KÖ»bwÌO Š#wŽ\C6©(Î\¨âüŠž]¡¨°ˆµ®òLkfúëî¯{zöç?Ÿüà ì¥1†kj¸®†7uŒã-#x[Ç;XÖ±„•s¸¡c7Õò- k:\WCIƆM†‘‡ÜÉ6¯ ?`¸üIÅó&oq«)LË;<ôÜÀlJÙ²[¸Ò¼ž\fД^= ‹½*M^÷¼–y˜Áçö4ÛÒv̪·÷KîpÙ$íaÙ´ƒìÂÊÊn¤¼+Y =PÊ+¶kËU†¯r±µŒ/vHùCªäÕÃxÅvÅÝöá¾ðïñ}‡V&+žÅ÷mõ»³˜R0€Á(»®ðKA+KqƒÊ^#RFÅ‘ô¹¥–ÛeeílnJ^ë‹-!›^}9_yÀrÓánÃ\÷I[%ózÕkû–¸m+î§ÿtEéP‘nº–ã¶Ûˆ@5Ü1PÆ{¦qÞ@&1e`[¸«aÛÀ>0°‹*µ B´)HJª†{î£fàC|D\žS¯ã²—q‘:Aìªg(ý,RŠBmÏ,ow •.Ð3°E·ž# eW ßåÕ½=fLÿ(wyÒYý­oŠ­0`CÌïiø˜áz쎛] oáhCHÅN;ˆšd2—/3Lœ*àèai‹í†™Ü^ÿ¾¤sKå.{u®ö—ñ¯÷:ÄÀ Õžï~ìS«Þ}ߎäé\o¸UéÓí¤h3¼^ߟµÉÿÈ=Ó¹ÓO¯(bߪE\¤ÏƒqúÂHf2ª4S+gj´3Dÿ3xŽž®çIº‰R4Ÿ/Á sÇHæ‘,°úŽ6¸@ã0GòÌ’l(™ ^ÀKP/ 5:¥ÀviNÐÆð#åSÔ¯!ÔLt¼¥¤Wð*í¿ú™»Å%1EK—h¤~Ó±T#0¥:[ø¬HCâ1´âä9’’'HŸ@êûEò7Éßq!ùGáÙ®áÙŽá¿­å2d-ß±v£WºPü #'0ú¦» iPì½”Y#²æþtôÑ¥q>Ô4±΋ôy©ö–p/†±%‰ƒèoœV¯¤t¼ýü PK kL[DÑ7¾¦ý 7org/apache/hadoop/fs/swift/http/SwiftRestClient$6.class¥UmoG~6vrø8b7J ºàœK ¤Ð„´q0MRCBŒ"Ué—Íy±ïÌÝ ?¡ÿ¦•H¨Z©êçþšJýD‘Pgï"׸iãKž›g^vgfóó¯®ãQ£˜Sä3En)r[GŸë8Žy :4ÜQ›E³øB×:é,)RѰ¬á.ñgÜ-6¸ä §jOø3n9¾µêu"Y—àíy]©¸ÂkÊ[cS‚@W¦Xdž‡Ød°R^¹Ë(¸€‹T|©[€¡òþ§Ì0ÞëÝZfXúÓ¢[ò‰ƒUOŠÀãn]r)zlX)ãVŒ:®ÏG ³H:nNJrƒËÖû3 u†¹Ô3´87a¾)”°CÖEÅo<§aWЇËDéð¸¨*POA1Ütl¼…‹Cй­®´8;Móàûwx  ŠùDêÊžÁ$½ÈyzÔ³…‚ê^€Vjàx¥ŽWjmÒ¦ÿLЛr–¸*2¡õœYÞ3¯ìaȼ°‡Œ91¼‡¬É^bøGú<„sDG„ÌøˆxCñ$ý˜\«ŠZ’´”ÉÛ´RŽÊûù¡üg >(€÷ ŠôýS•­—ˆR£ü–°J²ü ´oÌâ d^âØ>rı2‘ìOÐÿ t”²BæÆ3á|æuÏÉ®ÏÉ>ŸS…»äÓ<ðy‡V¥Ÿ3Ë¿áø>ŒÃO!1šëÍ¡Œ+½‰–èd®þ‡Ñý§3 Q¢×bä4fâõ:nÄßfqç‰;Kù"ùåIjeu¬%Û¿PK kL[Dý«ÇÂY6 7org/apache/hadoop/fs/swift/http/SwiftRestClient$7.class­U[SÛFþÖ¨r0·ÜÚ”´u‹±!BCBåæwHa Ãôe‘[DH®´Jù ùyËk:“ÐL:yîô7uzVr\OK bêïž=»ßwΞ=çè·?~ùÀ ¾ÉÁ€¥†)}˜6 cÆÀ]̘ė:î´7§†û:è˜g|ÆÝb ~ˆD(ׯ‹ dÛßðƒ†Å[Ün Ëö}/´šR¶lמ´’“zeÓ ‹S Ó݈&¯û~Ë: ­ðGçPÆHkG‰Ûdf5æPàÇsä"Ã~)5ú‚Žï2h«~]0ôo8žø6:>ÁwüÀ%Íà†osw—ŽZ·•šºÌšç‰`Õåa(Hs7­Å9ºc^œÈ€ÛJ¹’¡V:Çñc!›~=´Ö„+¤x¯ŽoñgÜr¹×°V|ßÜSñKÎ2Ì^†“|Û‘Ü~ú„·ÚW7ª'¶hI‡Ð:h¿'2j1Ì]Îç] “ç×Iì@2'”Må‹cÓXè äæÁ‘°Uævœ†ÇeP Fºï°ø¶C?`ø>õëŸI´pÆC.’ ÆŽ¶xì¨wþÕ…¡®z¶ë‡Ž×HHu<2±ˆ¯L\A¿ y,›XÁªŽ5U<6ñ5ÖfÒ¿Ÿ‰ÛøÄÄ Ü¤âNù Õÿ%^ CqÀߪmvÒ“aù?ØE'‹5OŠÀã.e»]VJ©àê±ç5¬H:nNRl‹Ëæ…ÖžŽýÔ±8—O¾!¤ºh&]/['ºdƒN?›‡ WK{g7ƒÛêгST“ÏÓ7é ×å Eš–ž:!§)›Õ7È ªTš©nHÓCÿ¨ã’4 ²JS®¼+O¼A¦Ì~Fö'…ŽtZ?†I6•Lè\‡úfP¹Ð)ųMs†æ¡Ê)´üþºöZö-z^‘:ÛE5S]MŽ·©”ô!>¢ý[±‹™+K “¨‘j”FªÎ¶¥*‘)h¾ü¬BCæ-zÿr×P›Úu˜Ú.;ùŽ|ÛÎ{òO kDþY›üQû¹råôS|pvÒ\‡4‡">ï"-,Sl¾øÒܫˑÒ8#ÇQŽç &â½IÜÁ5’nRh $¿>Ò–4KÉòOPK kL[D~@Äž 7org/apache/hadoop/fs/swift/http/SwiftRestClient$8.class­VmWE~†,,ÙnÚ”ÒÖªõPMmh——[BQHAÐð"iQÄ·a3%[—ݸ;©ü?ëð'Ôs,xì9êçþ¨ïì¦H³=æœÌܽsïs_æÎyþâÏ¿ŒÂNãn«áŽ~LÈ¢¤cRÇ]=˜êŇjþHñ§5£†²nÜS¼Yã˜Óñ±AójXÐñ‰ŽOús7ˆš"”ó‚×D2\Û¨øÁ–ÅÜ® Ëö··}/´êR6lמ´bÉC¬;an˜aä F×|¿a= ­ðG硌4­ª"WÉL9ÂPÊ“ŽçÈ)†|bí=,¬1he¿&ÎTO,5·7EpŸoºÄ髸6w×xà¨ïSS1€Á\ð<”]†‚8cI}Ìݦ3bGÜVì¦+‡zŒãÛBÖýZ°Ñ¥BçÛk3Œ&·B¾V%·¿_äV*ŒÙ[4¤Cº:* Ý¡Íí׉aMÇ"ÃõW©ÎÙVaèÚ¡Ú*v®Ã0•ÄBåÌ-—{[Öòæ#a«ªLW-Ëf@8·ÁJàÛ" ý€ÁK\ ÇÀLv¸©Sä’Qõ›-æµ+ý‡Ào¨¨¼g=ÛõCÇÛŠ éX2±Œñ†‰ó¸`âVMTqŸár¸ã[ sÉ—s~Ó«µ·›áâ~bª2 Й¦ã’; ã9D*‘΀‰5|nâ ¬+êKøJI}mâ|kâ;sÃIëÅDEïâ=j-‰ÏCùØ!Úþ—YZX>›é°EûÄÄ<)»t²¤8€a%ôáT<\5ÃÏùÊá*T•Mé¸1f\ç+\Ö;î¡ÊéØdOÜÊs#ÑIÏl ©²Ô ãvÊ¢‘Özx£!¼ZÔ>Ž„]8ÂjÕªê‡ú`•p®æ ¯‚ÄJ‡ ´ZÂIz¥³è\æ ’Äùã\¦˜úÈ5JBƒÜí‹·ï¼×ÜVuš»9L-ò§ä7hÇmr†‡"É}›ø¼Ž`€ž$ýôÔéÊfU§h¦¶Eœnú_›t¿EÔº¢9[| VzŠ®"ÛCê7¥‹·iì!IhC¸L´©hzþ¼ƒ+P:u’R84¢€ëƒ»Ðβ_~Å•gè^†žõ=è}½»HÓŠ±‹S»0÷ù›¾N?!…Tdä´rB³`hÃÈkc‘± 1`˘¢r¸JFß´X׈Ј‘§‘š[Ë“YZTŠ™âï`ƒ4tý3ûájQ»S›8`%Ó¶’iYy >˜½GàC-ð»4+ùtqðdwqöø<Å é6hšœ¿q4Ÿ¦ÜYÿÚ÷äõ@iŽ4G1Í7q+ZÇT MOQYÄ¿~âŽh>‹?ÿPK kL[D•õAˆ+ÿ7org/apache/hadoop/fs/swift/http/SwiftRestClient$9.class¥UmOA~¶ΞW)ˆoø®UK (¢ò¦¼Jb ‰áÓr,í™ã®Þí)Á£‰ ñƒñ³?Ê8{GjSPzÚ¤»³³ó<;;;3÷ãç×oî¡œŽ»j0uœÄ° #:mÜ×1ˆQ tÚSÃC 4·”:t$ÃbáÇw„¬y[¹Êç‘81°üš¿å¦ÃݪYv¥¨Æ‘ ["ß!ƒ¾°k‰º´ÉTÃ8Cg dXgýïÖ5ãÐqÈ%†Ô.%M±} Ãt’šB¶²ùZX*Ý2»êrúêéXõ=Kç3¼Jü‡Y&x¬iñŠú–X´UŽõµðÜUÊÐ×r¼Àv«1©†IS˜6p Ý d äðÔÀ f5̘ǂE›«¸fàú)ç5ÃìÿGŠát*Û3Ë+|d˜ù µh¤m|€Š°ïr§"¹MfB÷è}ê!­£6x\? ¥íÄqj­rYk»•5,1Œ%nxù‘¨l²U!Õeà nj…2CÏ¡œcШ±‡be›ál¡|t ѹ¥ž"?:Lµø>yn»gy ’tìÄ9‚kPŸ< •Ë©Bh¦ª!M'ýs衆ÞKÒ8RèPšbi¬8¸‡T‘}FÇG…Åi»Èé2úH6”Lè38õI z!+Å3I«Í'JûHAç‡üó6¶9À+é".Ñþe’Ó4_¡‘ ñ€süÒ‡_Éׯ}øOêø­¿3ÁSÂñïáþˆ?ø3þb`=þªão2ÿ]†ÈðOÿÒñoþãÃÓ.ÆíÆx&èD¬GÒd¨ÐIéTI˜cf‡ìT6³² 'µÍ¾bÔÎd çw;éÁ°5bņìðwœ‘ð®L8³7±+œpûdˆkUÙ¡DfAaÉt$†²Ù‘p¿€Û­#™`R‚¼6‘Jd×v7•}Š7o'¨'nfu'RvÏèðN;=`íLòJ Û‰YÉíV:!ßùE%òÞxz}ŽèÍìJ¥ìtGÒÊdl>³¡\)œH”ÕR™±³£#„ZÙ•µ-vvȉ:ÊVÓ‚ñ$\ñ«†óä.z¶ÔF羘="¼gtªb´³íɤ³×Ž÷g­ìhFTÍšQMÍ—ujì}Ù´’£IÖß¾g/ÐIý@Ô›MØ™p{,fg2În;Åœ×Ú©øˆ“H1íAf_Œ¹µÇ '­Ô`¸?›N¤ùTõÈèÎd"vɶnÆð¤ìlø’m]¼égl;²’@°læLu摘œ–X@˜í]2šM$Ã]LÝÊ:iÞS)kØõÅý#.y\/¸(éTÍuN'C'?× Â¢qs†‡Ù4n¤ÅÜø ob°$Yhû8··Ì‡°®œJÌÙëšhN&„ëÿo%¡ì´$¯™%2,=I¸OASKb0Å)2ÍÑ œy:éK;âQ‡÷+N)9N S†o•ú²ø—ÑcöÆ„øìì w-Ë™Øç›¸Û¢ýЩ֤YTgR=Lj Ù„MÓð`+ˆ'Îx#m´øöx1é4Ǥ¹ÔÈ1Ö×Û?`Òi”«O7é Y]Qž ìH[##vZ§3MšGg™t6…8‚LzÍ'4N‘käÂs8ÊLZ@çr„™ô\:Ϥ&j&´•É&µP£I­Â½qÙ¹™HèÜÌ ×„85N Žb“Ò"“Sؤ6Z¢ÓR“–ÑyœàòÔBL>½?äêȤå´B^r¹I+é|¾!–tFãbQN䦗e¸Æ°…7:£©xÈ=Ê'á•aVTè‚uLs•(eI٥ͤÕÂL„Ö˜´–دêYL¡Èc¸ muá4ßs­3éBºˆÐÚáŒ&ã¡”“ íJyË‘Ю´3’;C\€C¢–vNŠ!‹z¥ Äš©8o1“7Y?ö “f¾"|r½8eµwˆX/=N‡õÒQ†·v¥öXÉD¼P㊎Ê&)}k§Åi:LÚ@&m¤FÎÇ&]L›tê2i³èçyâfi 6©[6¶ÐlAî!œ1M…&,/ƒí´ùž„ “•ÙJdB‰TÌI§yGôSj ½&õÑV.m%ÂÙñ5h%R±Ý6êçðìu&^‚&¸œ°þÙ¿¨§7èLó('iWû 'ÜÕ[¢¾•§öbÿÿàè‰/¿qYÀËAÓÛyÊ„>}àN^¤X¸òßç^q —‰V|-XÒÖÆ/Žÿÿ“¼V¼f)“×~«S4¼ŸSY·jãÖøy|²÷OâxZ¢µ·MuMÍ]e×´bSïË:—™ÝÔûOY fnèql­Õâ̧ͰâyƒÊsî¸k@gq—­¨ÇC½QUOïèVÖãáÞhU=éêõøRoÔWG{£u¼ôežxí1žxñqžªëé@ïmE¦ª„¨z½Ëé]gˆäï WµÕ˜Ã(ÿhÍAçK?Œ3Â)s¨"òaø[Á<ÆGjP{³îEmu[æPßS±RE z ‘ÊPE"Õ‹Ž€{ºˆ?è?‚9„ë(!Ð\½hŒ˜A3‡Ó"52#µÓ+ïÆÑŠ`MTkûãÌȬ`epVóvÆYGp6SÀÙ æo „xËÝÏá9G0_C°&pN Þ‡F†Îu¡š`m๠]‡'i.!rž©á!Ð$Pm)9æ¬Ù¡.X牀rEh‰ÔëshdZiiQdv`q•'I@$ièVgçÅ©šNœðDq<Ý·=€@0‡]IæyK®ƒ/ØpL>¢ë#ÕQup,zð™k‚úƒØS »4W@ðç°,°¼øÙ&B¯a¥ÜÌ.½,X•Ãù×ãôî*ÙàµeÌ_XíqñE$ô³«ÖäiÂÚcìDŒ<²¬ç°N(9\˜ÃEä žC{Ä<ŠõQ;(hʼ!è'ÛÎˆï «±æ(.ŽøØ$Tta¿K ¿G)Xs›É=A=è ‡ñ¼É5Û]@¯Ù-Aã6–Ö®Ò®ÆFŽB ›/ ÃæÝœ%ÞƒsÔû±\]Õê#X«>ŠNuzÔØªn€úÒêfìW·àJu+®UŸÆÕm¸‰3Îýê³ø©:„§ÔajPGi…º‹Öª»©CÝC›Õ1Š«û(©î§½ê‹tznVÑ1õ0ݧ¡Ô£ô°zŒUÓê+Úbõ-mƒú‘¶YýXÛ­~¢¥ÕÏ´«Ô´«Õ“Úõ[íˆú£ö€ú³ö„ú‹ö”ú›öWõw7äŸFÖâ-Ø$Ô×b)4Ðf¼œ3Nнz`W žÎÆ'‘F†óÔ~êC–1jp%ua”¡Z\K˱‡¡YHÓìÅ>N'/ ï-æl·ŸwýÚ4á¥xãr:)ä>†¼Ü·\»o~§ÕÚ=x%^ÅYu­vWâÕ¬ùNí3x óX‰íxCUت݈×3§ºvKñ†|ÐnÀ™çjm3nÁ›ðfÚn\År i>ÿV†L^ñ’žw×Ûø®·sÞ{+ÎÃ*˜:®Öñt¼“t\ãþ½‹+ʵüÿÝÀ§çºééxŽ÷zÀ¼¶§áçÕ1ÖÉœ™#ºŸ×ºÀ»äN«mçyŒ%­/‹Š°¬\®‰=vðêëÆ˜æ´æÂï7†Í¨™úœŽëòàõ÷¡yªÃî_Éq,;^ ë6p±}¾Ø^À³Ïê–Ö{Ñ“Cïí Ú%¶ºXa«ñ|°´ÂÖµs‘»~ ¢}«äL‰‚É—í^v*%Ï–Ö…‡@“óÙâ)’Ô™äv·ëø¨ûÒè|Œ‰Uàã.ƒ¸É?ñ<Ÿ)lÅ­x!¯ôðlñkãCðþmãï•ñ?PK kL[DŒú¼“R²Iorg/apache/hadoop/fs/swift/http/SwiftRestClient$AuthMethodProcessor.class­•[oQÇÿ‡]X»b[KÅK¼ÔÚ*íZM|B¢¨i•iâ㲜Â6°‡ì´~?‚¾ØD?€Ê8¶Ä@»;—sæ7ÃÌ¿ÿüøà¶LÄpÇ„ wMh¸§<)i†XÞõ\Y`ÐRé=½(œa©äzü]¯Sç~Í®·É³RŽÝÞ³}WÙ¡S—-7`H<ëÉÖ[.[¢Qö…Ã@ø ñ]Ïã~±m§K/KÂoZv×vZÜjÙ !ºÖ~`ŸÜ}iµ¤ìZU¥Vx ‹m—{rc 5Ç/äs÷¸˜×g‘ _«ä ”e±!Š>·%œ“C].‹@;j©ÒýѶڶ״ªÒw½f.=W 'TJ¬õ|Wuû6CáÑé/ègrú|k‡Ô!;røÐ@– OR3—ºS òxŽ8†…ªÛôlÙóiN‰“šF¶åK¾òtä㽯pGæ 3g›ÏŸv*áN˜UÑóþÊUÛ¶:³¥jãÔëºû4°ÓåU q$p‰¡x»=eꨶgÞkƆ)Û>×Îêkô{g@=:"jB´ÿ&YÏÉ¢/bv„È7Ò4œW¶ò2‰8éñÁ \À2É‹cž’«jŠž†œ6=/‰ jÙ Û©q«fW„hX[¡î¸[ÒªIÙ°Êj¹ÊCYô\îËÙ>ÔÃÔa!»b^œF‚üÚj®@Y†*¢p[òöw†ÇéÒ¶ýζ<Û¯Ze¸~5—96ãAbjÍÀU…<Âa˜¬rùÄóᔥ-›¡jMOgÞ.3ŽKîˆz]øa+©ÓºœµDËNêØû{î0€áazà:-äÔh<8Á9†dÙ­út£€:4~ ©kN>äWu]oes›;2W8[x~E7Ë¢8|ÑU35Ñ“d^éLá ’êu™!ûÿ Ô‰»)ŒcÂÀ7Ç0~²™/­£†JC‘_[͈g æ}n­3,gJ[æ'ÓpL·nTBßvë¹lœÓòšMÏ :\V‡ÁhvÎFœ‰°¥¶o YOᑞu¾po›×*¡¶ñ($ œÉ~\a(œ£‰ešS'>?Ñðˆ Ï2}+6ŸFyzŽs ÉŠ]wéF>½ÕÈ¿žbžÙɯ>]¯¼±Å­0Wè›í ðüy*r‚^ñÚ¾Å_ÛÂe£=ds¢ß.!)†› ý‰“SÁ¨†9†Wâc†âèÆ0ß÷[“‰Ïr)ÇèKž¢ß•ñ)H‘ɶ:e/)#CaHìÑLÂe‘‹*ÛEŠæ©î\ÁÅô‰Ê0ÅQ!x„W¦ó ŠÉ荒¿ö@îu Ç»›"H1£‹'#pWq-ÿ@™X›Ô~èU)_媜Æa¹ª¤ñ»\UÓl§ÜË·»Âä©+LvY†é ×#–õ–èH2»¥o7Ö¿zÜ¿Š‰žþ#qh™Ì!¿¡(Î)3Bˆÿ+ÇÀ ß:¬DÂH¸Mcwp·ïaâÕUd1 ë¸î§Q>%ëPK kL[D¹¨IF4Horg/apache/hadoop/fs/swift/http/SwiftRestClient$GetMethodProcessor.class­”[sÒPÇÿ' ‰DlkKë]«V墯êŒÂðPª¶#ZZFC8…t a’ƒ—á7Ñ™Ñ?€ÊqOˆ´R›‡ìåœýífw'?}ûà!î™ÐqÄ›ÖL¨¸%=· d dô¢ë¹¢Ä f²u­ì79Ã|Åõø«~·Áƒ]»Ñ!ÏbÅwìNÝ\iÇNM´ÝΞsñ’‹¶ß|øC?`Hm{Ê; 9ÝÙ¬øA˲{¶ÓæVÛnú~ÏÚ­ð½»/¬¶=«&Õ*E¹ãrO¬†V&ËøØûSÊö ð‹»ÕB‰’Ì5ýrÀmÁ‡ç O3•ûmul¯eÕDàz­Bv<¡ãw»¾F‰œou£ØÐ¥!°Ú\ÙÊC0†Ò1Rl‘:b+È1€áqfæf¬äô#Ž!Ys[ž-úaéoMc»ð©X}2öy;îˆBiælGÀ‹3!²Yóûß¹r{ÒS™îËbS8…¤|]b°fÌ"Ãò),!mà.ÃÆÿ/'CùÚŰ>óˆ–ZÎúdWþ…¥­Ò¯Ç€|4(²½´­&YdÑêBϱ”/¤©8-mée{H‘žÞÀ,<;áY$™–ÝŽy;¯L~‡ú6?€öy ù&B® /ÅH©-“Á“CøÂ&ÁÏÅð E)Q±ù¯HL“÷ÆÈúˆ¬ãü9.›ŽiÁbò ’2.‘€Ý…!81'|yœˆû¡â ½\Ãj$¯cžäEò븃9ÍÄ ƒì«šùPK kL[Di·XG;Iorg/apache/hadoop/fs/swift/http/SwiftRestClient$HeadMethodProcessor.class­”]oÒ`ÇÿO)­TÜæÆæ»NÊ‹ÚM/„p!ºlŠÎÀ²Äˇò º@KÚâË×ð“è$š¸ÄK?”ñ<´"0nJÖ‹ž—>çwçù‡ß¾ÿð h¸e@Çm¸#3wuduä´’íØA™!‘Í0¨·)«¶#Þô» áíóF‡2ËU×âîÙ2Ž’jж}†•Á›¯EÐv›o=×¾ïz é]Ç^¥Ã}_СU×k™¼Ç­¶0Û¼éº=óÐ7ýöa`¶ƒ gÖ¥[~PéØÂ 6fP‹ k“ƒ|êýæåi4(í׊eê²Ðt+žà¿3lg«Gü=7;Üi™õÀ³V17ÞÑr»]×ñ‡¬!ßìk}ó"'úž-×y‚ÆPž£Ç¹#¶òqSGž O²±×±U” xþPK kL[DyOŠ8!¢Iorg/apache/hadoop/fs/swift/http/SwiftRestClient$PostMethodProcessor.class­”[oAÇÿtW{ƒ¢Ö[k…6q[|ƒh£Ûš&&M̰LaØ!³ƒ—¯á'Ñ›hâðCÏ,´Päe›îÞËÌüþgÏÌΟ¿?xŠ,ܳq?…¤0ƒm¬1X%Ï÷t™!ž/2$*²)檞/vû݆P¼Ñ¡ÌbUº¼sÈ•gâa2¡Û^ÀÙ—~'t[6÷•tEHÅ~ãûBU:<MzU•ªåðwÛÂió¦”=ç8p‚OÞ±vÚZ÷œºqk"ЕŽ'|½>…ZdÈ],äK﬘·W!P:¨ˤ2Û”%¸ƒq†|õ„äN‡û-§®•ç·Š…qEWv»ÒB%7ä;ÝpmàŒtˆï+Ï´ó?Cù¯É=gÇ>o1$ë^Ë纯¨'™ÑðØÎ|-Õž)í5N„«‹åÈý›/Eoɰ㩺ì+Wìxf/³ROLµiØxİUÁ¬{œÆ"26ò •+8&— Li–žå#·~»hþÖ¥i†òNDÃvä+tX0Oœn×èK’½ ˆ> Ö;ûާLl²ì×ÉOf#;!³@6kvkÈÛ ù@òbï7Oÿ6ü"sƒIC¤ñ²X¢ñ<9€Ï¿$xn¯ÒªXXìæ$&ÉGcdëœláæù¬ì8nÑ;†eÜ í]Ì’]¦ü Vq#a®áUÜ&»’ðPK kL[DNOg “¹Horg/apache/hadoop/fs/swift/http/SwiftRestClient$PutMethodProcessor.class­T]oA=Ã.»²b[ú¥ÖªU«B«]«‰BH,ÕHDiJÓH|Ú.SØvÉî`5iâ³þ}±‰Ÿµñ7ïÀŠ@û µû0÷cæžsçÌÍüúü À]˜4ÌÐq]Ç  nÊLRÇ‚ŽE-㸎È2(ÉÔ&ƒšó*œa´à¸üy«±Åý k«N™ñ‚g[õMËwd&UQsÚ[k‰g\ԼʚïÙ<<Ÿ!žw]îçêVp:³Zðüªi5-»ÆÍšUñ¼¦¹˜Á®³-ÌšM³$Ýuˆ\Ýᮘ? šf˜îoãMóO+ùÀÏl¬§³D2Rñr>·ïì3½ÒÄßžzFå}fýAÏõŠ[;ÜéìÐlG€g†~¥pŒ’×òmþØ‘Ã59À´$›ãbr¹È`É"ËnÇ1IK +ÿ?» ¹‹ayè'¦Ù=j87ûUù,uŽþL:äEDÊKÓjP´B.´¶ÈGòœ–±Ì²=ÄÉwNà ÆÈ&ú2ãd'¥Ú!^‘ê#dc_ ”÷¡~€|Û†œî !¥7E9Ö ÁUœÅ¹üEroVÿj”•¾Ëj?Šåh?‹e-ƒâ ß»ž+̺Âl‡el•®p>d)Po‘¶$‹ŸÄÛëé_ëö¯af ÿPÚ¦1‘Ÿ’•uÑ…}°£µîG»ÀQ¾Ô …Qp™Ö®àjÛ^Ã(ÙÊkHaD5pO§xN5~PK kL[DÇeLU1t~5org/apache/hadoop/fs/swift/http/SwiftRestClient.class½<`TEÚß̼–Í !›R6H ›F“:!HM8À`Á%Ù„…$w7R{Ãrö³œ½v±€¹=ÏzžåÔ;=ËyÞy§ž]φÊÿ}óÞ¾-YB’;…yóf¾ùæëßÌ<žùñ¡‡`‚Råbùüçôsaþ\D?ÓÏ%ôs)½¸Œ~.§Ç_ÐÏôse:¿Š_mð_êü¿–_GÏ×Ó›\ÐßHµ›t~³ r¬‡[èçVÿßaðÛâí¿ƒï4ø].~7¿‡î5øN*ï3øý€ª|•íï r·Á÷ü!Òið_Óì¿1øÃ:ß«óG\0ŽßhðG©ü-ýá‚Rþ¤ÎŸÒùÓ.(ãϸ`ÿ=|VçÐùs.˜M³øó:Aç/º œžçñ?R§— þ2•¯üO.þgþ*=¼fð¿PùºÁßKoü-*ÿjð·©çßèáƒÿÊü]ìy ÐÃ? þ/Bí=ƒ¿/`!ÈÆ þoƒHÕ þ±Î?qA½…ý§ÿÌàŸ» ƒá‚§ø—Ôé?:ÿÊMükcðo þÎ÷ü{üƒÁ4ø~C€!˜ ŽãO‚B¸à\¡B5ØïBÓ…nÃ'‹4)\Ô˜N?&ýdì9]ôsÁÙ"“žûë"‹ ¸é!›~rè'×`¢r€.ºàb1HyºlˆC 1D %4q!¯ˆa†ŽœzȧÚÄNj°·ÂHjEÍ£éq =b¬!¼´ˆOuQè‚ÛÄ ƒ½MŠ ÖdˆbC”PïRƒ½«‹q.¸:¼gˆñº˜à‚]b¢.&!éÄaûH“]ð˜âSÅ4ƒ}¦‹2<ŒÄt]ÌpÁ£ˆ={Ù`_éb¦ ƒò,]ÌF²‹9†˜k𥺘ç‚gE¹.æë¢ÂÏ‹.q¸XhˆEÔ{1ý,ÑÅRÄž?c°ýô¼L•TVéb¹!~fˆ.x]TSS.Vb•!VâCÔºàmëŢݑ†8ŠÊ£©åúYëbıéÂ'Ö¢Nõ.øDø]¢A4b=•úÙ`ˆi¢I4SÏ]]ð ¿‘´RËq.a¡¶6ª]Ä&ClFõ[¨×VCœ@剆Øfˆ“\ðÿÚ'î§Pû©$lØr=œnˆ3H”Î4ÄYô|¶!ÎÑÅvËAçâ<šé| NÑņ@û3XœBëÅç ]l˜¸Hëâêw©..3Ä冸…‹*®ÐÅ•†¸ÊWÓ»_ºÄ5âZC\gˆëInp‰ÅM†¸Ù·Ðó­†øõÛá·‰Û©v‡!î4Ä].xNÜMH{\l:dº¸×;uqŸ‹Í÷»Ä¢‘F<¨‹]†h§…tb·‹•‹=.ñ‰p§!~mˆßЫ‡ ±—J´@'£ˆ#°G ñ[C<¦ Ô°jһljšO¸Ä“â)C<ÍÀ=·-²Þß Ôù"`Ëò`8ÂÀ\ÔÒâ•7ùÂaXÏ0È^è÷Õ/óGÖë—‡‚uþp8ÒÅïäÎ÷7ù#þ.¯žÅ1åÁÖ-]^üç\ÞéÒþ \’^0èG­„—õFÏcÏØsˆôáþ®í/þ‘HkÐÀ@,­:œgi0ÔXêkõÕ­÷—Ö››ƒ-áÒ¦`cc ¥±ti°q:­²buEu ƒÑ©ú®GðuM$d)Qʺ_ à–nðï+mñGJW®XDÀBþF¤6âk½iòá<Õ‘NGo#þ_ òÁh ûC-¾f?V[‘›‚¡z|ík ,ñoaN¬ð׿'¦M‰Çk½¯>l-m—†7"¥Ô³4‘Õö@œ/{£K8lñχ—‡“/‰{É€­aVl‰ø((¸ Äóø@¾?\ ´ZÔHp£ËÉEŒ@GþpéÜ:bM C¼Òý-õ­Á@K‰Æ +¸nƒ¿.²4h!!ÛÜ4¼Âîµ<äolfÑhò‡q³ì”i#‡Œ?>PO蚸´åmëšu+W,eà ù#¡-åÁ6¢9Ãýpi-8WM ÙlÃÆŒp°n£?î9²>ŒDšpÁM>äDZk(¸yËB©4ù[#ë‘"£´úhÞn5>åthm.6 ¥6úYI»X.Ù8©÷ ¢Áçõ~Ø‘=sá}!V¿å?-(qcz8ƒõI”ù ±Ìêb‰õÓÑñHth¹ÖâÁÒEU1뀙š<°š`2Bsû ɾ-³µ2ÎÛ`"„ñNC ¥Þž·%jI5˘3ÜGæûý­v—E-ó!¬C¨ïG÷©5½!“°.Vg^[ ‰zêü âVÁ`f ?лyº»ê0n¯²‰q~ï×û“‰"ªÃßz ´ôõQÓÚZ›‚> ë¶ŽX¶´¶Er~_óôÅ=E ݤR/ nJ@ˆ[½LùÓé&ˆáÄî—¦†ŸN»{jà0ëñ…¥„!\Bº`piúb© 2!×—*ÒÅŸ ´²Ä·„a¤[ »I­ÆÏf™†mÙ£*(ðÜ‚äP‡ôŠ`ŠÑàkk’nÔζDÉ]¦õ:®}¸#‚ 0ª F`JSŠiaLÚPc¸Ÿ%'ƒ«”9¦+èX•6í5Túýõ~T®þ­!ÿ sælnöµ`SM¶ÙбڀÁ!ƒÆƒ‡œÝ¦ #S$ë 1˜% Ó­\+šÌWüO¦¡°3!ëÙÒÍ|–ý/&˜Q³lzÍŠé³ùê@c ²ƒÛg,++;˜T;Ц¯(ëJY?ÕlÇâ/ƒ-©ÌÊÿ«E H"j °•±º}MMÁMþúj§$ûHÔú ŒÊhíHCìÉB‰[-TÍRAޞõÜò5‡“v^RŒm•ýâ@XÍò“è6Á§­¥p›ÌJ‘,’1ù¤‹uñª.^ÓEPç¹ výïdê§ex’”õ[GaŽc¶,ïF²»_Ó¢±)£NÔ=( Õø7£yϲCöX¢B~âx_ ‰ôÞn]a%]ª´Å(rü2Dß׈¹½ÎàˆÞQ;Y[‘DV­Ni`:z¨E-¨-uþYÙT‰}öm´: W .Fny(U©öþ+—7Ú¡ÀÊ–p[kk0ñ×W´Ôë±s‚#óÛ8Lº›OWêâ2ò"2ÿ›QU} |69Ióþþ•”ÑNI±› .Ìûòˆ‚ËÕ<_Øßãðˆ¶‰¬£X_(4‹& <€?%½›ÅÃÞ£²7À{‰vß·äÒ}±\ƒ³“„âžO/#Å~¾¤Mî9½¶m‰'Rs䋞¹[ßtqƒÃzkMÐ~Ð"²ií«´ º®­±ª-ÒJÛÕ«{AŠž#Jóé묔”ÁÚ>`Ü»¹LìV·ÃÚʶ¦d]²ýUê |È/¦¨©Á–a3jR:;|9Vv˜šv÷šë̦7[NÁ¸Ï’–œ7'Ž3›[qEÞZÁ¢#Q–§ÔÉÀÇð;¯ŒHЂ`ím®°Ï•¨^cŸ"‘ÛXé$™ö–œ3dÂKÏ(0»XcÌ‹?ŸÀæÅXaóò¸3 l¥êœp|A³,ƒ8ÖI†Ä4væ’eá”päÒ/^™pJC{‰5‰Ç.än®?vî£ú¢ÅtG7Ñ¥Á^Ç\å]gôîPEFí.ËÖŒœ4nƒíÝz‹”îô'3å|ó8úO?›8rü8BôÖÿÑÿ·œ×Áz®Ûæ‰Ê’GÊ’Õ–H l«÷,™·­(~QÑF‹ ‰M¶Pxl¡¡v,¥Ñ#¥Q¶x,ÑJèå‰Ê‘GÊ‘l£t¡uc¬)¾_ÒÏLv2ÉïímãtH7w7”Ô¬÷{º¸O«uYb‹g“/Œœª †è|TòzeK=Úæ½Ó‰š'õÂPÇMß?*ÖEK<¿ÂüÄþr17wÞV€Œmš‰éO‘=ë®"uuƒllT«™ðK_·%âÏU_<ªž(öµu+‰à—™\'BìI ޾LnÚæ7ÈZMö;öŠÉg°×©öx÷¦4å¦ÂxÆÏ5yϬ(&ûKƒ-¦ø†o0Ùsìy“§yùz“ý…½Þ‡y&ÒßÒ:æoÙ)v"‘î |ŒÕcDDÔGáóÇÔ!Ø€täÄ¢ïÄ>éþÚ¥¦ø^ü`ŠÑŠýdÈîQÀd—³_˜ Ã6…+Âä³ðo i¥³-ßN<œç«÷ Pz½˜‰¦¢ˆPäG7Hã0sƒŒ™FíJ Å¢–ˆ¿Ñ2oÂä>ñÑö¯'y_n^_¶å¯8ÑÒ»7KLxÌèSf¼:äkEƒHS›º’a*ý”L4˜ñ}[=›0ŒÄèñ0ŽŒÝOv§´—ÛÑ&û‚}‰¡s·GÁ&ûŠ¡›Õ ³<Ï‘8ƒPF:j»gOq"£< h,ÚB~ ‡×a7Ç{(S™Í@,_YCÔèoòcøZå½ÀÆ6áѾ8”—Ûšê[Æ`d.ÏÇãæEo'£x )Ð÷ÐÅ ò$b ƒcz÷®Å¾Û’[g ­ÑDÀºxÍT)yˆÊiì€éá¢Ê`åS¢ Eœi3°xn#ÂÖ•a¦2³VÅ#>`PÐÓ„Õä‹ùSÉgwêÊS9Ti*£”Ñ&/àcMeŒR`*cÉ1¤ŠW)4•"zšj¡ïö xàoƒt'EëP×C››ežQ±†&$>6„M¥XAþ¸œ½>+Í¢"'^}È.,‰(¥¦2Ž\J?û¯Øº€h*ãÚ‹ŽAûb*©cF´£åÁ•IV?ç &‹*UEÕ‚8‘[Ø ÁM4 –?5.¶Ä–`Dš>”Õ­þúbO]FY-Ø&M‘µ+á™i²kÙu æ÷…è&j }.²,¦'ö¬ó“PGãËú2œ0H:°%J¹p÷©c™qĚú·j©î,ö`DÒ\ßÞ›˜Jön7E4ØM6§J­B]Y³ x*-b2†„=9¬Ä¨IZ$ÉOë(Ð2<:K` ¼t á‰Þìö×9LˆÊ@ôS=™–#ueŠ©L¥Í’Á±€Ì¶Y0yBÁf”V½Ä{T¸°Äk*ÓĺRf*Ói +DÓ;jÂ8S™¡Ì¤ìÌacGÉŸÁgêÊ,´ Æf‡ó…è¨+“‘ÁxêˆbèbéMeéQÁAeçP‡,ã\Ì­¥É ¡‘¢lXšºb™ß2}pP mo~ˆ®Ì3ù ^ݽWëÁ±úΘÚÄ‘^Ù6J¦RNáIžÇ‰ø›[¥í~@#éc*ó• y¥L¡G”£~_‚—ÇÑÒ†ÓPTïÆ@‹¯)ªV¦²@9¹ä2•…Ê"SAC‹F¬:f!=3‘·ý¤È¬ˆlõf+Œâ¬NI@ã]g?ùÖcQ#ye oÓ•¥¦²L©Ä„èˆâJÿ&¹?Ó§N}é£ë×c *z %å§f}À&Åwi}ˆ*»~ÄÖTR~ñÖk()¾†ëº~:×N§úÎ.;ã“óÙdݧÙv× +Q§›ô‘FÂ^|ô4ûÈT²‡õ)Zï‘‹ŒÃºïz€û_èº?Š’‡ ò ¿¢…HF'ÁV:‹~-å}ïnÄ“(µž`&};°ªn=Æïyòv¸e]€n„.íã­¤|ê“nˤ¤cù„ëh½P£![}ôYXÊE¯‘_U¶úiÅ=º£½ŠŸÔÿ€dõÓ³E ÷ò÷$°…}Öe]‚ZÞPܲ·î!É«m(×RÀ^Dw D#]R°¨;Úz»§­c|äœ:¦mþªštM«7·YS…ÁFC %^O’)V æôå1ç):Y!ÌÈ|a¹³ßhÝí³Dÿžƒ¢ƒŠh¬7¢¹¶”îý÷„t Aëúy1ArÊú¢£Ñ¯ÜèàжÃY´µE…øoïlõÂa¤Jz|z{ä$ÉÌÒgõ‚r•aºë‹à pÔ_Yx/•cDÍÚfÂŽ!k“4‘óãñ,_ï U?ZêüÓÔžÒvõæj%ƒ±U©%UŠOë{ip-ö9·çYbüÀxµ>{êv6ºS9õ`K‰îs&Þ·ŒFÞòˇQ]šu¡2+Y_ÂDÊžšá—G&ŽC$C.z‡wT÷ÉñŸýE6-@&ˤˆþñ¢ÔJÙƳ|èÃa0]ÄZÛcé¦;„² ³ˆ,Ûìòx»Üd—›ír ÛŠ%§ÛG²<‘m“í'ÙïO–%§+ˆ²Û†sÛ.Ësíò<»<_–œ. Éòçvû…ö¸‹ìç‹íò»¼”]†åá0‚ä±+ð©þ z„´Ü•ø«ÉÆ/ØUøkZØÕì—XŽ`פìJü]ÊÁײëR NOÌXŠÁœ.~Úƒÿý4,—Üê0\¬º3´ß@¿ZQX]«UתÅÕ»!sôçà­-S~ë-ÜiÞ¢]àòï‚ô<å±^¦b™§î½8Ãl˜‡`ç!•¨¥` JYÎÜPÀr¡˜ € l Laƒ`˃Ål°D× ¡d£Kµ‘³LÖn—µ›Q»EÊ (Øy êìVýJþÙ¡³Û€éìöï@ǵÞÁîLA®¬dr MA.ÆîbwÛƒgáÜ4»FȺÏí¢VÔ•X½œhRFö¾‡Ý‹¿Ky©¡#iy;Àí» ÛÛ 9µ» ü…E»aPe±;o7 .S¼ÅîC°² †`mh; Ûñ摵|¬5êw¨¬ÄÚ(Yµ1²VPìëö¶Ca™êuå©í€ øºD‚ÎB¢ä? Ó°þ$Lè„ÒÚN‡8wOh‡‰…í0É}Vhþ‰îÉôƒóOl‡)0u/z§í„iµXÙ eT¡ö˜Þ3vÁÌ0;̲;ÌŽvÈï€90;àÛyöÛò¸·ó; b,@¬‹Ý‡Ë%HÜz‹Ü‹´vX¼ –`uiVÙ•òa>Tº«d}9Ö&duV«Û¡f¬ÄÚª¬áÉæÕøp„»VÖ×à|k®…üøÅ‰‹¤æv8*ºÖ"÷Ñîc䀵øfmÒ€c­køúÏ‘Öá›uIê¬ëbv@N™F]òpyþ<­özÝý3¥fbßѨœX ~æ)ÕµѬºV'’U×X(O ‚b±z}umê!vC«Ôk¡U¥žkwì¦V•®±ªéX]fUM¬.±ªX]nUûauUÑHì‚1S’y¿T³–å脦Ú<¥šw¶µ]ן§X²d­í›ÅòÁPÏÿÅ¿ä_C‹mGž€ñ¨m/¢•ø¤Á‡ÃþˆÞç%Î^†1ìôM‚éìÏPÁ^ƒ*ö:Ô²7¡Ž½Í쯰‰½ §±¿Áùìïp5û<Í>€Ø¿á3ö)Ëd_°bö›Å~`‹Øl¶Ž ÖÌu¶‰ì îbq“]Á3Ø.žÅåÙì)žÃþÉóØçüö-‚øŽäƒ¹k‡òB>š¿Àçò×ù<\A®a1ÿš/àßòÃÅ@¾DZŠ`ð!Év²û@ãßÂ0¬ÝO&Y ŠZ¬iÙ?¬=ÀDÛ2œËv±v0†Æ:ж(°‰ßÈvcM•6ñ$ÈØOƒ5‰{töÎ:ñÀ˜éûàìqãæéì×ý÷ÃÔ”,[ŠÕßà¶gOÕKKè…µ}pJ•ÎÎö" Ž*]º·lœŒmË1^ÔÞaËŠž„añBÐjÙ˜¨=)Úé¸7¹¾It¸x-ŒâGJòy,p’ k¿•†Vƒ|öÖ¸$†ì{…èÅ¡ÅèZ¡…kC1BS ŸxÒ.»fwÂqJ!DM\؉ڿ¶hËñX PmSçÍ{(i—] sâAlqúõ†{«*uܳN@÷ P|.?ÕQ¿Ê´˜z Çjt’I4 †¥nîmp’·A\+:ñÇ} )Ö©ñ~8 àé1òŽFoBò6Ã<‚E< gñ-p ?îåÛà÷üx‘Ÿ*I> T"•ãÛ>aOØÞùö${Jz¾OØÓXSàôÓÏ 4Ɇ~ öÁ¼}Pþ=ÌgÆe’¿·ed­’dÛ+ðvÀ"Ò™1w-.¿ ¾<.dÈvÊfϲ?H¤²¤²-¤¦}Ž=oO[ƒ“R¯!^÷Y4­û¬ÂN8»ó9°ý8·λ;iþ›pþ›ãþgþ!ì"JÒ\/ÚsÍB©¥þ.kqíp~ ² Døî8¨.ªËZUÔS‘p´¡V#TÒ†¬xÂÙ|ŽGÍ&N}²œ ²²e±—Кò¤©^±§ª²§ÊˆJYQ\<Íë8ÍqÓd8Ód8"“‘bš?¡é¶¦ùj%ñµ ÝýϯF˶Óë¾P2çBZÜEpq­§¾DrèÒ•q qyT×~•b¬ bï+°8Q&ÊüœÎ’ÿa¤þü}Ðù¿!™ü#² @¸a™ÈvD¬ ÌNöª\N•³œ*‡jUörHÚuàiGë쵸µqºbl¯í}BŠ´bþv¸²‘½Ê}u;üýf1>\Ó×Vbj¼6Ú@N´}è²h 23äR¯#[ÉÐÎM2ÇMJ¥ÃÁù#F€GŒ„1¦/”‰˜!ÆÂQè(õ ”/r>Dñ…ÎâÚ‹›‚"ñ‚ ¾ ˜ÃÞ@¥¶‚ï|P~@£>g,AË¿ÒA‘^áMé,ÐÝ88™ûV”¹lRˆô|»×}=ñ´n@2”)¸Ò1—PmZ¤ç©¨ú7QËn¸™’Œ(=nÁÂ}+þP{-ÃŽ/*)ºPãMð¯â£ µv`$u}q'Ü&Åçv[|¤X¬€•'`*¢ 412Å,ȳa¸˜ƒ¢1)Y ÅBX!–ÀJ±V‹ÅÐ&–Ij.Ä¥åÂLöWrǰÚvêÄíí]·;B³ÝœyФ°@,¶J +’ÂÓ@üˆ²j¹Üý˜[©6}á1j¤,É><®¹ åI­ÞƨǢü|œ‚ÜÝ…^÷ñ”·©E¤–ÄZ-Y¡åi+4É »åΫÁ´j×F›Š¯Ý}>IîM´˜4‘˜d9'÷ÝøS¦¹šØê¾‡æð÷¦ HßsG³ Úî›î›a؂һF‹`œ8¦Š“`ž8–ˆÓö§C­8 6‹ àDñs8C\ìð—èÐûB‡Þ²w0Áç²F”'zŸ‚)&Qƒ0@|CCµ' ùg¹®,P~„©’ÿÀ_w†7A­ßeÿ´Õúqœ–]áuïDÚ’LZ"9ÂR×x‘´cZ’È(‘÷I‰¼?&‘bçv$r° ¯CÊÜ€”¹ÜâFÔí›`¼¸ÕÑãêèq…³þ ö/QÍ’·L˜lÉ›\¡ | Ò÷0 ¥'ApÞcïÛ‹kAÝ'¨#¼î¤~Pú˜NØUë-F´Û%Úí°û®D_ þ€?ç©G8x`Ø|Áþ5.k_°øˆ}lcq?Â$Þ–zÝ{${l,BÚå“zw"«Ñóýz˜eUò4¹/¡¡¥…CG¡>rÄÍ"êGˆâ§0 um„Ø£°,QX\ÄSê \êˆR)û„}*Q.•Á…5"¯õ~Ýn$¬ä3ö¹½’:ì@Cò½îßÈ•Üe¯äa){DN%4%3εæ;¸å;¸å§p­_°/í©·Ú®µÄÈGâƒoté„¢ò(²¶~+Qy,ÉÅ+…0P)Šck‰ƒG û-n%#.k/uaëWìk#rÖ2Üëþ¾tÂã’OtÀ“ÀSíðt2)")Å‘b¸ƒÂp‡û‚Ñ]t{âKQª ½É艳ÑÊa±`é ¼xæRÚ {ƨ-˜´>[ë-dðI’çbVk‰‰â]i€Le=ä(a˜Òäàfb¿oÙV‰ÑdË”uÙ&Å% xæ÷ä>¿K°.û0{±P½ÐŽ»¦tÂódW^Ø /.‹%ÃââŸ?ÚñO4(Ú /%åSÊ÷`¢Y¨ì‡ *‹£á‡†SlŒM(DQOȧ&$†ð#è›rÍ)HOZeš—ö'Úáå¤25 ÕŒÛkK³&L`Ð~v·í¯Î±#…;:á•ÚB÷]ð§e^÷Ÿ‹™ÚXÉÌ««wÀù•b²â-ØÜ™¬ìW™šÅ.›¬ä*Yì¢ÀÝ ¯×ºßÈUÜo"ßÚëÍUô_»j1Ð{ªªVÉ‚§«jÕ,x¦ªVË‚g«:à¯WÃäNxÛ±Ù³UÄýýü=úôKkl°ïî-–¶&†"9_ƒ‘˜lNÀ2ε©!] CÕéàQ|uŒT‡A©z(2b8LRGÀ4u$ÌUGÁrµ"ê8Ø¡NƒÛÕ’f³“(&R"ü39cÛ‘nùpçdm‰V ïàBjb:\ÏÒDÉÂI í'nGø71Œàê~ttNæÿ&60Ù¬ Ø«¾MH<8}°a³{¡½¥šA‘Ä“¨8íðÏÕ÷9²ÖR+@W€©.Œc»H„0.7àëQþCÄN— ¥¤ÿ+YÒß‹®=ð>*j|›]JºZ¯Ñê*ðª«ãr«B‡L…Ü 2Eq€Kw!8Â¥ÖÞÁHCcMYÏö˜NÉÌM ÅéQš=ÍÊ©ÆÓ9úZž~lég['z¼Îªámxg’Úðþ, N;öýfÃQ¡ÿ½ºXþ ‡»²>,Sò”=ð>Ü ŸÐþéÑŸ¶ÃgØô9íäµÃy´+·¼¾Ä`ú?÷ÃW{àkð ’óÛ2MLÖÅdÃKQÉw“0 ,& Øûä¥í†ï÷bBõCYš7×ÈKC5™ìÊu]ygQž’ktÀeéyé{‹)†Ü_–®LÖóÒË0Rlg€Ó°ÙfƒC¾e3 i¹z;ãyæÎ2»03{Ë2z9wª½êCð ëÛê.xƒ“÷1ùÛ¾P;¥¨\‰¡ “+Ê+ÓàZܵtÄþTžzmÀép4Ïá¹øö¨æ0xuÁm0•䃰ß;2m{Už­8Îk¶ÿÅš•¶aMŠž€–Îóø`¤ÐPÆù!| ]-<Á‡òa(\GÁc|8ΡK%A¶æòà\Œ~Œ|;E©4 -j:~¥ó:?Tç#u>Š‘a¯¹lÍöÃl0Rw”ý0‹¼ý€º0ÿð±–aàw º˜ã²íÙLÑå–Y‘”ŠêZ¥°ºV*·§µ¢= í˜Ü\Ë Y+¦Ä.8‘a4ÂI¸±f›`¶]J?—5Ò0·À»¼uH±ìX^Bi{Ì@Ëþù¶ßœ®Ð£ðN+ó”R>ÈK¿þ~ý½ö£Ç‰n•óCI e*nËï88>8wvì é'rÉ™t²aµ´‘ƆS`ÉÑÙrT¬ÍÄN–Ót°Éðg'l9ZgéXã“Øƒ žßEÙÚP·£ŠÊü£“ZëÍf#³Ù¨6£çþË(ÙÍfE”Ûî”(b&ËÆÄGËZdj+À­-‡líg0\«v™jº“O‘èäËØSŽçSù4ÛKËŒ•—a«óéq›Liô– ¹JLæf³±»™—¶4®\V„Ê5»áæÝ¬peèˆcóÎʸì<›‘2[>¥˜V€30À‘˜‰kÇ‚¡­ƒ ÍyZŒÐáPm½“jgâD¢92Y;M²,&Ê]K°r-ã È¢g´¯·,}ïo/ë"››¥HóâvVrŒÆÚ¨Ý¬t –ã²Ùøv6a ra©Å…‰…r‡¸à¢°Ôá‚ÜøÔ"àÒŽG.l‚¡ÚVðh›!_ÛÅÚ6G$òmnXIïÄh‚{,òÀŒÃ“Ó?Ú`ã9Ó>/O+¤Ðp7›”,tçÄ ]|´”¸ÏM@çð¹6Ð]ööUQQeñÓ1YÍUs•› ·8WP¦R¨pØ*Û±ÿÌbÑ®u°É±xÐ’»Ka€v Ár„ö (Ô®pöŒ€Æç¡ñ Ó/—ûC áóeˆy¼‚m—E¾Eh HÚÅmgÿ]"<†É& Š.ݧíŬ±3º´áS¤€f³©EílZ;+‹eM2„ÖnFa»åÖ8K㠑㠑ñSc–@θ˜/±Hö$.‡Ðß×ɦ×v°Ë0ªZù ÁIgf³YLžÉ¯Ä¿«;Øœv6W¥y³†f±‹¯×P™I­+|”•_GRÊ|5ÔÄܪ7˜g³ùäâö° òˆGe³ñÇSÊ.È.|„•“Û 4qA ;;$ìýh‰€­Jµ=°Dû ¬Ò†mÚoá4í18G{.מ€k´§áfíY¸C{Õ^†gµWàÏÚ«ð†öüKû |®½ßjo9ÛFç 3–òeÒísXµO: Ú¯ÿ‚Wò*”´RxÖrEqyÁ>év,[çã{(Ðùòï‘㳇̬ÅL²í[~†Wðj›Í'à;t€pcŒˆ16s “Âç=l)=NÊfËì= fVbk%fÈ¡‡'«b²–«åbà;£|mB™^LñG…gh¨lïâDÅÖVCa;[>+å9e6ûjÝr°‚¯3¼hœ@ûÒµ`ºö ÔjŸB–ísi_Âfí?pºö\¯}ãøÿZ´‹¤®:”Ãh©® ªq¾TW–À0©®L‡)¼ãÒ™FÜh3‚j#6ÃØ.Œ¸1Ž#AÙ`uòð·.§€Tàú±©ˆuaÍJÄr_mçóíímõ Ôù¤RWãRdƒQ>ÄaC-µ±½VmϱFÎq¤=Gƒ3gzŸ³“U×¢ÆÔìõÆ4iRt(z:dèqv8ÓšRÖŽ¢$"nòL{r;KÎè‚ÅÑàÃ×Ú÷¥v"@2Õ£“Lõ@4Õ%“µCrµ@Uî’Æú=qW¢‘ÖÁ=†èƒaÖGêC㌴Á•|å0‚—cnDFz˜c¤qŒôh‡×£å. “5ŽàÑ3i2Ò˜—°œop£ÇÆ~žÍ§AŽð®´nél´T$ə釯ñ-z{ƒñº”WÏ”äÁ…©®žñú”ƒG&ž˜r°?åà!Ƀ§§ÜrpAòàò”ƒùúƒÞ•4xYÊÁ¾Á<Îf]\ZsfÒ¹¾2n´æŒÞÈ›RL½6yê£SNÝœrðºäÁþ”ƒ[R^<8rp0%¹W&“»9åàÖ”3/Ižù¸”ƒK9xYòà¶”ƒC)-’oI98œræɃ·¥ámv5ÁLöÚÙªdšoN­ñ¶±:¾¿Á7ñÍ6œ ;Óåuã¾D0pL\Ô¥ó-¶UÑùV;êÒù ñù«Fà'òm6ð²ØÍQ t’ïøkÔ­ñ­òhÀñ&ñ$~r ši©Ú4³âàSø©öà!ò ‚E×LcYüXìy?½FëÓœÁÏìP`ðYül{ðr;w3˜ƒñÔ ±ž ‡h-d;ÊìËÂ?‡. #ôíü\›!1Ô:€¥€Z\lðóØ6’‡Ééò²Yþn¶úŠtNÆŒë7ÊЩÀi÷9Ô•G°ÝÓz”½ó¥^ÀÖ`yäÀ³,ŸÕ"Gby”âbGcy –k±<K–밬òËr,ý˜¨7`ÙˆÏë± `¹ËXë¿ÇíòQ»|Ø.²Ë]vyŸ]Þf—ÛåÙVñPK kL[D\VH1³Dorg/apache/hadoop/fs/swift/snative/StrictBufferedFSInputStream.class¥R]kA=“Äl²&6MMýª&ý°Ùn´ ¾VŠ-¤Äöa¥ÐÇéf’ŒMw–ÝMýø>ú'! `”xgB…ZwaæÎ;çœ9sþúúÀ<²qu –md°R„U k<´‘ǺÞoZp,l0äŸÊP¦Û ŽÓUñÀã†ÂòžR‘×O¼¿FãÔOcÁO¶: ¹¶ê †¹® ÅÞøäHįùш2Õ® øè€ÇR¯§É\:” óËá“·²ŸzIÈSy*)RÃvÊ~ʃãW<š µ_¾ D”JRí«qˆ©wWèÚ|ÃOy %”tìIåuögX ó{b`®× j©sºÜ-a-†WØ"f‚<_'Ú* E í‘JDïIù\M#0S‚[¸Í°ý¾3¸—ü¥¸æt # l8ØÒ¶×º3o.¼–¡»Q0m"7hÕA–~Àr[K°Ï¦`ŽFÝÑÀ õñ**5ÎË0ª±ÈÕ›¢£ ,Ù@³†}œ©oÖ‹?¡ü ™Ãjv‚Ü™û×~èDþ°jMP8së5mÖÐÞ%B:F£C´.A¶Hûc¬cÓÈX4²š3MÜ¡3 ÷(Î!S©`ÉD̺o®óà7PK kL[DµXÑ)Ú8org/apache/hadoop/fs/swift/snative/SwiftFileStatus.class•V[oEþ&¾¬í¬ãÜjcÒC€: !”«›R’6mŒs‡¤ -amoìMlïjw“ª þOý©ðâH ñÆ ? qÎìjãæb©‰4ç:ß|gΙ•ÿýïÏ¿,à( ÜååK^–xYNàuÜãeYÁýNXÁƒb•ÍR _ÅPæŒ5¶×cØ`¹Ã×,¿‰¡ÂÁ­¾e¹Ãû)ØU°'½mt ÷Ž@(?³-^6ëº@ªltôõãvU··´j‹<£e³¦µ¶5Û`Ûw†Ý¦áÜ*›v£ YZ­©šZÝ4­ÂSpžnÁéh®q¢*l­-½âjî±S˜Î—öVK¥Ë7ojn³ÈŒ¢-½Óp›¢$1œºa“NÌGª-³v´oëV˨Ñf‡ü«qéwŒ‰àHÛ¬~xß5ÚLÚÒn¢Ï¹'·+ÈYºÝ6‡0 +Îf`ˇډVhiF¡âÚF§q™§¹ƒZ­¦;ŽÏ5qv’Àü«‘¡Û2Ÿvt›{w¶ylÉ+½ÇWJý§KMRwjGkšåw8j8Ü3â%ÓôškÚÏb®é Œçg.ƒpªÙ ‘¥c£U×mJHTÌc»¦{èãç¦cž÷©¸ŽI^¦y™Wñ>Vñ d®ÀUðDÅ÷ØWðƒ EUÔˆÌO9*”û¾¨¢ÝCÅ\OU‹ì¥‚âÅœ7rÒA“¢sÁH‘Ï`ãÅÜ…Ñ¢Ø!ÇÄÏ*>•¸ðêÏB`êÒM½цî–õŽl½‰á³»Ø¨R-ÔÊXni%½Ôé$f©s.jZß ëC’‚ddöâR0Ek‚$H #Gÿ€ê%Ì[$`ÚxF¹ ’©Ù©Ékát$­dã§8õ.†pCB=$”îC±ö6Þ‘ð)ʺÔò˜¡ÃY›¥hDjs¤y{ßÃMÄ%yŸÆoäM‘Ì4Òñì`VÍ&³C§§óÒT:Ó±0ÈÛ:™€N& “ èd:™€NÒâR{Ÿ=¼ÂW¥v‹´¤Ô˜öaÓk÷i— •ÏJÓõ‡ŸCí"ÿå9”ð¯‡^P$$YGeÖbO/Ò>ãOd44ºËü4@^ò›$ÈèU€z“=€ J¼Ï¼B0(„÷"¸NgãÒ!¡/…¿ù:•¯½ý”ÝSÄÊs„ï"ÑÅàÎܨê‰$ òu‘"kس¢]Œ5êY‘.ÆÈ÷¬kž•ö¬Œg½&ÁºÈþœ•ºú¥€qlÓCØ¡~>¢Rwq{¨Ð§WÃc4ñD’öÈú…Ä ã6Ý=×Gb~ñ?PK kL[DO"1{¯g;>org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.class­Z `å~oÌìdÂB`‘cAŽA• h8¢ÐpÄâ’Ý$+›Ý¸ˆV­­VÛzT¥¶^­–¶bE+ 4Uì%­­ÕÞ‡½ìݪÕÚCkµô{ÿÌÎΆMLÄVfÿ™ùÿ÷ï~ïŸ<ù¿/=FD <†A×ñIº–è|²Ü,4p³HãS *±Þœªói/æ:—è|ºA¥|†\–ÊåL™q–\êKy/×x…Acø$WÊoƒ\ÎÖùùm”«„îBWëÜ$[­Ñ¹Y­•Ë:yr®Áçq‹Æë5Þ óFƒf`+ÞÄçkÜ*¿›eæo1hŸdð…ü.¹l•Ç•r˜·É¥MçˆÆQÛuîйӠÙ“Ëe{)ǹK. “ÁÝr¹Df¥tN ŽŒLÌ “;äv§\.•Ë./ÓùrßmÐB¾Â søJ¯Òù=:_­ó{Èût¾Fçk5~¿Î×´ž¯×ù‚þƒ:H&Ü ñ:ßdÐ|½½Yãë|‹A[­û[5¾Mĺ۠6þˆ<ù¨,û˜Æ—›=‚ëvÁÜhP'BÞ}ÒOê>’wh|§Îwt _¯ñÝߣñ§ ÚÁŸ6(Ë÷”áûdÕgdò Bn¯\>+Ï>'Ï>/„ï—í÷Éè¹|A.j¼ßà‡øa¿ÈäoÙÔØ°ž©¼éâðŽpm<œè¨mɤb‰Ž%LeË“‰t&œÈl dzQ&oÓÚ³™BMÉTGm¸;ÜÖ­mKvuaRm<ÙÑEµMIYhìL¦¶ãvE,Åt‚{~g8’Lv×¶§k×…3˜êͦbL#¬íÑLí†óñØŸÎ$SØòÌâ‹Ó;cí™Út"œ‰íˆÖ¶È]³7ÄâÑ–]éL´«E€RÉé±D,³;UÍÞÈä[žŒ€îȦX"ÚœíÚM­o‹GEɶp|c8“{û¡/ÓK3Õ½m°¬ê8yØzG4Ób‰¤¾jöq % ä°º ô*@¯ˆò [,]†9§V*¨€¶d¢½ÓëȦ°m2¡€ûÛÓJÃ>yÏ4sh+Áp6M%°¬%nÛ¾&Ü­”‚°p+/m‹vËÔ´Æ@Éàgƒl3*ÇLÞ”ôLÒbŠi ¦mrL3Úiìbš4@m“.¶jòjU‹„soD¬¿ [*ùg™,Lié ËŠ¿Ë¯_7f2™áméd<›‰®S·ÂÙŠh{8Ï,‹'Û¶·(µÁäW½RL`(Æ×Ê’XZv}«Å›¡ŠvwS‡À€èoŠ¥¤XR½jNf’ÙDÄQê L!e!‘0•ÆÒ.Áµ%ªÐŠÓ*3`Z5R׿«V; ø¤b"åp*5iÚÓ 5&0—™<1 |–ÙŽ¹ ˜[ù:“éŒ2Êî$‚&øÑãö¶pEkE6‹×6f¢p†dJÖ´+U &†SXËP™{ÂÚã±tfm»# H!ÏKd”‹\fJ\lxåøeøLeaäÛÕ‹~¡~ëO/t­¥K4ú¦Æ==¥q/l¦k;LD›³™â¯º£©®X: ܵ éuβ0å'0Í %’7¿ø@Î=†¹1/±4ìô¼d2£¼ÃøËðžtg2,OEÃŽ•_$SmÑ5Ö{C•s’3u©¡X\I¸»;šˆ0Å#ÕXäbw]*Ù‘Š¦Ó¢åcNËŠp&¼6›éÎ8ø’Tfl˶·GSVˆÐ»m2L3†´p·Ùºã³Í-«ÞqFKºÃ©h $wDS;S1]šŠvÇc9_å¼Ý抵É,æ`E¦æ‚}5>¨ñ!¦q–˜Ô4÷¦Ö·‘=†¶·¸¢‹ð²zBN²[Làö5¨DnâfÞr„|G\éæìA÷Y5ŒJ 7~ßÄœœ¶r0IFÒÃæÚî¨Uµ4„ᣮ$›H§Ú¤HgÐ[$Š%M ÛfA@¼Ê¦ÒÐd×Þ­·Ó=ÓioÃìJ¦©Ó+ÑQ)Ö.Ï•Mî\h‰uÀj²Rt.?f –®f­Èæ7J«:ZÂàyáî‚è¬Áe1¤p˜ˆîŒJV=á¼l"ëŠ6&vÄÒ18}}"‘Ì„í:±I[—ŠíPÈlL$¢©åñp:-ù°7mò2Ön{|mc¹=Ü­ÏFbÑD[tºM * ØupŸ˜ÚjâEFÜgÎ@%ɱ1g6ª3Yàìà X¾t" 6&H]_ÜÙ:†æøBIÙΔd¨HnçJ§Üë7q”[nV(ŽÌP”ºuW&šú¨Ü“ Ýñd8EšÔOo‹Û] Ñ’Ì"/[Åí„¢MÒ<kÒm´Û¤[èV)¼dšIwÑÝ&™e:ã¸0“n&„ƒ€´8ó$Ziü˜É‡ùq¦à€¾µ²Àø+rùªÉ_㯛t#ÝdÒ‡è¿aò|ù£ݹ,‹G¢¨C`ÄvS‡§¡¢CáŽp m%^ȤWø›rùœ7NDBvgBYƒ·÷Ó&?É߆ë)b.2ùö ­Þ†\¯MÃL³ú-›W¤× e’˜þ~ÊäïRIÑÃ?mÒ×ù4oƒ6¨àó²hŒÇ£áx}ª#Û…äîštZ3^ˆBªÄ%S!Ô×!$D(^/ŠvèSLþõ`Ÿ·n+Lþ>=`òø‡hòÁ´>• ï’ˆªñLþ1ÿÄäŸòÏLþ9?‹âÚä_ð/Mþ•èuBs2ä”÷¡T¡9„Ph†`öý•ŽMj#VK87…Ú2O»Cˆ[mȈ\¼NlO$w&Ĭ~mòoø9“Ë¿CUoòïùˆÉà?"<¹«“ÿ$ºÿ3.Xô¦ñý5iuu°’©Ïä¿òó&¿Ð_qK/у&=OˆÉg #×BÁŽY¸õWj 'feBjÏP:†èŠeBÑK!Ö´²ÛX:„T ‡œV .4#] ¶µÛ.Æ“_ä¿ O/!¸¤·Çº»ÅÜá*D²Ù¡pÚM:.• Ú sMØñ‹Y3³à–ƒ™ü2=ˆfª¿|ò-h°*¬ì?%Wæ7 CDélww2•‰F¢*{»5 9ïëBÖU³U<)À*DMþ»˜_íZ»>£±L'#œE»º3»BÒ¢Šß¸lò(Í´tà%ЭT’u.“§¶ÀÅœ~+V³ü¨Ð`´ª?ƒÔ qs‹cÄ{ž†˜ŠL~…ÿ%Bü·IŸ OšüªÄ„º·ŸDM~ÿcòëôy¡ú_“ߟôs€©~œœª¹ø®\Ž €t%ÁªU™Ãi$ûÎZ¡=¸VÞ‡Ú2Ë}XcKÅÃŽk­HäM¹üO.GM6$ž#—¹¢™ÓÞnþÖ<~ÓSâÑPÃ\—ª™¹ÔÔ¸Ö%úAÖÙñˆÂBÕù±§B¨“†Ó%38„YóèEv¨…0êk¤ïåê wv±2n€3bwê²eW*…a Mh*ƒà:³êØ£´¢‡Ö³ŠL,²{Ž-FÏˤÙ:7‹Â¥=&•KÃ[Ÿ|­÷š<øW¨=–^Ý–íX™]DÔé-:Á¹CâÝ®ð–ô›o¥´Açû#²i?äÖA £¥gT·k¢™pq€éÜw梠‰s ­@TýVÝ¥•ì0‚ÇLë£_¥|%h’ASm‡:FWY8»mu(X?h3:´n´vHÈ-‘Z ôöT²Ën9nsx#锣—À´Óh{³[ÛMΩôÜÁµ}̹AI8©ÇÑÌT¹^.O¢ünËäuµXz¥òE¤ÈAùЭZ¥>é¥s;ÐCë1'BVœY¸ÎÓµÎpº9ziFŽêÕOá'0[–™œ£êd1Îúl¦3‰êk†ÙÂç…<Mcq X=°:n,A%ØΨӫA7б{þ '_E£ÞH«ìq}eñ«o.(dÓÎ1‘æ Hÿì:6š3ˆ;¨@>=?ÀVV?£íSƒ´}š(z²²NUÑOyE?0ú3IõÅðò!&ªãüæZÄ8”à%ê~ðx?GšƒŠ‚W5àaŒûÔµQ¦ŽÏ+×ýN^½í£Øò:?ú”£M…GKãs®{ìáÒH»e_gŸk3Í<7„mãW‰BròÉn7T‡……a·øŸ?,~ 5PÂ+ª6šJ×ÒuDä§ ]O X î<ä¥ r¤Æ7ÒMê÷fú°ú½…nÅoæÞF»qýîNÆ/ ê^â‡Ô´âjà—¨™|´Ž>†‘iM¢Óü Ûm ˜)s !P]s€<»¥ºfN/ùªçôXŽÐˆê>*ií% ü “zyà ÍxZÚZnÎí¥2ÌÓåã=4òz˜F¢rÆ.s1ì£1X_Q>¶‡*en+oGaÜCÁC4ÁC²¼ÆÖ'`UùDõô!XYøDI½„J)E唦I”¥i´šÝI+iTvþ½Ÿ®„æ¯RLW[Œ9b¿…îÁ;V£Oѧ!€*ŠÑ½°$/݇§òLzƒ¶hôÚ«þ$‰é³ø÷9ú|éý}]QÝï訳åݸœT&Y’ðX’èOî¹q¹}ô@,£ú/þhQ,_ íÅ»À±˜äüêêšš\ ¸É.ÅMÄ—¦ò:)W²½2»‹*`LµhÞÔç;2ŸOûᑬäë%ÔÑÖíѹÊy‰F*kD°PžË%÷«õ5Ý¡>’¾¨¨—Ç#y w€zl~ز×-yO} Ÿ¼ºä¥;ò:@½öúÓm‡hý×\¼ë:toRj:Jž!Qúæ”,>ÝÜ}‰úlš·Úb¬1N[3ç x;i¾½äóî[ãݧè–Óš^ »ïbǧñôèï{Ž¿”P}™Å ü{Œ+$•’J›'Ñçò¾A£5zü,>*lhô¾J¬An¤_wî°‘ŽÈ#uðíB„?šŸãé³ÐÌ/\ý6BÓ…p„ƒp„ !dºº?šoÐï|P(ƒ¢¯¨9B~Þ ìþ•¾ÀîORYÍh-ŸÙK³iUh÷¤{¥ÙûEmxRíõõPM]I̭ͣӂ%ùî¥ÍÁ’:=¨?vJÀ{Š1Öø­êcu¥2‚¥XyæÉ¸ÆöPí*ÙK゚V3zhþ!:ÉC›®1xïѾ½4*·+Þ`+í-OÝâòÔ“-O­v9«ïQZØê-_ÔR§ÛÃSZêöðÔ–:ÃÙ­Þ>\‚z04Âj/Ö²_¶XÌòB°“ Wr[ÇTW,=DKÆlÎô8Qgc ÙCK[„oÙ¢ØøAánö^gµìWª¥_F“q} w/Cqÿ€iþaþUšI¯Q ý»2-g­b/EØGWs ]ÃpÜßÃ¥ô†êy$ý˜ÇÐsÈk/ðXz•+y4ŸÀ“x"ÏÆuOæÕ<…7pˆ·òTÞÆÓ8Ê'*ãêÙEh¡2®RÔëèÜQ§ZMߢ'b6Ò9ômŒ zí;ôÞ¾@3àJOáí«pˆ§12x5"ö3H5&/¼oýJÅÌr&ŠÑ÷±«Ñè‡bÀý#¸\N?ƾ%ൎ~B?%'ÑEpìÆ³i=\»ñDÏg–_ء˚ÿKÒ”é·PY‰(; Ÿ (Ç|ÜçÓèWýWH’Þ ¹xø:yüó¿´í(HFÿ©¿Ápèçˆ^GŒwûÑoéwv ± ^«áÍÕÅ“H}¿$¢‚}sÎs|Õ¨–í¡Š Ï;÷-÷ÒÜZѼ÷è_ƒ>ËÒüA¿miSe¤,­´®¤:XÒK+÷P@ ö}ַϱ%U2ð*åÓi&/¥9\O‹ô ^A Ü@Í¼Šº §ÜLïæut¯W6péBZù=BH ª·*¥=ùå«sµŒeôú#„0õÞŸðÌK‹¨‰þ ù­XOþ‚'÷©L^2óM0 鎃tˆvúëyË#_Šì¼³ÆÎˆ—½t¶Ô,¬rŸïÊî8øˆýä1ÏbLxÞ!ÝD¬”A+Ž @y‹‹r™C¹¬r©¢ü‚ä¯Ä¯„ïqvVGY¹lé¤=Hy5LÊïQ)‚å*áN¸JŒFñÅ4–ã®Úmœ³ç8'ÓV¢–è_J/ÒßœÚߤ±+ˆòÅ*Ä]é±M‰rù½ûÜé2ØGç´Ö”7"J­jõÎi9H«{©épÞ¢OtYôۢ˛1È™µwé^Òê|¾¥“$Ÿ´™.tò™eŽWÀ¯Bxx8½–Æñu4—o ÓøF:Ÿo¦Í|]È·ÐV¾•¼[Ia! oµ³œÈ£Í‘G›’zC8º• 7CyyL!¿ñMƒ/WÅRwn~S ±)½„ÂÐÒÝva¸°8ãksŒ¯s1n'§sû‡|7¼ïªá{éd¾ÏeH &º ˜j âËËôwÔeÂ~/çõ/ÒNˆé©½ßA¥:3ô{¥ü0bÂ#.D‹D‹l3³Í,@ô ’’…è"¼}TH»Ò¢ví£õ­åziãa*¨^ùQhü1—MW8›UØ•ŠŒþ‰PîQ£Ñ¿Aݽï«H€–K‚^è€ì»IíËuÅlQ4`Ïu½äGv*è’=»‚·æl¤ÙñÔíOcÐXÏv‘°önäÿšýrt%§'(®è—&<ÒÉÙo¢ÐÜänôd#>A Ý‘ O´ÖrHÔŠ&hš |™]hL¯j†ïÉËÝŠ-G«°Ç}¤É'òt’*ÐÏ3ø¸€¶!Ž`K?O»‡®nªyœ:ûÇ­;Aû®hW¡ña`­FÃê)ÄúÞúÑ»ôö +ªi¿¯?탠}h´çLûšþ´Ÿí#C¦­ó<ç\{šÒ7VŒ¡kÒû¿H£¥$c×éÒbY­2µùrQÍ&®”§`ù²¨=q9 ‘k3CZòÎ0Ó¡i:#Ôjê=—án¼C&ò˜¢&L3Õ%•Õh¥ÒÑûÔC_Cd&iJ L¦ÎÀ’é3(ÉèùF*#ô¿9k%Y—29Ã}È‘¥B¶Ü|¥Ó½´N'¿CIý„ªà½»r—þ¿[QçÆÕszÇN/Hl¶‰–Ã{€,5µ@¤ôàiL¤Ð­{xŒIº;>â: 5š£`ž´‚yOÉ)’L]@KÓwŽ¾ÓŽŒ¦š/Á"xŠgX1’¨+oÎ`6XãHЪ"‰… Í¡à.þPK kL[DŽRà#¥,_hCorg/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.class½{ `TÅùø÷ͼì{»y9È –;ä $àPN ’€ ‡à¹$›d%Ù»$j½ªÖ³õ¼cÕV”š@©WÛÚÓ¶µ­µö§=í]m-šÿ÷Í{ûò’lb í¿Å·óæÍ|óÝÇÌäå?ÿTÉG}ð”мðÈ0„Gº. Ÿð ŸOd “û³|"[äp+Wã|)4Cäño>? øQ˜êðà"~ŒçÇ~ø}4w¢.&ñËd\#Š}°YúÄ1ÕlÓø1]3xìLÄ,~ÌæÇþV¹º(õÁQfˆrLè—ùàq9cB«}\\áWŠ«tñ ]\í׈ky­2þrƒ½žsƒ.ŸäÇü¸I7³¬«x¡[2ŧĭº¸o×ñwÞÁ«ßÉÞýüz—.îöÁmL÷âî¹×ûøå>~ÜoàÓ†xÀ2è…ºxÈûà>ñí1Ä£ÜÿiCcˆÏâIÆùVž±ßOéâiy€ß?gˆg˜ò^Cô1N qÈŸgÙ6ÄxȳLúsÜó¼!^0Ä‹†ø¢!¾Ä_6ÄWxÈK†ø*¿Í_Ï/‹ LÇ7 ñMC|‹5ëC|Ûß1ÄwYº¯ðã{üø>?~ÀxþP¯âG†ø1cñÖ¥×|â§âg>ñsñ:?~Áh¼Á~©‹7 ñ+~ÿ?^Ýïo‰·uñk]üÆo±2¿ÍRx[ü–ñø!~ïïðØ?âO†ø37ÿbˆ¿2þfˆ¿â†x×ï⟆ø—!Þ7Ä¿ q„Qø@úD¿úgÖF£áøÊöP"N dnn¨={ãòM›V7Ô#ø×ŸÚªìJFÚ+ãáÖðîÊ¡d2Ö èΨF"j\@P• fVèM±hK%I¤%ÒÚ'˜±h cŸÑ’  $¬&÷'„ÙcAx¬ÞÝîä—„.ÁHÆ, @((™›N/ÌÖprE{¬igBáN\\GëSçÆP<É}§¬ vò ®––p¼Ñî¡q„g.õoj‹Ç’Éöðªp{¨›0èêl…š™EÇ—¤ç,im[…M$VYíìJBáPGÍ:%ÄNúŽ0i”Éd‘i…i¡‘z´‡£­ ‰¬ìܘ:„…£áW;2‚¾Nšn©A&FøŠŸÞãøk„…À+ECĉ¬Æd¨i'©­Ò<ŠfD^L†ëBÑH ÙÄšXœ‘Ú¬PD˜2j6ɶ”pÇ‘ 6ì87Ü”¬ 'CÍ¡dáÔQ!ŒY啲'CÉ®D%pkµáPs8NƳz´e¶Ï==‡aoФ|@¥‡% ŸOl ~n'âbj‹Ÿ£:å ¢œ)­at¬Žz$AN´´"Þ$`ñDر8fO"ÒÑI¦@b%qv„ˆšÉ®ñC¾²Á¶)úë•’x¬—!ö>Ó´P<>aΘ¹¬‘BÒ¡‡Þ–’aF$±*Bëšv’u±æHK$L:˜Û‹Ç‰§ŠÌ9J'u\KñŠRJ?)ï£,€R`!ROç£ê(ôÃë¨Bý1«¶ŠCké±"ÖÜMÞ3iÅ¡B—kt ”‚_;¹NWØ­¥p"¯Ï‘ÛTŽ—8˜ë²>’`˜Eƒ#Nwg*êÌ2vÉp¿¼¬†SC)(£%äˆrRš‚·:ÚÜ‹D“´fÅè\†´3ÒNø$ÉOm±D²|I"üúRLòe®×¬Yä¾8´¦ŽN"ÒÒ“ ËhHõàIi(võÔ’TZIæÄ)‰Í‘V2É®8±nùÑœ–¯VpktqbNÉ¨ØØ=î2âįùöoYQœ^òOOVEœ¾cDÇ]©hÁµ·§º¸Ýåon¥ËdÑ,onއ ª9ÈP €5yv\Ó†õ£1jݺc±‘ìÝÉp­Â‘£gV;1™ü™d,Þ=r$Íþ·§ÑÓ©Nàeï[£ÐØmvshZú…9ëÕ£!vÒ$ ¥Úè.JÞš)7í¤¾A>=;6$®lÁÓP-„à˜ç[ŒÅP<‡É3XOœ¾‚3G•p™bçòx<Ômó4»)ÖÞNÀˆ&öDóM”-·…º•ç’z%bÑÊŽPge’PféÏZ<&ÂâtK-ùHɰ ®f©ÇÌö”!]Ge6UÃ:Þ K*õ)F]ÐößPȱQd²\»v0ÄGø©íC¢Þ ì%ÃêŠ'ˆt?À|–r‘ ÇAÑœÒ~ 9V>ê2肈ªE‡v”ÕÂVNX­Šqƒ+±ó&cÖj{HO» .3ˆkɘ{¹$9ÞëèXz ÆéÌæp{8Ne*£§ñÛ9–t4«ÔËBdõn›864OÜ®7Nµò½²0š)·Xækj‹´7¯$ÿK´øÈí«Â¿CéìnŒuÅ›ÂVGÖ@‡ªasø}Uر "1Õc½:"Q›÷¾-§ŒÊZÍ-@O"Þ¤`çvÆÃ-‘Ýí:mä$}£g3{¦\Å „¢›báöR5úœ’‰éK½d҇ߞIóÞxåøjwÃÛÌx[m %)/¿ÔZù³Å.¦ aÍ®MX…ð‚ct*±V—ÝmO—¨­–¸ó,Ne)#UB×;(Ûµ®FÂñ3éF#,JÓ=6Š>ö?"a,««Ö9K‘N sƒ¤ªüaS[8ºjù†Þ É7úxùÔ¢–%u55…ÃÍ\†å“¬R‰Qø=‡iÇzKW{ûfÞÖ+äÀ»£ÉÐîäG¯Žì˜G£hô*…)N¼÷6f±¬±âV³"h¦[9(þµ†Ú—Ç[»:ÈnÝIÒÊÚ¹ v::â´P<Ýîjõ©F9ÖDW{R—]¼‰°á?Ã:MÜ0’ö>šSO9µP<ÞÕIô¹ñÏJºwÝtñkÊ ØYv†­­@VzZ'ÒDt敤aÚÊa£æ`jÔª®à–9{¬`G)ûèÛ-™tk(©³—dï¶”8SµS.úXÒ"3´#kïJÚaÇ›0 3”ع&¯·7’¼Nš¬ð`,ij·7Š}V²óÔ‘·uç1LÌÆ ôRÕ‘ŒuÆÈ%wÏ‹RÚŠŸ6åT ³eÀÆrÛÎÐ'§ÊÅT©JĵaËSfPîwZ…°"µßjÃZ„'š¸·’/µê$½¡¾€ê ðÜ™¦œ…ÅN«à9¡T9µ†ÊIL9[Î!»tg ›lÙvi]±^mQ›²‹u9×”¥²Œ,|}(AÈ:{qGÞF$¬^½º<Ш«« tÓÿmmÁŽŽ`"¸à‚ LY.+t9Ï”•8•ðiÿ’’Σ¨˜]ÓrׄˆcÍd,ÐÉ{ŽÓå|SV‘*áëø „ª£/šMœÂÓ³åÂzØw…µ_lÊãq)Êj›1lÊEr±.…)OASÖÈ%„ßPÛ0уº.—šr™ ’oh Å¥î ´ÄcÔÆ^ŒãDyy·¿6årI%w~š-CBg<¶»;À~‚G¯4ñÖÞ'x9?MÙL :л6”`Ó7å*žs1^bâyHvµZÎÑ%Ñz²\«KiÊZ¹Žâçtt5s>ó÷]®7e¬7å¹Q—§š²A6šrëwñ¨ÛUäA¤O2R¬kç ¬@ÈÞÁ’›M¹En5åir›)·ËÓ™#gðãLÖ_™ò,y6E_F­ÒIw‰wÉ@‹emò<`ÊþƒâÖaSÇ”;Ø¥­—A]6™²Y†uÙbÊVÉeÍà HiÖ±îLè2bÊsåNS¶ËSFYcü˜ÅÞ«“}ìy’³Ž§‰—ãÇu™0e§£?"6š²Kî2ña|Ä”çËݦì–ðãBÒ;y¥¸nÒ¢;ÃÍL ‰ÿ‡o™øžüÅ7vp*L³1ÄcüæˆCjŸÄý&>¨ÔêØEƒ£“‰“¡XZãÒ£¢¦P”enU°Îò–«%7½˜| «Z 4Ç ¥$a.ÄLy±¤ÐRÜìÑ©Ê5 $^5-:Ç™u©¼ŒW¸aþàõCα]áx  ’m!Rév )ÍÝÖdJXæ6Ø…¦=8JF z3HˆUÄ:K—œÇ'È—½†?E¨µrk8E÷0ùK¶œá€shnžà À"pÄ¢*LÂq[³t}ÕÇMü/T¼*5Á×7+Aus«*Ù$,Œ" "ÙaUy`GWRq—‡¸˜jÊ+Yø3mfuÜód¯Ä°H2¡ªnt„»Š]hÞ€iÁL ʼnïâ+&–’5ÊO°'Ö­)W´Ÿ@]yV—kK:s:U)¯æÅòwFTî”g¬…Vû3æ ̶E0€³™š»â3Ly¼–Âp0À;{ÜO tyñW^O‰ëÊk‘"ƒò`KôJVr$ÚªÇӢݔ7°Ù’WX‰Ìü¸ aÒ(%ØØnR9@€| bË*µÏæL<Á! ]Þlâïä-T£˜òSìinE•"ÝfÊÛå”^ti„°”æöf%ú¦X”´?˜>+1$rBBqž­2AÖ$R©8ùúuêùÝ”wÊ=ºÜkÊ»äÝSF/dRÔ»ó¹vƒËSß#ï5ñïìù§º4šø0gVbNÀÙ!]:k‡.÷qh¼ÏÄ{qBÙFgT[ˆEÀÚÖTD’ªgN+›uèüP7%i“FÏjNd— 'æ˜øî'WGÙ÷ýòʵL¬ãÞ”Yóê¨FŒp—2ÓeLʪ•9{ B‰ñÞmž”b®rùª–³rpÉî õˆz̈·–âÚy6†(aÖeÝ£ JA¢Øm%ë줰ÌG:ƒ9Dí *ņÉ4y¦ŸÑXv"Yèƒò!¢ð?J7tù°)‘=©$ˆ¿Á¥yÃÏYIß"”ÍÁÂò£ˆNƒ®Ó¸ð?ÂÆÀà’hÄ‹9äx ‡ït!œ<úq£5¼cî°òdÃ×V(6„[ll¦W¾DF©r­ºýÕD 9¥dørc»¡´~hÄ©½˜KwéË:À¦õØ.KÃ]3ÇÛh³:7Ã!ª]@× o>Úøl÷•*¾,•ºN•º<åé²/ùœw Û‹i/%ñ…ºc–Œ©²ÜÔÐÀ¾Ò1!c+Õ¤a·œÖPᓺUXR›ö\Ú $¶Fx·¦0ø ƒäÝÄU­©ö=íÒyÌçô[Ôm,¾¼f-ãMtíHØ×¤2JN_ÁÒ.½Å:h·¯â„Ïë 1™…鄿ݲhû*eöÀ€õ1^Æ«Ê`«v)ªÛ2Ôr #+£µéFe~Í º„{,T7EÕÒÑùÂ÷óÎw¶ùݵ°òËTyR åD™¡ÕEÚÛyWvFɺíµëÖ}ÄA‘§~õÖÕ›øÆéW±Lí»}꿱É?Vóý!#âD”¢A›Ÿ®=:…çúðßHBÖ¢êeðÕIÇõMýî, -’XÞÑÕº:Êü'{Ïhæ×”ÊÖ¿Uêè©mˆ[+;Žá^ÇuGï„þ‡ØèöeA¨¹9µ/â¾Dcv±ámhaIr3©›C4™œ£:#ÆçÑ}¹#Ü‘nOŽùRðkRÊí¤;œç“ˆDŒïÓÌ-ºO?è4¿ƒ<†ÒSåÅí‘dz§87í¤GŸjéa_/²œ¬ºXf1}‚;*¸ÚamíÀͤ”/ØÈâd1d_™UrúˆŒvŸCè*ÖÎW¦£>ÍpI8 .ŽÌV1šc»çÔrôöpl MLÝå± %j£®ÃÞ3áîʘ}&å ÞH¢!Kªck¯ª0,Á©Ûqªþ±nÒ§9¥Œ—~œXS×Q.ÎÇl¦sKæŽ|ÉÈÞÌUC‰³Æ4ÐRBÎëׄl>UŒ¶ˆ*\Ãi¡¹cLiÿÍD¼«)9´¨hH“COÙò†•‘º®‚°i¤:fpi D$V³¨ÉP#öÍÎcVXÉør7Ÿ:œÝáœ: >;´s‘‚tnŒ½˜ú =SÞ€T'­HÓÇ—Ù»’Nþp ÷ˆÇªùµD0ïÏ[ç…g~tí1¶+ã#\B?cìðüÿ,«Zñ?Ãvë:wê>MqÉGÜP×ÓSžqæx¹N‰yã>úoO†¦ÿkKÒáïþs¨áóÒ©¸ÉYJ–O6ǵĦ؆¸E0'.Z$ÚSâ¦x¨)Ð)Ùîn÷óàÔ5?þÓ?»2_[òÑw?Æxõ]' ìT±m”ž´B9ktëwòÊ™[‡*‰+êâB`|O€&ò¶=µ&¢D ùo¾œ@ïÔ¹Í÷NÔ¯3Õ¯‰Yô[Fc³1‡ž¹ôV È: ¥}€O©aãxî b=Mkæcý"bâÉ’~g—–=¢´¾èy Ñs=Í®ƒÉp*Ì‚µRÀ‚f¯Ä­ñDªÖôÓì‰ê¯Ú'Ñ“±Ø&b9ΣÆ†Ìm}`æeõBvéçÀ× 9½»Y[\dwÈš‚SmpÕ¼(€q”üÈg†À8ÇÃt` l”¼€‚+‡hvð:¦át@•-ÃPøÄùm®ù†3ÆHó‹†ÎoO;&βe{6c•Êçù¥e}0¾|š|&ô‚@>` ÜMíVàJ•`óæãlœ£–ÈÇœ«Fçc)–$Êr¬°—ý-kÐocY/Lì…IAÏäƒ09¨û=yŽ0eäø=~½¦==0Îêjw0¦‡!°±Ö Óýž>˜QQ¤¹Ð– í)Àr5xáÈëÀד~’„v34ÀmŠ”µjL£CJ£CJ#ÎÃJEJ£"Jª¥Äux™£‡à®ÂùÔÒ±JñY|³t\@ÿ²‘~<.´Iï¥Q ëTÅí^˜YW®hÞ >Õ˜j½ÏÚÞr­f×9’9 s¶É/}PâÕˆÖmò0=òæ–÷AicÂ÷š÷‘Ìî'„éð”@l„Ç]FxªCø©6á^X‹ÕDPyA?Q“=ˆšE¸Ø¦¦†À0ôÒ2B¶l¨ öªµŠ¬!ÎZºµÖ '`й@²~e)¢ëK+&÷Aù`K¿L |Å¥ŒYð,G‚YXƒKh´‡\ŠËleôœIP¡¨8ÑZRÜ ™@°Q¸— j~í¹½0í0TlKyœyäq,•eÓ•ÏËê ï"w‘áׂ>¿ï¹êLYmš…™À~_¡YÌò“Ÿ:.˜íÏÎ›ß U{!‹Z ¨E¦©ZuFÞñþlõ®†.<Õ‹Ùk·L¢¬@15-W`5å‹‹`+ÅšžX \ —ãÅÙ"hÌSü4(žOÁ‚ä%›]¨Rbb}½â1+ß•¯´yœ—b+:ò`q[’%Ÿ‡jiPL™×¹¸“0;Û–Ö¨vgÔ9ΨÅR™ ²ŸÕ-+W6Rÿ¢$&€ú÷ݹ-ÆTLë¤ÿÎøÏ=Gðßš2NέW¹À¹A¢DaFÅs@ 0ƒÂÒ0¡¢0£Ê¯ÑÊ$Wd—ÿ9óæ'cŸÈ °Ø·äàfÈÃ-py’jÜî('-å°¦håÄ5˜T¬ÑáxìÂ]„T©úùÔÒÈ…ÌÅÝJhLtˆ!OÇnúGº/ø«íª® Á¬ß”Úíì…öõ=SÇÑ*/ZÎq‰ÍmÇ~µœN²ì°‘.â|”¸K MÞ6Dz±аÙIZ2©ç$¼Pe“mÔuòΩòÈÎÂÖéø±#ì–¤øMÄ ¾'kó¹VaSÈð ²ù ÛdYãAˆ‚Nòœ'Ø% Ig°Í•ÎLqØ6Eé *§w©íôΧlp)§g™ÛY‘m[§qËö|ÆX<Ÿ‘ÎóQ6¸%PDN1¾?èíéèX\žšŸ`Ilœ;Ä×í"Iì¦Ê¬æãTT} ÎÅ‹Éß]1ü8Ùİ?¡øÒ@vØnû:/…»5¶mæÂÙÊ9)}ÌáÚcNžö^F1V¨–•_ëð òÇÿY=çPãÿrSþo5‰¹1ÿ»œTþO^EúKÕ1®M9½ ¦ƒß."_ÉSÂÇzáâ|¸$.í…Ë‚úaÒFi—ûiôÇ_€Kxö!¸‚1ššèõ{{áÊ=IÀ¼)`†¥dôÍQ2¯­d—rvÚ W½ËýœùñÛ'ÔƒCHפʹïjÊ/KÃ5jÔU}p-%Œ”ò:¤d´‰+¦RUQ¡~Oƒn[i.'#¼›ŠÝ{ ÷ÁTüLÇû`&Þ•øNoÍø(œ†Áv|ÂøYR¨'ÉÅŽ*Lâ$7}nÀ/ÀMø"%u_‡ðkЇ߂ñÛ”ž}¾‡¯Àkøø9þˆ’¹£†?¡dñ JßTÊx'I|&y¯“HAuÂn<åPW’2n‡¨ï*ê{™jÔO —‡GHU¯ÁkIÝß¶UÕ˺â$kIUU D­ëT Ä-N†$Ñ7¯Ç¸(Äñ“x#9–\\¬’EÕ\áMx3­Ö7ã-¤Ò†Riv!ÖøOÑxëÛ­ô•û5Èé'ÆZadåj:ÞFñ„³ÃÛÀzúúÙ5§ù~ǯž:Þù> §ã{$|ktÜ«ìç®÷¹Ä3Èú)gœ·C'¡RÂz›ú§õ¦1:c`4¾3Ò¿{¨}/i‰0®¥o¼1½TÕö“‹ûà:®ÂüÚ!¸áY¸á|’ó ­*ñTì½ë$²Áq3Ó7CZhÉŽZ–ì²(lÜO:©9nÆêyzܹìCNñ¾Â.w}†}pÓ]úçè >ß°ªWðE|ò²Úв3÷›·…žY"Û=Ûž=¬£ƒ ?ŠŸ¶¡/sðþˆ2ynÚDÞ˜p~Œæ<î@]J£y¼WA=Ÿ t¼ ¨×êù¯^,W Wªœ5T:ñÉbÆ­uå/QNyæa¸\òíûÙQR*q9-}ÇGL‡qbL³`º˜ãÚñ¨tШ´5cðÏ»v<&*…ª¬Ÿ¥ö“¸ß&þÏù Ǻròù{!—‘.·¾ó‰@’Eͼ=*nhCò3Q±€äzå`´Åäá¼ïB±Æ%#³”'ÍÍÓÄeƒÌ`|/Ü5jÃM€¡°¡î#7ÉÏPåë݃ËÚºòçöØQê‰:ù„²è©Ä¡ŠAR? ED3Lag{¨RF.X`BEFk†ƒÖ Ë¿+nNy¦P5{²‹–‹SÎwÚŸÃg,´õ~"y‰´j@Üs\yÃ=©Üë^j”»zÙ·L10¨•–ó¥ö*uÈ: ÷mË»¿xžeö å)åü£Sm¯’ż‡ êôùá QÎO û/ÁR¿—“‹}{aA)'ní¹AŸ ¡G8™ô2F=®lÐï{Þ¯¿ds¸ÚWèã=XÊ?Mq½:³Ô¯i»ä,¿§>Jª³ ³÷À³<£º0sLP‰C9!öX<Îd™=0£x¯MÕL“ðµîSå¦?ë|¡:§0g/9Arúf<Ù;xè~²¨§ÂÓv÷̉N®?×Nt*¸¥‚ãxÊçH{žÚ&ýãÕ¼q½ð )?e/ßaíòÛ»ÅwníÁ‹ó¼ÂÌ‚½Œ ÷¸éæô€§<˜SêÏa|s “Ü^èKÉê MUŠÉ‡ì½ Kìþœ”Ü?OywÞašœÕ_Hm~côžÕΩÎK–ïÏ· ëã–MX¿€Èú í‰Eþ¢Â¼^˜¯rµÿxÆt‚RÄçxöRJÏ3"´PvÞ Ü*âÖ‹ÜÏ­/rkB Í/šþ L¨ŸêýBjøýJÓ‚] ¿Ì“ Ý´õÂWzúW+5ÞKqœ¹¤ææ³T^bj¾*AíPìÔà h¢R¾n‚;`/TˆWÅ›â-¨yr¢,¦~•UbêÉ»!S\@Þñ"X(.†%âX).ƒSÄå°Y\gˆk I|ZĵÐ&®‡¸¸ºÄÍp¡¸.·ÃMâN¸CÜ{Ÿ[ÜÏŠ»áq|SÜoˆGàw¢þ%…Å“(ÅShˆX,zqšèÃ2qŠ/`µx—Š/áñU¯I*C‡ Êš9ÎÅ5TâõQ¥ŸƒKáB{Ü/hÆArâ9â×T~3”‰"÷â!ê+ ¸?OE_Éè:O3rEÞÃÞ8±þˆ/VyìsS>œZ_´òF9£ÍG«/Dœ:hÖdÊÖ߇ïCW 1¬™ù…G826I4äCXÇ-ª„¸ŒO‹>‚zÙ`O¢R%µ]¶@U'ý”7Lü¡¤bv13$ÿ¨€Pëö~È#3TZ`šUÀ}*íþ.•¦VFó{7oÛ@j°qÀ}–±ßýÚ¬ â|€ªyT˜Ê-;dÐdÅ5´§ÿç\h¹¤¬‚L¹Jåñ°V.„ͲN“‹½¾µ¤•luP³•ÕñIÂ6'چ߳÷ÿ¶©M¡Zß§*[:úmõüÐÞüÏäܳJiÐ,;K~•’£¥’#¸Ð.éŠí“µr+Û¡\‘ŠÁ/»ª•ÅÉðÈ%%—Âd¹ÌUÕ;˜SÅoUuÅT÷ÏQG:ã•sU—«l\T ½†?µQÚn'î¥e„ÍW½u§ “l#7& ä›]ùxƒEZUë%•»Wü™³â+ )N·qÊG)Ã÷™Ý7Rùá74¡¾õ¼ÂË.%^^FéS¹{Ö·S³¾cÏÊû®•ÌñÔ“`K+ÎCžåre,“ÍÐ*[]‡§mEmEmET<Ìß³UòÖA”àà þu ¡OÛü:ޛ䢰^±~¾gå:ßGìƒԩĪ~¸¿ÎMÓR4ýØ"Eí+dæ@ ¼êNôe¦Ên(‘À\y¡CÊTª-ß@*ˆ`®:Eü¥Ráuyëìú£ÊªSÚt|sÐþðëä°-Jž´)99-%?962f¬!d\C~#Ì’7Ály³CÆd‡ŒÙ.2NvÈ8ÙU¯áÁdxùÏõísŸðªm×OÃk”›ý´ÎÞÿ¬'•ùÙØL??g_£q®ú¬ÖÜ ¯S’q~ôTðž5éô@iУ²Ü_z¬¤:£q›VÖ¸-£TŽC 9”c¿É pOÿc¼\KàDxÃvMU|+÷‘Þ¹ò~*è€bù )äð@>Kä~8QöÀrù´È§ "Ÿvö“9o+<°œ<ÌIøkjC-þF¢•“*hy˜ø[Ô{vÚe²Mø;ü½Úª<ÿÀ'<޳zÚmµ.Mí'¿C½æ'DÞQ»YS8šüñ_0Û±qrö–ÃûIâϘc+Î]¶w™MìþÕ’I÷ÂÔIW„ÿ£ ˜xÿö6bø¯ûà7ôóÛàm²T¾q‘ ãà-÷Ý ùyȇÁKÖŸ-¿ãäs'_ =ù²ã‡ò(O: ÿ¢4#u•*ƒ´å¯”Yš1D??(ˆ!cÿ·|õ·iÈßñ6ÊCîýnÈ.ŽüÆð;D6éïœ÷ðŸJààZL«ÖJ+ ù²oÂkÜz‡ÏG¾ “yƇÏOø=ÏUó¶|ÞŸ¾RqæÏ…ú^Èä%ÿ¼Usm¹p¬ôäýå üÕ® uí>˜Kj-«T5¸oLb Ól \)ò Áá(Kn.ÿ»m¡jò^˜Çv²ôpù“Á¦þ.·Y°‰ÓàDü7%,™˜Iöz2e«iÚuÛËùШn{ð)ÕÅÎfýÅø}Ûãb{Ã׃»ñCê“À³°Ÿú4˜‹›)ºšLfžB ÍÒÝ-X"ø4ÚºËu*döÓCwîlðwép#'ÎsqRYÁ¼!“ÞÕ]N?WPâ˜×Om ›•J­ itîfÎS†E!ïßáÈx->8€ÇÑu驆\ÚR‚ÛBH5ÍG³[Õ|øPK kL[DËBòëü«?org/apache/hadoop/fs/swift/snative/SwiftNativeInputStream.class¥W |[w}ÿþdÉïYQbG‰;±œ£±'ÎÑ8‰í¤s:uâôp2ºÉò³­VÖsu$¸l Ö–m…”£ ×`óØRH¶B Ú»×Á6έ° FÇ6¶Á¬Ù÷÷’,;Nh‹>½÷?Ç÷w¾/¾ð‰OØ(»‚x.Wà>¸–ÿéìIŸÒ÷}|ZŸÑÇSúø¬…ÏÙx:ˆžÑùÕ >/ñEü±NÿDjãÏ,üy¿ â¯ð¬¿rýKA|cáo-ü]u¸lã+úþª>¾fãëúþÆ|¯ÇÿAÏáà[6¾Ä?⟔Ú?ëã;Jê»6þÅÆ÷ôÂó6þÕÆ÷õÆ¿éãß-ü‡ØøOÝý/ÿ­ïê­YøŸ Úðcÿ« bã§6þO·^°qÍð¨ˆ%>[Ê‚ØÍƒâ¢Ïذ¥ÜËÛ– [‚¶Ì!7 Ù2×–yz¯Ò–*[æÛ¶e±‘…ÄFªõT%‹‚ò YlÉAY÷á}‚H·›l‰ÄâCNKÜvS™–¤;8˜H ¶t»ƒí‚`_n`ÀI÷$tr€â¥bÙÄI§'릹r[)…¡X¿ëŽ´ dZ2§Ù–Œw´¥Gg‡Ìxo"éôŒf²Î°!  2Yîd²‰xÆ’:Î{ŠsA¨+•rÒ»’±LÆá´yvnSDWN]VÒCÙìHO6íĆ7“T¶ìç£+5’ËzWîNd‡Ž:I'–Q1ý#±ì vv*G¸ÉCe#.¥œwSY'•ívRƒz'Dj7µ+éfœ~A¸û¾ØÉXK2FŒÉ‰Póæœ4§Îጓ”w$R‰ìÁN|_ `7Qí@Ó]„`—ÛO“Wv'RΡÜpŸ“>ëK:ª‘%:Ï/ú³C ‚qª7T Ä KFµðv,…ß06bXП-YJŸ±dm½çÕqg$› ¿D*~DMáoìRÉËÝ<°ÒEæFúcY:p,=<ÐiÜ›‡zWh<`”µ2Žs¿Gƒ²ÐleM¼\6œ$ Sí™1á¶t.ò¦Èåi'“Kf™gH¤ñDgW—Þ“>ïD' P>“NÊÂÛŒo¸#NÊ“Â09ΧãòLM‘ÁžÃ{K9X™\<îd(šg¢¡Ìqu-s›’^̘¥êÆë½Í âù¤%õ¼pÒI'FSu½c‚…M×_fò´$Âs‰T,É¬à‘ˆ¹Ã#£YÇ€~À¨œ d}ÞR îæRÄ߯¨z'(DM0èd ÊÁD2Y€¡<™_Ò%r^Tìvºý£»¼ÐS;(¦­Jå|Ì=äœêqsé¸ã A°íŽx2mAoK£ƒn2»S®SDBHãLÃà(…‰äRjF§?#!< ;}ˆ‡Ð¡îGR°| Ë®dÒŒ%w¦sÔ¶h\Á‚®ÔIÂÚñÒn$C„•Û{Cø%Äèï!iå–¬ÉJYB÷…p/~‘òv¥âm& í« ·½t¹ jŠùá¾ûœxÖ’[B²Z-i ITÖZº'…ö™$Ú#Ó²WBøÎXÒ¯ã7(ù,á X4Ó:s‰d¿ÚxIɹȩ!bÑ@ã‘HHÖÊ:}´–¶EVGÖ8Í‘X–銧¹Ÿu#^ج áwñAKÖ‡dƒl á4 áux}H6á#t`%Øè)ÒYÕ¯ÿ&KnUÅ7ë•5!i•-ªÀ£‚•G¸Cc÷Gn,ÙjÁâÕ³ËÒF±·Ê6f)^ð¦aÞ$Xáíëá¼Qy‡X r˜v˜/è"ïQŸ3RÏöL…{Rýw 2`¼³Ü„|HÚð˜ N£]ì?vìH$¡ÞÉw¨Xd[¿ƒ3‚Ý7‰#§˜@½|ÌPJÑQ…ÊU”Åø#›‡ÙDÈäÒíjAbe©/G¶3wHÄØ…#’r§‰ÛÆÝÛ»JåOd",Á÷«rC±TÒÉDúFy-éstÑá±)2’&Êéä¨*Ü¡v¸õ%èZ‚scÑ œtÚMÓíû¼˜$§LÈŒèCŒbU?ÒÃäñŠŒ‘-“qÓY-÷¶Ù£É‰ƒ(—N3ˆ¤ÈvfÀöü-Žçj†R}SîZwD0¿0ïcÑ;K÷3—õÖN±9I¤Šä<°ÚMðÒÉ#í‘bj5±›gâm]—*ŠŠ…ðø0¡/£»8Ο\Róq®,n Äx‹ô–õ$wFÝÔ´\¦¬eòÑCGÚëdãC:ÕÌ¥ƒú°8ëJ,ËEÍyô¦áa<"Øö²{ Kn Éí²“Æ›½3ê™v˜;úâ;(ÁöŸ«[cr¹a VH°&•J»§¼6«jf‚¥9ix/É Ý 4]]ÓË(«-/ñŠ`Å)U°£ñÅ1¾a/>oJçnW5¶X3sìž ‹z÷ôíöÙÐc—Ë&’žMîäH{•l:¦ýÁþÆŸõut}wt¢{f5ÝVÛËW—©>6ÂòÁ°Z;[/wÝR¾À¶Ï8_æ&ç—Ý\]"ÓïôåøŽÎB¸{ï4º‡Ù‹§msL{xÔ4ÓóKz!ºÉ 65ìÕ»¯[§\U¦#bË®wÙ3eе~”$RîŒ>·ùy‰ÌnzOJ"çºé2ƒ€Iú‚#7‹~!¼ŒÈ©bìšþe¸êæˆïÅ™F½^ÖX`S©šæ³xzÏþ _ÿ3R"8Wb‰6“…µÝ4ov±æÍF–oŸvœfΞÖ̇Í9ŸöÁæÍþ×¼0óùv γœ=?Êù>„D+Æá‹.DÙ8Ñpù8,]±£kÆQmG0ºvsjü§ßƒ¹—ê Ïļ+Ñÿ8*£kÖN jóÇ>OŠeÈñÙŠ¹|ÞK.w GP…;Qƒã¨¥Nj´Š2E©ÑêÔ;1ˆƒÔè$oÝŠ€J†Sx5`F£<%f¤ºûÌHµ.3#ÅÇ9®\#¹€…×Xøe ¿báµ~®ÿ*ÿ~R¨ÅrmJ9R NÁ2´¶D{?_í#1ECÛ í X^xÕ½eQî_DMO¯Ÿ#[G±è¼7gL LÀ\Cˆ[Iþ¤Ç/*²¿†‡È|¹ÖμPˆ€¢^ÙÜhåþŠòpï 4\'D!óZ>_Gïy=*ɬo0‚D R¥‚¼˜‰©·”¡A?<ß?çªÐ }c±¯ƒLÃXÒ1†m/£®wKÃË&PÏõ9ˆ„8ž`F‰×6_Š2ðÞÊwcNáb­ï}TÊ?‰U\ >ƒ*]ÐÝ`p·Ôž¥ å˜G­Í;OgÀ)Ä›¸óÎãî£Äù­X‡ßÂVü6nÃÛ±ï@7Þ‰WâÝ šÇéFgŒºQÞ\‡ÅøM@om&­7[ æ!(§Òo6¶P·Yÿ52ðÑmØ[xËæŠ%>£Ø<šÇÆåU›sjŸÕžwÔ]D#³T_Ñ´¦Ž£¨¯Õ¯ð)\˜ÀšVÿîmS»@l.‘1T¿8À‘¼„u>”œk)Ù[(A¸²”zµÿ½X­ž‚y®¾ªýãj¿‚¼›è;“à‡ã~DðAîþVã÷yæ£<5F?ÌÈú#zÔY:é„ýá>Oˆ/ðôǸ2iÀÞA<¶¢×@¬x®èeçh®·/;ÇÛœ£é¢+hV5€·‹QvL½¸}»ˆg…Ÿàxm‰|úQ™O_ï$ uéeÑðúIl(ëˆjLb£Ÿn< ßÊ$f<˦ý«©”§ôrÎOSäÏpôYBð9îž'žf]E¾P ˜žz9©ØËòJÙ\}½ÍóºÌ5^-úÍãåÓ¼†ßyyq7ó­8XÑðfŠ;´ACýKäôeÃ7äËs+Õ_¥ybOæu? Ö›5M[t—Khж¬Äs¶ÐsÖxáy [}ð"r[T˜«Ç0ç 4_9o¤ØÆ$ þ;ÍÛÃë&oà+”è«”íkX„¯c#¾É³ÏÑðßâÉçÐ…o—d»EóÀûRðo,ßgáýÖ4¥>PDè0ªê UÄ«¨¼Œ¶^­I´_™ª*óŒâߣ8Ï3h¿_ÙÂ<ÛÍY™ ß6‚_¸ùlû®<›£„:¼ôM`{x‡7®ÔñmÞ8ì…A»} ‹ —vz›–‡¬÷4Jy‰MÓj,"4F®üÖÌó~–k–†uÙ{kÅ–@M þô4çºæš3Úìî ï™ÄÞ+—‰Iï5`lK`ìÚ³KÏŠÒÇ!ÔZ¡Bc,A0<¤kdÖI=6q½C–‹ãÏK!¬Ç2WL¢)xÖ¦%-_>t09i:É'‘ÍåËL*ñ K~Ì /Ê'ðOU_e.—ŽúŠÓFŸÚÞð¾Iì¿b<Â霊©M%è"îõê"J\çŽY·­¡âô㘧5¢ÛÌÏ7(‹*]8hÆpV‹yCLŽ[9Ý:dͨ÷õ%ž·|ª÷hÒÓ‡½n˜Ä‘»Ç0ß”–ð+óÂS¡(ÅÑKè¡: uå>VH3Êe-ªdÉF,—M8*­¸G¶ O¶".íHHNÊv<,·ã‚ìÅ'ež’;ð´ìÇU9€g¥߃øŽÂwå~(GYz¤^î–V¹GÚ¥·Xì¯ÒQòõ‡TónÂÑG›,’Uì…by7Y€ŠªŸb§¿*ü#b•™[jœ–”Îï÷» "1uˆðÝ^Ío(1ÐE¨ZÊÕ îi Dµ\kC5…›àŸÊùKŒÓÅ¢ýŒëeÛd»d¨¨y½}w^ ÛèªÆz¦» ,ÁiŽýqŒçånÉ—›€:ÁEÆ&1I²$'®+4‚ ÖuL+gšo}eggÐ-Iø¾¢ ¾¼ SÔl\dRóRïrÑ%oN\Ä/| )I™kIá’¡ó 6jZSÅÿ?PK kL[Dæ[ iÌL @org/apache/hadoop/fs/swift/snative/SwiftNativeOutputStream.class¥X |çuÿ?íjg´Œ®’’`„‘+ÀØà–„d/H#,p€•vV¬Yíª{pÄM“&8ݺmR× ÂqâÄTic7¾$a›6qpë9z%mÒ¦­“:iÒ$Mzĵã¾÷Íì!±(¤Ÿ¾ùæ;ßñÿ÷f_ùÉó/¸ŽxÅË%ˆà¥ùi^‘æO½Üü™ôþ\š¿Ððy _ð‹/zQ‚/Éà_JóWÒüu1Ô?7äío4ü­Ž/{ñüUø{i¾*Í×4üƒ†ôb9^Öñuyþ“4ÿ,Í¿xQ‹Wu|CÇ7eñ¿Jóšø-9õÛ:þM^¾£á»^¬•S#øw/¾‡ïKïÒü‡,ü¡†éøO/6à¿äý¿eæ4üX^^×ñ¿Kðޔâã-à%¢"/¹Èí¥bòh¤y±Côüé:•ðròê´D'ƒ7Q©Ne|"•kT¡S¥% ùtªÒh©NÕ¬ Õèt{ðªL-Ó©V§å,=•³T§S½Œ7è´BnY©“)›uZ¥ÓõÒ]­S“<›uZ£S‹NkuòëÔªS›Fë¼´žÚ½´6jt¡´kx¸w`ïð‘þÀ@`˜@ë‰Ç’©`,µ?M[.vŽ‹`„#Qko0‘мÃâu»®þÁ[ f<1Þœ ޳ÚÇâ¼¹=ÄÆÛûãã[ î±x,LhÎ_y,ŠÇ'Ûe¦ï GÆÓ‰`*ñz×që4Á×wðD°=äs†R >ާ¼£Á±ãéÉ>†Pf¯ˆÄÛåg {–W[Á BMv~0šL§ìq^·$ÆW°†Rñ³£€\ád{òd$œjOÚKÛ‡ämêËeC§“)kBÀçyÆ¢ñ¤b³d'ÙJ{Ò£V‚oÆÇކÃI+%òNYɉH*eÅØÐêõŽÉh<’Ýj§ýJÐâ1«›ç E‡º U«ƒÉ¤• %wF’ÁѨìðl‹Ä"©í„{Z®Í¸?§ªWºd×Úýìàžxˆ-ïÄ,[óaP| F÷ywÝ©c‘$aÛÏ(Ë*c w³u¼½§Æ¬Ie6kÌ:Ù“Š–µ â EØ9ž„•LGÙ/¥C)^?œT2ŠÃÑtò¯l½'¬D$|zpRÜV¬¼Íb˜NS¼¨®°*{yr+sŽF›øŽ´rì`¬ÇÞ½¢Ç2›äNâEÕYÁƒY·fJYIVY 2¢&&SLU„J{RTí²‡ +»-°–£¹ÄÞÕ/êáH,eÛ*É=!+j +[æÛQ¹>¬l\|’A-¾m (ÙGY¦–CÝõ¶$™Ð[êpÏh:–ØðİpEe¼J1ïžÃî–ƒrD©Øœñ–‰ŽêœŽBJY=å~V¦|ÜJõÍ#,ö#/åq‹(¨d‘'ö¦îyZ‘ÉŨžŠÛÈçÃ2¸šOQú¶±¨•Þ¡x:1fÙ0¬¿ –×ËîÃY÷âÝÞ‡_1Æ 'qÊÀñ[NãÞ‰_âc2.èVf´Bùg–å{(Jnx‚ºÑÀ=øE¿Œ˜÷ aàÀÞ0ò72·ÚYŸš˜\ÏÁ¢ÑMm¦-rÀͽ: ÚJ„ b+”vgL¸n¡áºÓ‘hHPÑÜŒÅâ)sŒ…NY¦ÚhÚ1YkŒ‰èt‡iÐ6ê”f»Ïà³J.!M¥ñ:‚HnкŠ.1퇤yÞÍwÛf1“Ê.f$iÚômàÎj¯(⟠uÔƒG |TšÇŨï—c—åTêÊPuo"gVÚy)sã±`Òµ¬˜™°âÌ(ró}´ÓÀqÆ4>Á±P ì $Á/³±h†ƒì§©Q¯A}t«A·áÂ-=vœ˜*˜ÌxØ”ø4›’[ͨP±¼u6%e‚ƒnø…AŸÓcpôn¶·Fƒóýnúå9 ÑƒioÆ‹f±ÖèvÚÇ|áè«Üi›˜†h§Ðš<.RŠ]ä©„d¢PL#MI6­à«²'žŽ†Ô6 ™>…§ ×çÄ ÄBÖ)vê`¸;žŽ…’y8[ˆ`& ™6×´;J‡ã ÛB¦”Ïí·{UgS¨MÌÃϵ„®ŸáXÊJ0ur>IYybÔdÄ`&´=ÃÇÛ æ¢«Ì\çÜÅœ(¸"aM#¬ZÂLÅ2VcÖ¸¢Â³cPÁÀ{%À×&G&'Ÿ6w 6˜R‚˜ŠîÍÉ 1x ½6Òd‡L R® OóeÐľò*ìwO›Éi´s˜âó|›™«ð:9+µ9ÈQ±~'“t›™_‰ªáµ/Wvñ¾2>'WyufØ*Ù òŠÀNup~uÆ#ÎH†üy¨ãÿ_²htРCÔ¡Ñ]½sþ(Tfb_{øX"~Ò.šj[ ³’W_KéÇYn\’msË•I«`«.´oóLgNª¬Éå®Ç:I¦øÕœd2cYwM78„¿uÁz›r]_f§†a˜«º \xµ‘lírÅ·ƒM/ƒ±ÞS6SçÏU(sIh 9ŒEÂV2ÕOìÍ+î« ©Ì¶]±ø·§µH,'ø ìï/€Ûavd²ár+úãRÁh'ä o¿ÍªZvå›\¦Ù&•ù›²Æ%?×p…ÕÅ8/ö/b±t*µÍu÷,Å!k4ÍÜÖòÓ>¯tä¡Â¶óæJB.K«wsEu.e92'ƒ Ž”úù:^ÀùÅ'á¦E«ë«_WßÒ³Ø=õ\û.:½k±éòÜTO4(_ 5!+ÉT([žHšJóDÓâ>è ª¢‹‘Ä ¢`¸)êt¨ŸâH˜ˆ  µ˜@ „8¿a’ÿ¸åþr)qÔ—¿êɰzrý«ž\¹ª'—ÂêÉzr1«žïQçIÆSÏ3Îó^µ®Bjk~,“Šß—² ÷á~žÿU~û<üØçŸù]3(ò»?­ÏÀåoÛ¿nÅ~žòÌ@ó·Í@÷_DÉÈExGüÏB›Ã’93(õ»gPæ/™A¹)*jÜ‘øžR2üšünƒÛ]pc+J°%Újô³æX…=XƒAtb/ºq;nÅ>n‡ðï¸ò;Ù>ü:~Cý\¶OYˆTOlQ¤zb-—êý&ëíÆÔkŒOžçÏGÏßã1Y¸ˆ*‘Ýí[:‹ê9Ôô·Îâº)4ñcÙV\D-«·|du¾úY4ððŠY¬œƒyÉ×è[Õz×´ÍbuÛ“ò‘Ro¹2à~TâVâNtà zpˆU¹K©aÚ;jTâf<ÈŽ%^ãÁos¯ñ¸z~'O’ÞQ`3¿‰ê: _:‹¦œy½ÊahW÷ö:ç¾Üi«¤ìsN`™d•Ok>‹RQÛ·F´|*«W™’;ÎçN²Ô‰¼³}ÎÙ©µT™½¤ò=a_BÕ<)Gœ¶/q?%0i¶åoñÓŒ\»VœQ<??‹Î¢ÊïžC+÷u¶|Û4«Û:‡u~mëe“6ªym—žFû–ÊÉ—åÐ #¾s¸u)b;t1 ÀûÔÓÖ­5–Æ¡¥sX”r-åªåàiâÀiã ÙÈa³C¥‹wXû=¬õíÜ?Èý ëŠ'6©á[šP‡ñaeÓyÖñÁ£¿MyîA )¯™Pþœr<ò£Aà¾ÛÕéÚ^?…Þ3.ò<‹MþÖ†9ÜœÁ&wçô[¯u¸ŸÆæ B·0BkÝ6DkÝÏáf<ÁSë¾4ýÖ Oñ%: ¶’¯²ÕÞÌŠBùgŠE8dzÆ<Âþ¯;Ï+?Ê|;ð1ìÄÇÂç9n§³Ñ؈fžù8ïeA³q¹ÛQ½ˆÍô³ˆçegŒo#”Âý&ÜD5üyú󇟣ÿûBôïÏÎâmݬ¦¯C{[G\‘¡·Ì 7^À¶¡Oýt]Àv…•‹Øá0Ò-³èj|">%ì'YÕDZϰZ³J¿ïË*Ñ—õ_Ÿ]³i#ÁJä$vã÷³1tC[ÞS(»ŸCOD’ËhÌsÔNv”­˜¸J-ÊZRü"àÓ Ë± —ØS”p[ò åFQåê¬HŸdaøsœ{"Ò‘Á–Ö˸Iäéu+¶Š¡8 ú˜áò¤º5GpEz‰¯øÇÅeÔáeé•,Þs"IïŸ9l Dõ)<éiPäågkïrÕ}`د:ŒÙÛr f3ÍçÙâ_d})ï–ªì-U,¤Ïœ/囨¡šn'zÎןCyÃ9,©o8ÚúâcBß®9ì¾ÄY¬_L2P¬L¢à3ânÈ¡¨¼áÞ ?|žƒ`I¨±ÜzÿfwµûJªÝ ¡å"öd³Ã ¯šõØë»{ ÜQybß%k}5“Ø·G;yIh»ßÅœ6ýÖ#<ñ<”3óv”sûeΈ_áü*Vàkœ¾ÎÑó*çÀoà0¾É¤ômVñ;L6ßåtñ}”0PÈ‘ö£l\ñŽgØ<éóY“dzLa¤z3 Òқ͋KOEIÃh¢Õ¯3óåÇå\Ʋø [VŽV[Ùð°C°ÃB°bÏ; Ú³aJøž ™UžÅ~a2±¯TÜ= ºr†Xž¹±”бž<¸tl"T†Û¨CT‘µÃY‡³*gUÎS‘Á·tÅ<ÍîÅ39˜yÉÎ^'1Õå|£rQ³CE%Na¹¨z§‹U°8Ùg÷KJ+r({g–²ÙË#ÆþÖµ3ö‹Ù´Æ§”¹fŠªº)4f«°‚˜P}€ë#Yª³GAË¡QÊh;ùzl¤ÕØFÍè¢5è¡ôÒZ¨ýÔ†ƒ´wÑ&¡õR;ÞEp†8òØýt>Èý0mÁ£t3ÎSž ­ø,íÈB0˜M =œì4ÐËEš„K2>{ Ïñ,9ùQ+[÷&t¢•~ÌðÏ÷ÖóL‹6?Áô(¾?ì8¥KrPÏä‡2~v¦(¶3…ÇT™B«ËËy‡ìÊËæ,™ òõ°h;ÑA»Ù6ƒyÕÜá¬6‡•†’h÷«l‘ŸßÀ„þ‚CÍí<#8Ù|¹k<2Hò¸¾Ø9ÚäLpÉÙ¼ÑÙ¬‰ÌîÐã ¶ÍÛ®e%ûÂwXxw¤ÀÝàìShó¦…›6“ü6ílžsräüße'Âb~T¬™ÅÛ}‡s™ÑwÄî7Ïâ¨/h÷}³õÙý2!ÛÝ/—qËîÉú°Ý×åœq»@Ö³û›¸_ñN¾ãîÉJœÌSâG azÖQ¢[ÍuUˆÌân®TÜÓp»žA;\Àñg°ya5ÞÆ‰¼]!¼E¥·Héþ9µäòÿPK kL[Dƒ”©˜£>org/apache/hadoop/fs/swift/snative/SwiftObjectFileStatus.class”[oGÇÿc¯×6$â&´å~ql’-áÚ@ $À a$ˆÆö&ÞàìïÄáQ)•júÐ×JýPUÏÌn6ÉfüÂËÌ™Ù3çüþçÌì¿ÿýõ7€y<-` ·ò°p[¿ÊaAwäp·@ÃoÒªåp¯€ûx ‡Å,–²xÈi~NÀÀê VË÷„ã‰uñ©ç0W¶ønw¹·i7Dßõ6ï0tƺ<ëÛ~ÛÝp6ÃDè;n×~È…#==¾MQÌ`Ðl»}2+j éòÌ}^òÛôy|ÅõœÕÁvÓé¿âÍ®Jë·xw÷]¹Ž6 Ñq‰raÅïoÚ¼Ç[Çîð¶ï÷ìÀ>ºÂ<.ÜŽÝ«çÍ-§%–Ý®Ó\ ª—ë‡ÚÙSrØCâç6±–ŽôPñrA¼a”ëÒeœ\–TôDyFWÓñ é8YÖgÍRÈ'ªÙ`ך ½•ƒÝ(îæÙ߉àßñrÒm7ɪêLZyÚkDÌ{öÑWþKß/Èb¸¤#×^¤žò£ö´Þ=㽨х†?è·Ù8†“ÚVÎÉh~À”…£·0cŠr8Ž&¥uóJ8F7ÛÎâ‘…e<¶ðó ·¿÷Q ÷„„l…ÚHC x_¯])PÛÊ7ôè±µ¸ÀzšcôM¤¤²RRš‹ÑL²Ô<­Kj–¾T0L“ufFs¦ò ìåv’ÆÍÀE¸„ɲB'ü„ŸàTà3ÒÈÒ|V¨œ"U™"]™6†0*Ó™!…ž6‡0÷‚ŸRf(xyT‰ÿ ‘ÎÒ·9Òe«„5RFAã„g•J¦,©3­,©ÔP–ÔšQ–Tk’çiŠ"Ú”7Ôø'R¿Ç¦Úœ×ècæ\t¸F©d2SiKVè&¹¥"”Ôڌ͘á<.Ä ,Iï$@´ ©» ©0E¥JõM‚Ôè̽ ²^2ñe-ˆ‘YÔ‚”õ Fd™Î<RŒ@f¨ñ‡A2Iº¤J·E’I‚¬Ò™ç#@äm‘‰gãŠ\‹îH¾B$æ7d“0/÷Áäc˜9}UÌ$ÌÁ¼3UÅÖÃä40o´0¿èarI˜·³>¦¤`RòGÅÚˆbMUÿA†íT‹ù¯(|A¦ºSÌW¿âˆdK«è%õ]z‘[i›®¿GÑÞïË4gš"‘2Óu² ¤ÌÙ7çÍÿPK jL[D|¶Jƒ·.org/apache/hadoop/fs/swift/util/Duration.classT]OA=C·Ýv³Eh ~ X”êR¥‹|X¡ˆXå¡„„ø´´ÛvIw·³ão2ÑD”˜èð—ø+Œw†¥-•÷áÎ;÷Ü9÷ÜÉþúóý'€%<ц¡aó)2Eaîj¸‡+ cÆ!¿ØïÈÁF‹*îk„_Ö°‚U‘ö@E™Aír+àvƒm1$›ŽçtÛb›Ø —o2ÄŒù]å™ß°.UÏ~ºûv°cíw(’©úu«³kŽØGA…·.C±ê-Ó:´êmÛl[ ß?4›]³ûÎir3äNÇ|w|¯"0ŽkËëˆÈxËægg58^‹!gÌW¬#ËìX^Ë<.Õ]ËÛ‘à cëµk×}¯A”T×ñBn“—äþYåø‘Õ ®Õü0¨Û/ÑCúìþ’(¨c“:®cZGY9dU<Ô±&"aò˜eÐ õÂâ’0ˤâXŸÌöþ]ç*ÖuT°¡â‘ŽM<Ö1ƒ Æÿ u®dí}—Û.©UƒÀö¸á•ÓéíGûiU_4©Ê&·› Ùó*‰ãÊùº‘.‰¦¸gX3þUõMu¸·ÊÚc–žZšÞàÉFÂI/'WLˆJö2í^"F0QüV$3rŒX±ø±c(Ÿ%ì Ù ²×ÈNAŒ#‡i\¥ˆ~ ¦“)™KƒŠ —h'ΔÄ?öê$d,?€U",=´©rD* ¢Ñ Ì!IÃïIõŠÐX£"Kdñ+'P? ‘0ðÉŸ¯) h‚ÖÕ™’îÓ|š[V2zâÒ{±Iêk´¶§ˆð[éÆEžtOp©Ù(õúQ$°@)ÉKó2¶Jje%7É¿Ü"O! s½FLZÉ8526ÜÅò@ñž …ÞVd1ÒFªˆ±ÚÃÃ(TÐznˬ;PK jL[D°á:oóÖ 3org/apache/hadoop/fs/swift/util/DurationStats.class•U]se~6_› Kúa¨hR›i´ü(…ÖV %mÅR*È6Ù$ I¶&›ê3\¨ÌpƒÎ0ê…í7:èŒÁŽ3úøMŽãsÞl“¶ô’™sÞ÷¼çœ÷9ç9»ûä¿¿þ0†J }8kàÎÅ(ÎËjNļˆ "²"D,ư„Åïbòûñû;²û(†.EqYÇ•>ÆÕÈÔý)åÃ2®‰ËJdzô«;mÓ'¸®ã†Ã]³ë–ç¸5 ýÙ[Öº•©XµR&çÕZiBƒÆmNC°Ñ¬r5ÏUÕ©‰´¾ÐªÚ–8ÌjTÇ4DN95Ç;­a_êéd#W0ãl =Y§f/6««vý’µZ±år7oU®XuGö¾1䕆†LÖ­—2Öš•/Û™²UpݵL±‘i|î½LÓs*™Ùf»ˆœgy ¢K=gˆÂæ•-•Y…‚†ãÏžAG ÍÍ>¦Ÿ=’—¦æ%\c3ûâYöRþö‚µæw!\·¶K©‹J¶7ã6km™‰Ðr¸%=}ÜNׯ\µ='¿ øá ÚÃÕàZžYMîfíuÇ]ïœD=·Í˜†Djd·¡ˆåÜf=oŸusÛ:9*î&^Ã;&≷eõ2^11ˆ¤‰Ã"ŽˆxGM ᨎOMÜÄ)Œ›8ƒÓ:˜ay 3Cd^Š*$=׳*“C£ãÅFRæÎ_6¼BÁ^ß´;›fÎg{¥¡·[ÂÒê-;ïé(˜°QÔQ]61%·:&ná6¹®ÙÑzVr¹nUšlZ¼ iÁòÊœƒÆguOÆaV¨êëÏÕ<»d×5è*t©¨a 5·•ßcb[¡³nS Ï>&Ìî4owÝ$;RtëUyÞÝåá]ÉîlâÄ.£AzðeèèúÕ ˆä+Mþ•>ìë#¾æ (=¤ôhœžaÊw‹ðÄÓBKk!~Œàï´0BÙŠ 1Æ`Ô^F¥ihGá^W¹ã ¦r÷s‡ òšlçþIOj!$:ÜBXt¤…ˆh½]t´…¨h££ E€3Â0ñ&|µÂ8‹z‹ NPžÄ)>]x“x“åJàe݆7íÃ3ÒÇ#ö{ºWÅÔÁiDù´tÓt¯óÓûéžÐ*_Œ•ô2+ Ý”¹ Z¬rð^še•æÅá:t×ÍK•Ü#þémZŒA.#~âÑ »";ýÁÃöN€A0MˆÀYÊsìÃyöaŽÐæ ù-Yþq™²k¸¨ H2ŽÐ:¬°[I?ÄO°ƒAöN†*ð/²Á½<à[ů«ÂCMÚŸJI†p¦D¦ðfiq©%¾®#ìø¡ ãªúDöÌ/>Èoc×9S74³Ý‡Æ‡ïù7gT‡Á«Ø¯_:‰#Êho w‚'8›ÁNpø×ÁÕ]ƒ'Ù¾§ƒ£v7w >Ó ^à„ÊÙ~ýýT†,C…»îÏÅuXlçûrK¾ý~¾“*w``Jz?ÕI}©…ÈÒøùáøÞ@Ïö´íéýš ¾Ù2½ ?u˜LL“rM]Êûƒêž÷1ãßó-í§ØßÛ«ÿ¾å ä–CRÇús˪­÷xÁÝÀ¾ÜrDÕÙé„7À‹í]T&ºëÚ+¾zg¿—~ÛÑàû[Rì4xVy}ð?PK jL[D‘wub 8org/apache/hadoop/fs/swift/util/DurationStatsTable.class¥VmSW~n²ÉB²ˆ ¢±Z_ª61_ Ô‚¨ hÅbÁ¾-Éš,ÆlºÙU±/¶ú#ÚOýÆ×:ÓÊPfj§Ngú#ú½Á©¥Ï½01|€é—{ϹçžsžóÜsïîŸÿþô3€~|Ã.Œé¸C™V$pYÇ•"“C¶ãržˆã*®éx'†8®KeRJ7t¼+•©Ú‘‘ÒÍ8ÞÃt3¸%‡÷å®âøIéc¦ŽYXÕ3½ê s¶d ´gçÌ{fÚ÷ìRzܬ ´NÚ…²éù.­×­CZ2Ë…ô¤çÚåÂ`Öq i³bæŠVºhæ§’¾]MWïÛ·½Àí¼ïšží”'eÒÁ³L²Ë¶wV œì™ÐF¼b—­ ÿî¬åÖ ufœYš2][êµEÍ+ÚU›Ë«œ™:læóÃÉÿQÇàŒÝêT¬@—8›¢ ´äýU{jã±ôªŸËYU–(fÚòõU¤7I7+¾cÍ3ÕÜž "BGNõBž=ÅŽÐA^·,¯Þ×®zvŽ;“=u=å2ƒj²ƒBöA®:sÆ“¤8®¬Ãµª~IbÞÚìßÝx²ó•ÕÓyiïЦÛKÇmÑ&Ü›%a+M:¾›³.ÚäŽæÖ:*SØ‹}/Ò^2«E’. E¯`7_î•s¾]Ê[®sLØwq$“5p%w ”áôm ¸ÌÆ”Ÿ€1ƒ°í ×5ç%<øîᾎæñ¬øŸø_ÈH¥Œð¥ó²®þÍß8þ‹r¯ÎÎY9¯a)`@ ¹ÑЫÅ(ïE×¹¤ÙÒxÔlÛæ¦dÛ&3òîFÍJÅ*óè[çèiZª¼Óž³Š¸+Ù¼‘;ÚžKÞ¾‚ìžÃõyòÔ–x ×DÄáŠÏ0gÖ ³ÁÀ¬tÃoz“«öCK=ÐòuÏ,ùigÃuJ%Fž­®õÖɘ½F~wƒsýKQ4«ÖO¥ã›§••ÒÈðZ%ý¯$¸Æ²œíëÀ öó‹»‹_b¾dòrrÞC-„W¡Qæe渟+ÃJ:S‹©e„¦·FþÚcµÿGù ý5ú½FÉAc²¤ókmý5ç°ÌÖXI¶·VIWïj%ãl¢ +?^ëŒNEJ†òe¶öžXV!ß§ìšBb»r¡j Ktq„þA»‚– ñ‚wñ?PK kL[Dó”FËÞ .org/apache/hadoop/fs/swift/util/JSONUtil.classµVmWE~†· !i)mSÑZ¡¯!PÖjŠZl+!4¥I©Xß–eI–n³9›MiûÉÿã—VÏs䨟õ/ø_Ôgf7! IO8¿Ì¹s÷Þç¹wæîüù÷/¿xv£XÖ‰¢˃x+Q$pWª³îiXBC6ŠîkÈE1$ ä5<ˆâ–å°ÁC)ÕΗ¬Kù•†GRÒû×¾ˆnWÜÒŠQ.[žÀ•ŒëtÓÝ´ŠFµ¢oæcîêOŒ²žÝضL?0œ˜·K¶¿ ЛœXè[ä7Ç3vɺ[}²ayycáf$㚆³fx¶\‡Ê>¿hWR*šQ6Ì¢¥M×-ë[½²coùzÕ·=ËÞ}À‰ è»r%p9™Ù6žºc” !ª¹‰UÎ÷ìR_FšnO(ÍáŽU²<÷ÝÒ­g¦U–~¤wÎ@:L½7~1à* ’ê!\¤*Ó+0ìÙ®þг}•ÁxΧs:TIa-4|«j›cQê*¾ˆøn6Œ2“<̵A³è•JS>êPúŸNÕjÆYOVÿcÇxñBàÄ!_‰æ>/×ê˜h5žÏççøÅ`Î.” ¿êÑhq>?{ÎB4o²íìÛ–Ê'>]‚¼omYžU2­öéˆû6]»ëx¬ÛÖò¨û$Ésœ|×~ä9V¾]Ç!ž0éµ}†Ž™MFiwrúßiw„«hÇ –Ÿnè^É5lÓ¿"ó¦v°hέz¦uÛ–G:^k9Ób gñ¶À©Úå ÀW8†7ñV ߃V0¦ŽÒjx/^Óü¬úõ×sR°ô*EÃqÛÄ·)põH=+ ×δÖàfÕv6%¯-x…Ç**-³cÒþ†Àè-Ïs½±BÈ¥Tó¬J™-iP¸ÔUî’Ýv}¶ÍÖ³ÃCÙÜHNÖKÙ†ìFwäöZÐñ.%[Ûo›_Èšj³A*XòdÛJêõm+_ôÜÙ•»AÏ26C’|IÖª‡Ã¡ÂZ0búh="MÛN7ã|IŒò…ÁôÈÃË™Æ85ç¸zŸRPö§ö ^qÒƒw8ÊWp}HbŒ³X`Dwç)qi!üJ›^ÊkûèYßCoæ'ôM¦vÑ?¹‹—+ûˆ¬OíbpjÑßk«˜ZÑA§pšˆ¤¼ÀM¯ ~Žàjg0Œhñ1Ÿ/³´˜cØy(Ű Äq —é碚]Q ¯òe„c “>ÅU=ÿÐQ†«¦!4è+Þ¥5èIð6È\\'Iê¯Ô*ɤH$ÙÇ1R<.ùìbxä2ÞÅȽ›X;1Â`š’WIºO‘N0»À2w2Ìþ wîçjl”ô²Ój]eš>Ôl†©LU­ZR3Òg=—HÙl¦<§ê=¯¨ß©ÿR©Ÿ ¨œ: Î[¨œn¡bpgƒMîlRg5P ©$Ô, 2^§2Ž|ÒDEj>m¦²ÜŽÊgŠÊÍÊ-U<ýV±·z…;>V¹óŒõz~„*.âó.jv‹nãNx‰Ïª$Ñ‚D_ªË.± (íÄ÷Eýº_Pñ€¡}$HðÌÏè{ÕòÁyŽK*Hú_PK kL[D âÛ?O 5org/apache/hadoop/fs/swift/util/SwiftObjectPath.classUmSW~6 l’.‚DÄV©ˆ^R-µð¨[”VŠÕ%,ÉbÈÆÍFÐjíû»}ÿ¢Ÿ;µ3eZ­5:u¦ÓöKgúOúl§ÏÝ,!ÆÈ 3p÷Üsï9ç9Ï=çäïÿ~ý Àn|ãÇ6LøPŽ£byÅÏå˜&e¼*6S~Çkb9áÇI¨^L‹MTÆŒ˜‹&–Y¡Ž “¸:æü8…„ð4ïG†)qá´P›^¤ÅÆò#ƒ3ÂdÁ‹Eqí¬ŒsÖôŽ œéã26vh4,¡npN=£†2–ž™ZL[ ¨–¥™Én ¾¨‘´T=©™¹{ 5 E,SOÆx¡Ü˜žÓ¢–9cê´‹SÕ£'uk¿„Í÷[ܯi™àé7f4 •ƒŒÎÌOkæ˜:ÐDL#ª&&TS{Gé±âzZ®AÃŒ…Ô”k¡¸:c©Ðl:”^Ðg­\6!Û4ÂUbšÕ¿’RusK©¤|¼5ìäU®Î¨ †«)Ì&wÚÝ2)A2î¥Æ9²aªtP±Ôè©!5å ÷ÆÕt<—¯»¹å#˜Z:“àM‰e:£'fÆ—¹ôYF^öZF!}æXR-æ^Œm™Wbó¤lË zúÈbÌ6GTSKZó:›š@á³"e¤Ó:Sé)a¦sÐF…JÈ2^çɬiÌçN¦€IÍ é.9ãžÕÍâ“°îÞêW #Áh1ª¥,ÝH¦eœ—p|í'd•:33fªz‚¯I°øâ“6ö¹ê‰&œ¶aU°%ü#cFµÃº¨™ê"o¨‚ˆ(ØŽF;ÄÒ‚&4J=$:=ñÍìÂnÏb¯Œ ÞÀEb¤Ó7q@Bmq‰õ‰RÕLoámûDôwp@Á ËxWÁ{x_Æ >' xRÆG >FH|‚‹ >©CÁ%|&ãs_àK‘V3¡#êç°šZÒü+|M‹»ìÕrsô®’¼–òœ³z,cªB—¯ Á58N‰Ýç›3ô¤ ‘ò޵ͼRSÇ«%gÒGu{„ª©wÚK5ö}*ç!DM§-Õ´/u,` eö`!Ëp¥k”Â=;‡aΦö¿¼“Z–ü¦–J¨Q­7‘È™i>“×ù!Cõ…)õÇU3 «%£š=]v¯B|ܲR¡Q-mõ'tΰ>=9Ãäé²Z[´L5º2ÙÃê<»¦iMä1c9j̧ìF[…ï¿ì–rþÈlN) úÑþîp¾Mö·’¨k®-ÜipÁÍomð¤`ëM¸‚m7ár빉²ë‡Ÿ@Þê¼D¡ƒX¾<û>>}£·}Ž«?yG”lŸÀ1Ôn<–…rµwPqìÖµgQ)T•YT…JîòÔyøþÞ+Pê<<]²ç*<î•jØLÀý.²"β"Î1îyž]°quæ"æqõ¡Ÿg{Þn™O%° ›XWvCÕðü‹FVÿ«û ç9+n€V"ŸnÚˆ¬å`+ß4PÌì¥bdÁ‹6™Á^âù ]ŒëI¸ ;.ÿ¡Nžfú²¨þ5Yl Ôf±)¼å2¶­Ji­”ý ¶‹DU×y›³¨¿ ç6¶tyî ü©¼…ǯå)l"à[lÀU6Ñ÷¤ìÖð+àG¶ØOÁu–Ø:§óÙL;Ùi˜7]ô2Œ—m:»8Ò¶ç鬄û.:m.×s­* À‹Q6@®Flg¬ŸÀ·±õj®Ûõ¸Âê6®c6ÓãÿPK jL[DÏóOlâ24org/apache/hadoop/fs/swift/util/SwiftTestUtils.class¥Y `TÕÕ>g¶÷fòؘ„ &3Y(ШA¶@"¡!,‰P -¾$“00™Iga©¨µb7ÛÚŶj+mÅZÛ‚ÚÄBk­k[»/Ú}Ñ.VkµVáÿÎ}o&BúƒòÞûî=ç;û¹—ÇNÜwŒˆæ9t]Í•:Ïññ\¥qµÎa¹8¢q4®Õ¸NãzùdÕ\y¿Nó|<Ÿèüz|t%_¨óEò¾ØGMÜèã…|‰ü\$ÅòX"sKu^&?št^.ï:7 µ/•ï+un «äÇ|ÜÆ«åÑ.k×ÈÜZ¯ãõòh×¹C¦;u¾Lã >ÞÈoôñ&ÞìÅä–~¿ÙÇoá­ò¸\š:wù¸›{tŽÊï^÷ñ6c`»lÝ¡s\¾ôË#¡qRd©•Ç€ÆoÕ9¥sZçŒ¬Ï ƒòØ¥ón÷Èðm:_!ï½_é£u|•y¼_çè|ƒ ?¨ó‡tþ°? ||£|ú¨¼?¦óÇu¾Iç›eÕ-Bñ:RçOéüioÕø32w›êa5`ß/œ?+b}NçÏë|»Î4¾ÃÇ_à;5þ"“³mÍ¥L¡¶dª¯Þ0»·E뻓ýýÉDº>žìë‹%úêÛ’} ™Æu6wtnméØÚ±±µ¥“Éß¶ÝÜiÖÇM¬èȤ°P-ÇÆŒ™Èl0ãÙ(ÓäÖ5[[ZÛš·6]ÖÒÒ¼~kGëæf&Ï%±D,³Ì«ª70¹–'{°vB[,mÏöwESfW<*,’Ýf|ƒ™ŠÉo{Ò•ÙK3Í-D¼ÍìI&ê{Óõé]±ÞL}6‹×wȰ3šÎ\†_i×ÍtDS;cÝÑËÖ·25T¡ÑLôÖCŠÞX_6efbÉÄÂjKÒD4S} Ä€Q>b²cO"cînÞÝP[S1Í &=¦× Òã:2f÷ŽÕ怒Xã»–L¾„a&™í† ÆY*Y‘+çf•]½J$/TœM¥câFΞRÂ$I·ðéΤ…“iÝ9º@±åD6RIÒò²R´@+¢½f6žÙ˜LíÀšóO©*JÂ!ÝÁÚssbt½=ˆñƒtñ„²åìÞSŒ¬.4bÊàçù)Ô±v×%D%&B¢§g<ÿ[³$rÃÔ/ð?P¤6¦’ ¦Œe[ 6S?áÔ!«Ëi iüOƒÿÅÿ6èŸô"SYǶd6Þꂆ^Bµ]ÙL(–N\_äçÄÁ^2è'ôSq±— z‰^fšnváÔ$N‹ö¡Q13xO7ØË>S}°!”í+ þ W¯Ï"­¢§÷ÊÛÎIÿÑøtî°÷ùU´Í9?°š‡h·BÒ]¨;nÆúÓ6%…U›ÿ5~Ðàü ðm ¡©Ü‰.Ë–ÆP6‘ó°øžügwÈL ƒòQ‹‚Á'/p°ápðqÃát¸°WÜ {§0³aXªÑ6õØob4‡Ûpx¼lØG–©r4"smÑäòÖ5=ó,e ™bl'Ï“ïú™.ã!}îX®^m}Ò7Ì)v|)ÖÇxÌhªvL;r'J¤L¾£œRì*Ç É]ÿ ÞZ­@; ¤ýäd¥HUÌí~)w}tñ˜6'å<8ó' Ÿòª3\_Øw/žþð?0¬8C§bu›á!£-Ø,+lÉ;SrUYìÈ­zø²ª"­õ°º5€_®îúªŠB/zÛçÁ¦V·×ÖêV¬è†sÉŪQÅhíX5öëD´})¥UÉ‘º¶ëŽ'¥}uÁg¾~³Ûø1^æ¨ÙÓ—‚ƒyªVmi^ÓŠ»j•|™4<µDûÔýÉN¹C^Ó+Ja{…¨±7™ê—–²˜÷m)âlÅ­bô«o¨f—™:õâ¡À{ÇÏ®V÷$Î~s7‹‚ÒÙ/÷`‹šÔ½Óä‘×M2)÷IR`Fd$¹œ2»Õ W jyñK¨*eÉë&˾ArUUoYŽbÓoCˆÆzcr¦]|öþ~ôø«-¦…QÂíº¯S®dNÏ#¯dÆÇÒ+¢]Ù¾æ„Lõ¨(@¼º{d’)RŒ–€Þfö#lÒ1Ч¬ûÄZKÁµÄ™T0ª‹žG)=6 ï6 'à^Ü1¹eaºtôŒô¿Äq•Q…åÒ.{á1UcU‰w¹ûš‚kˆ•Ug?.±†•ØW]¨rBŽÚÇa_<ì<ó¥“eޱœÜËŠÅÅÈm]`z¢v±*Êp°ÊÑSA‹iß”(ûJj˜_HhyâŽô bÿ”FçÑ•t59©TÎét-1íÃØÿ‚t]ÁïÙØÁræÅóݘ©Ç›ñv‡RKÞƒ§GMVÓ{ñ4¬t=½o/½Ÿ>€U²ùvr1QcØï$g[äaw”\›üîÃä9~”´M‘ä\-sGÉ·é0•øAÁßA/‹@;@åTºNʼ‰.".¦jÄ×Åøº„fÒR…&„u3iÝ@ ð¦чñÖh>}D z£Âìx•}t¥F£cÆ…/.p¹‰n¶EߤÔC4)‡þ!Ò\Èå¼KéSÀøÔ‚foQ̧Zò,},}ÂrX.EÅ¢[è6c¢5aˆ=¡½æaš“ÓÅD[þIxÔZJ©"ÿ:?·dr‘%5X3åPäT؆h=ôÖTK¨6+Àa‹5}’>E¤FŸV€et+Fìj¥Ï`äTBxEˆZÂð6Úoú,óâ½Dd ÒÔáHMyÀpÑ´p¤|ˆ¦7j5A ƒ`Às3Ãç •6èþ}ˆÊÂþòÈU cž£ÈíÁó š.ƒφûVÁyçÍ…p×ÅpS‘áMX5—ÆÓgés¤ ú<<ÔèÝ¡¤YB_ ;•q–Ðé.å•KèK¹ÔèËØëV£¯`ä¿Ùt;4%µNü*-ÖèPÞSîÏ{è^[ü¿c†wGÎ*3aˆòA ùϳ-r¬ü6ÛyDÚHÀ¤YþÙ“&Âü绮멲ÑôÔ8±mNÐ3H(µU «Ärü› šOæ~º·ìzˆ+jXiAÈ ß‘¾#/|G^øŽ¼ð¶ðK è¯ŽG‰\NŽ.Ÿ”X×hH£#Ÿ'HÌò=ÜGGm=Çœ€¬7¨n/»_otÕzƒ®A Ã5#pœtË è>tĦò#ÿHNš¡Þ–¤J’C u7VÜ‹5ð(:ŒP )i(sU業ÌK[©dÇI÷Ó×@³ºcº”d¥ä8 RnK*Öè˜ïãú±¾Nß°"ÔÑ ^¾¼ì¯)».rìº!ªu.v4¸œ l?Í ÜóaÓù·Òøò[H ¸\ûœ¼ÏÍNþ¶üÓô€¿Îý5ªßä„Açvlr•Ékˆ^×è¾›æÝGh¾ƒ&ú<΀k¢¯gˆ4hMÆ—cåë÷Ó•‘€6¿AËÓ+ñÒÐèó_èQ$šES¦/êØäúå’ÞJóýëgZ¶É𪚼¬m¨DÚ§ýApã°ö*+=ˆ4ò4ò0M Gh:=Š`yŒ. Ç)Bß¡:Ì- ï#Eÿ€ÖÑh ý”¶ÒÏh;ýœvÓStýÁûK”‹_ÁR¿†®C¿Cy–ž¤Пé%ú«²ð>ËÏ¢`M7v%è›à¬c~RÓƒ°ÐõdÒ· | ³ ˆ®…¥·Ãb²CïÛðZìÝŠ¨ysâš/«X 5Ÿ`5²<ƆGU|LŽÇT|ˆÇ´’~‚üpþéŽWÈ}‚^üÇ|’v‘×±âBM©þ': ÚˆDÿQõåqú¶øßh¾¼tbè6ò–~xˆô_âR+ÓÀ$óº¯âößâE }¹ ô–Cëâ}£Zˆâò]›Ëý˜ÁçKn ;L‹Ú«kŽ-v6 í”ï§`MÀ5¿Á‘TÝ€lµqŸ †<~˜·UB^ƒ¦O :O" Ÿ :$Á°ä+ðE0HœÌ¢'T¶™€øûžÒ¦éôûv¶™¯¸SUÃjú!¼ÃªÓ$&ý“ÐØ•ÒÄ,3 ÉOà>–$‡@@`„‘;·EŽ-B~iWb §Ö\‘€kÁ$:yÕ‡:• ©@¥v“›=¤Ãë`/Í»ŠKò)eüjN{­êl*áºO…ár®R¡qß ðaý§Xð½‚ÈðÂ×iÃ_ŠÅ’ƒu?ªä-n¦TÁIãI]„Žø°r›Ž(‘¢ì•«Y›Ú7Q@®Ñ2t&ÈQjÚTs˜–Ãr+qK7 yíˆç\MjqÃÒ)\ŠG VZ-C«tGh•SÒ ˆ–¡šGèm*¹x:ô¤R® r.£ .§6ž™ï!*ö7 ŒÅEWÓï•Þd$"°ýZuV5u–6iô‡:ôGzÚ¨O\ï@>Bo`´a³ зåЯ¶*êjwÐpbRÁÁ³¨„gS„+ ‚£ÆFcGLŸ¡?ÙLvƒZUÀiÍ L"Gh­Õmݦ¦õ«kŽçmf1¬§2žKsx^¾õ-•uHþL§9ØóWäxÿ-/ð|[`¯ÐGwÝ1L\u×¼°@ ¯MÔKÏÀŠ¢žG»b•ë½pmñ¡)×í9\fµ”­þ ÃþQvQø¢uäáõPù†‚v²&_•kòU¹9ÛªÊsèäyçˆZûOú—iޢŅiuͽñT–—ƒ¥I>î*PzIže‰ÍÒáþ FŽS½hk¯_ØŒœØ›,»‹´xmFŸvj„Æàþ; t:C1¾„deáÿ¢nç }¤-.'š wÍC4¢¼©Ñt݃ôfé‰.ºOMPW ÈöÒ8¾’&ñU4¯F ]C³ùùL+­qNÔ9yíÎAq¹Ve9ª›s"È*!ø8¤´ÿ"U¹U”¿F½zj7ûZÞm˜RÕGÂF×ú–§ÚáýHX  |CAÒš˜7ÑWÈãJ‡Åã(H._öoE2¼<)«Åêø0Df£+g•®œ3J²­I\–«#CwîÈàîùF€ú(çÑtþ8]Ì7ÑR¾¹ c\–¸ г´· Õæs 2!õY¥½éTÍhÔUŸì²áÆœô¨Õ§7ºA×õäÚ܃5À9V\?o¥Þ¥•o‡OßìõôwRïÊC.Ar«æUÎÛ9ðÕyÓW£¢Y5ÛnwoTTÑŸ„È5Üäªw„8jŸ s•툦ç´•LÛ;D}–¾ÅM¹ ±Ân/ûle4Úçd”µmOIB_-ðO¾¤yN÷.aÃÆÓf—ÊIÇlˆ¶ã)äq¤€Çtg§ßé<ž'HÙDiÍùöMöa{iØ¿Ãò;É26ågq«:JÈgÄþ‚³µJ ‰a·³ì÷Üîaî#TÉÒ~¢ .ÍK½^¤G{Aå¦JšÁ~uº.ÔÄdžbkâFP· •æzÒý4½Î t=—Ûγµ§´jü$*öSÀó há)$‘_åýIC·’ëpBª›‘?!`Äj4Uù“ ¥mr¾Ã1ÈñšjhùpðÊ¿VÚ½ÞNØ…¡šä-ù:ª.%leúßj –bq¦Õé‡k¸_0ñ³HÏ¡¹|ž.ä 2üEy-^¤J«‘x‘#WÛ/¡Ä`âç•8DuÂ4_çÀT`z¸Öêí‡ì+®ÊŒPíIòÀϪ.ªq0Õ:ÜyÕJZµ­3·r©^—^—^W\#G\ã4|ú+žÛðöáË^qÕ¬ø€9Fn Ž5hÎ= ´ýTôôyÞ ;è¤i¢ovm”_»!æž›ÉãÂÙT‡Ï|/çÜoË©àŠ\ªÝ+©Öm)#ಠ¡¥‚°`qø¡‚ÉäuL!¿#@ÇTZè˜F—:¦Ó*ü^ë˜AW8*”Zz±>B3Ñ5þù?@ é ĤɺA4ôšóÕá@nnöæóÛ^¤_+¿íE—y‡Êo;U<;A'Í!¤i©ó²$0LMùg3-±t;—q¬˜EÝ–32IÝ"S¹‘µuËéÎhö å:¤ÛÑÙ§ K›vAdï2,}À1­ëbÞæí12:yœÝRG9¡Tá]Áš[²(XeØâ}¿¥;z6I”=f›^¤·Ì°˜¦Ã›çÀp³‹»ú’]øVb*·§4ݱ :7©÷çÒ î›`Àb%2f:Mà‰Ó+j}EG®Ð‘b>r)ˆÞJžÈ.ýaû5+M—.ö§@•øvŸP°÷ÿ@Xt²ûÄ€>ìfVb,Ñ)±Êóš>q,m€Ù_=¬YŽ!E³Ú¾Ç2‡Üª!w¬Ý3mC‚kŠHi§® ­Û?_’ØŽ>ÔâMàÆÚK‘Bˆw¹†)ušÎ3—MSU hï*>wÓYjž?]p “(C:e˜t[Cs×öÝF…D‘*Ϋ¸ â ¶GöFÌŸP,ÅUNÕžÝcºö‹Œ½µŽÛì4ŠÁj´uNci2ƒûs)‹Ži#©9ú]Kî9ªÆúÚÚÛE•’$%"\Ò×F3•‰¯g ¼ÛÌYº@Á>;Ý7 Åa|›,›£Ña\Ä›‚ì·aŒc‚àa\Âå0ÞËüNÅÛaLâj÷aw¿Ç@ÅÃø.«x'Œ?‹Ëwq-Œ}xOˆù‹Ø½Æuqq?Þc4×oº;Io.9„ñWüMXcG˜0þŽÙÌæÍ3 Š6ŠE³¦M‰;´GâÑ=$ÔH™Ü¥m¡0†ñEz¿oý¦æäCÑõtòêiL¹Lf?‡‰£[E"6Šãhÿˆ£G5ËÒF¢FÖbˆbnH¢ †­; Q†#í Ö g~¤ vZl{6©ŸèÊ9]©6Õ.’½¬SOkŽqLºÒò¢‹ÿ Û¨‹¸kæÓÇš+tùs²U¶Íi¦x¶´_1³É)u¶ñ¨î^Äoæ²Ü¢~94ç,ÛY0U]¥.hùÙÀæ¹Y¯ ë"¶c}viMå ¹ìÆXåÅ*æ+[‡å䉂N¡Î—Ä#M_ÌÛVR–PJÛ±ƒ:Úx_ßǨS±S …hØ. r Û›÷Ê:PüÊG¡úÇà÷½^€è:iO‘“–ðí³|¶É[_…o;»¾˜`y¹»PÐ-¤ÜpÙ³§çùù"‘ ‹DP²@Jl—…ÄN%È–Ç# ÆQqaéðйÐ>P$z9>‡Ži´öiÎÐμì$9}|VÅ/c¡‚QTÄë.aÑeT–àBAp…$JVš0¤‚¨Ëˆ.ì—‘¨¢MMRi › Û$¿²„l=8˜×hå5ÖNi¬‰‹Ôš@Ddd1s¤á2–øfé·(ÈÆZäŠô×ô×ô×âOë_K‹ï÷´¸j†ÅKg[|’Ÿ ÅO޹ŽŒƒ«±…AD•‘eþ·±¼×·òVtSåù›jñ™¢ÀU¢ÊZ|_ )M”àÙPý5”FJÏçýYIÇÕ»ŽëŒsS½o ›·úó×¼¬â"joæé]\Vø]ÎóE ‘è%¬É³È’Zì‘Ï­XwQ!®Ï3ß_ägÛK¼=Ãû1R¾ÌŸúgIù Ëô#ÿª4¬•fn 3ÆòlcÉ? M–óú1 ]:ÄøètÀfmŠÉ&tÀ šòå¼~õ‘øU*ŽRï#PTdªº•Sç*²â0ù/êq5É´|D^#ŸÐ»]6µÉJœýŒ +>y±†ÈÚ€ Ô*¨^|U³—ëåÀá§|ÊØÏleKe`Þ`Ú\äþMj¹È¸ZÖ"Ô3}” ´]Z¹³eU°Ý9²yµÉ(ù‹DËúy˘îÇ\ÜŠ°EH8¿†WPÛY7õW«_@ì 6ôNåÃ'ˆ²z±HrÜãê¯ÖÜDH5^„Õñɳh.&mœ’¹‘›ø¤Ótñl jð+ÞwhÛ»ˆà=qù€±ÿö„ç㤜:…ã8!}r #ô“"wrW’ï]þŠXLL´/É¢ùòÿPK lL[DMETA-INF/maven/PK lL[D!META-INF/maven/org.apache.hadoop/PK lL[D2META-INF/maven/org.apache.hadoop/hadoop-openstack/PK »T]C®îóÑvh9META-INF/maven/org.apache.hadoop/hadoop-openstack/pom.xmlÍWmoÛ6þî_ÁùÓD’dC—).¼$EŒ¥v9ëŠa(h‰V˜H¤FR±ýïw|‘#ÙrìtëÐ|‰EžŽwÏ=Ïé¾]æz"BRÎλ}¿×E„Å<¡,=ïÞMßyoºoð;Ïë tCcÂ$IPÉ"º'hXàþ¹#ô»u…Žýú^tÝV÷‡_ÀÊ—(Ç+ĸB¥$à‚J4§Ad“B!ÊPÌó"£˜Å-¨º7Ç8'>¸øä\ð™Â`Á¾€§yÝaÕS„î•*΂`±XøØDês‘™µ’ÁÍèâj]y­¶¿c‘ òwI¤9[!\@,1žA„^ .N=Åu¬ A@u„$Ÿ«¼$T*Ag¥jUEùÖ *ÌPw¡QÔE¿£Qt>>ަד»)ú8¼½ާ£«MnÑÅd|9šŽ&cxz‡†ãOè·Ñøò€ Ž!ËBèø!Hª!$‰Æ+"¤ÀœÛ€dAb:§1äÅÒ§¥ˆÀ T‘S© )!¼¼d4§ +³²•”oÁ±®f+íÁ!kjëw<8Tþ@b…€qLžw]irüDX½8&ïƒS¿L„cíÙRÒµ½.åâÄX÷zýà÷7¼šc2©4kÌ{’žI³|Ãcö¢]fÁ{:ýÜûÜó—2éàˆ0ç É߯A4Ö,ÃTð²%ðT9½Ç çET[Ö Eç8V°` <^TüµMkíD;8Ñ'{Ñxø!ºžLàZ·V çdàDzmœ¢ 8´S$Ë¢àB…±²/$DÆ‚jíBÿMµL!·ds¦…'áGB´ œƒ"©0`[á®ò׎.J!SÙÊ*|IƒÔòņ+r%ÉQ ìeJ{'(Á ¶Ï׎€•F|Äî!Èè9³hAç Z„!œT\Clå88Ò°ƒçG]?ÍZPƒ¢D:sÃèªGL‹ ƒæ¢C’/XÆqñRÄðº%ó7V] Vþ¬¤YâK³wµqÆ‹6:Ô «‹Ý`ú|ˆ~¬ÊÒd  ïñù< h²ÞšÑ'ܤ@XÕ,˜GKÚOÍÇ&àRÝ{d%}0¨ÂY>; Ûnán4ÊÔ‡ùò‘ÒÍÕµÛ­Ú ^„}d©©|Îð?GcŽ3ù…ptjOŽ<†fPY Š®£­j±´49‡{u£×YXŸ[ší°ôìÙ¶žÕ‰' ´õU}ù¥À¾¤¸%¬ÍÐŒg›Ÿ—cñ¨Üžyµê¿}ÿd£Wuh‹µ¶Ú,2´È9MK±A?ÍVV”j³I4ëæ¼T-ö«µxZ’4*ÿ•ˆàZ|ÒçÜD^d+1ÁéîK_®L^1=Ï%•^vî·;©ŠÝn¶,¾Tõxú/Kl7¶± ÷»¾­°H‰²Ûî÷믞¤ýš¹ÔÙ’ýº µ‹þàk'7vÅ0/ìRxMßÇ~KáMu¿"P=MÝãRú0HýÚ«³ƒ–T@W÷°Ìùg¿ÜÞ•$\£ÈÀQêk[“úrÒø&3ªèí˜3˜RÛÄ·A+mn&Ú½ «~ü·ùT‹pkGwú¾®ï|zèÔ73= õlbiÐæ“ `£wú6óôl¥ÈŸ!}u; ˆ,‡¦çQ¾ºé@ŽýÓ¯GÎãGp½µ¼"YMãBíÏÖÙy8;„Ðo^®¿.ßAÙîIà¡dTí¤µÚKÚþv'|e‘星ržÂm+-ñSûüVOÀYíH`'¤ÍšèËý­~s†[_ÙPK kL[Dtïhly@META-INF/maven/org.apache.hadoop/hadoop-openstack/pom.properties%ÉK Â0Ð}N1ÐuCL±Ð…? µ\`šL›"dB𼽂Û÷ª+EÊXÈÃðnEeà @AÚæÐj ½± Õn/6ÊË̱k¤’ª6Ïson/+¦ÌkºûŽó$1¡ $zæ$0—yDW~ù—šÅ¥ {‹/PK lL[D íAMETA-INF/PK kL[D Ÿ6Éf}¤'META-INF/MANIFEST.MFPK jL[DíA¿org/PK jL[D íAáorg/apache/PK jL[DíA org/apache/hadoop/PK jL[DíA:org/apache/hadoop/fs/PK kL[DíAmorg/apache/hadoop/fs/swift/PK kL[D íA¦org/apache/hadoop/fs/swift/auth/PK jL[D)íAäorg/apache/hadoop/fs/swift/auth/entities/PK kL[D&íA+org/apache/hadoop/fs/swift/exceptions/PK kL[D íAoorg/apache/hadoop/fs/swift/http/PK kL[D#íA­org/apache/hadoop/fs/swift/snative/PK kL[D íAîorg/apache/hadoop/fs/swift/util/PK kL[DÑ«#|‚A¤,org/apache/hadoop/fs/swift/auth/ApiKeyAuthenticationRequest.classPK kL[DÝó‰G˜j7¤org/apache/hadoop/fs/swift/auth/ApiKeyCredentials.classPK jL[D­Ì‡×”;¤ôorg/apache/hadoop/fs/swift/auth/AuthenticationRequest.classPK kL[DY¿cs6B¤$ org/apache/hadoop/fs/swift/auth/AuthenticationRequestWrapper.classPK jL[DÕ‘÷0Œ^<¤÷ org/apache/hadoop/fs/swift/auth/AuthenticationResponse.classPK kL[D‡g¤žQÒ;¤Ýorg/apache/hadoop/fs/swift/auth/AuthenticationWrapper.classPK jL[D¸â¿f:¤‡org/apache/hadoop/fs/swift/auth/entities/AccessToken.classPK jL[D{ÒUº6¤žorg/apache/hadoop/fs/swift/auth/entities/Catalog.classPK jL[D¼ß±\lã 7¤¬org/apache/hadoop/fs/swift/auth/entities/Endpoint.classPK jL[DéöŽõI5¤morg/apache/hadoop/fs/swift/auth/entities/Tenant.classPK jL[Dص$Fâ’3¤ org/apache/hadoop/fs/swift/auth/entities/User.classPK jL[Dó~(jŽM?¤¤‘ùorg/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.classPK kL[Di BÅE¤œorg/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore$1.classPK kL[DŽRà#¥,_hC¤Äorg/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.classPK kL[DËBòëü«?¤ÊAorg/apache/hadoop/fs/swift/snative/SwiftNativeInputStream.classPK kL[Dæ[ iÌL @¤#Qorg/apache/hadoop/fs/swift/snative/SwiftNativeOutputStream.classPK kL[Dƒ”©˜£>¤Maorg/apache/hadoop/fs/swift/snative/SwiftObjectFileStatus.classPK jL[D|¶Jƒ·.¤Aeorg/apache/hadoop/fs/swift/util/Duration.classPK jL[D°á:oóÖ 3¤Dhorg/apache/hadoop/fs/swift/util/DurationStats.classPK jL[D‘wub 8¤ˆmorg/apache/hadoop/fs/swift/util/DurationStatsTable.classPK kL[Dó”FËÞ .¤ìrorg/apache/hadoop/fs/swift/util/JSONUtil.classPK kL[D âÛ?O 5¤ûworg/apache/hadoop/fs/swift/util/SwiftObjectPath.classPK jL[DÏóOlâ24¤~org/apache/hadoop/fs/swift/util/SwiftTestUtils.classPK kL[D¬î8§Ä0¤K–org/apache/hadoop/fs/swift/util/SwiftUtils.classPK lL[Dÿÿ]ŸMETA-INF/maven/PK lL[D!ÿÿŠŸMETA-INF/maven/org.apache.hadoop/PK lL[D2ÿÿÉŸMETA-INF/maven/org.apache.hadoop/hadoop-openstack/PK »T]C®îóÑvh9¤ META-INF/maven/org.apache.hadoop/hadoop-openstack/pom.xmlPK kL[Dtïhly@¤æ¥META-INF/maven/org.apache.hadoop/hadoop-openstack/pom.propertiesPKZZ #½¦sahara-8.0.0/sahara/plugins/mapr/services/mysql/0000775000175100017510000000000013245515027021655 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mysql/mysql.py0000666000175100017510000002046513245514472023410 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import collections as c from oslo_log import log as logging import six import sahara.plugins.mapr.domain.configuration_file as cf import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.general as g import sahara.utils.files as f LOG = logging.getLogger(__name__) db_spec = c.namedtuple('DatabaseSpec', ['db_name', 'user', 'password']) class MySQL(s.Service): METRICS_SPECS = db_spec('metrics', 'maprmetrics', 'mapr') HUE_SPECS = db_spec('hue', 'maprhue', 'mapr') METASTORE_SPECS = db_spec('metastore', 'maprmetastore', 'mapr') RDBMS_SPECS = db_spec('rdbms', 'maprrdbms', 'mapr') OOZIE_SPECS = db_spec('oozie', 'maproozie', 'mapr') SENTRY_SPECS = db_spec('sentry', 'maprsentry', 'mapr') SELECT_DATA = 'mysql -uroot --skip-column-names -e "%s"| grep -E "\w+"' GET_DBS_LIST = SELECT_DATA % 'SHOW DATABASES' GET_USERS_HOSTS = ( SELECT_DATA % "SELECT Host FROM mysql.user WHERE mysql.user.User='%s'" # nosec ) DRIVER_CLASS = 'com.mysql.jdbc.Driver' MYSQL_SERVER_PORT = 3306 MYSQL_INSTALL_SCRIPT = 'plugins/mapr/resources/install_mysql.sh' INSTALL_PACKAGES_TIMEOUT = 1800 def __init__(self): super(MySQL, self).__init__() self._ui_name = 'MySQL' @staticmethod def _get_db_daemon_name(distro, distro_version): if distro.lower() == 'ubuntu': return 'mysql' if distro.lower() == 'suse': return 'mysqld' if distro.lower() in ['centos', 'redhatenterpriseserver', 'redhat']: if distro_version.split('.')[0] == '7': return 'mariadb' return 'mysqld' return None @staticmethod def _execute_script(instance, script_path, script_text=None, user='root', password=None): with instance.remote() as r: if script_text: r.write_file_to(script_path, script_text, run_as_root=True) LOG.debug('Executing SQL script {path}'.format(path=script_path)) r.execute_command(("mysql %s %s < %s" % ('-u' + user if user else '', '-p' + password if password else '', script_path)), run_as_root=True) @staticmethod def _create_service_db(instance, specs): f_name = 'create_db_%s.sql' % specs.db_name script = MySQL._create_script_obj(f_name, 'create_database.sql', db_name=specs.db_name, user=specs.user, password=specs.password) MySQL._execute_script(instance, script.remote_path, script.render()) @staticmethod def _create_metrics_db(instance, databases, instances): if MySQL.METRICS_SPECS.db_name not in databases: MySQL._create_service_db(instance, MySQL.METRICS_SPECS) MySQL._execute_script(instance=instance, script_path='/opt/mapr/bin/setup.sql') MySQL._grant_access(instance, MySQL.METRICS_SPECS, instances) @staticmethod def _create_hue_db(instance, databases, instances): if MySQL.HUE_SPECS.db_name not in databases: MySQL._create_service_db(instance, MySQL.HUE_SPECS) MySQL._grant_access(instance, MySQL.HUE_SPECS, instances) @staticmethod def _create_rdbms_db(instance, databases, instances): if MySQL.RDBMS_SPECS.db_name not in databases: MySQL._create_service_db(instance, MySQL.RDBMS_SPECS) MySQL._grant_access(instance, MySQL.RDBMS_SPECS, instances) @staticmethod def _create_metastore_db(instance, cluster_context, databases, instances): hive_meta = cluster_context.get_instance('HiveMetastore') if not hive_meta: return db_name = MySQL.METASTORE_SPECS.db_name if db_name not in databases: MySQL._create_service_db(instance, MySQL.METASTORE_SPECS) MySQL._grant_access(instance, MySQL.METASTORE_SPECS, instances) @staticmethod def _create_oozie_db(instance, databases, instances): if MySQL.OOZIE_SPECS.db_name not in databases: MySQL._create_service_db(instance, MySQL.OOZIE_SPECS) MySQL._grant_access(instance, MySQL.OOZIE_SPECS, instances) @staticmethod def _create_sentry_db(instance, cluster_context, databases, instances): sentry_instance = cluster_context.get_instance('Sentry') if not sentry_instance: return if MySQL.SENTRY_SPECS.db_name not in databases: MySQL._create_service_db(instance, MySQL.SENTRY_SPECS) MySQL._grant_access(instance, MySQL.SENTRY_SPECS, instances) @staticmethod def start_mysql_server(cluster_context): LOG.debug('Starting MySQL Server') instance = MySQL.get_db_instance(cluster_context) distro = cluster_context.distro distro_version = cluster_context.distro_version with instance.remote() as r: r.execute_command(('service %s restart' % MySQL._get_db_daemon_name(distro.name, distro_version)), run_as_root=True) LOG.debug('MySQL Server successfully started') @staticmethod def get_databases_list(db_instance): with db_instance.remote() as r: ec, out = r.execute_command(MySQL.GET_DBS_LIST) if out: return out.splitlines() return list() @staticmethod def get_user_hosts(db_instance, username): with db_instance.remote() as r: ec, out = r.execute_command(MySQL.GET_USERS_HOSTS % username) if out: return out.splitlines() return list() @staticmethod def get_db_instance(cluster_context): return cluster_context.oozie_server @staticmethod def create_databases(cluster_context, instances): db_instance = MySQL.get_db_instance(cluster_context) databases = MySQL.get_databases_list(db_instance) MySQL._create_metrics_db(db_instance, databases, instances) MySQL._create_hue_db(db_instance, databases, instances) MySQL._create_rdbms_db(db_instance, databases, instances) MySQL._create_oozie_db(db_instance, databases, instances) MySQL._create_metastore_db( db_instance, cluster_context, databases, instances) MySQL._create_sentry_db(db_instance, cluster_context, databases, instances) @staticmethod def _create_script_obj(filename, template, **kwargs): script = cf.TemplateFile(filename) script.remote_path = '/tmp/' script.parse(f.get_file_text( 'plugins/mapr/services/mysql/resources/%s' % template)) for k, v in six.iteritems(kwargs): script.add_property(k, v) return script @staticmethod def _grant_access(instance, specs, instances): f_name = 'grant_access_%s.sql' % specs.db_name ips = [i.internal_ip for i in instances] user_hosts = MySQL.get_user_hosts(instance, specs.user) script = MySQL._create_script_obj(f_name, 'grant_access.sql', hosts=set(ips) - set(user_hosts), db_name=specs.db_name, user=specs.user, password=specs.password) MySQL._execute_script(instance, script.remote_path, script.render()) @staticmethod def install_mysql(instance, distro_name, distro_version): g.run_script(instance, MySQL.MYSQL_INSTALL_SCRIPT, 'root', distro_name, distro_version.split('.')[0]) sahara-8.0.0/sahara/plugins/mapr/services/mysql/__init__.py0000666000175100017510000000000013245514472023761 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mysql/resources/0000775000175100017510000000000013245515027023667 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mysql/resources/grant_access.sql0000666000175100017510000000033513245514472027052 0ustar zuulzuul00000000000000{% for host in hosts %} CREATE USER {{ user }}@'{{ host }}' IDENTIFIED BY '{{ password }}'; GRANT ALL PRIVILEGES ON {{ db_name }}.* TO {{ user }}@'{{ host }}' WITH GRANT OPTION; {% endfor %} FLUSH PRIVILEGES; FLUSH HOSTS;sahara-8.0.0/sahara/plugins/mapr/services/mysql/resources/hive_schema.sql0000666000175100017510000000004513245514472026667 0ustar zuulzuul00000000000000USE {{ db_name }}; SOURCE {{ path }};sahara-8.0.0/sahara/plugins/mapr/services/mysql/resources/create_database.sql0000666000175100017510000000055513245514472027511 0ustar zuulzuul00000000000000CREATE DATABASE {{ db_name }}; CREATE USER {{ user }}@'localhost' IDENTIFIED BY '{{ password }}'; CREATE USER {{ user }}@'127.0.0.1' IDENTIFIED BY '{{ password }}'; GRANT ALL PRIVILEGES ON {{ db_name }}.* TO {{ user }}@'localhost' WITH GRANT OPTION; GRANT ALL PRIVILEGES ON {{ db_name }}.* TO {{ user }}@'127.0.0.1' WITH GRANT OPTION; FLUSH PRIVILEGES; FLUSH HOSTS;sahara-8.0.0/sahara/plugins/mapr/services/spark/0000775000175100017510000000000013245515027021630 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/spark/__init__.py0000666000175100017510000000000013245514472023734 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/spark/spark.py0000666000175100017510000003202313245514472023327 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.services.hbase.hbase as hbase import sahara.plugins.mapr.services.hive.hive as hive import sahara.plugins.mapr.util.general as g import sahara.plugins.mapr.util.maprfs_helper as mfs import sahara.plugins.mapr.util.validation_utils as vu import sahara.utils.files as files SPARK_SLAVE_UI_PORT = 8081 SPARK_HS_UI_PORT = 18080 SPARK_HISTORY_SERVER = np.NodeProcess( name='spark-historyserver', ui_name='Spark HistoryServer', package='mapr-spark-historyserver', open_ports=[SPARK_HS_UI_PORT] ) SPARK_SLAVE = np.NodeProcess( name='spark-master', ui_name='Spark Slave', package='mapr-spark', open_ports=[SPARK_SLAVE_UI_PORT] ) class SparkOnYarn(s.Service): JAR_FILE_TARGET = '/apps/spark/lib' MFS_DIR = '/apps/spark' SERVLET_JAR = 'javax.servlet-api.jar' def __init__(self): super(SparkOnYarn, self).__init__() self._name = 'spark' self._ui_name = 'Spark' self._version = '1.5.2' self._dependencies = [('mapr-spark', self.version)] self._node_processes = [ SPARK_HISTORY_SERVER, SPARK_SLAVE, ] self._validation_rules = [ vu.exactly(1, SPARK_HISTORY_SERVER), vu.at_least(1, SPARK_SLAVE), ] self._ui_info = [ ('Spark History Server', SPARK_HISTORY_SERVER, {s.SERVICE_UI: 'http://%%s:%s' % SPARK_HS_UI_PORT})] def _get_packages(self, cluster_context, node_processes): result = [] result += self.dependencies result += [(np.package, self.version) for np in node_processes] hbase_version = self._get_hbase_version(cluster_context) hive_version = self._get_hive_version(cluster_context) if hbase_version: result += [('mapr-hbase', hbase_version)] if hive_version: result += [('mapr-hive', hive_version)] return result def get_config_files(self, cluster_context, configs, instance=None): hbase_version = self._get_hbase_version(cluster_context) hive_version = self._get_hive_version(cluster_context) # spark-env-sh template = 'plugins/mapr/services/' \ 'spark/resources/spark-env.template' env_sh = bcf.TemplateFile('spark-env.sh') env_sh.remote_path = self.conf_dir(cluster_context) env_sh.parse(files.get_file_text(template)) env_sh.add_property('version', self.version) env_sh.add_property('servlet_api_jar', self.SERVLET_JAR) # spark-defaults conf = bcf.PropertiesFile('spark-defaults.conf', separator=' ') conf.remote_path = self.conf_dir(cluster_context) if instance: conf.fetch(instance) conf.add_property('spark.yarn.jar', 'maprfs://%s/%s' % (self.JAR_FILE_TARGET, self._assembly_jar_path(cluster_context) .rsplit('/', 1)[1])) # compatibility.version versions = bcf.PropertiesFile('compatibility.version') versions.remote_path = self.home_dir(cluster_context) + '/mapr-util' if instance: versions.fetch(instance) if hive_version: versions.add_property('hive_versions', hive_version + '.0') conf.add_properties(self._hive_properties(cluster_context)) if hbase_version: versions.add_property('hbase_versions', hbase_version) conf.add_property('spark.executor.extraClassPath', '%s/lib/*' % self._hbase(cluster_context) .home_dir(cluster_context)) return [conf, versions, env_sh] def update(self, cluster_context, instances=None): pass def post_install(self, cluster_context, instances): pass def configure(self, cluster_context, instances=None): pass def post_start(self, cluster_context, instances): self._create_hadoop_spark_dirs(cluster_context) self.copy_assembly_jar_to_mfs(cluster_context) self._copy_hive_site(cluster_context) self._copy_hbase_site(cluster_context) self._copy_jar_from_hue(cluster_context) def copy_assembly_jar_to_mfs(self, cluster_context): target = self.JAR_FILE_TARGET hdfs_user = 'mapr' with cluster_context.get_instance(SPARK_HISTORY_SERVER).remote() as r: mfs.copy_from_local(r, self._assembly_jar_path(cluster_context), target, hdfs_user) def _copy_hive_site(self, cluster_context): if not self._hive(cluster_context): return hive_site = self._hive(cluster_context).conf_dir( cluster_context) + '/hive-site.xml' path = self.conf_dir(cluster_context) + '/hive-site.xml' hive_instance = cluster_context.get_instance(hive.HIVE_SERVER_2) spark_instance = cluster_context.get_instance( SPARK_HISTORY_SERVER) g.copy_file(hive_site, hive_instance, path, spark_instance, run_as='root', owner='mapr') def _copy_hbase_site(self, cluster_context): if not self._hbase(cluster_context): return hbase_site = self._hbase(cluster_context).conf_dir( cluster_context) + '/hbase-site.xml' path = self.conf_dir(cluster_context) + '/hbase-site.xml' hbase_instance = cluster_context.get_instance(hbase.HBASE_MASTER) spark_instance = cluster_context.get_instance( SPARK_HISTORY_SERVER) g.copy_file(hbase_site, hbase_instance, path, spark_instance, run_as='root', owner='mapr') def _create_hadoop_spark_dirs(self, cluster_context): home = '/apps/spark' libs = self.JAR_FILE_TARGET run_as_user = 'mapr' with cluster_context.get_instance(SPARK_HISTORY_SERVER).remote() as r: mfs.mkdir(r, home, run_as=run_as_user) mfs.mkdir(r, libs, run_as=run_as_user) mfs.chmod(r, home, 777, run_as=run_as_user) mfs.chmod(r, libs, 777, run_as=run_as_user) def _hive_properties(self, cluster_context): hive_version = self._hive(cluster_context).version hive_conf = self._hive(cluster_context).conf_dir(cluster_context) hive_site = hive_conf + '/hive-site.xml' hive_datanucleus_libs = self._hive_datanucleus_libs_path( cluster_context) hive_libs = self._hive_libs_path(cluster_context) hadoop_libs = self._hadoop_libs_path(cluster_context) hive_datanucleus_libs.insert(0, hive_site) mfs_paths = self._hive_datanucleus_libs_path(cluster_context) return { 'spark.yarn.dist.files': ','.join(mfs_paths), 'spark.sql.hive.metastore.version': hive_version + '.0', 'spark.sql.hive.metastore.jars': ':'.join(hadoop_libs + hive_libs) } def _hadoop_libs_path(self, cluster_context): cmd = 'echo $(hadoop classpath)' with cluster_context.get_instance(SPARK_HISTORY_SERVER).remote() as r: result = r.execute_command(cmd, run_as_root=True, timeout=600) return result[1].replace('\n', '').split(':') def _hive_libs_path(self, cluster_context): cmd = "find %s -name '*.jar'" % ( self._hive(cluster_context).home_dir(cluster_context) + '/lib') with cluster_context.get_instance(hive.HIVE_METASTORE).remote() as r: result = r.execute_command(cmd, run_as_root=True, timeout=600) return [x for x in list(result[1].split('\n')) if x] def _assembly_jar_path(self, cluster_context): cmd = "find %s -name 'spark-assembly*.jar'" % ( self.home_dir(cluster_context) + '/lib') with cluster_context.get_instance(SPARK_HISTORY_SERVER).remote() as r: result = r.execute_command(cmd, run_as_root=True, timeout=600) if result[1]: return result[1].strip() else: raise Exception("no spark-assembly lib found!") def _hive_datanucleus_libs_path(self, cluster_context): cmd = "find %s -name 'datanucleus-*.jar'" % ( self._hive(cluster_context).home_dir(cluster_context) + '/lib') with cluster_context.get_instance(hive.HIVE_METASTORE).remote() as r: result = r.execute_command(cmd, run_as_root=True, timeout=600) return [x for x in list(result[1].split('\n')) if x] # hive installed service instance def _hive(self, cluster_context): hive_instance = cluster_context.get_instance(hive.HIVE_SERVER_2) if not hive_instance: return None hive_version = cluster_context.get_chosen_service_version('Hive') return cluster_context._find_service_instance('Hive', hive_version) # hbase installed service instance def _hbase(self, cluster_context): hbase_instance = cluster_context.get_instance(hbase.HBASE_MASTER) if not hbase_instance: return None hbase_version = cluster_context.get_chosen_service_version('HBase') return cluster_context._find_service_instance('HBase', hbase_version) def _get_hbase_version(self, cluster_context): return (self._hbase(cluster_context).version if self._hbase(cluster_context) else None) def _get_hive_version(self, cluster_context): return (self._hive(cluster_context).version if self._hive(cluster_context) else None) # hue installed service instance def _hue(self, cluster_context): hue_instance = cluster_context.get_instance('Hue') if not hue_instance: return None hue_version = cluster_context.get_chosen_service_version('Hue') return cluster_context._find_service_instance('Hue', hue_version) def _copy_jar_from_hue(self, cluster_context): if not self._hue(cluster_context): return jar_path = "%s/apps/spark/java-lib/javax.servlet-api-*.jar" % \ self._hue(cluster_context).home_dir(cluster_context) path = '%s/lib/' % self.home_dir(cluster_context) + self.SERVLET_JAR with cluster_context.get_instance('Hue').remote() as r1: for instance in cluster_context.get_instances(SPARK_SLAVE): with instance.remote() as r2: mfs.exchange(r1, r2, jar_path, path, 'mapr') class SparkOnYarnV161(SparkOnYarn): def __init__(self): super(SparkOnYarnV161, self).__init__() self._version = '1.6.1' self._dependencies = [('mapr-spark', self.version)] class SparkOnYarnV201(SparkOnYarn): def __init__(self): super(SparkOnYarnV201, self).__init__() self._version = '2.0.1' self._dependencies = [('mapr-spark', self.version)] def get_config_files(self, cluster_context, configs, instance=None): hbase_version = self._get_hbase_version(cluster_context) hive_version = self._get_hive_version(cluster_context) # spark-env-sh template = 'plugins/mapr/services/' \ 'spark/resources/spark-env.template' env_sh = bcf.TemplateFile('spark-env.sh') env_sh.remote_path = self.conf_dir(cluster_context) env_sh.parse(files.get_file_text(template)) env_sh.add_property('version', self.version) # spark-defaults conf = bcf.PropertiesFile('spark-defaults.conf', separator=' ') conf.remote_path = self.conf_dir(cluster_context) if instance: conf.fetch(instance) # compatibility.version versions = bcf.PropertiesFile('compatibility.version') versions.remote_path = self.home_dir(cluster_context) + '/mapr-util' if instance: versions.fetch(instance) if hive_version: versions.add_property('hive_versions', hive_version + '.0') conf.add_properties(self._hive_properties(cluster_context)) if hbase_version: versions.add_property('hbase_versions', hbase_version) conf.add_property('spark.executor.extraClassPath', '%s/lib/*' % self._hbase(cluster_context) .home_dir(cluster_context)) return [conf, versions, env_sh] def post_start(self, cluster_context, instances): self._create_hadoop_spark_dirs(cluster_context) self._copy_hive_site(cluster_context) self._copy_hbase_site(cluster_context) sahara-8.0.0/sahara/plugins/mapr/services/spark/resources/0000775000175100017510000000000013245515027023642 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/spark/resources/spark-env.template0000666000175100017510000001461713245514472027323 0ustar zuulzuul00000000000000#!/usr/bin/env bash # This file is sourced when running various Spark programs. # Copy it as spark-env.sh and edit that to configure Spark for your site. # Options read when launching programs locally with # ./bin/run-example or ./bin/spark-submit # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - SPARK_PUBLIC_DNS, to set the public dns name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # Options read by executors and drivers running inside the cluster # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data # - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos # Options read in YARN client mode # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2) # - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1). # - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G) # - SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 1G) # - SPARK_YARN_APP_NAME, The name of your application (Default: Spark) # - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests (Default: "default") # - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed with the job. # - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job. # Options for the daemons used in the standalone deploy mode # - SPARK_MASTER_IP, to bind the master to a different IP address or hostname # - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master # - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y") # - SPARK_WORKER_CORES, to set the number of cores to use on this machine # - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g) # - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker # - SPARK_WORKER_INSTANCES, to set the number of worker processes per node # - SPARK_WORKER_DIR, to set the working directory of worker processes # - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y") # - SPARK_DAEMON_MEMORY, to allocate to the master, worker and history server themselves (default: 1g). # - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y") # - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y") # - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y") # - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers # Generic options for the daemons used in the standalone deploy mode # - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf) # - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs) # - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp) # - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER) # - SPARK_NICENESS The scheduling priority for daemons. (Default: 0) ######################################################################################################### # Set MapR attributes and compute classpath ######################################################################################################### # Set the spark attributes export SPARK_HOME=/opt/mapr/spark/spark-{{ version }} # Load the hadoop version attributes source /opt/mapr/spark/spark-{{ version }}/mapr-util/hadoop-version-picker.sh export HADOOP_HOME=$hadoop_home_dir export HADOOP_CONF_DIR=$hadoop_conf_dir export SPARK_LIBRARY_PATH=$MAPR_HADOOP_JNI_PATH # Enable mapr impersonation export MAPR_IMPERSONATION_ENABLED=1 MAPR_HADOOP_HBASE_VERSION=$(ls -1 /opt/mapr/hadoop/hadoop-0.20.2/lib/mapr-hbase-[0-9]*.jar 2> /dev/null | head -1) MAPR_HADOOP_CLASSPATH=`hadoop classpath` MAPR_HADOOP_JNI_PATH=`hadoop jnipath` MAPR_SPARK_CLASSPATH="$MAPR_HADOOP_CLASSPATH:$MAPR_HADOOP_HBASE_VERSION" SPARK_SUBMIT_CLASSPATH=$SPARK_SUBMIT_CLASSPATH:/opt/mapr/spark/spark-spark-{{ version }}/lib/{{ servlet_api_jar }}:$MAPR_SPARK_CLASSPATH # Load the classpath generator script source /opt/mapr/spark/spark-{{ version }}/mapr-util/generate-classpath.sh # Calculate hive jars to include in classpath generate_compatible_classpath "spark" "{{ version }}" "hive" MAPR_HIVE_CLASSPATH=${generated_classpath} if [ ! -z "$MAPR_HIVE_CLASSPATH" ]; then MAPR_SPARK_CLASSPATH="$MAPR_SPARK_CLASSPATH:$MAPR_HIVE_CLASSPATH" fi # Calculate hbase jars to include in classpath generate_compatible_classpath "spark" "{{ version }}" "hbase" MAPR_HBASE_CLASSPATH=${generated_classpath} if [ ! -z "$MAPR_HBASE_CLASSPATH" ]; then MAPR_SPARK_CLASSPATH="$MAPR_SPARK_CLASSPATH:$MAPR_HBASE_CLASSPATH" fi # Set SPARK_DIST_CLASSPATH SPARK_DIST_CLASSPATH=$MAPR_SPARK_CLASSPATH export SPARK_DIST_CLASSPATH # Security status source /opt/mapr/conf/env.sh if [ "$MAPR_SECURITY_STATUS" = "true" ]; then SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dmapr_sec_enabled=true" fi # scala export SCALA_VERSION=2.10 export SPARK_SCALA_VERSION=$SCALA_VERSION export SCALA_HOME=/opt/mapr/spark/spark-{{ version }}/scala export SCALA_LIBRARY_PATH=$SCALA_HOME/lib # Use a fixed identifier for pid files export SPARK_IDENT_STRING="mapr" ######################################################################################################### # :::CAUTION::: DO NOT EDIT ANYTHING ON OR ABOVE THIS LINE ######################################################################################################### # # MASTER HA SETTINGS # #export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url= -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.client=false" # MEMORY SETTINGS export SPARK_DAEMON_MEMORY=1g export SPARK_WORKER_MEMORY=16g # Worker Directory export SPARK_WORKER_DIR=$SPARK_HOME/tmp # Environment variable for printing spark command everytime you run spark.Set to "1" to print. # export SPARK_PRINT_LAUNCH_COMMAND=1 export SPARK_HISTORY_OPTS=" -Dspark.history.fs.logDirectory=maprfs:///apps/spark -Dspark.ui.acls.enable=true"sahara-8.0.0/sahara/plugins/mapr/services/hue/0000775000175100017510000000000013245515027021271 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hue/hue.py0000666000175100017510000003450213245514472022435 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import itertools import random import string from oslo_log import log as logging from sahara.i18n import _ import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.services.hbase.hbase as hbase import sahara.plugins.mapr.services.hive.hive as hive import sahara.plugins.mapr.services.httpfs.httpfs as httpfs import sahara.plugins.mapr.services.impala.impala as impala import sahara.plugins.mapr.services.mapreduce.mapreduce as mr import sahara.plugins.mapr.services.mysql.mysql as mysql import sahara.plugins.mapr.services.oozie.oozie as oozie import sahara.plugins.mapr.services.sentry.sentry as sentry import sahara.plugins.mapr.services.spark.spark as spark import sahara.plugins.mapr.services.sqoop.sqoop2 as sqoop import sahara.plugins.mapr.services.yarn.yarn as yarn import sahara.plugins.mapr.util.event_log as el import sahara.plugins.mapr.util.general as g import sahara.plugins.mapr.util.password_utils as pu import sahara.plugins.mapr.util.validation_utils as vu import sahara.plugins.provisioning as p import sahara.utils.files as files LOG = logging.getLogger(__name__) HUE = np.NodeProcess( name='hue', ui_name='Hue', package='mapr-hue', open_ports=[8002, 8888] ) HUE_LIVY = np.NodeProcess( name="livy", ui_name="Hue Livy", package="mapr-hue-livy", open_ports=[8998] ) class Hue(s.Service): THRIFT_VERSIONS = [5, 7] THRIFT_VERSION = p.Config( name="Thrift version", applicable_target="Hue", scope='cluster', config_type="dropdown", config_values=[(v, v) for v in THRIFT_VERSIONS], priority=1, description=_( 'Specifies thrift version.') ) def __init__(self): super(Hue, self).__init__() self._name = 'hue' self._ui_name = 'Hue' self._node_processes = [HUE] self._ui_info = None self._validation_rules = [ vu.exactly(1, HUE), vu.on_same_node(HUE, httpfs.HTTP_FS), vu.on_same_node(HUE_LIVY, spark.SPARK_SLAVE), ] self._priority = 2 def get_ui_info(self, cluster_context): # Hue uses credentials of the administrative user (PAM auth) return [('HUE', HUE, {s.SERVICE_UI: 'http://%s:8888', 'Username': 'mapr', 'Password': pu.get_mapr_password(cluster_context .cluster)})] def get_configs(self): return [Hue.THRIFT_VERSION] def conf_dir(self, cluster_context): return '%s/desktop/conf' % self.home_dir(cluster_context) def get_config_files(self, cluster_context, configs, instance=None): template = 'plugins/mapr/services/hue/resources/hue_%s.template' # hue.ini hue_ini = bcf.TemplateFile("hue.ini") hue_ini.remote_path = self.conf_dir(cluster_context) hue_ini.parse(files.get_file_text(template % self.version)) hue_ini.add_properties(self._get_hue_ini_props(cluster_context)) hue_ini.add_property("thrift_version", configs[self.THRIFT_VERSION.name]) # # hue.sh hue_sh_template = 'plugins/mapr/services/hue/' \ 'resources/hue_sh_%s.template' hue_sh = bcf.TemplateFile("hue.sh") hue_sh.remote_path = self.home_dir(cluster_context) + '/bin' hue_sh.parse(files.get_file_text(hue_sh_template % self.version)) hue_sh.add_property('hadoop_version', cluster_context.hadoop_version) hue_sh.mode = 777 hue_instances = cluster_context.get_instances(HUE) for instance in hue_instances: if instance not in cluster_context.changed_instances(): cluster_context.should_be_restarted[self] += [instance] return [hue_ini, hue_sh] def _get_hue_ini_props(self, cluster_context): db_instance = mysql.MySQL.get_db_instance(cluster_context) is_yarn = cluster_context.cluster_mode == 'yarn' hue_specs = mysql.MySQL.HUE_SPECS rdbms_specs = mysql.MySQL.RDBMS_SPECS result = { 'db_host': db_instance.internal_ip, 'hue_name': hue_specs.db_name, 'hue_user': hue_specs.user, 'hue_password': hue_specs.password, 'rdbms_name': rdbms_specs.db_name, 'rdbms_user': rdbms_specs.user, 'rdbms_password': rdbms_specs.password, 'resource_manager_uri': cluster_context.resource_manager_uri, 'yarn_mode': is_yarn, 'rm_host': cluster_context.get_instance_ip(yarn.RESOURCE_MANAGER), 'webhdfs_url': cluster_context.get_instance_ip(httpfs.HTTP_FS), 'jt_host': cluster_context.get_instance_ip(mr.JOB_TRACKER), 'oozie_host': cluster_context.get_instance_ip(oozie.OOZIE), 'sqoop_host': cluster_context.get_instance_ip( sqoop.SQOOP_2_SERVER), 'impala_host': cluster_context.get_instance_ip( impala.IMPALA_STATE_STORE), 'zk_hosts_with_port': cluster_context.get_zookeeper_nodes_ip_with_port(), 'secret_key': self._generate_secret_key() } hive_host = cluster_context.get_instance(hive.HIVE_SERVER_2) if hive_host: hive_service = cluster_context.get_service(hive.HIVE_SERVER_2) result.update({ 'hive_host': hive_host.internal_ip, 'hive_version': hive_service.version, 'hive_conf_dir': hive_service.conf_dir(cluster_context), }) hbase_host = cluster_context.get_instance(hbase.HBASE_THRIFT) if hbase_host: hbase_service = cluster_context.get_service(hbase.HBASE_THRIFT) result.update({ 'hbase_host': hbase_host.internal_ip, 'hbase_conf_dir': hbase_service.conf_dir(cluster_context), }) livy_host = cluster_context.get_instance(HUE_LIVY) if livy_host: result.update({ 'livy_host': livy_host.internal_ip }) sentry_host = cluster_context.get_instance(sentry.SENTRY) if sentry_host: ui_name = sentry.Sentry().ui_name sentry_version = cluster_context.get_chosen_service_version( ui_name) sentry_service = cluster_context. \ _find_service_instance(ui_name, sentry_version) result.update({ 'sentry_host': sentry_host.internal_ip, 'sentry_conf': sentry_service.conf_dir(cluster_context) }) return result def post_install(self, cluster_context, instances): hue_instance = cluster_context.get_instance(HUE) @el.provision_event(name=_("Migrating Hue database"), instance=hue_instance) def migrate_database(remote, cluster_context): hue_home = self.home_dir(cluster_context) cmd = '%(activate)s && %(syncdb)s && %(migrate)s' args = { 'activate': 'source %s/build/env/bin/activate' % hue_home, 'syncdb': '%s/build/env/bin/hue syncdb --noinput' % hue_home, 'migrate': '%s/build/env/bin/hue migrate' % hue_home, } remote.execute_command(cmd % args, run_as_root=True, timeout=600) def hue_syncdb_workround(remote): cmd = 'printf "/opt/mapr/lib\n$JAVA_HOME/jre/lib/amd64/server\n"' \ ' | tee /etc/ld.so.conf.d/mapr-hue.conf && ldconfig' remote.execute_command(cmd, run_as_root=True) def centos7_workaround(remote): cmd = 'ln -s /lib64/libsasl2.so.3.0.0 /lib64/libsasl2.so.2' \ ' && rpm -ivh --nodeps http://yum.mariadb.org/5.5.49/' \ 'rhel7-amd64/rpms/MariaDB-5.5.49-centos7-x86_64-compat.rpm' remote.execute_command(cmd, run_as_root=True, raise_when_error=False) with hue_instance.remote() as r: LOG.debug("Executing Hue database migration") # workaround for centos7 if cluster_context.distro_version.split('.')[0] == '7': centos7_workaround(r) # temporary workaround to prevent failure of db migrate on mapr 5.2 hue_syncdb_workround(r) migrate_database(r, cluster_context) self._copy_hive_configs(cluster_context, hue_instance) self._install_jt_plugin(cluster_context, hue_instance) self._set_service_dir_owner(cluster_context, instances) def _copy_hive_configs(self, cluster_context, hue_instance): hive_server = cluster_context.get_instance(hive.HIVE_SERVER_2) if not hive_server or hive_server == hue_instance: LOG.debug('No Hive Servers found. Skip') return hive_service = cluster_context.get_service(hive.HIVE_SERVER_2) hive_conf_dir = hive_service.conf_dir(cluster_context) g.copy(hive_conf_dir, hive_server, hive_conf_dir, hue_instance, 'root') def update(self, cluster_context, instances=None): if self._should_restart(cluster_context, instances): self.restart(instances) def post_start(self, cluster_context, instances): self.update(cluster_context, instances=instances) def _should_restart(self, cluster_context, instances): app_services = [ impala.Impala(), hive.Hive(), hbase.HBase(), sqoop.Sqoop2(), spark.SparkOnYarn(), ] instances = [ cluster_context.filter_instances(instances, service=service) for service in app_services] return bool(g.unique_list(itertools.chain(*instances))) def jt_plugin_path(self, cluster_context): path = ('%(home)s/desktop/libs/hadoop/java-lib' '/hue-plugins-%(version)s-mapr.jar') args = { 'home': self.home_dir(cluster_context), 'version': self.version, } return path % args @el.provision_event(name="Install Hue Job Tracker plugin", instance_reference=2) def _install_jt_plugin(self, cluster_context, hue_instance): LOG.debug("Copying Hue JobTracker plugin") job_trackers = cluster_context.get_instances(mr.JOB_TRACKER) if not job_trackers: LOG.debug('No JobTrackers found. Skip') return jt_plugin_src = self.jt_plugin_path(cluster_context) jt_plugin_dest = cluster_context.hadoop_lib + '/jt_plugin.jar' for jt in job_trackers: g.copy(jt_plugin_src, hue_instance, jt_plugin_dest, jt, 'root') def _generate_secret_key(self, length=80): ascii_alphanum = string.ascii_letters + string.digits generator = random.SystemRandom() return ''.join(generator.choice(ascii_alphanum) for _ in range(length)) class HueV360(Hue): def __init__(self): super(HueV360, self).__init__() self._version = '3.6.0' class HueV370(Hue): def __init__(self): super(HueV370, self).__init__() self._version = '3.7.0' class HueV381(Hue): def __init__(self): super(HueV381, self).__init__() self._version = "3.8.1" self._dependencies = [("mapr-hue-base", self.version)] self._node_processes = [HUE, HUE_LIVY] self._validation_rules.append(vu.at_most(1, HUE_LIVY)) class HueV390(Hue): def __init__(self): super(HueV390, self).__init__() self._version = "3.9.0" self._dependencies = [("mapr-hue-base", self.version)] self._node_processes = [HUE, HUE_LIVY] self._validation_rules.append(vu.at_most(1, HUE_LIVY)) class HueV310(Hue): def __init__(self): super(HueV310, self).__init__() self._version = "3.10.0" self._node_processes = [HUE] self._validation_rules.append(vu.at_most(1, HUE_LIVY)) class HueLivyV310(s.Service): def __init__(self): super(HueLivyV310, self).__init__() self._name = 'hue-livy' self._ui_name = 'Livy' self._version = "3.10.0" self._node_processes = [HUE_LIVY] self._validation_rules = [vu.at_most(1, HUE_LIVY), vu.depends_on(HueV310(), self), vu.depends_on(spark.SparkOnYarnV201(), self), vu.on_same_node(HUE_LIVY, spark.SPARK_SLAVE)] def get_config_files(self, cluster_context, configs, instance=None): livy_conf_template = 'plugins/mapr/services/hue/' \ 'resources/livy_conf_%s.template' livy_conf = bcf.TemplateFile("livy.conf") livy_conf.parse(files.get_file_text(livy_conf_template % self.version)) livy_conf.remote_path = self.home_dir(cluster_context) + '/conf' livy_sh_template = 'plugins/mapr/services/hue/' \ 'resources/livy_sh_%s.template' livy_sh = bcf.TemplateFile("livy-env.sh") livy_sh.remote_path = self.home_dir(cluster_context) + '/conf' livy_sh.parse(files.get_file_text(livy_sh_template % self.version)) livy_sh.add_property('hadoop_version', cluster_context.hadoop_version) livy_sh.add_property('spark_version', spark.SparkOnYarnV201().version) livy_sh.mode = 777 hue_instances = cluster_context.get_instances(HUE) for instance in hue_instances: if instance not in cluster_context.changed_instances(): cluster_context.should_be_restarted[self] += [instance] return [livy_sh, livy_conf] def post_install(self, cluster_context, instances): self._set_service_dir_owner(cluster_context, instances) def post_start(self, cluster_context, instances): self.restart(instances) sahara-8.0.0/sahara/plugins/mapr/services/hue/__init__.py0000666000175100017510000000000013245514472023375 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/0000775000175100017510000000000013245515027023303 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_3.6.0.template0000666000175100017510000010212513245514472026353 0ustar zuulzuul00000000000000# Hue configuration file # =================================== # # For complete documentation about the contents of this file, run # $ /build/env/bin/hue config_help # # All .ini files under the current directory are treated equally. Their # contents are merged to form the Hue configuration, which can # can be viewed on the Hue at # http://:/dump_config ########################################################################### # General configuration for core Desktop features (authentication, etc) ########################################################################### [desktop] send_dbug_messages=1 # To show database transactions, set database_logging to 1 database_logging=0 # Set this to a random string, the longer the better. # This is used for secure hashing in the session store. secret_key={{ secret_key }} # Webserver listens on this address and port http_host=0.0.0.0 http_port=8888 # Time zone name time_zone=America/Los_Angeles # Enable or disable Django debug mode. django_debug_mode=false # Enable or disable backtrace for server error http_500_debug_mode=false # Enable or disable memory profiling. ## memory_profiler=false # Server email for internal error messages ## django_server_email='hue@localhost.localdomain' # Email backend ## django_email_backend=django.core.mail.backends.smtp.EmailBackend # Webserver runs as this user server_user=mapr server_group=mapr # This should be the Hue admin and proxy user default_user=mapr # This should be the hadoop cluster admin default_hdfs_superuser=mapr default_jobtracker_host={{ resource_manager_uri | default("maprfs:///", True) }} # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # Number of threads used by the CherryPy web server ## cherrypy_server_threads=10 # Filename of SSL Certificate ## ssl_certificate= # Filename of SSL RSA Private Key ## ssl_private_key= # List of allowed and disallowed ciphers in cipher list format. # See http://www.openssl.org/docs/apps/ciphers.html for more information on cipher list format. ## ssl_cipher_list=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2 # LDAP password of the hue user used for LDAP authentications. For example for LDAP Authentication with HiveServer2. # ldap_password= # Default encoding for site data ## default_site_encoding=utf-8 # Help improve Hue with anonymous usage analytics. # Use Google Analytics to see how many times an application or specific section of an application is used, nothing more. ## collect_usage=true # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER. ## secure_proxy_ssl_header=false # Comma-separated list of Django middleware classes to use. # See https://docs.djangoproject.com/en/1.4/ref/middleware/ for more details on middlewares in Django. ## middleware=desktop.auth.backend.LdapSynchronizationBackend # Comma-separated list of regular expressions, which match the redirect URL. # For example, to restrict to your local domain and FQDN, the following value can be used: # ^\/.*$,^http:\/\/www.mydomain.com\/.*$ ## redirect_whitelist= # Comma separated list of apps to not load at server startup. # e.g.: pig,zookeeper app_blacklist=spark,search,zookeeper{{ ",impala" if not impala_host }}{{ ",hbase" if not hbase_host }}{{ ",sqoop" if not sqoop_host }} # The directory where to store the auditing logs. Auditing is disable if the value is empty. # e.g. /var/log/hue/audit.log ## audit_event_log_dir= # Size in KB/MB/GB for audit log to rollover. ## audit_log_max_file_size=100MB # Administrators # ---------------- [[django_admins]] ## [[[admin1]]] ## name=john ## email=john@doe.com # UI customizations # ------------------- [[custom]] # Top banner HTML code # e.g.

Test Lab A2 Hue Services

## banner_top_html= # Configuration options for user authentication into the web application # ------------------------------------------------------------------------ [[auth]] # Authentication backend. Common settings are: # - django.contrib.auth.backends.ModelBackend (entirely Django backend) # - desktop.auth.backend.AllowAllBackend (allows everyone) # - desktop.auth.backend.AllowFirstUserDjangoBackend # (Default. Relies on Django and user manager, after the first login) # - desktop.auth.backend.LdapBackend # - desktop.auth.backend.PamBackend # - desktop.auth.backend.SpnegoDjangoBackend # - desktop.auth.backend.RemoteUserDjangoBackend # - libsaml.backend.SAML2Backend # - libopenid.backend.OpenIDBackend # - liboauth.backend.OAuthBackend # (Support Twitter, Facebook, Google+ and Linkedin ## backend=desktop.auth.backend.AllowFirstUserDjangoBackend # The service to use when querying PAM. ## pam_service=login # When using the desktop.auth.backend.RemoteUserDjangoBackend, this sets # the normalized name of the header that contains the remote user. # The HTTP header in the request is converted to a key by converting # all characters to uppercase, replacing any hyphens with underscores # and adding an HTTP_ prefix to the name. So, for example, if the header # is called Remote-User that would be configured as HTTP_REMOTE_USER # # Defaults to HTTP_REMOTE_USER ## remote_user_header=HTTP_REMOTE_USER # Synchronize a users groups when they login ## sync_groups_on_login=false # Ignore the case of usernames when searching for existing users. # Only supported in remoteUserDjangoBackend. ## ignore_username_case=false # Ignore the case of usernames when searching for existing users to authenticate with. # Only supported in remoteUserDjangoBackend. ## force_username_lowercase=false # Users will expire after they have not logged in for 'n' amount of seconds. # A negative number means that users will never expire. ## expires_after=-1 # Apply 'expires_after' to superusers. ## expire_superusers=true # Configuration options for connecting to LDAP and Active Directory # ------------------------------------------------------------------- [[ldap]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication # If nt_domain is specified, this config is completely ignored. # If nt_domain is not specified, this should take on the form "cn=,dc=example,dc=com", # where is replaced by whatever is provided at the login page. Depending on your ldap schema, # you can also specify additional/alternative comma-separated attributes like uid, ou, etc ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" # Create users in Hue when they try to login with their LDAP credentials # For use when using LdapBackend for Hue authentication ## create_users_on_login = true # Ignore the case of usernames when searching for existing users in Hue. ## ignore_username_case=false # Ignore the case of usernames when searching for existing users to authenticate with. # Only supported in remoteUserDjangoBackend. ## force_username_lowercase=false # Use search bind authentication. # If set to true, hue will perform ldap search using bind credentials above (bind_dn, bind_password) # Hue will then search using the 'base_dn' for an entry with attr defined in 'user_name_attr', with the value # of short name provided on the login page. The search filter defined in 'user_filter' will also be used to limit # the search. Hue will search the entire subtree starting from base_dn. # If search_bind_authentication is set to false, Hue performs a direct bind to LDAP using the credentials provided # (not bind_dn and bind_password specified in hue.ini). There are 2 modes here - 'nt_domain' is specified or not. ## search_bind_authentication=true # Choose which kind of subgrouping to use: nested or suboordinate (deprecated). ## subgroups=suboordinate # Define the number of levels to search for nested members. ## nested_members_search_depth=10 [[[users]]] # Base filter for searching for users ## user_filter="objectclass=*" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName [[[groups]]] # Base filter for searching for groups ## group_filter="objectclass=*" # The group name attribute in the LDAP schema ## group_name_attr=cn # The attribute of the group object which identifies the members of the group ## group_member_attr=members [[[ldap_servers]]] ## [[[[mycompany]]]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" ## [[[[[users]]]]] # Base filter for searching for users ## user_filter="objectclass=Person" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName ## [[[[[groups]]]]] # Base filter for searching for groups ## group_filter="objectclass=groupOfNames" # The username attribute in the LDAP schema ## group_name_attr=cn # Configuration options for specifying the Desktop Database. For more info, # see http://docs.djangoproject.com/en/1.1/ref/settings/#database-engine # ------------------------------------------------------------------------ [[database]] # Database engine is typically one of: # postgresql_psycopg2, mysql, or sqlite3 # # Note that for sqlite3, 'name', below is a filename; # for other backends, it is the database name. engine=mysql host={{ db_host }} port=3306 user={{ hue_user }} password={{ hue_password }} name={{ hue_name }} ## options={} # Configuration options for specifying the Desktop session. # For more info, see https://docs.djangoproject.com/en/1.4/topics/http/sessions/ # ------------------------------------------------------------------------ [[session]] # The cookie containing the users' session ID will expire after this amount of time in seconds. # Default is 2 weeks. ## ttl=1209600 # The cookie containing the users' session ID will be secure. # Should only be enabled with HTTPS. ## secure=false # The cookie containing the users' session ID will use the HTTP only flag. ## http_only=false # Use session-length cookies. Logs out the user when she closes the browser window. ## expire_at_browser_close=false # Configuration options for connecting to an external SMTP server # ------------------------------------------------------------------------ [[smtp]] # The SMTP server information for email notification delivery host=localhost port=25 user= password= # Whether to use a TLS (secure) connection when talking to the SMTP server tls=no # Default email address to use for various automated notification from Hue ## default_from_email=hue@localhost # Configuration options for Kerberos integration for secured Hadoop clusters # ------------------------------------------------------------------------ [[kerberos]] # Path to Hue's Kerberos keytab file ## hue_keytab= # Kerberos principal name for Hue ## hue_principal=hue/hostname.foo.com # Path to kinit ## kinit_path=/path/to/kinit # Configuration options for using OAuthBackend (core) login # ------------------------------------------------------------------------ [[oauth]] # The Consumer key of the application ## consumer_key=XXXXXXXXXXXXXXXXXXXXX # The Consumer secret of the application ## consumer_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # The Request token URL ## request_token_url=https://api.twitter.com/oauth/request_token # The Access token URL ## access_token_url=https://api.twitter.com/oauth/access_token # The Authorize URL ## authenticate_url=https://api.twitter.com/oauth/authorize ########################################################################### # Settings to configure SAML ########################################################################### [libsaml] # Xmlsec1 binary path. This program should be executable by the user running Hue. ## xmlsec_binary=/usr/local/bin/xmlsec1 # Entity ID for Hue acting as service provider. # Can also accept a pattern where '' will be replaced with server URL base. ## entity_id="/saml2/metadata/" # Create users from SSO on login. ## create_users_on_login=true # Required attributes to ask for from IdP. # This requires a comma separated list. ## required_attributes=uid # Optional attributes to ask for from IdP. # This requires a comma separated list. ## optional_attributes= # IdP metadata in the form of a file. This is generally an XML file containing metadata that the Identity Provider generates. ## metadata_file= # Private key to encrypt metadata with. ## key_file= # Signed certificate to send along with encrypted metadata. ## cert_file= # A mapping from attributes in the response from the IdP to django user attributes. ## user_attribute_mapping={'uid':'username'} # Have Hue initiated authn requests be signed and provide a certificate. ## authn_requests_signed=false # Have Hue initiated logout requests be signed and provide a certificate. ## logout_requests_signed=false ## Username can be sourced from 'attributes' or 'nameid'. ## username_source=attributes # Performs the logout or not. ## logout_enabled=true ########################################################################### # Settings to configure OPENID ########################################################################### [libopenid] # (Required) OpenId SSO endpoint url. ## server_endpoint_url=https://www.google.com/accounts/o8/id # OpenId 1.1 identity url prefix to be used instead of SSO endpoint url # This is only supported if you are using an OpenId 1.1 endpoint ## identity_url_prefix=https://app.onelogin.com/openid/your_company.com/ # Create users from OPENID on login. ## create_users_on_login=true # Use email for username ## use_email_for_username=true ########################################################################### # Settings to configure OAuth ########################################################################### [liboauth] # NOTE: # To work, each of the active (i.e. uncommented) service must have # applications created on the social network. # Then the "consumer key" and "consumer secret" must be provided here. # # The addresses where to do so are: # Twitter: https://dev.twitter.com/apps # Google+ : https://cloud.google.com/ # Facebook: https://developers.facebook.com/apps # Linkedin: https://www.linkedin.com/secure/developer # # Additionnaly, the following must be set in the application settings: # Twitter: Callback URL (aka Redirect URL) must be set to http://YOUR_HUE_IP_OR_DOMAIN_NAME/oauth/social_login/oauth_authenticated # Google+ : CONSENT SCREEN must have email address # Facebook: Sandbox Mode must be DISABLED # Linkedin: "In OAuth User Agreement", r_emailaddress is REQUIRED # The Consumer key of the application ## consumer_key_twitter= ## consumer_key_google= ## consumer_key_facebook= ## consumer_key_linkedin= # The Consumer secret of the application ## consumer_secret_twitter= ## consumer_secret_google= ## consumer_secret_facebook= ## consumer_secret_linkedin= # The Request token URL ## request_token_url_twitter=https://api.twitter.com/oauth/request_token ## request_token_url_google=https://accounts.google.com/o/oauth2/auth ## request_token_url_linkedin=https://www.linkedin.com/uas/oauth2/authorization ## request_token_url_facebook=https://graph.facebook.com/oauth/authorize # The Access token URL ## access_token_url_twitter=https://api.twitter.com/oauth/access_token ## access_token_url_google=https://accounts.google.com/o/oauth2/token ## access_token_url_facebook=https://graph.facebook.com/oauth/access_token ## access_token_url_linkedin=https://api.linkedin.com/uas/oauth2/accessToken # The Authenticate URL ## authenticate_url_twitter=https://api.twitter.com/oauth/authorize ## authenticate_url_google=https://www.googleapis.com/oauth2/v1/userinfo?access_token= ## authenticate_url_facebook=https://graph.facebook.com/me?access_token= ## authenticate_url_linkedin=https://api.linkedin.com/v1/people/~:(email-address)?format=json&oauth2_access_token= # Username Map. Json Hash format. # Replaces username parts in order to simplify usernames obtained # Example: {"@sub1.domain.com":"_S1", "@sub2.domain.com":"_S2"} # converts 'email@sub1.domain.com' to 'email_S1' ## username_map={} # Whitelisted domains (only applies to Google OAuth). CSV format. ## whitelisted_domains_google= ########################################################################### # Settings for the RDBMS application ########################################################################### [librdbms] # The RDBMS app can have any number of databases configured in the databases # section. A database is known by its section name # (IE sqlite, mysql, psql, and oracle in the list below). [[databases]] # sqlite configuration. ## [[[sqlite]]] # Name to show in the UI. ## nice_name=SQLite # For SQLite, name defines the path to the database. ## name=/opt/mapr/hue/hue-3.6.0/desktop/desktop.db # Database backend to use. ## engine=sqlite # mysql, oracle, or postgresql configuration. {%- if db_host and rdbms_name %} [[[mysql]]] # Name to show in the UI. nice_name="MySQL DB" # For MySQL and PostgreSQL, name is the name of the database. # For Oracle, Name is instance of the Oracle server. For express edition # this is 'xe' by default. name={{ rdbms_name }} # Database backend to use. This can be: # 1. mysql # 2. postgresql # 3. oracle engine=mysql # IP or hostname of the database to connect to. host={{ db_host }} # Port the database server is listening to. Defaults are: # 1. MySQL: 3306 # 2. PostgreSQL: 5432 # 3. Oracle Express Edition: 1521 port=3306 # Username to authenticate with when connecting to the database. user={{ rdbms_user }} # Password matching the username to authenticate with when # connecting to the database. password={{ rdbms_password }} {%- endif %} ########################################################################### # Settings to configure your Hadoop cluster. ########################################################################### [hadoop] # Configuration for HDFS NameNode # ------------------------------------------------------------------------ [[hdfs_clusters]] # HA support by using HttpFs [[[default]]] # Enter the filesystem uri fs_defaultfs=maprfs:/// # NameNode logical name. ## logical_name= # Use WebHdfs/HttpFs as the communication mechanism. # Domain should be the NameNode or HttpFs host. # Default port is 14000 for HttpFs. webhdfs_url=http://{{ webhdfs_url | default("localhost", True) }}:14000/webhdfs/v1 # Change this if your HDFS cluster is Kerberos-secured ## security_enabled=false {% if yarn_mode %} # Configuration for YARN (MR2) # ------------------------------------------------------------------------ [[yarn_clusters]] [[[default]]] # Enter the host on which you are running the ResourceManager resourcemanager_host={{ rm_host | default("localhost", True) }} # The port where the ResourceManager IPC listens on resourcemanager_port={{ rm_port | default("8032", True) }} # Whether to submit jobs to this cluster submit_to={{ "True" if yarn_mode else "False" }} # Change this if your YARN cluster is Kerberos-secured ## security_enabled=false # URL of the ResourceManager API resourcemanager_api_url=http://{{ rm_host | default("localhost", True) }}:{{ rm_api_port | default("8088", True)}} # URL of the ProxyServer API proxy_api_url=http://{{ rm_proxy_api | default(rm_host, True) }}:{{ rm_proxy_port | default("8088", True)}} # URL of the HistoryServer API history_server_api_url=http://{{ hs_host | default(rm_host, True) }}:{{ hs_api_port | default("19888", True)}} # In secure mode (HTTPS), if SSL certificates from Resource Manager's # Rest Server have to be verified against certificate authority ssl_cert_ca_verify=False {%- endif %} {%- if not yarn_mode %} # Configuration for MapReduce (MR1) # ------------------------------------------------------------------------ [[mapred_clusters]] [[[default]]] # Enter the host on which you are running the Hadoop JobTracker jobtracker_host={{ jt_host | default("localhost", True) }} # The port where the JobTracker IPC listens on jobtracker_port={{ jt_port | default("9001", True) }} # JobTracker logical name. ## logical_name= # Thrift plug-in port for the JobTracker thrift_port={{ jt_plugin_port | default("9290", True) }} # jobtracker kerberos principal name jt_kerberos_principal=mapr # Whether to submit jobs to this cluster submit_to={{ "False" if yarn_mode else "True" }} # Change this if your MapReduce cluster is Kerberos-secured ## security_enabled=false {%- if fo_jobtracker_host %} # HA support by specifying multiple clusters # e.g. # [[[ha]]] # Enter the host on which you are running the failover JobTracker jobtracker_host={{ fo_jobtracker_host | default("localhost-ha", True) }} {%- endif %} {%- endif %} ########################################################################### # Settings to configure the Filebrowser app ########################################################################### [filebrowser] # Location on local filesystem where the uploaded archives are temporary stored. ## archive_upload_tempdir=/tmp ########################################################################### # Settings to configure liboozie ########################################################################### [liboozie] # The URL where the Oozie service runs on. This is required in order for # users to submit jobs. oozie_url=http://{{ oozie_host | default("localhost", True) }}:{{ oozie_port | default("11000", True) }}/oozie # Requires FQDN in oozie_url if enabled ## security_enabled=false # Location on HDFS where the workflows/coordinator are deployed when submitted. ## remote_deployement_dir=/oozie/deployments ########################################################################### # Settings to configure the Oozie app ########################################################################### [oozie] # Location on local FS where the examples are stored. ## local_data_dir=..../examples # Location on local FS where the data for the examples is stored. ## sample_data_dir=...thirdparty/sample_data # Location on HDFS where the oozie examples and workflows are stored. remote_data_dir=/oozie/workspaces # Maximum of Oozie workflows or coodinators to retrieve in one API call. ## oozie_jobs_count=100 # Use Cron format for defining the frequency of a Coordinator instead of the old frequency number/unit. ## enable_cron_scheduling=true ########################################################################### # Settings to configure Beeswax with Hive ########################################################################### {%- if hive_host %} [beeswax] # Host where Hive server Thrift daemon is running. # If Kerberos security is enabled, use fully-qualified domain name (FQDN). hive_server_host={{ hive_host | default("localhost", True) }} # Port where HiveServer2 Thrift server runs on. hive_server_port={{ hive_port | default("10000", True) }} # Hive configuration directory, where hive-site.xml is located hive_conf_dir={{ hive_conf_dir | default("/opt/mapr/hive/hive-0.13/conf", True) }} # Timeout in seconds for thrift calls to Hive service ## server_conn_timeout=120 # Path to HiveServer2 start script hive_server_bin=/opt/mapr/hive/hive-{{ hive_version | default("0.13", True) }}/bin/hiveserver2 # Set a LIMIT clause when browsing a partitioned table. # A positive value will be set as the LIMIT. If 0 or negative, do not set any limit. ## browse_partitioned_table_limit=250 # A limit to the number of rows that can be downloaded from a query. # A value of -1 means there will be no limit. # A maximum of 65,000 is applied to XLS downloads. ## download_row_limit=1000000 # Hue will try to close the Hive query when the user leaves the editor page. # This will free all the query resources in HiveServer2, but also make its results inaccessible. ## close_queries=false # Default thrift version to use when communicating with HiveServer2 # thrift_version=2 [[ssl]] # SSL communication enabled for this server. ## enabled=false # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Path to the private key file. ## key=/etc/hue/key.pem # Path to the public certificate file. ## cert=/etc/hue/cert.pem # Choose whether Hue should validate certificates received from the server. ## validate=true {%- endif %} ########################################################################### # Settings to configure Pig ########################################################################### [pig] # Location of piggybank.jar on local filesystem. ## local_sample_dir=/opt/mapr/pig/pig-0.12/contrib/piggybank/java/ # Location piggybank.jar will be copied to in HDFS. ## remote_data_dir=/oozie/pig/examples ########################################################################### # Settings to configure Sqoop ########################################################################### {%- if sqoop_host %} [sqoop] # For autocompletion, fill out the librdbms section. # Sqoop server URL server_url=http://{{ sqoop_host | default("localhost", True) }}:{{ sqoop_port | default("12000", True) }}/sqoop {%- endif %} ########################################################################### # Settings to configure Proxy ########################################################################### [proxy] # Comma-separated list of regular expressions, # which match 'host:port' of requested proxy target. ## whitelist=(localhost|127\.0\.0\.1):(50030|50070|50060|50075) # Comma-separated list of regular expressions, # which match any prefix of 'host:port/path' of requested proxy target. # This does not support matching GET parameters. ## blacklist= ########################################################################### # Settings to configure Impala ########################################################################### {%- if impala_host %} [impala] # Host of the Impala Server (one of the Impalad) server_host={{ impala_host | default("localhost", True) }} # Port of the Impala Server server_port={{ impala_port | default("21050", True) }} # Kerberos principal ## impala_principal=impala/hostname.foo.com # Turn on/off impersonation mechanism when talking to Impala impersonation_enabled=False # Number of initial rows of a result set to ask Impala to cache in order # to support re-fetching them for downloading them. # Set to 0 for disabling the option and backward compatibility. querycache_rows=0 # Timeout in seconds for thrift calls ## server_conn_timeout=120 # Hue will try to close the Impala query when the user leaves the editor page. # This will free all the query resources in Impala, but also make its results inaccessible. ## close_queries=true {%- endif %} ########################################################################### # Settings to configure HBase Browser ########################################################################### {%- if hbase_host %} [hbase] # Comma-separated list of HBase Thrift servers for # clusters in the format of '(name|host:port)'. hbase_clusters=(HBase|{{ hbase_host | default("localhost", True) }}:{{ hbase_port | default("9090", True) }}) # Hard limit of rows or columns per row fetched before truncating. ## truncate_limit = 500 # 'buffered' is the default of the HBase Thrift Server. # 'framed' can be used to chunk up responses, # which is useful when used in conjunction with the nonblocking server in Thrift. ## thrift_transport=buffered {%- endif %} ########################################################################### # Settings to configure Solr Search ########################################################################### [search] # URL of the Solr Server ## solr_url=http://localhost:8983/solr/ # Requires FQDN in solr_url if enabled ## security_enabled=false ## Query sent when no term is entered ## empty_query=*:* ########################################################################### # Settings to configure Solr Indexer ########################################################################### [indexer] # Location of the solrctl binary. ## solrctl_path=/usr/bin/solrctl # Location of the solr home. ## solr_home=/usr/lib/solr # Zookeeper ensemble. ## solr_zk_ensemble=localhost:2181/solr # The contents of this directory will be copied over to the solrctl host to its temporary directory. ## config_template_path=/../hue/desktop/libs/indexer/src/data/solr_configs ########################################################################### # Settings to configure Job Designer ########################################################################### [jobsub] # Location on local FS where examples and template are stored. ## local_data_dir=..../data # Location on local FS where sample data is stored ## sample_data_dir=...thirdparty/sample_data ########################################################################### # Settings to configure Job Browser. ########################################################################### [jobbrowser] # Share submitted jobs information with all users. If set to false, # submitted jobs are visible only to the owner and administrators. ## share_jobs=true ########################################################################### # Settings to configure the Zookeeper application. ########################################################################### [zookeeper] [[clusters]] [[[default]]] # Zookeeper ensemble. Comma separated list of Host/Port. # e.g. localhost:5181,node2_ip@:5181,node3_ip@:5181 host_ports={{ zk_hosts_with_port | default("localhost:5181", True) }} # The URL of the REST contrib service (required for znode browsing) ## rest_url=http://localhost:9999 ########################################################################### # Settings to configure the Spark application. ########################################################################### [spark] # URL of the REST Spark Job Server. ## server_url=http://localhost:8090/ ########################################################################### # Settings for the User Admin application ########################################################################### [useradmin] # The name of the default user group that users will be a member of ## default_user_group=default sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/livy_conf_3.10.0.template0000666000175100017510000000662113245514472027641 0ustar zuulzuul00000000000000# Use this keystore for the SSL certificate and key. # livy.keystore = # Specify the keystore password. # livy.keystore.password = # What host address to start the server on. By default, Livy will bind to all network interfaces. # livy.server.host = 0.0.0.0 # What port to start the server on. # livy.server.port = 8998 # What spark master Livy sessions should use. livy.spark.master = yarn # What spark deploy mode Livy sessions should use. #livy.spark.deployMode = yarn # Time in milliseconds on how long Livy will wait before timing out an idle session. # livy.server.session.timeout = 1h # If livy should impersonate the requesting users when creating a new session. # livy.impersonation.enabled = true # Comma-separated list of Livy RSC jars. By default Livy will upload jars from its installation # directory every time a session is started. By caching these files in HDFS, for example, startup # time of sessions on YARN can be reduced. # livy.jars = # Comma-separated list of Livy REPL jars. By default Livy will upload jars from its installation # directory every time a session is started. By caching these files in HDFS, for example, startup # time of sessions on YARN can be reduced. # livy.repl.jars = # Location of PySpark archives. By default Livy will upload the file from SPARK_HOME, but # by caching the file in HDFS, startup time of PySpark sessions on YARN can be reduced. # livy.pyspark.archives = # Location of the SparkR package. By default Livy will upload the file from SPARK_HOME, but # by caching the file in HDFS, startup time of R sessions on YARN can be reduced. # livy.sparkr.package = # List of local directories from where files are allowed to be added to user sessions. By # default it's empty, meaning users can only reference remote URIs when starting their # sessions. # livy.file.local-dir-whitelist = # Whether to enable csrf protection, by default it is false. If it is enabled, client should add # http-header "X-Requested-By" in request if the http method is POST/DELETE/PUT/PATCH. # livy.server.csrf_protection.enabled = # Whether to enable HiveContext in livy interpreter, if it is true hive-site.xml will be detected # on user request and then livy server classpath automatically. # livy.repl.enableHiveContext = # Recovery mode of Livy. Possible values: # off: Default. Turn off recovery. Every time Livy shuts down, it stops and forgets all sessions. # recovery: Livy persists session info to the state store. When Livy restarts, it recovers # previous sessions from the state store. # Must set livy.server.recovery.state-store and livy.server.recovery.state-store.url to # configure the state store. # livy.server.recovery.mode = off # Where Livy should store state to for recovery. Possible values: # : Default. State store disabled. # filesystem: Store state on a file system. # zookeeper: Store state in a Zookeeper instance. # livy.server.recovery.state-store = # For filesystem state store, the path of the state store directory. Please don't use a filesystem # that doesn't support atomic rename (e.g. S3). e.g. file:///tmp/livy or hdfs:///. # For zookeeper, the address to the Zookeeper servers. e.g. host1:port1,host2:port2 # livy.server.recovery.state-store.url = # If Livy can't find the yarn app within this time, consider it lost. # livy.server.yarn.app-lookup-timeout = 30s # How often Livy polls YARN to refresh YARN app state. # livy.server.yarn.poll-interval = 1s ## Check sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_3.9.0.template0000666000175100017510000012534713245514472026371 0ustar zuulzuul00000000000000# Hue configuration file # =================================== # # For complete documentation about the contents of this file, run # $ /build/env/bin/hue config_help # # All .ini files under the current directory are treated equally. Their # contents are merged to form the Hue configuration, which can # can be viewed on the Hue at # http://:/dump_config ########################################################################### # General configuration for core Desktop features (authentication, etc) ########################################################################### [desktop] # Set this to a random string, the longer the better. # This is used for secure hashing in the session store. secret_key={{ secret_key }} # Execute this script to produce the Django secret key. This will be used when # `secret_key` is not set. ## secret_key_script= # Webserver listens on this address and port http_host=0.0.0.0 http_port=8888 # Time zone name time_zone=America/Los_Angeles # Enable or disable Django debug mode. django_debug_mode=false # Enable or disable backtrace for server error http_500_debug_mode=false # Enable or disable memory profiling. ## memory_profiler=false # Server email for internal error messages ## django_server_email='hue@localhost.localdomain' # Email backend ## django_email_backend=django.core.mail.backends.smtp.EmailBackend # Webserver runs as this user server_user=mapr server_group=mapr # This should be the Hue admin and proxy user default_user=mapr # This should be the hadoop cluster admin default_hdfs_superuser=mapr default_jobtracker_host=maprfs:/// # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # This should be the Hue admin and proxy user ## default_user=hue # This should be the hadoop cluster admin ## default_hdfs_superuser=hdfs # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # Number of threads used by the CherryPy web server ## cherrypy_server_threads=40 # Filename of SSL Certificate ## ssl_certificate= # Filename of SSL RSA Private Key ## ssl_private_key= # SSL certificate password ## ssl_password= # Execute this script to produce the SSL password. This will be used when `ssl_password` is not set. ## ssl_password_script= # List of allowed and disallowed ciphers in cipher list format. # See http://www.openssl.org/docs/apps/ciphers.html for more information on # cipher list format. This list is from # https://wiki.mozilla.org/Security/Server_Side_TLS v3.7 intermediate # recommendation, which should be compatible with Firefox 1, Chrome 1, IE 7, # Opera 5 and Safari 1. ## ssl_cipher_list=ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA # Path to default Certificate Authority certificates. ## ssl_cacerts=/opt/mapr/hue/hue-3.9.0/cert.pem # Choose whether Hue should validate certificates received from the server. ## validate=true # LDAP username and password of the hue user used for LDAP authentications. # Set it to use LDAP Authentication with HiveServer2 and Impala. ## ldap_username=hue ## ldap_password= # Default encoding for site data ## default_site_encoding=utf-8 # Help improve Hue with anonymous usage analytics. # Use Google Analytics to see how many times an application or specific section of an application is used, nothing more. ## collect_usage=true # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER. ## secure_proxy_ssl_header=false # Comma-separated list of Django middleware classes to use. # See https://docs.djangoproject.com/en/1.4/ref/middleware/ for more details on middlewares in Django. ## middleware=desktop.auth.backend.LdapSynchronizationBackend # Comma-separated list of regular expressions, which match the redirect URL. # For example, to restrict to your local domain and FQDN, the following value can be used: # ^\/.*$,^http:\/\/www.mydomain.com\/.*$ ## redirect_whitelist=^\/.*$ # Comma separated list of apps to not load at server startup. # e.g.: pig,zookeeper app_blacklist=search,zookeeper {{ ",security" if not sentry_host }}{{ ",impala" if not impala_host }}{{ ",hbase" if not hbase_host }}{{ ",sqoop" if not sqoop_host }}{{ ",spark" if not livy_host }}{{ ",hive" if not hive_host }} # The directory where to store the auditing logs. Auditing is disable if the value is empty. # e.g. /var/log/hue/audit.log ## audit_event_log_dir= # Size in KB/MB/GB for audit log to rollover. ## audit_log_max_file_size=100MB # A json file containing a list of log redaction rules for cleaning sensitive data # from log files. It is defined as: # # { # "version": 1, # "rules": [ # { # "description": "This is the first rule", # "trigger": "triggerstring 1", # "search": "regex 1", # "replace": "replace 1" # }, # { # "description": "This is the second rule", # "trigger": "triggerstring 2", # "search": "regex 2", # "replace": "replace 2" # } # ] # } # # Redaction works by searching a string for the [TRIGGER] string. If found, # the [REGEX] is used to replace sensitive information with the # [REDACTION_MASK]. If specified with `log_redaction_string`, the # `log_redaction_string` rules will be executed after the # `log_redaction_file` rules. # # For example, here is a file that would redact passwords and social security numbers: # { # "version": 1, # "rules": [ # { # "description": "Redact passwords", # "trigger": "password", # "search": "password=\".*\"", # "replace": "password=\"???\"" # }, # { # "description": "Redact social security numbers", # "trigger": "", # "search": "\d{3}-\d{2}-\d{4}", # "replace": "XXX-XX-XXXX" # } # ] # } ## log_redaction_file= # Comma separated list of strings representing the host/domain names that the Hue server can serve. # e.g.: localhost,domain1,* ## allowed_hosts=* # Administrators # ---------------- [[django_admins]] ## [[[admin1]]] ## name=john ## email=john@doe.com # UI customizations # ------------------- [[custom]] # Top banner HTML code # e.g.

Test Lab A2 Hue Services

## banner_top_html= # Configuration options for user authentication into the web application # ------------------------------------------------------------------------ [[auth]] # Authentication backend. Common settings are: # - django.contrib.auth.backends.ModelBackend (entirely Django backend) # - desktop.auth.backend.AllowAllBackend (allows everyone) # - desktop.auth.backend.AllowFirstUserDjangoBackend # (Default. Relies on Django and user manager, after the first login) # - desktop.auth.backend.LdapBackend # - desktop.auth.backend.PamBackend - WARNING: existing users in Hue may be unaccessible if they not exist in OS # - desktop.auth.backend.SpnegoDjangoBackend # - desktop.auth.backend.RemoteUserDjangoBackend # - libsaml.backend.SAML2Backend # - libopenid.backend.OpenIDBackend # - liboauth.backend.OAuthBackend # (New oauth, support Twitter, Facebook, Google+ and Linkedin backend=desktop.auth.backend.PamBackend # The service to use when querying PAM. pam_service=sudo sshd login # When using the desktop.auth.backend.RemoteUserDjangoBackend, this sets # the normalized name of the header that contains the remote user. # The HTTP header in the request is converted to a key by converting # all characters to uppercase, replacing any hyphens with underscores # and adding an HTTP_ prefix to the name. So, for example, if the header # is called Remote-User that would be configured as HTTP_REMOTE_USER # # Defaults to HTTP_REMOTE_USER ## remote_user_header=HTTP_REMOTE_USER # Ignore the case of usernames when searching for existing users. # Only supported in remoteUserDjangoBackend. ## ignore_username_case=true # Ignore the case of usernames when searching for existing users to authenticate with. # Only supported in remoteUserDjangoBackend. ## force_username_lowercase=true # Users will expire after they have not logged in for 'n' amount of seconds. # A negative number means that users will never expire. ## expires_after=-1 # Apply 'expires_after' to superusers. ## expire_superusers=true # Force users to change password on first login with desktop.auth.backend.AllowFirstUserDjangoBackend ## change_default_password=false # Configuration options for connecting to LDAP and Active Directory # ------------------------------------------------------------------- [[ldap]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Execute this script to produce the bind user password. This will be used # when `bind_password` is not set. ## bind_password_script= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication # If nt_domain is specified, this config is completely ignored. # If nt_domain is not specified, this should take on the form "cn=,dc=example,dc=com", # where is replaced by whatever is provided at the login page. Depending on your ldap schema, # you can also specify additional/alternative comma-separated attributes like uid, ou, etc ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" # Create users in Hue when they try to login with their LDAP credentials # For use when using LdapBackend for Hue authentication ## create_users_on_login = true # Synchronize a users groups when they login ## sync_groups_on_login=false # Ignore the case of usernames when searching for existing users in Hue. ## ignore_username_case=true # Force usernames to lowercase when creating new users from LDAP. ## force_username_lowercase=true # Use search bind authentication. # If set to true, hue will perform ldap search using bind credentials above (bind_dn, bind_password) # Hue will then search using the 'base_dn' for an entry with attr defined in 'user_name_attr', with the value # of short name provided on the login page. The search filter defined in 'user_filter' will also be used to limit # the search. Hue will search the entire subtree starting from base_dn. # If search_bind_authentication is set to false, Hue performs a direct bind to LDAP using the credentials provided # (not bind_dn and bind_password specified in hue.ini). There are 2 modes here - 'nt_domain' is specified or not. ## search_bind_authentication=true # Choose which kind of subgrouping to use: nested or suboordinate (deprecated). ## subgroups=suboordinate # Define the number of levels to search for nested members. ## nested_members_search_depth=10 # Whether or not to follow referrals ## follow_referrals=false # Enable python-ldap debugging. ## debug=false # Sets the debug level within the underlying LDAP C lib. ## debug_level=255 # Possible values for trace_level are 0 for no logging, 1 for only logging the method calls with arguments, # 2 for logging the method calls with arguments and the complete results and 9 for also logging the traceback of method calls. ## trace_level=0 [[[users]]] # Base filter for searching for users ## user_filter="objectclass=*" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName [[[groups]]] # Base filter for searching for groups ## group_filter="objectclass=*" # The group name attribute in the LDAP schema ## group_name_attr=cn # The attribute of the group object which identifies the members of the group ## group_member_attr=members [[[ldap_servers]]] ## [[[[mycompany]]]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Execute this script to produce the bind user password. This will be used # when `bind_password` is not set. ## bind_password_script= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" ## Use search bind authentication. ## search_bind_authentication=true # Whether or not to follow referrals ## follow_referrals=false # Enable python-ldap debugging. ## debug=false # Sets the debug level within the underlying LDAP C lib. ## debug_level=255 # Possible values for trace_level are 0 for no logging, 1 for only logging the method calls with arguments, # 2 for logging the method calls with arguments and the complete results and 9 for also logging the traceback of method calls. ## trace_level=0 ## [[[[[users]]]]] # Base filter for searching for users ## user_filter="objectclass=Person" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName ## [[[[[groups]]]]] # Base filter for searching for groups ## group_filter="objectclass=groupOfNames" # The username attribute in the LDAP schema ## group_name_attr=cn # Configuration options for specifying the Desktop Database. For more info, # see http://docs.djangoproject.com/en/1.4/ref/settings/#database-engine # ------------------------------------------------------------------------ [[database]] # Database engine is typically one of: # postgresql_psycopg2, mysql, sqlite3 or oracle. # # Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name. # Note for Oracle, options={'threaded':true} must be set in order to avoid crashes. # Note for Oracle, you can use the Oracle Service Name by setting "port=0" and then "name=:/". # Note for MariaDB use the 'mysql' engine. engine=mysql host={{ db_host }} port=3306 user={{ hue_user }} password={{ hue_password }} name={{ hue_name }} ## options={} # Configuration options for specifying the Desktop session. # For more info, see https://docs.djangoproject.com/en/1.4/topics/http/sessions/ # ------------------------------------------------------------------------ [[session]] # The cookie containing the users' session ID will expire after this amount of time in seconds. # Default is 2 weeks. ## ttl=1209600 # The cookie containing the users' session ID will be secure. # Should only be enabled with HTTPS. ## secure=false # The cookie containing the users' session ID will use the HTTP only flag. ## http_only=true # Use session-length cookies. Logs out the user when she closes the browser window. ## expire_at_browser_close=false # Configuration options for connecting to an external SMTP server # ------------------------------------------------------------------------ [[smtp]] # The SMTP server information for email notification delivery host=localhost port=25 user= password= # Whether to use a TLS (secure) connection when talking to the SMTP server tls=no # Default email address to use for various automated notification from Hue ## default_from_email=hue@localhost # Configuration options for Kerberos integration for secured Hadoop clusters # ------------------------------------------------------------------------ [[kerberos]] # Path to Hue's Kerberos keytab file ## hue_keytab= # Kerberos principal name for Hue ## hue_principal=hue/hostname.foo.com # Path to kinit ## kinit_path=/path/to/kinit # Configuration options for using OAuthBackend (core) login # ------------------------------------------------------------------------ [[oauth]] # The Consumer key of the application ## consumer_key=XXXXXXXXXXXXXXXXXXXXX # The Consumer secret of the application ## consumer_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # The Request token URL ## request_token_url=https://api.twitter.com/oauth/request_token # The Access token URL ## access_token_url=https://api.twitter.com/oauth/access_token # The Authorize URL ## authenticate_url=https://api.twitter.com/oauth/authorize ########################################################################### # Settings to configure SAML ########################################################################### [libsaml] # Xmlsec1 binary path. This program should be executable by the user running Hue. ## xmlsec_binary=/usr/local/bin/xmlsec1 # Entity ID for Hue acting as service provider. # Can also accept a pattern where '' will be replaced with server URL base. ## entity_id="/saml2/metadata/" # Create users from SSO on login. ## create_users_on_login=true # Required attributes to ask for from IdP. # This requires a comma separated list. ## required_attributes=uid # Optional attributes to ask for from IdP. # This requires a comma separated list. ## optional_attributes= # IdP metadata in the form of a file. This is generally an XML file containing metadata that the Identity Provider generates. ## metadata_file= # Private key to encrypt metadata with. ## key_file= # Signed certificate to send along with encrypted metadata. ## cert_file= # A mapping from attributes in the response from the IdP to django user attributes. ## user_attribute_mapping={'uid':'username'} # Have Hue initiated authn requests be signed and provide a certificate. ## authn_requests_signed=false # Have Hue initiated logout requests be signed and provide a certificate. ## logout_requests_signed=false # Username can be sourced from 'attributes' or 'nameid'. ## username_source=attributes # Performs the logout or not. ## logout_enabled=true ########################################################################### # Settings to configure OPENID ########################################################################### [libopenid] # (Required) OpenId SSO endpoint url. ## server_endpoint_url=https://www.google.com/accounts/o8/id # OpenId 1.1 identity url prefix to be used instead of SSO endpoint url # This is only supported if you are using an OpenId 1.1 endpoint ## identity_url_prefix=https://app.onelogin.com/openid/your_company.com/ # Create users from OPENID on login. ## create_users_on_login=true # Use email for username ## use_email_for_username=true ########################################################################### # Settings to configure OAuth ########################################################################### [liboauth] # NOTE: # To work, each of the active (i.e. uncommented) service must have # applications created on the social network. # Then the "consumer key" and "consumer secret" must be provided here. # # The addresses where to do so are: # Twitter: https://dev.twitter.com/apps # Google+ : https://cloud.google.com/ # Facebook: https://developers.facebook.com/apps # Linkedin: https://www.linkedin.com/secure/developer # # Additionnaly, the following must be set in the application settings: # Twitter: Callback URL (aka Redirect URL) must be set to http://YOUR_HUE_IP_OR_DOMAIN_NAME/oauth/social_login/oauth_authenticated # Google+ : CONSENT SCREEN must have email address # Facebook: Sandbox Mode must be DISABLED # Linkedin: "In OAuth User Agreement", r_emailaddress is REQUIRED # The Consumer key of the application ## consumer_key_twitter= ## consumer_key_google= ## consumer_key_facebook= ## consumer_key_linkedin= # The Consumer secret of the application ## consumer_secret_twitter= ## consumer_secret_google= ## consumer_secret_facebook= ## consumer_secret_linkedin= # The Request token URL ## request_token_url_twitter=https://api.twitter.com/oauth/request_token ## request_token_url_google=https://accounts.google.com/o/oauth2/auth ## request_token_url_linkedin=https://www.linkedin.com/uas/oauth2/authorization ## request_token_url_facebook=https://graph.facebook.com/oauth/authorize # The Access token URL ## access_token_url_twitter=https://api.twitter.com/oauth/access_token ## access_token_url_google=https://accounts.google.com/o/oauth2/token ## access_token_url_facebook=https://graph.facebook.com/oauth/access_token ## access_token_url_linkedin=https://api.linkedin.com/uas/oauth2/accessToken # The Authenticate URL ## authenticate_url_twitter=https://api.twitter.com/oauth/authorize ## authenticate_url_google=https://www.googleapis.com/oauth2/v1/userinfo?access_token= ## authenticate_url_facebook=https://graph.facebook.com/me?access_token= ## authenticate_url_linkedin=https://api.linkedin.com/v1/people/~:(email-address)?format=json&oauth2_access_token= # Username Map. Json Hash format. # Replaces username parts in order to simplify usernames obtained # Example: {"@sub1.domain.com":"_S1", "@sub2.domain.com":"_S2"} # converts 'email@sub1.domain.com' to 'email_S1' ## username_map={} # Whitelisted domains (only applies to Google OAuth). CSV format. ## whitelisted_domains_google= ########################################################################### # Settings for the RDBMS application ########################################################################### [librdbms] # The RDBMS app can have any number of databases configured in the databases # section. A database is known by its section name # (IE sqlite, mysql, psql, and oracle in the list below). [[databases]] # sqlite configuration. ## [[[sqlite]]] # Name to show in the UI. ## nice_name=SQLite # For SQLite, name defines the path to the database. ## name=/opt/mapr/hue/hue-3.9.0/desktop/desktop.db # Database backend to use. ## engine=sqlite # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} # mysql, oracle, or postgresql configuration. {%- if db_host and rdbms_name %} [[[mysql]]] # Name to show in the UI. nice_name="MySQL DB" # For MySQL and PostgreSQL, name is the name of the database. # For Oracle, Name is instance of the Oracle server. For express edition # this is 'xe' by default. name={{ rdbms_name }} # Database backend to use. This can be: # 1. mysql # 2. postgresql # 3. oracle engine=mysql # IP or hostname of the database to connect to. host={{ db_host }} # Port the database server is listening to. Defaults are: # 1. MySQL: 3306 # 2. PostgreSQL: 5432 # 3. Oracle Express Edition: 1521 port=3306 # Username to authenticate with when connecting to the database. user={{ rdbms_user }} # Password matching the username to authenticate with when # connecting to the database. password={{ rdbms_password }} # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} {%- endif %} ########################################################################### # Settings to configure your Hadoop cluster. ########################################################################### [hadoop] # Configuration for HDFS NameNode # ------------------------------------------------------------------------ [[hdfs_clusters]] # HA support by using HttpFs [[[default]]] # Enter the filesystem uri fs_defaultfs=maprfs:/// # NameNode logical name. ## logical_name= # Use WebHdfs/HttpFs as the communication mechanism. # Domain should be the NameNode or HttpFs host. # Default port is 14000 for HttpFs. webhdfs_url=http://{{ webhdfs_url | default("localhost", True) }}:14000/webhdfs/v1 # Change this if your HDFS cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} #Enable mutual ssl authentication mutual_ssl_auth=False ssl_cert= ssl_key= # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs # have to be verified against certificate authority ## ssl_cert_ca_verify=True # File size restriction for viewing file (float) # '1.0' - default 1 GB file size restriction # '0' - no file size restrictions # >0 - set file size restriction in gigabytes, ex. 0.5, 1.0, 1.2... ## file_size=1.0 # Directory of the Hadoop configuration ## hadoop_conf_dir=$HADOOP_CONF_DIR when set or '/etc/hadoop/conf' {% if yarn_mode %} # Configuration for YARN (MR2) # ------------------------------------------------------------------------ [[yarn_clusters]] [[[default]]] # Enter the host on which you are running the ResourceManager resourcemanager_host={{ rm_host | default("localhost", True) }} # The port where the ResourceManager IPC listens on resourcemanager_port={{ rm_port | default("8032", True) }} # Whether to submit jobs to this cluster submit_to={{ "True" if yarn_mode else "False" }} # Change this if your YARN cluster is secured ## security_enabled=false # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY ## mechanism=MAPR-SECURITY # URL of the ResourceManager API resourcemanager_api_url=http://{{ rm_host | default("localhost", True) }}:{{ rm_api_port | default("8088", True)}} # URL of the ProxyServer API proxy_api_url=http://{{ rm_proxy_api | default(rm_host, True) }}:{{ rm_proxy_port | default("8088", True)}} # URL of the HistoryServer API history_server_api_url=http://{{ hs_host | default(rm_host, True) }}:{{ hs_api_port | default("19888", True)}} # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs # have to be verified against certificate authority ## ssl_cert_ca_verify=True # HA support by specifying multiple clusters # e.g. {% for rm in ha_rms %} [[[ha{{ loop.index }}]]] # Enter the host on which you are running the failover Resource Manager resourcemanager_api_url=http://{{ rm }}:8088 logical_name=ha-rm-{{ loop.index }} submit_to=True {% endfor %} {%- endif %} {%- if not yarn_mode %} # Configuration for MapReduce (MR1) # ------------------------------------------------------------------------ [[mapred_clusters]] [[[default]]] # Enter the host on which you are running the Hadoop JobTracker jobtracker_host={{ jt_host | default("localhost", True) }} # The port where the JobTracker IPC listens on jobtracker_port={{ jt_port | default("9001", True) }} # JobTracker logical name for HA ## logical_name= # Thrift plug-in port for the JobTracker thrift_port={{ jt_plugin_port | default("9290", True) }} # Whether to submit jobs to this cluster submit_to={{ "False" if yarn_mode else "True" }} # Change this if your MapReduce cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- if fo_jobtracker_host %} # HA support by specifying multiple clusters # e.g. # [[[ha]]] # Enter the logical name of the JobTrackers ## logical_name=my-jt-name {%- endif %} {%- endif %} ########################################################################### # Settings to configure the Filebrowser app ########################################################################### [filebrowser] # Location on local filesystem where the uploaded archives are temporary stored. ## archive_upload_tempdir=/tmp ########################################################################### # Settings to configure liboozie ########################################################################### [liboozie] # The URL where the Oozie service runs on. This is required in order for # users to submit jobs. oozie_url=http://{{ oozie_host | default("localhost", True) }}:{{ oozie_port | default("11000", True) }}/oozie # Requires FQDN in oozie_url if enabled ## security_enabled=false # Location on HDFS where the workflows/coordinator are deployed when submitted. ## remote_deployement_dir=/oozie/deployments # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} ########################################################################### # Settings to configure the Oozie app ########################################################################### [oozie] # Location on local FS where the examples are stored. ## local_data_dir=..../examples # Location on local FS where the data for the examples is stored. ## sample_data_dir=...thirdparty/sample_data # Location on HDFS where the oozie examples and workflows are stored. remote_data_dir=/oozie/workspaces # Maximum of Oozie workflows or coodinators to retrieve in one API call. ## oozie_jobs_count=50 # Use Cron format for defining the frequency of a Coordinator instead of the old frequency number/unit. ## enable_cron_scheduling=true ########################################################################### # Settings to configure Beeswax with Hive ########################################################################### {%- if hive_host %} [beeswax] # Host where HiveServer2 is running. # If Kerberos security is enabled, use fully-qualified domain name (FQDN). hive_server_host={{ hive_host | default("localhost", True) }} # Port where HiveServer2 Thrift server runs on. hive_server_port={{ hive_port | default("10000", True) }} # Hive configuration directory, where hive-site.xml is located hive_conf_dir={{ hive_conf_dir | default("/opt/mapr/hive/hive-1.0/conf", True) }} # Timeout in seconds for thrift calls to Hive service ## server_conn_timeout=120 # Change this if your Hive is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} # Path to HiveServer2 start script hive_server_bin=/opt/mapr/hive/hive-{{ hive_version | default("1.0", True) }}/bin/hiveserver2 # Choose whether to use the old GetLog() thrift call from before Hive 0.14 to retrieve the logs. # If false, use the FetchResults() thrift call from Hive 1.0 or more instead. ## use_get_log_api=false # Set a LIMIT clause when browsing a partitioned table. # A positive value will be set as the LIMIT. If 0 or negative, do not set any limit. ## browse_partitioned_table_limit=250 # A limit to the number of rows that can be downloaded from a query. # A value of -1 means there will be no limit. # A maximum of 65,000 is applied to XLS downloads. ## download_row_limit=1000000 # Hue will try to close the Hive query when the user leaves the editor page. # This will free all the query resources in HiveServer2, but also make its results inaccessible. ## close_queries=false # Thrift version to use when communicating with HiveServer2. # New column format is from version 7. thrift_version={{ thrift_version }} [[ssl]] # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Choose whether Hue should validate certificates received from the server. ## validate=true {%- endif %} ########################################################################### # Settings to configure Impala ########################################################################### {%- if impala_host %} [impala] # Host of the Impala Server (one of the Impalad) server_host={{ impala_host | default("localhost", True) }} # Port of the Impala Server server_port={{ impala_port | default("21050", True) }} # Kerberos principal ## impala_principal=impala/hostname.foo.com # Turn on/off impersonation mechanism when talking to Impala impersonation_enabled=False # Number of initial rows of a result set to ask Impala to cache in order # to support re-fetching them for downloading them. # Set to 0 for disabling the option and backward compatibility. querycache_rows=50000 # Timeout in seconds for thrift calls ## server_conn_timeout=120 # Hue will try to close the Impala query when the user leaves the editor page. # This will free all the query resources in Impala, but also make its results inaccessible. ## close_queries=true # If QUERY_TIMEOUT_S > 0, the query will be timed out (i.e. cancelled) if Impala does not do any work # (compute or send back results) for that query within QUERY_TIMEOUT_S seconds. ## query_timeout_s=0 [[ssl]] # SSL communication enabled for this server. ## enabled=false # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Choose whether Hue should validate certificates received from the server. ## validate=true {%- endif %} ########################################################################### # Settings to configure Pig ########################################################################### [pig] # Location of piggybank.jar on local filesystem. ## local_sample_dir=/opt/mapr/pig/pig-0.12/contrib/piggybank/java/ # Location piggybank.jar will be copied to in HDFS. ## remote_data_dir=/oozie/pig/examples ########################################################################### # Settings to configure Sqoop2 ########################################################################### {%- if sqoop_host %} [sqoop] # For autocompletion, fill out the librdbms section. # Sqoop server URL server_url=http://{{ sqoop_host | default("localhost", True) }}:{{ sqoop_port | default("12000", True) }}/sqoop # Change this if your cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- endif %} ########################################################################### # Settings to configure Proxy ########################################################################### [proxy] # Comma-separated list of regular expressions, # which match 'host:port' of requested proxy target. ## whitelist=(localhost|127\.0\.0\.1):(50030|50070|50060|50075) # Comma-separated list of regular expressions, # which match any prefix of 'host:port/path' of requested proxy target. # This does not support matching GET parameters. ## blacklist= ########################################################################### # Settings to configure HBase Browser ########################################################################### {%- if hbase_host %} [hbase] # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'. # Use full hostname with security. hbase_clusters=(HBase|{{ hbase_host | default("localhost", True) }}:{{ hbase_port | default("9090", True) }}) # HBase configuration directory, where hbase-site.xml is located. hbase_conf_dir=/opt/mapr/hbase/hbase-{{ hbase_version | default("0.98.12", True) }}/conf # Hard limit of rows or columns per row fetched before truncating. ## truncate_limit = 500 # 'buffered' is the default of the HBase Thrift Server and supports security. # 'framed' can be used to chunk up responses, # which is useful when used in conjunction with the nonblocking server in Thrift. ## thrift_transport=buffered # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- endif %} ########################################################################### # Settings to configure Solr Search ########################################################################### [search] # URL of the Solr Server ## solr_url=http://localhost:8983/solr/ # Requires FQDN in solr_url if enabled security_enabled=${security_enabled} ## Query sent when no term is entered ## empty_query=*:* # Use latest Solr 5.2+ features. ## latest=false ########################################################################### # Settings to configure Solr Indexer ########################################################################### [indexer] # Location of the solrctl binary. ## solrctl_path=/usr/bin/solrctl ########################################################################### # Settings to configure Job Designer ########################################################################### [jobsub] # Location on local FS where examples and template are stored. ## local_data_dir=..../data # Location on local FS where sample data is stored ## sample_data_dir=...thirdparty/sample_data ########################################################################### # Settings to configure Job Browser. ########################################################################### [jobbrowser] # Share submitted jobs information with all users. If set to false, # submitted jobs are visible only to the owner and administrators. ## share_jobs=true # Whether to disalbe the job kill button for all users in the jobbrowser ## disable_killing_jobs=false ########################################################################### # Settings to configure the Zookeeper application. ########################################################################### [zookeeper] [[clusters]] [[[default]]] # Zookeeper ensemble. Comma separated list of Host/Port. # e.g. localhost:5181,node2_ip@:5181,node3_ip@:5181 host_ports={{ zk_hosts_with_port | default("localhost:5181", True) }} # The URL of the REST contrib service (required for znode browsing). ## rest_url=http://localhost:9999 # Name of Kerberos principal when using security. ## principal_name=zookeeper ########################################################################### # Settings to configure the Spark application. ########################################################################### {%- if livy_host %} [spark] # Host address of the Livy Server. livy_server_host={{ livy_host }} # Port of the Livy Server. ## livy_server_port=8998 # Configure livy to start with 'process', 'thread', or 'yarn' workers. livy_server_session_kind=yarn # If livy should use proxy users when submitting a job. ## livy_impersonation_enabled=true # List of available types of snippets ## languages='[{"name": "Scala Shell", "type": "scala"},{"name": "PySpark Shell", "type": "python"},{"name": "R Shell", "type": "r"},{"name": "Jar", "type": "Jar"},{"name": "Python", "type": "py"},{"name": "Impala SQL", "type": "impala"},{"name": "Hive SQL", "type": "hive"},{"name": "Text", "type": "text"}]' {%- endif %} ########################################################################### # Settings for the User Admin application ########################################################################### [useradmin] # The name of the default user group that users will be a member of ## default_user_group=default [[password_policy]] # Set password policy to all users. The default policy requires password to be at least 8 characters long, # and contain both uppercase and lowercase letters, numbers, and special characters. ## is_enabled=false ## pwd_regex="^(?=.*?[A-Z])(?=(.*[a-z]){1,})(?=(.*[\d]){1,})(?=(.*[\W_]){1,}).{8,}$" ## pwd_hint="The password must be at least 8 characters long, and must contain both uppercase and lowercase letters, at least one number, and at least one special character." ## pwd_error_message="The password must be at least 8 characters long, and must contain both uppercase and lowercase letters, at least one number, and at least one special character." ########################################################################### # Settings for the Sentry lib ########################################################################### {%- if sentry_host %} [libsentry] # Hostname or IP of server. hostname={{ sentry_host }} # Port the sentry service is running on. port=8038 # Sentry configuration directory, where sentry-site.xml is located. sentry_conf_dir={{ sentry_conf }} {%- endif %} ########################################################################### # Settings to configure the ZooKeeper Lib ########################################################################### [libzookeeper] # ZooKeeper ensemble. Comma separated list of Host/Port. # e.g. localhost:2181,localhost:2182,localhost:2183 ## ensemble=localhost:2181 # Name of Kerberos principal when using security. ## principal_name=zookeeper sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_3.7.0.template0000666000175100017510000010625013245514472026357 0ustar zuulzuul00000000000000# Hue configuration file # =================================== # # For complete documentation about the contents of this file, run # $ /build/env/bin/hue config_help # # All .ini files under the current directory are treated equally. Their # contents are merged to form the Hue configuration, which can # can be viewed on the Hue at # http://:/dump_config ########################################################################### # General configuration for core Desktop features (authentication, etc) ########################################################################### [desktop] send_dbug_messages=1 # To show database transactions, set database_logging to 1 database_logging=0 # Set this to a random string, the longer the better. # This is used for secure hashing in the session store. secret_key={{ secret_key }} # Webserver listens on this address and port http_host=0.0.0.0 http_port=8888 # Time zone name time_zone=America/Los_Angeles # Enable or disable Django debug mode. django_debug_mode=false # Enable or disable backtrace for server error http_500_debug_mode=false # Enable or disable memory profiling. ## memory_profiler=false # Server email for internal error messages ## django_server_email='hue@localhost.localdomain' # Email backend ## django_email_backend=django.core.mail.backends.smtp.EmailBackend # Webserver runs as this user server_user=mapr server_group=mapr # This should be the Hue admin and proxy user default_user=mapr # This should be the hadoop cluster admin default_hdfs_superuser=mapr default_jobtracker_host={{ resource_manager_uri | default("maprfs:///", True) }} # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # This should be the Hue admin and proxy user ## default_user=hue # This should be the hadoop cluster admin ## default_hdfs_superuser=hdfs # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # Number of threads used by the CherryPy web server ## cherrypy_server_threads=10 # Filename of SSL Certificate ## ssl_certificate= # Filename of SSL RSA Private Key ## ssl_private_key= # List of allowed and disallowed ciphers in cipher list format. # See http://www.openssl.org/docs/apps/ciphers.html for more information on cipher list format. ## ssl_cipher_list=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2 # LDAP username and password of the hue user used for LDAP authentications. # Set it to use LDAP Authentication with HiveServer2 and Impala. ## ldap_username=hue ## ldap_password= # Default encoding for site data ## default_site_encoding=utf-8 # Help improve Hue with anonymous usage analytics. # Use Google Analytics to see how many times an application or specific section of an application is used, nothing more. ## collect_usage=true # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER. ## secure_proxy_ssl_header=false # Comma-separated list of Django middleware classes to use. # See https://docs.djangoproject.com/en/1.4/ref/middleware/ for more details on middlewares in Django. ## middleware=desktop.auth.backend.LdapSynchronizationBackend # Comma-separated list of regular expressions, which match the redirect URL. # For example, to restrict to your local domain and FQDN, the following value can be used: # ^\/.*$,^http:\/\/www.mydomain.com\/.*$ ## redirect_whitelist= # Comma separated list of apps to not load at server startup. # e.g.: pig,zookeeper app_blacklist=spark,search,zookeeper,security{{ ",impala" if not impala_host }}{{ ",hbase" if not hbase_host }}{{ ",sqoop" if not sqoop_host }} # The directory where to store the auditing logs. Auditing is disable if the value is empty. # e.g. /var/log/hue/audit.log ## audit_event_log_dir= # Size in KB/MB/GB for audit log to rollover. ## audit_log_max_file_size=100MB # Administrators # ---------------- [[django_admins]] ## [[[admin1]]] ## name=john ## email=john@doe.com # UI customizations # ------------------- [[custom]] # Top banner HTML code # e.g.

Test Lab A2 Hue Services

## banner_top_html= # Configuration options for user authentication into the web application # ------------------------------------------------------------------------ [[auth]] # Authentication backend. Common settings are: # - django.contrib.auth.backends.ModelBackend (entirely Django backend) # - desktop.auth.backend.AllowAllBackend (allows everyone) # - desktop.auth.backend.AllowFirstUserDjangoBackend # (Default. Relies on Django and user manager, after the first login) # - desktop.auth.backend.LdapBackend # - desktop.auth.backend.PamBackend # - desktop.auth.backend.SpnegoDjangoBackend # - desktop.auth.backend.RemoteUserDjangoBackend # - libsaml.backend.SAML2Backend # - libopenid.backend.OpenIDBackend # - liboauth.backend.OAuthBackend # (Support Twitter, Facebook, Google+ and Linkedin ## backend=desktop.auth.backend.AllowFirstUserDjangoBackend # The service to use when querying PAM. ## pam_service=login # When using the desktop.auth.backend.RemoteUserDjangoBackend, this sets # the normalized name of the header that contains the remote user. # The HTTP header in the request is converted to a key by converting # all characters to uppercase, replacing any hyphens with underscores # and adding an HTTP_ prefix to the name. So, for example, if the header # is called Remote-User that would be configured as HTTP_REMOTE_USER # # Defaults to HTTP_REMOTE_USER ## remote_user_header=HTTP_REMOTE_USER # Synchronize a users groups when they login ## sync_groups_on_login=false # Ignore the case of usernames when searching for existing users. # Only supported in remoteUserDjangoBackend. ## ignore_username_case=false # Ignore the case of usernames when searching for existing users to authenticate with. # Only supported in remoteUserDjangoBackend. ## force_username_lowercase=false # Users will expire after they have not logged in for 'n' amount of seconds. # A negative number means that users will never expire. ## expires_after=-1 # Apply 'expires_after' to superusers. ## expire_superusers=true # Configuration options for connecting to LDAP and Active Directory # ------------------------------------------------------------------- [[ldap]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication # If nt_domain is specified, this config is completely ignored. # If nt_domain is not specified, this should take on the form "cn=,dc=example,dc=com", # where is replaced by whatever is provided at the login page. Depending on your ldap schema, # you can also specify additional/alternative comma-separated attributes like uid, ou, etc ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" # Create users in Hue when they try to login with their LDAP credentials # For use when using LdapBackend for Hue authentication ## create_users_on_login = true # Ignore the case of usernames when searching for existing users in Hue. ## ignore_username_case=false # Ignore the case of usernames when searching for existing users to authenticate with. # Only supported in remoteUserDjangoBackend. ## force_username_lowercase=false # Use search bind authentication. # If set to true, hue will perform ldap search using bind credentials above (bind_dn, bind_password) # Hue will then search using the 'base_dn' for an entry with attr defined in 'user_name_attr', with the value # of short name provided on the login page. The search filter defined in 'user_filter' will also be used to limit # the search. Hue will search the entire subtree starting from base_dn. # If search_bind_authentication is set to false, Hue performs a direct bind to LDAP using the credentials provided # (not bind_dn and bind_password specified in hue.ini). There are 2 modes here - 'nt_domain' is specified or not. ## search_bind_authentication=true # Choose which kind of subgrouping to use: nested or suboordinate (deprecated). ## subgroups=suboordinate # Define the number of levels to search for nested members. ## nested_members_search_depth=10 [[[users]]] # Base filter for searching for users ## user_filter="objectclass=*" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName [[[groups]]] # Base filter for searching for groups ## group_filter="objectclass=*" # The group name attribute in the LDAP schema ## group_name_attr=cn # The attribute of the group object which identifies the members of the group ## group_member_attr=members [[[ldap_servers]]] ## [[[[mycompany]]]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" ## Use search bind authentication. ## search_bind_authentication=true ## [[[[[users]]]]] # Base filter for searching for users ## user_filter="objectclass=Person" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName ## [[[[[groups]]]]] # Base filter for searching for groups ## group_filter="objectclass=groupOfNames" # The username attribute in the LDAP schema ## group_name_attr=cn # Configuration options for specifying the Desktop Database. For more info, # see http://docs.djangoproject.com/en/1.4/ref/settings/#database-engine # ------------------------------------------------------------------------ [[database]] # Database engine is typically one of: # postgresql_psycopg2, mysql, sqlite3 or oracle. # # Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name. # Note for Oracle, options={'threaded':true} must be set in order to avoid crashes. # Note for Oracle, you can use the Oracle Service Name by setting "port=0" and then "name=:/". engine=mysql host={{ db_host }} port=3306 user={{ hue_user }} password={{ hue_password }} name={{ hue_name }} ## options={} # Configuration options for specifying the Desktop session. # For more info, see https://docs.djangoproject.com/en/1.4/topics/http/sessions/ # ------------------------------------------------------------------------ [[session]] # The cookie containing the users' session ID will expire after this amount of time in seconds. # Default is 2 weeks. ## ttl=1209600 # The cookie containing the users' session ID will be secure. # Should only be enabled with HTTPS. ## secure=false # The cookie containing the users' session ID will use the HTTP only flag. ## http_only=false # Use session-length cookies. Logs out the user when she closes the browser window. ## expire_at_browser_close=false # Configuration options for connecting to an external SMTP server # ------------------------------------------------------------------------ [[smtp]] # The SMTP server information for email notification delivery host=localhost port=25 user= password= # Whether to use a TLS (secure) connection when talking to the SMTP server tls=no # Default email address to use for various automated notification from Hue ## default_from_email=hue@localhost # Configuration options for Kerberos integration for secured Hadoop clusters # ------------------------------------------------------------------------ [[kerberos]] # Path to Hue's Kerberos keytab file ## hue_keytab= # Kerberos principal name for Hue ## hue_principal=hue/hostname.foo.com # Path to kinit ## kinit_path=/path/to/kinit # Configuration options for using OAuthBackend (core) login # ------------------------------------------------------------------------ [[oauth]] # The Consumer key of the application ## consumer_key=XXXXXXXXXXXXXXXXXXXXX # The Consumer secret of the application ## consumer_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # The Request token URL ## request_token_url=https://api.twitter.com/oauth/request_token # The Access token URL ## access_token_url=https://api.twitter.com/oauth/access_token # The Authorize URL ## authenticate_url=https://api.twitter.com/oauth/authorize ########################################################################### # Settings to configure SAML ########################################################################### [libsaml] # Xmlsec1 binary path. This program should be executable by the user running Hue. ## xmlsec_binary=/usr/local/bin/xmlsec1 # Entity ID for Hue acting as service provider. # Can also accept a pattern where '' will be replaced with server URL base. ## entity_id="/saml2/metadata/" # Create users from SSO on login. ## create_users_on_login=true # Required attributes to ask for from IdP. # This requires a comma separated list. ## required_attributes=uid # Optional attributes to ask for from IdP. # This requires a comma separated list. ## optional_attributes= # IdP metadata in the form of a file. This is generally an XML file containing metadata that the Identity Provider generates. ## metadata_file= # Private key to encrypt metadata with. ## key_file= # Signed certificate to send along with encrypted metadata. ## cert_file= # A mapping from attributes in the response from the IdP to django user attributes. ## user_attribute_mapping={'uid':'username'} # Have Hue initiated authn requests be signed and provide a certificate. ## authn_requests_signed=false # Have Hue initiated logout requests be signed and provide a certificate. ## logout_requests_signed=false ## Username can be sourced from 'attributes' or 'nameid'. ## username_source=attributes # Performs the logout or not. ## logout_enabled=true ########################################################################### # Settings to configure OPENID ########################################################################### [libopenid] # (Required) OpenId SSO endpoint url. ## server_endpoint_url=https://www.google.com/accounts/o8/id # OpenId 1.1 identity url prefix to be used instead of SSO endpoint url # This is only supported if you are using an OpenId 1.1 endpoint ## identity_url_prefix=https://app.onelogin.com/openid/your_company.com/ # Create users from OPENID on login. ## create_users_on_login=true # Use email for username ## use_email_for_username=true ########################################################################### # Settings to configure OAuth ########################################################################### [liboauth] # NOTE: # To work, each of the active (i.e. uncommented) service must have # applications created on the social network. # Then the "consumer key" and "consumer secret" must be provided here. # # The addresses where to do so are: # Twitter: https://dev.twitter.com/apps # Google+ : https://cloud.google.com/ # Facebook: https://developers.facebook.com/apps # Linkedin: https://www.linkedin.com/secure/developer # # Additionnaly, the following must be set in the application settings: # Twitter: Callback URL (aka Redirect URL) must be set to http://YOUR_HUE_IP_OR_DOMAIN_NAME/oauth/social_login/oauth_authenticated # Google+ : CONSENT SCREEN must have email address # Facebook: Sandbox Mode must be DISABLED # Linkedin: "In OAuth User Agreement", r_emailaddress is REQUIRED # The Consumer key of the application ## consumer_key_twitter= ## consumer_key_google= ## consumer_key_facebook= ## consumer_key_linkedin= # The Consumer secret of the application ## consumer_secret_twitter= ## consumer_secret_google= ## consumer_secret_facebook= ## consumer_secret_linkedin= # The Request token URL ## request_token_url_twitter=https://api.twitter.com/oauth/request_token ## request_token_url_google=https://accounts.google.com/o/oauth2/auth ## request_token_url_linkedin=https://www.linkedin.com/uas/oauth2/authorization ## request_token_url_facebook=https://graph.facebook.com/oauth/authorize # The Access token URL ## access_token_url_twitter=https://api.twitter.com/oauth/access_token ## access_token_url_google=https://accounts.google.com/o/oauth2/token ## access_token_url_facebook=https://graph.facebook.com/oauth/access_token ## access_token_url_linkedin=https://api.linkedin.com/uas/oauth2/accessToken # The Authenticate URL ## authenticate_url_twitter=https://api.twitter.com/oauth/authorize ## authenticate_url_google=https://www.googleapis.com/oauth2/v1/userinfo?access_token= ## authenticate_url_facebook=https://graph.facebook.com/me?access_token= ## authenticate_url_linkedin=https://api.linkedin.com/v1/people/~:(email-address)?format=json&oauth2_access_token= # Username Map. Json Hash format. # Replaces username parts in order to simplify usernames obtained # Example: {"@sub1.domain.com":"_S1", "@sub2.domain.com":"_S2"} # converts 'email@sub1.domain.com' to 'email_S1' ## username_map={} # Whitelisted domains (only applies to Google OAuth). CSV format. ## whitelisted_domains_google= ########################################################################### # Settings for the RDBMS application ########################################################################### [librdbms] # The RDBMS app can have any number of databases configured in the databases # section. A database is known by its section name # (IE sqlite, mysql, psql, and oracle in the list below). [[databases]] # sqlite configuration. ## [[[sqlite]]] # Name to show in the UI. ## nice_name=SQLite # For SQLite, name defines the path to the database. ## name=/opt/mapr/hue/hue-3.7.0/desktop/desktop.db # Database backend to use. ## engine=sqlite # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} # mysql, oracle, or postgresql configuration. {%- if db_host and rdbms_name %} [[[mysql]]] # Name to show in the UI. nice_name="MySQL DB" # For MySQL and PostgreSQL, name is the name of the database. # For Oracle, Name is instance of the Oracle server. For express edition # this is 'xe' by default. name={{ rdbms_name }} # Database backend to use. This can be: # 1. mysql # 2. postgresql # 3. oracle engine=mysql # IP or hostname of the database to connect to. host={{ db_host }} # Port the database server is listening to. Defaults are: # 1. MySQL: 3306 # 2. PostgreSQL: 5432 # 3. Oracle Express Edition: 1521 port=3306 # Username to authenticate with when connecting to the database. user={{ rdbms_user }} # Password matching the username to authenticate with when # connecting to the database. password={{ rdbms_password }} {%- endif %} # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} ########################################################################### # Settings to configure your Hadoop cluster. ########################################################################### [hadoop] # Configuration for HDFS NameNode # ------------------------------------------------------------------------ [[hdfs_clusters]] # HA support by using HttpFs [[[default]]] # Enter the filesystem uri fs_defaultfs=maprfs:/// # NameNode logical name. ## logical_name= # Use WebHdfs/HttpFs as the communication mechanism. # Domain should be the NameNode or HttpFs host. # Default port is 14000 for HttpFs. webhdfs_url=http://{{ webhdfs_url | default("localhost", True) }}:14000/webhdfs/v1 # Change this if your HDFS cluster is secured ## security_enabled=false # Default umask for file and directory creation, specified in an octal value. ## umask=022 {% if yarn_mode %} # Configuration for YARN (MR2) # ------------------------------------------------------------------------ [[yarn_clusters]] [[[default]]] # Enter the host on which you are running the ResourceManager resourcemanager_host={{ rm_host | default("localhost", True) }} # The port where the ResourceManager IPC listens on resourcemanager_port={{ rm_port | default("8032", True) }} # Whether to submit jobs to this cluster submit_to={{ "True" if yarn_mode else "False" }} # Change this if your YARN cluster is secured ## security_enabled=false # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY ## mechanism=MAPR-SECURITY # URL of the ResourceManager API resourcemanager_api_url=http://{{ rm_host | default("localhost", True) }}:{{ rm_api_port | default("8088", True)}} # URL of the ProxyServer API proxy_api_url=http://{{ rm_proxy_api | default(rm_host, True) }}:{{ rm_proxy_port | default("8088", True)}} # URL of the HistoryServer API history_server_api_url=http://{{ hs_host | default(rm_host, True) }}:{{ hs_api_port | default("19888", True)}} # In secure mode (HTTPS), if SSL certificates from Resource Manager's # Rest Server have to be verified against certificate authority ssl_cert_ca_verify=False {%- endif %} {%- if not yarn_mode %} # Configuration for MapReduce (MR1) # ------------------------------------------------------------------------ [[mapred_clusters]] [[[default]]] # Enter the host on which you are running the Hadoop JobTracker jobtracker_host={{ jt_host | default("localhost", True) }} # The port where the JobTracker IPC listens on jobtracker_port={{ jt_port | default("9001", True) }} # JobTracker logical name for HA ## logical_name= # Thrift plug-in port for the JobTracker thrift_port={{ jt_plugin_port | default("9290", True) }} # jobtracker kerberos principal name jt_kerberos_principal=mapr # Whether to submit jobs to this cluster submit_to={{ "False" if yarn_mode else "True" }} # Change this if your MapReduce cluster is secured ## security_enabled=false # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY ## mechanism=MAPR-SECURITY {%- if fo_jobtracker_host %} # HA support by specifying multiple clusters # e.g. # [[[ha]]] # Enter the logical name of the JobTrackers ## logical_name=my-jt-name {%- endif %} {%- endif %} ########################################################################### # Settings to configure the Filebrowser app ########################################################################### [filebrowser] # Location on local filesystem where the uploaded archives are temporary stored. ## archive_upload_tempdir=/tmp ########################################################################### # Settings to configure liboozie ########################################################################### [liboozie] # The URL where the Oozie service runs on. This is required in order for # users to submit jobs. oozie_url=http://{{ oozie_host | default("localhost", True) }}:{{ oozie_port | default("11000", True) }}/oozie # Requires FQDN in oozie_url if enabled ## security_enabled=false # Location on HDFS where the workflows/coordinator are deployed when submitted. ## remote_deployement_dir=/oozie/deployments ########################################################################### # Settings to configure the Oozie app ########################################################################### [oozie] # Location on local FS where the examples are stored. ## local_data_dir=..../examples # Location on local FS where the data for the examples is stored. ## sample_data_dir=...thirdparty/sample_data # Location on HDFS where the oozie examples and workflows are stored. remote_data_dir=/oozie/workspaces # Maximum of Oozie workflows or coodinators to retrieve in one API call. ## oozie_jobs_count=100 # Use Cron format for defining the frequency of a Coordinator instead of the old frequency number/unit. ## enable_cron_scheduling=true ########################################################################### # Settings to configure Beeswax with Hive ########################################################################### {%- if hive_host %} [beeswax] # Host where HiveServer2 is running. # If Kerberos security is enabled, use fully-qualified domain name (FQDN). hive_server_host={{ hive_host | default("localhost", True) }} # Port where HiveServer2 Thrift server runs on. hive_server_port={{ hive_port | default("10000", True) }} # Hive configuration directory, where hive-site.xml is located hive_conf_dir={{ hive_conf_dir | default("/opt/mapr/hive/hive-0.13/conf", True) }} # Timeout in seconds for thrift calls to Hive service ## server_conn_timeout=120 # Path to HiveServer2 start script hive_server_bin=/opt/mapr/hive/hive-{{ hive_version | default("0.13", True) }}/bin/hiveserver2 # Set a LIMIT clause when browsing a partitioned table. # A positive value will be set as the LIMIT. If 0 or negative, do not set any limit. ## browse_partitioned_table_limit=250 # A limit to the number of rows that can be downloaded from a query. # A value of -1 means there will be no limit. # A maximum of 65,000 is applied to XLS downloads. ## download_row_limit=1000000 # Hue will try to close the Hive query when the user leaves the editor page. # This will free all the query resources in HiveServer2, but also make its results inaccessible. ## close_queries=false # Default thrift version to use when communicating with HiveServer2 # thrift_version=2 [[ssl]] # SSL communication enabled for this server. ## enabled=false # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Path to the private key file. ## key=/etc/hue/key.pem # Path to the public certificate file. ## cert=/etc/hue/cert.pem # Choose whether Hue should validate certificates received from the server. ## validate=true {%- endif %} ########################################################################### # Settings to configure Pig ########################################################################### [pig] # Location of piggybank.jar on local filesystem. ## local_sample_dir=/opt/mapr/pig/pig-0.12/contrib/piggybank/java/ # Location piggybank.jar will be copied to in HDFS. ## remote_data_dir=/oozie/pig/examples ########################################################################### # Settings to configure Sqoop ########################################################################### {%- if sqoop_host %} [sqoop] # For autocompletion, fill out the librdbms section. # Sqoop server URL server_url=http://{{ sqoop_host | default("localhost", True) }}:{{ sqoop_port | default("12000", True) }}/sqoop {%- endif %} ########################################################################### # Settings to configure Proxy ########################################################################### [proxy] # Comma-separated list of regular expressions, # which match 'host:port' of requested proxy target. ## whitelist=(localhost|127\.0\.0\.1):(50030|50070|50060|50075) # Comma-separated list of regular expressions, # which match any prefix of 'host:port/path' of requested proxy target. # This does not support matching GET parameters. ## blacklist= ########################################################################### # Settings to configure Impala ########################################################################### {%- if impala_host %} [impala] # Host of the Impala Server (one of the Impalad) server_host={{ impala_host | default("localhost", True) }} # Port of the Impala Server server_port={{ impala_port | default("21050", True) }} # Kerberos principal ## impala_principal=impala/hostname.foo.com # Turn on/off impersonation mechanism when talking to Impala impersonation_enabled=False # Number of initial rows of a result set to ask Impala to cache in order # to support re-fetching them for downloading them. # Set to 0 for disabling the option and backward compatibility. querycache_rows=0 # Timeout in seconds for thrift calls ## server_conn_timeout=120 # Hue will try to close the Impala query when the user leaves the editor page. # This will free all the query resources in Impala, but also make its results inaccessible. ## close_queries=true # If QUERY_TIMEOUT_S > 0, the query will be timed out (i.e. cancelled) if Impala does not do any work # (compute or send back results) for that query within QUERY_TIMEOUT_S seconds. ## query_timeout_s=0 {%- endif %} ########################################################################### # Settings to configure HBase Browser ########################################################################### {%- if hbase_host %} [hbase] # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'. # Use full hostname with security. hbase_clusters=(HBase|{{ hbase_host | default("localhost", True) }}:{{ hbase_port | default("9090", True) }}) # HBase configuration directory, where hbase-site.xml is located. hbase_conf_dir={{ hbase_conf_dir | default("/opt/mapr/hbase/hbase-0.98.4/conf", True) }} # Hard limit of rows or columns per row fetched before truncating. ## truncate_limit = 500 # 'buffered' is the default of the HBase Thrift Server and supports security. # 'framed' can be used to chunk up responses, # which is useful when used in conjunction with the nonblocking server in Thrift. ## thrift_transport=buffered # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY ## mechanism=MAPR-SECURITY {%- endif %} ########################################################################### # Settings to configure Solr Search ########################################################################### [search] # URL of the Solr Server ## solr_url=http://localhost:8983/solr/ # Requires FQDN in solr_url if enabled ## security_enabled=false ## Query sent when no term is entered ## empty_query=*:* ########################################################################### # Settings to configure Solr Indexer ########################################################################### [indexer] # Location of the solrctl binary. ## solrctl_path=/usr/bin/solrctl # Location of the solr home. ## solr_home=/usr/lib/solr # Zookeeper ensemble. ## solr_zk_ensemble=localhost:2181/solr # The contents of this directory will be copied over to the solrctl host to its temporary directory. ## config_template_path=/../hue/desktop/libs/indexer/src/data/solr_configs ########################################################################### # Settings to configure Job Designer ########################################################################### [jobsub] # Location on local FS where examples and template are stored. ## local_data_dir=..../data # Location on local FS where sample data is stored ## sample_data_dir=...thirdparty/sample_data ########################################################################### # Settings to configure Job Browser. ########################################################################### [jobbrowser] # Share submitted jobs information with all users. If set to false, # submitted jobs are visible only to the owner and administrators. ## share_jobs=true ########################################################################### # Settings to configure the Zookeeper application. ########################################################################### [zookeeper] [[clusters]] [[[default]]] # Zookeeper ensemble. Comma separated list of Host/Port. # e.g. localhost:5181,node2_ip@:5181,node3_ip@:5181 host_ports={{ zk_hosts_with_port | default("localhost:5181", True) }} # The URL of the REST contrib service (required for znode browsing) ## rest_url=http://localhost:9999 ########################################################################### # Settings to configure the Spark application. ########################################################################### [spark] # URL of the REST Spark Job Server. ## server_url=http://localhost:8090/ ########################################################################### # Settings for the User Admin application ########################################################################### [useradmin] # The name of the default user group that users will be a member of ## default_user_group=default ########################################################################### # Settings for the Sentry lib ########################################################################### [libsentry] # Hostname or IP of server. ## hostname=localhost # Port the sentry service is running on. ## port=8038 # Sentry configuration directory, where sentry-site.xml is located. ## sentry_conf_dir=/opt/mapr/sentry/sentry-1.4.0/conf sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_sh_3.10.0.template0000666000175100017510000000561513245514472027126 0ustar zuulzuul00000000000000#!/bin/bash bin=`dirname "$0"` bin=`cd "$bin"; pwd` export HUE_HOME=${bin}/.. export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-{{ hadoop_version }}/etc/hadoop # # activate the Python virtual environment # source $HUE_HOME/build/env/bin/activate # Look for installed JDK if [ -z "$JAVA_HOME" ]; then sys_java="/usr/bin/java" if [ -e $sys_java ]; then jcmd=`readlink -f $sys_java` if [ -x ${jcmd%/jre/bin/java}/bin/javac ]; then JAVA_HOME=${jcmd%/jre/bin/java} elif [ -x ${jcmd%/java}/javac ]; then JAVA_HOME=${jcmd%/bin/java} fi [ -n "${JAVA_HOME}" ] && export JAVA_HOME fi fi export LD_LIBRARY_PATH=$HUE_HOME/build/env/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/mapr/lib:$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH # get arguments command=$1 shift startStop=$1 shift #get spark_home if [ $command == "livy_server" ]; then if [ -z "$SPARK_HOME" ]; then if [ -d /opt/mapr/spark/spark-* ]; then cd /opt/mapr/spark/spark-* SPARK_HOME=`pwd` export SPARK_HOME=`pwd` else echo "Cannot export spark home!" exit 0 fi fi export PATH="$PATH:$SPARK_HOME/bin" fi #file client impersonation MAPR_IMPERSONATION_ENABLED=" " export MAPR_IMPERSONATION_ENABLED HUE_PID_DIR=$HUE_HOME/pids if [ "$HUE_IDENT_STRING" = "" ]; then export HUE_IDENT_STRING=`id -nu` fi # get log directory if [ "$HUE_LOG_DIR" = "" ]; then export HUE_LOG_DIR="$HUE_HOME/logs" fi mkdir -p "$HUE_LOG_DIR" if [ "$HUE_PID_DIR" = "" ]; then export HUE_PID_DIR="$HUE_HOME/pids" fi mkdir -p "$HUE_PID_DIR" log=$HUE_LOG_DIR/hue-$HUE_IDENT_STRING-$command-$HOSTNAME.out pid=$HUE_PID_DIR/hue-$HUE_IDENT_STRING-$command.pid secure_sh_log=$HUE_LOG_DIR/secure-sh-log.out case $startStop in (start) if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo Hue web server running as process `cat $pid`. Stop it first. exit 1 fi fi cd $HUE_HOME # Generate secure keys nohup $HUE_HOME/bin/secure.sh >> "$secure_sh_log" 2>&1 < /dev/null & nohup $HUE_HOME/build/env/bin/hue $command >> "$log" 2>&1 < /dev/null & echo $! > $pid echo "`date` $command started, pid `cat $pid`" >> "$log" 2>&1 < /dev/null ;; (stop) if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo stopping $command kill `cat $pid` echo "`date` $command stopped, pid `cat $pid`" >> "$log" 2>&1 < /dev/null else echo no $command to stop fi else echo no $command to stop fi ;; (status) if [ -f $pid ]; then if kill -0 `cat $pid`; then echo $command running as process `cat $pid`. exit 0 fi echo $pid exists with pid `cat $pid` but no $command. exit 1 fi echo $command not running. exit 1 ;; (*) echo $usage exit 1 ;; esac # # deactivate the Python virtual environment # deactivate sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_3.8.1.template0000666000175100017510000012126213245514472026361 0ustar zuulzuul00000000000000# Hue configuration file # =================================== # # For complete documentation about the contents of this file, run # $ /build/env/bin/hue config_help # # All .ini files under the current directory are treated equally. Their # contents are merged to form the Hue configuration, which can # can be viewed on the Hue at # http://:/dump_config ########################################################################### # General configuration for core Desktop features (authentication, etc) ########################################################################### [desktop] # Set this to a random string, the longer the better. # This is used for secure hashing in the session store. secret_key={{ secret_key }} # Webserver listens on this address and port http_host=0.0.0.0 http_port=8888 # Time zone name time_zone=America/Los_Angeles # Enable or disable Django debug mode. django_debug_mode=false # Enable or disable backtrace for server error http_500_debug_mode=false # Enable or disable memory profiling. ## memory_profiler=false # Server email for internal error messages ## django_server_email='hue@localhost.localdomain' # Email backend ## django_email_backend=django.core.mail.backends.smtp.EmailBackend # Webserver runs as this user server_user=mapr server_group=mapr # This should be the Hue admin and proxy user default_user=mapr # This should be the hadoop cluster admin default_hdfs_superuser=mapr default_jobtracker_host=maprfs:/// # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # This should be the Hue admin and proxy user ## default_user=hue # This should be the hadoop cluster admin ## default_hdfs_superuser=hdfs # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # Number of threads used by the CherryPy web server ## cherrypy_server_threads=40 # Filename of SSL Certificate ## ssl_certificate= # Filename of SSL RSA Private Key ## ssl_private_key= # SSL certificate password ## ssl_password= # Execute this script to produce the SSL password. This will be used when `ssl_password` is not set. ## ssl_password_script= # List of allowed and disallowed ciphers in cipher list format. # See http://www.openssl.org/docs/apps/ciphers.html for more information on cipher list format. ## ssl_cipher_list=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2 # LDAP username and password of the hue user used for LDAP authentications. # Set it to use LDAP Authentication with HiveServer2 and Impala. ## ldap_username=hue ## ldap_password= # Default encoding for site data ## default_site_encoding=utf-8 # Help improve Hue with anonymous usage analytics. # Use Google Analytics to see how many times an application or specific section of an application is used, nothing more. ## collect_usage=true # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER. ## secure_proxy_ssl_header=false # Comma-separated list of Django middleware classes to use. # See https://docs.djangoproject.com/en/1.4/ref/middleware/ for more details on middlewares in Django. ## middleware=desktop.auth.backend.LdapSynchronizationBackend # Comma-separated list of regular expressions, which match the redirect URL. # For example, to restrict to your local domain and FQDN, the following value can be used: # ^\/.*$,^http:\/\/www.mydomain.com\/.*$ ## redirect_whitelist=^\/.*$ # Comma separated list of apps to not load at server startup. # e.g.: pig,zookeeper app_blacklist=search,zookeeper,security{{ ",impala" if not impala_host }}{{ ",hbase" if not hbase_host }}{{ ",sqoop" if not sqoop_host }}{{ ",spark" if not livy_host }}{{ ",hive" if not hive_host }} # The directory where to store the auditing logs. Auditing is disable if the value is empty. # e.g. /var/log/hue/audit.log ## audit_event_log_dir= # Size in KB/MB/GB for audit log to rollover. ## audit_log_max_file_size=100MB # A json file containing a list of log redaction rules for cleaning sensitive data # from log files. It is defined as: # # { # "version": 1, # "rules": [ # { # "description": "This is the first rule", # "trigger": "triggerstring 1", # "search": "regex 1", # "replace": "replace 1" # }, # { # "description": "This is the second rule", # "trigger": "triggerstring 2", # "search": "regex 2", # "replace": "replace 2" # } # ] # } # # Redaction works by searching a string for the [TRIGGER] string. If found, # the [REGEX] is used to replace sensitive information with the # [REDACTION_MASK]. If specified with `log_redaction_string`, the # `log_redaction_string` rules will be executed after the # `log_redaction_file` rules. # # For example, here is a file that would redact passwords and social security numbers: # { # "version": 1, # "rules": [ # { # "description": "Redact passwords", # "trigger": "password", # "search": "password=\".*\"", # "replace": "password=\"???\"" # }, # { # "description": "Redact social security numbers", # "trigger": "", # "search": "\d{3}-\d{2}-\d{4}", # "replace": "XXX-XX-XXXX" # } # ] # } ## log_redaction_file= # Comma separated list of strings representing the host/domain names that the Hue server can serve. # e.g.: localhost,domain1,* ## allowed_hosts=* # Administrators # ---------------- [[django_admins]] ## [[[admin1]]] ## name=john ## email=john@doe.com # UI customizations # ------------------- [[custom]] # Top banner HTML code # e.g.

Test Lab A2 Hue Services

## banner_top_html= # Configuration options for user authentication into the web application # ------------------------------------------------------------------------ [[auth]] # Authentication backend. Common settings are: # - django.contrib.auth.backends.ModelBackend (entirely Django backend) # - desktop.auth.backend.AllowAllBackend (allows everyone) # - desktop.auth.backend.AllowFirstUserDjangoBackend # (Default. Relies on Django and user manager, after the first login) # - desktop.auth.backend.LdapBackend # - desktop.auth.backend.PamBackend - WARNING: existing users in Hue may be unaccessible if they not exist in OS # - desktop.auth.backend.SpnegoDjangoBackend # - desktop.auth.backend.RemoteUserDjangoBackend # - libsaml.backend.SAML2Backend # - libopenid.backend.OpenIDBackend # - liboauth.backend.OAuthBackend # (Support Twitter, Facebook, Google+ and Linkedin backend=desktop.auth.backend.PamBackend # The service to use when querying PAM. pam_service=sudo sshd login # When using the desktop.auth.backend.RemoteUserDjangoBackend, this sets # the normalized name of the header that contains the remote user. # The HTTP header in the request is converted to a key by converting # all characters to uppercase, replacing any hyphens with underscores # and adding an HTTP_ prefix to the name. So, for example, if the header # is called Remote-User that would be configured as HTTP_REMOTE_USER # # Defaults to HTTP_REMOTE_USER ## remote_user_header=HTTP_REMOTE_USER # Ignore the case of usernames when searching for existing users. # Only supported in remoteUserDjangoBackend. ## ignore_username_case=true # Ignore the case of usernames when searching for existing users to authenticate with. # Only supported in remoteUserDjangoBackend. ## force_username_lowercase=true # Users will expire after they have not logged in for 'n' amount of seconds. # A negative number means that users will never expire. ## expires_after=-1 # Apply 'expires_after' to superusers. ## expire_superusers=true # Configuration options for connecting to LDAP and Active Directory # ------------------------------------------------------------------- [[ldap]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Execute this script to produce the bind user password. This will be used # when `bind_password` is not set. ## bind_password_script= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication # If nt_domain is specified, this config is completely ignored. # If nt_domain is not specified, this should take on the form "cn=,dc=example,dc=com", # where is replaced by whatever is provided at the login page. Depending on your ldap schema, # you can also specify additional/alternative comma-separated attributes like uid, ou, etc ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" # Create users in Hue when they try to login with their LDAP credentials # For use when using LdapBackend for Hue authentication ## create_users_on_login = true # Synchronize a users groups when they login ## sync_groups_on_login=false # Ignore the case of usernames when searching for existing users in Hue. ## ignore_username_case=false # Ignore the case of usernames when searching for existing users to authenticate with. # Only supported in remoteUserDjangoBackend. ## force_username_lowercase=false # Use search bind authentication. # If set to true, hue will perform ldap search using bind credentials above (bind_dn, bind_password) # Hue will then search using the 'base_dn' for an entry with attr defined in 'user_name_attr', with the value # of short name provided on the login page. The search filter defined in 'user_filter' will also be used to limit # the search. Hue will search the entire subtree starting from base_dn. # If search_bind_authentication is set to false, Hue performs a direct bind to LDAP using the credentials provided # (not bind_dn and bind_password specified in hue.ini). There are 2 modes here - 'nt_domain' is specified or not. ## search_bind_authentication=true # Choose which kind of subgrouping to use: nested or suboordinate (deprecated). ## subgroups=suboordinate # Define the number of levels to search for nested members. ## nested_members_search_depth=10 # Whether or not to follow referrals ## follow_referrals=false # Enable python-ldap debugging. ## debug=false # Sets the debug level within the underlying LDAP C lib. ## debug_level=255 # Possible values for trace_level are 0 for no logging, 1 for only logging the method calls with arguments, # 2 for logging the method calls with arguments and the complete results and 9 for also logging the traceback of method calls. ## trace_level=0 [[[users]]] # Base filter for searching for users ## user_filter="objectclass=*" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName [[[groups]]] # Base filter for searching for groups ## group_filter="objectclass=*" # The group name attribute in the LDAP schema ## group_name_attr=cn # The attribute of the group object which identifies the members of the group ## group_member_attr=members [[[ldap_servers]]] ## [[[[mycompany]]]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Execute this script to produce the bind user password. This will be used # when `bind_password` is not set. ## bind_password_script= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" ## Use search bind authentication. ## search_bind_authentication=true # Whether or not to follow referrals ## follow_referrals=false # Enable python-ldap debugging. ## debug=false # Sets the debug level within the underlying LDAP C lib. ## debug_level=255 # Possible values for trace_level are 0 for no logging, 1 for only logging the method calls with arguments, # 2 for logging the method calls with arguments and the complete results and 9 for also logging the traceback of method calls. ## trace_level=0 ## [[[[[users]]]]] # Base filter for searching for users ## user_filter="objectclass=Person" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName ## [[[[[groups]]]]] # Base filter for searching for groups ## group_filter="objectclass=groupOfNames" # The username attribute in the LDAP schema ## group_name_attr=cn # Configuration options for specifying the Desktop Database. For more info, # see http://docs.djangoproject.com/en/1.4/ref/settings/#database-engine # ------------------------------------------------------------------------ [[database]] # Database engine is typically one of: # postgresql_psycopg2, mysql, sqlite3 or oracle. # # Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name. # Note for Oracle, options={'threaded':true} must be set in order to avoid crashes. # Note for Oracle, you can use the Oracle Service Name by setting "port=0" and then "name=:/". engine=mysql host={{ db_host }} port=3306 user={{ hue_user }} password={{ hue_password }} name={{ hue_name }} ## options={} # Configuration options for specifying the Desktop session. # For more info, see https://docs.djangoproject.com/en/1.4/topics/http/sessions/ # ------------------------------------------------------------------------ [[session]] # The cookie containing the users' session ID will expire after this amount of time in seconds. # Default is 2 weeks. ## ttl=1209600 # The cookie containing the users' session ID will be secure. # Should only be enabled with HTTPS. ## secure=false # The cookie containing the users' session ID will use the HTTP only flag. ## http_only=false # Use session-length cookies. Logs out the user when she closes the browser window. ## expire_at_browser_close=false # Configuration options for connecting to an external SMTP server # ------------------------------------------------------------------------ [[smtp]] # The SMTP server information for email notification delivery host=localhost port=25 user= password= # Whether to use a TLS (secure) connection when talking to the SMTP server tls=no # Default email address to use for various automated notification from Hue ## default_from_email=hue@localhost # Configuration options for Kerberos integration for secured Hadoop clusters # ------------------------------------------------------------------------ [[kerberos]] # Path to Hue's Kerberos keytab file ## hue_keytab= # Kerberos principal name for Hue ## hue_principal=hue/hostname.foo.com # Path to kinit ## kinit_path=/path/to/kinit # Configuration options for using OAuthBackend (core) login # ------------------------------------------------------------------------ [[oauth]] # The Consumer key of the application ## consumer_key=XXXXXXXXXXXXXXXXXXXXX # The Consumer secret of the application ## consumer_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # The Request token URL ## request_token_url=https://api.twitter.com/oauth/request_token # The Access token URL ## access_token_url=https://api.twitter.com/oauth/access_token # The Authorize URL ## authenticate_url=https://api.twitter.com/oauth/authorize ########################################################################### # Settings to configure SAML ########################################################################### [libsaml] # Xmlsec1 binary path. This program should be executable by the user running Hue. ## xmlsec_binary=/usr/local/bin/xmlsec1 # Entity ID for Hue acting as service provider. # Can also accept a pattern where '' will be replaced with server URL base. ## entity_id="/saml2/metadata/" # Create users from SSO on login. ## create_users_on_login=true # Required attributes to ask for from IdP. # This requires a comma separated list. ## required_attributes=uid # Optional attributes to ask for from IdP. # This requires a comma separated list. ## optional_attributes= # IdP metadata in the form of a file. This is generally an XML file containing metadata that the Identity Provider generates. ## metadata_file= # Private key to encrypt metadata with. ## key_file= # Signed certificate to send along with encrypted metadata. ## cert_file= # A mapping from attributes in the response from the IdP to django user attributes. ## user_attribute_mapping={'uid':'username'} # Have Hue initiated authn requests be signed and provide a certificate. ## authn_requests_signed=false # Have Hue initiated logout requests be signed and provide a certificate. ## logout_requests_signed=false ## Username can be sourced from 'attributes' or 'nameid'. ## username_source=attributes # Performs the logout or not. ## logout_enabled=true ########################################################################### # Settings to configure OPENID ########################################################################### [libopenid] # (Required) OpenId SSO endpoint url. ## server_endpoint_url=https://www.google.com/accounts/o8/id # OpenId 1.1 identity url prefix to be used instead of SSO endpoint url # This is only supported if you are using an OpenId 1.1 endpoint ## identity_url_prefix=https://app.onelogin.com/openid/your_company.com/ # Create users from OPENID on login. ## create_users_on_login=true # Use email for username ## use_email_for_username=true ########################################################################### # Settings to configure OAuth ########################################################################### [liboauth] # NOTE: # To work, each of the active (i.e. uncommented) service must have # applications created on the social network. # Then the "consumer key" and "consumer secret" must be provided here. # # The addresses where to do so are: # Twitter: https://dev.twitter.com/apps # Google+ : https://cloud.google.com/ # Facebook: https://developers.facebook.com/apps # Linkedin: https://www.linkedin.com/secure/developer # # Additionnaly, the following must be set in the application settings: # Twitter: Callback URL (aka Redirect URL) must be set to http://YOUR_HUE_IP_OR_DOMAIN_NAME/oauth/social_login/oauth_authenticated # Google+ : CONSENT SCREEN must have email address # Facebook: Sandbox Mode must be DISABLED # Linkedin: "In OAuth User Agreement", r_emailaddress is REQUIRED # The Consumer key of the application ## consumer_key_twitter= ## consumer_key_google= ## consumer_key_facebook= ## consumer_key_linkedin= # The Consumer secret of the application ## consumer_secret_twitter= ## consumer_secret_google= ## consumer_secret_facebook= ## consumer_secret_linkedin= # The Request token URL ## request_token_url_twitter=https://api.twitter.com/oauth/request_token ## request_token_url_google=https://accounts.google.com/o/oauth2/auth ## request_token_url_linkedin=https://www.linkedin.com/uas/oauth2/authorization ## request_token_url_facebook=https://graph.facebook.com/oauth/authorize # The Access token URL ## access_token_url_twitter=https://api.twitter.com/oauth/access_token ## access_token_url_google=https://accounts.google.com/o/oauth2/token ## access_token_url_facebook=https://graph.facebook.com/oauth/access_token ## access_token_url_linkedin=https://api.linkedin.com/uas/oauth2/accessToken # The Authenticate URL ## authenticate_url_twitter=https://api.twitter.com/oauth/authorize ## authenticate_url_google=https://www.googleapis.com/oauth2/v1/userinfo?access_token= ## authenticate_url_facebook=https://graph.facebook.com/me?access_token= ## authenticate_url_linkedin=https://api.linkedin.com/v1/people/~:(email-address)?format=json&oauth2_access_token= # Username Map. Json Hash format. # Replaces username parts in order to simplify usernames obtained # Example: {"@sub1.domain.com":"_S1", "@sub2.domain.com":"_S2"} # converts 'email@sub1.domain.com' to 'email_S1' ## username_map={} # Whitelisted domains (only applies to Google OAuth). CSV format. ## whitelisted_domains_google= ########################################################################### # Settings for the RDBMS application ########################################################################### [librdbms] # The RDBMS app can have any number of databases configured in the databases # section. A database is known by its section name # (IE sqlite, mysql, psql, and oracle in the list below). [[databases]] # sqlite configuration. ## [[[sqlite]]] # Name to show in the UI. ## nice_name=SQLite # For SQLite, name defines the path to the database. ## name=/opt/mapr/hue/hue-3.8.1/desktop/desktop.db # Database backend to use. ## engine=sqlite # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} # mysql, oracle, or postgresql configuration. {%- if db_host and rdbms_name %} [[[mysql]]] # Name to show in the UI. nice_name="MySQL DB" # For MySQL and PostgreSQL, name is the name of the database. # For Oracle, Name is instance of the Oracle server. For express edition # this is 'xe' by default. name={{ rdbms_name }} # Database backend to use. This can be: # 1. mysql # 2. postgresql # 3. oracle engine=mysql # IP or hostname of the database to connect to. host={{ db_host }} # Port the database server is listening to. Defaults are: # 1. MySQL: 3306 # 2. PostgreSQL: 5432 # 3. Oracle Express Edition: 1521 port=3306 # Username to authenticate with when connecting to the database. user={{ rdbms_user }} # Password matching the username to authenticate with when # connecting to the database. password={{ rdbms_password }} # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} {%- endif %} ########################################################################### # Settings to configure your Hadoop cluster. ########################################################################### [hadoop] # Configuration for HDFS NameNode # ------------------------------------------------------------------------ [[hdfs_clusters]] # HA support by using HttpFs [[[default]]] # Enter the filesystem uri fs_defaultfs=maprfs:/// # NameNode logical name. ## logical_name= # Use WebHdfs/HttpFs as the communication mechanism. # Domain should be the NameNode or HttpFs host. # Default port is 14000 for HttpFs. webhdfs_url=http://{{ webhdfs_url | default("localhost", True) }}:14000/webhdfs/v1 # Change this if your HDFS cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs # have to be verified against certificate authority ## ssl_cert_ca_verify=True # File size restriction for viewing file (float) # '1.0' - default 1 GB file size restriction # '0' - no file size restrictions # >0 - set file size restriction in gigabytes, ex. 0.5, 1.0, 1.2... ## file_size=1.0 # Directory of the Hadoop configuration ## hadoop_conf_dir=$HADOOP_CONF_DIR when set or '/etc/hadoop/conf' {% if yarn_mode %} # Configuration for YARN (MR2) # ------------------------------------------------------------------------ [[yarn_clusters]] [[[default]]] # Enter the host on which you are running the ResourceManager resourcemanager_host={{ rm_host | default("localhost", True) }} # The port where the ResourceManager IPC listens on resourcemanager_port={{ rm_port | default("8032", True) }} # Whether to submit jobs to this cluster submit_to={{ "True" if yarn_mode else "False" }} # Change this if your YARN cluster is secured ## security_enabled=false # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY ## mechanism=MAPR-SECURITY # URL of the ResourceManager API resourcemanager_api_url=http://{{ rm_host | default("localhost", True) }}:{{ rm_api_port | default("8088", True)}} # URL of the ProxyServer API proxy_api_url=http://{{ rm_proxy_api | default(rm_host, True) }}:{{ rm_proxy_port | default("8088", True)}} # URL of the HistoryServer API history_server_api_url=http://{{ hs_host | default(rm_host, True) }}:{{ hs_api_port | default("19888", True)}} # In secure mode (HTTPS), if SSL certificates from from YARN Rest APIs # have to be verified against certificate authority ## ssl_cert_ca_verify=True # HA support by specifying multiple clusters # e.g. {% for rm in ha_rms %} [[[ha{{ loop.index }}]]] # Enter the host on which you are running the failover Resource Manager resourcemanager_api_url=http://{{ rm }}:8088 logical_name=ha-rm-{{ loop.index }} submit_to=True {% endfor %} {%- endif %} {%- if not yarn_mode %} # Configuration for MapReduce (MR1) # ------------------------------------------------------------------------ [[mapred_clusters]] [[[default]]] # Enter the host on which you are running the Hadoop JobTracker jobtracker_host={{ jt_host | default("localhost", True) }} # The port where the JobTracker IPC listens on jobtracker_port={{ jt_port | default("9001", True) }} # JobTracker logical name for HA ## logical_name= # Thrift plug-in port for the JobTracker thrift_port={{ jt_plugin_port | default("9290", True) }} # Whether to submit jobs to this cluster submit_to={{ "False" if yarn_mode else "True" }} # Change this if your MapReduce cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- if fo_jobtracker_host %} # HA support by specifying multiple clusters # e.g. # [[[ha]]] # Enter the logical name of the JobTrackers ## logical_name=my-jt-name {%- endif %} {%- endif %} ########################################################################### # Settings to configure the Filebrowser app ########################################################################### [filebrowser] # Location on local filesystem where the uploaded archives are temporary stored. ## archive_upload_tempdir=/tmp ########################################################################### # Settings to configure liboozie ########################################################################### [liboozie] # The URL where the Oozie service runs on. This is required in order for # users to submit jobs. oozie_url=http://{{ oozie_host | default("localhost", True) }}:{{ oozie_port | default("11000", True) }}/oozie # Requires FQDN in oozie_url if enabled ## security_enabled=false # Location on HDFS where the workflows/coordinator are deployed when submitted. ## remote_deployement_dir=/oozie/deployments # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} ########################################################################### # Settings to configure the Oozie app ########################################################################### [oozie] # Location on local FS where the examples are stored. ## local_data_dir=..../examples # Location on local FS where the data for the examples is stored. ## sample_data_dir=...thirdparty/sample_data # Location on HDFS where the oozie examples and workflows are stored. remote_data_dir=/oozie/workspaces # Maximum of Oozie workflows or coodinators to retrieve in one API call. ## oozie_jobs_count=100 # Use Cron format for defining the frequency of a Coordinator instead of the old frequency number/unit. ## enable_cron_scheduling=true ########################################################################### # Settings to configure Beeswax with Hive ########################################################################### {%- if hive_host %} [beeswax] # Host where HiveServer2 is running. # If Kerberos security is enabled, use fully-qualified domain name (FQDN). hive_server_host={{ hive_host | default("localhost", True) }} # Port where HiveServer2 Thrift server runs on. hive_server_port={{ hive_port | default("10000", True) }} # Hive configuration directory, where hive-site.xml is located hive_conf_dir={{ hive_conf_dir | default("/opt/mapr/hive/hive-1.0/conf", True) }} # Timeout in seconds for thrift calls to Hive service ## server_conn_timeout=120 # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} # Path to HiveServer2 start script hive_server_bin=/opt/mapr/hive/hive-{{ hive_version | default("1.0", True) }}/bin/hiveserver2 # Choose whether to use the old GetLog() thrift call from before Hive 0.14 to retrieve the logs. # If false, use the FetchResults() thrift call from Hive 1.0 or more instead. ## use_get_log_api=false # Set a LIMIT clause when browsing a partitioned table. # A positive value will be set as the LIMIT. If 0 or negative, do not set any limit. ## browse_partitioned_table_limit=250 # A limit to the number of rows that can be downloaded from a query. # A value of -1 means there will be no limit. # A maximum of 65,000 is applied to XLS downloads. ## download_row_limit=1000000 # Hue will try to close the Hive query when the user leaves the editor page. # This will free all the query resources in HiveServer2, but also make its results inaccessible. ## close_queries=false # Thrift version to use when communicating with HiveServer2. # New column format is from version 7. thrift_version={{ thrift_version }} [[ssl]] # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Choose whether Hue should validate certificates received from the server. ## validate=true {%- endif %} ########################################################################### # Settings to configure Impala ########################################################################### {%- if impala_host %} [impala] # Host of the Impala Server (one of the Impalad) server_host={{ impala_host | default("localhost", True) }} # Port of the Impala Server server_port={{ impala_port | default("21050", True) }} # Kerberos principal ## impala_principal=impala/hostname.foo.com # Turn on/off impersonation mechanism when talking to Impala impersonation_enabled=False # Number of initial rows of a result set to ask Impala to cache in order # to support re-fetching them for downloading them. # Set to 0 for disabling the option and backward compatibility. querycache_rows=0 # Timeout in seconds for thrift calls ## server_conn_timeout=120 # Hue will try to close the Impala query when the user leaves the editor page. # This will free all the query resources in Impala, but also make its results inaccessible. ## close_queries=true # If QUERY_TIMEOUT_S > 0, the query will be timed out (i.e. cancelled) if Impala does not do any work # (compute or send back results) for that query within QUERY_TIMEOUT_S seconds. ## query_timeout_s=0 [[ssl]] # SSL communication enabled for this server. ## enabled=false # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Choose whether Hue should validate certificates received from the server. ## validate=true {%- endif %} ########################################################################### # Settings to configure Pig ########################################################################### [pig] # Location of piggybank.jar on local filesystem. ## local_sample_dir=/opt/mapr/pig/pig-0.12/contrib/piggybank/java/ # Location piggybank.jar will be copied to in HDFS. ## remote_data_dir=/oozie/pig/examples ########################################################################### # Settings to configure Sqoop ########################################################################### {%- if sqoop_host %} [sqoop] # For autocompletion, fill out the librdbms section. # Sqoop server URL server_url=http://{{ sqoop_host | default("localhost", True) }}:{{ sqoop_port | default("12000", True) }}/sqoop # Change this if your cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- endif %} ########################################################################### # Settings to configure Proxy ########################################################################### [proxy] # Comma-separated list of regular expressions, # which match 'host:port' of requested proxy target. ## whitelist=(localhost|127\.0\.0\.1):(50030|50070|50060|50075) # Comma-separated list of regular expressions, # which match any prefix of 'host:port/path' of requested proxy target. # This does not support matching GET parameters. ## blacklist= ########################################################################### # Settings to configure HBase Browser ########################################################################### {%- if hbase_host %} [hbase] # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'. # Use full hostname with security. hbase_clusters=(HBase|{{ hbase_host | default("localhost", True) }}:{{ hbase_port | default("9090", True) }}) # HBase configuration directory, where hbase-site.xml is located. hbase_conf_dir=/opt/mapr/hbase/hbase-{{ hbase_version | default("0.98.12", True) }}/conf # Hard limit of rows or columns per row fetched before truncating. ## truncate_limit = 500 # 'buffered' is the default of the HBase Thrift Server and supports security. # 'framed' can be used to chunk up responses, # which is useful when used in conjunction with the nonblocking server in Thrift. ## thrift_transport=buffered # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- endif %} ########################################################################### # Settings to configure Solr Search ########################################################################### [search] # URL of the Solr Server ## solr_url=http://localhost:8983/solr/ # Requires FQDN in solr_url if enabled security_enabled=${security_enabled} ## Query sent when no term is entered ## empty_query=*:* ########################################################################### # Settings to configure Solr Indexer ########################################################################### [indexer] # Location of the solrctl binary. ## solrctl_path=/usr/bin/solrctl # Zookeeper ensemble. ## solr_zk_ensemble=localhost:2181/solr ########################################################################### # Settings to configure Job Designer ########################################################################### [jobsub] # Location on local FS where examples and template are stored. ## local_data_dir=..../data # Location on local FS where sample data is stored ## sample_data_dir=...thirdparty/sample_data ########################################################################### # Settings to configure Job Browser. ########################################################################### [jobbrowser] # Share submitted jobs information with all users. If set to false, # submitted jobs are visible only to the owner and administrators. ## share_jobs=true ########################################################################### # Settings to configure the Zookeeper application. ########################################################################### [zookeeper] [[clusters]] [[[default]]] # Zookeeper ensemble. Comma separated list of Host/Port. # e.g. localhost:5181,node2_ip@:5181,node3_ip@:5181 host_ports={{ zk_hosts_with_port | default("localhost:5181", True) }} # The URL of the REST contrib service (required for znode browsing). ## rest_url=http://localhost:9999 # Name of Kerberos principal when using security. ## principal_name=zookeeper ########################################################################### # Settings to configure the Spark application. ########################################################################### {%- if livy_host %} [spark] # Host address of the Livy Server. livy_server_host={{ livy_host }} # Port of the Livy Server. ## livy_server_port=8998 # Configure livy to start with 'process', 'thread', or 'yarn' workers. livy_server_session_kind=yarn # If livy should use proxy users when submitting a job. ## livy_impersonation_enabled=true # List of available types of snippets ## languages='[{"name": "Scala Shell", "type": "scala"},{"name": "PySpark Shell", "type": "python"},{"name": "R Shell", "type": "r"},{"name": "Jar", "type": "Jar"},{"name": "Python", "type": "py"},{"name": "Impala SQL", "type": "impala"},{"name": "Hive SQL", "type": "hive"},{"name": "Text", "type": "text"}]' {%- endif %} ########################################################################### # Settings for the User Admin application ########################################################################### [useradmin] # The name of the default user group that users will be a member of ## default_user_group=default [[password_policy]] # Set password policy to all users. The default policy requires password to be at least 8 characters long, # and contain both uppercase and lowercase letters, numbers, and special characters. ## is_enabled=false ## pwd_regex="^(?=.*?[A-Z])(?=(.*[a-z]){1,})(?=(.*[\d]){1,})(?=(.*[\W_]){1,}).{8,}$" ## pwd_hint="The password must be at least 8 characters long, and must contain both uppercase and lowercase letters, at least one number, and at least one special character." ## pwd_error_message="The password must be at least 8 characters long, and must contain both uppercase and lowercase letters, at least one number, and at least one special character." ########################################################################### # Settings for the Sentry lib ########################################################################### [libsentry] # Hostname or IP of server. ## hostname=localhost # Port the sentry service is running on. ## port=8038 # Sentry configuration directory, where sentry-site.xml is located. ## sentry_conf_dir=/opt/mapr/sentry/sentry-1.4.0/conf sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_sh_3.7.0.template0000666000175100017510000000502713245514472027051 0ustar zuulzuul00000000000000#!/bin/bash bin=`dirname "$0"` bin=`cd "$bin"; pwd` export HUE_HOME=${bin}/.. export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-{{ hadoop_version }}/etc/hadoop # # activate the Python virtual environment # source $HUE_HOME/build/env/bin/activate # Look for installed JDK if [ -z "$JAVA_HOME" ]; then sys_java="/usr/bin/java" if [ -e $sys_java ]; then jcmd=`readlink -f $sys_java` if [ -x ${jcmd%/jre/bin/java}/bin/javac ]; then JAVA_HOME=${jcmd%/jre/bin/java} elif [ -x ${jcmd%/java}/javac ]; then JAVA_HOME=${jcmd%/bin/java} fi [ -n "${JAVA_HOME}" ] && export JAVA_HOME fi fi export LD_LIBRARY_PATH=$HUE_HOME/build/env/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/mapr/lib:$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH # get arguments command=$1 shift startStop=$1 shift #file client impersonation MAPR_IMPERSONATION_ENABLED=" " export MAPR_IMPERSONATION_ENABLED HUE_PID_DIR=$HUE_HOME/pids if [ "$HUE_IDENT_STRING" = "" ]; then export HUE_IDENT_STRING=`id -nu` fi # get log directory if [ "$HUE_LOG_DIR" = "" ]; then export HUE_LOG_DIR="$HUE_HOME/logs" fi mkdir -p "$HUE_LOG_DIR" if [ "$HUE_PID_DIR" = "" ]; then export HUE_PID_DIR="$HUE_HOME/pids" fi mkdir -p "$HUE_PID_DIR" log=$HUE_LOG_DIR/hue-$HUE_IDENT_STRING-$command-$HOSTNAME.out pid=$HUE_PID_DIR/hue-$HUE_IDENT_STRING-$command.pid secure_sh_log=$HUE_LOG_DIR/secure-sh-log.out case $startStop in (start) if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo Hue web server running as process `cat $pid`. Stop it first. exit 1 fi fi cd $HUE_HOME # Generate secure keys nohup $HUE_HOME/bin/secure.sh >> "$secure_sh_log" 2>&1 < /dev/null & nohup $HUE_HOME/build/env/bin/hue $command >> "$log" 2>&1 < /dev/null & echo $! > $pid echo "`date` $command started, pid `cat $pid`" >> "$log" 2>&1 < /dev/null ;; (stop) if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo stopping $command kill `cat $pid` echo "`date` $command stopped, pid `cat $pid`" >> "$log" 2>&1 < /dev/null else echo no $command to stop fi else echo no $command to stop fi ;; (status) if [ -f $pid ]; then if kill -0 `cat $pid`; then echo $command running as process `cat $pid`. exit 0 fi echo $pid exists with pid `cat $pid` but no $command. exit 1 fi echo $command not running. exit 1 ;; (*) echo $usage exit 1 ;; esac # # deactivate the Python virtual environment # deactivatesahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_sh_3.9.0.template0000666000175100017510000000561513245514472027056 0ustar zuulzuul00000000000000#!/bin/bash bin=`dirname "$0"` bin=`cd "$bin"; pwd` export HUE_HOME=${bin}/.. export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-{{ hadoop_version }}/etc/hadoop # # activate the Python virtual environment # source $HUE_HOME/build/env/bin/activate # Look for installed JDK if [ -z "$JAVA_HOME" ]; then sys_java="/usr/bin/java" if [ -e $sys_java ]; then jcmd=`readlink -f $sys_java` if [ -x ${jcmd%/jre/bin/java}/bin/javac ]; then JAVA_HOME=${jcmd%/jre/bin/java} elif [ -x ${jcmd%/java}/javac ]; then JAVA_HOME=${jcmd%/bin/java} fi [ -n "${JAVA_HOME}" ] && export JAVA_HOME fi fi export LD_LIBRARY_PATH=$HUE_HOME/build/env/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/mapr/lib:$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH # get arguments command=$1 shift startStop=$1 shift #get spark_home if [ $command == "livy_server" ]; then if [ -z "$SPARK_HOME" ]; then if [ -d /opt/mapr/spark/spark-* ]; then cd /opt/mapr/spark/spark-* SPARK_HOME=`pwd` export SPARK_HOME=`pwd` else echo "Cannot export spark home!" exit 0 fi fi export PATH="$PATH:$SPARK_HOME/bin" fi #file client impersonation MAPR_IMPERSONATION_ENABLED=" " export MAPR_IMPERSONATION_ENABLED HUE_PID_DIR=$HUE_HOME/pids if [ "$HUE_IDENT_STRING" = "" ]; then export HUE_IDENT_STRING=`id -nu` fi # get log directory if [ "$HUE_LOG_DIR" = "" ]; then export HUE_LOG_DIR="$HUE_HOME/logs" fi mkdir -p "$HUE_LOG_DIR" if [ "$HUE_PID_DIR" = "" ]; then export HUE_PID_DIR="$HUE_HOME/pids" fi mkdir -p "$HUE_PID_DIR" log=$HUE_LOG_DIR/hue-$HUE_IDENT_STRING-$command-$HOSTNAME.out pid=$HUE_PID_DIR/hue-$HUE_IDENT_STRING-$command.pid secure_sh_log=$HUE_LOG_DIR/secure-sh-log.out case $startStop in (start) if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo Hue web server running as process `cat $pid`. Stop it first. exit 1 fi fi cd $HUE_HOME # Generate secure keys nohup $HUE_HOME/bin/secure.sh >> "$secure_sh_log" 2>&1 < /dev/null & nohup $HUE_HOME/build/env/bin/hue $command >> "$log" 2>&1 < /dev/null & echo $! > $pid echo "`date` $command started, pid `cat $pid`" >> "$log" 2>&1 < /dev/null ;; (stop) if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo stopping $command kill `cat $pid` echo "`date` $command stopped, pid `cat $pid`" >> "$log" 2>&1 < /dev/null else echo no $command to stop fi else echo no $command to stop fi ;; (status) if [ -f $pid ]; then if kill -0 `cat $pid`; then echo $command running as process `cat $pid`. exit 0 fi echo $pid exists with pid `cat $pid` but no $command. exit 1 fi echo $command not running. exit 1 ;; (*) echo $usage exit 1 ;; esac # # deactivate the Python virtual environment # deactivate sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_3.10.0.template0000666000175100017510000014506313245514472026436 0ustar zuulzuul00000000000000# Hue configuration file # =================================== # # For complete documentation about the contents of this file, run # $ /build/env/bin/hue config_help # # All .ini files under the current directory are treated equally. Their # contents are merged to form the Hue configuration, which can # can be viewed on the Hue at # http://:/dump_config ########################################################################### # General configuration for core Desktop features (authentication, etc) ########################################################################### [desktop] # Set this to a random string, the longer the better. # This is used for secure hashing in the session store. secret_key={{ secret_key }} # Execute this script to produce the Django secret key. This will be used when # `secret_key` is not set. ## secret_key_script= # Webserver listens on this address and port http_host=0.0.0.0 http_port=8888 # Time zone name time_zone=America/Los_Angeles # Enable or disable Django debug mode. django_debug_mode=true # Enable or disable database debug mode. ## database_logging=false # Whether to send debug messages from JavaScript to the server logs. ## send_dbug_messages=false # Enable or disable backtrace for server error http_500_debug_mode=true # Enable or disable memory profiling. ## memory_profiler=false # Server email for internal error messages ## django_server_email='hue@localhost.localdomain' # Email backend ## django_email_backend=django.core.mail.backends.smtp.EmailBackend # Webserver runs as this user server_user=mapr server_group=mapr # This should be the Hue admin and proxy user default_user=mapr # This should be the hadoop cluster admin default_hdfs_superuser=mapr default_jobtracker_host=maprfs:/// # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # Number of threads used by the CherryPy web server ## cherrypy_server_threads=40 # Filename of SSL Certificate ## ssl_certificate= # Filename of SSL RSA Private Key ## ssl_private_key= # Filename of SSL Certificate Chain ## ssl_certificate_chain= # SSL certificate password ## ssl_password= # Execute this script to produce the SSL password. This will be used when `ssl_password` is not set. ## ssl_password_script= # List of allowed and disallowed ciphers in cipher list format. # See http://www.openssl.org/docs/apps/ciphers.html for more information on # cipher list format. This list is from # https://wiki.mozilla.org/Security/Server_Side_TLS v3.7 intermediate # recommendation, which should be compatible with Firefox 1, Chrome 1, IE 7, # Opera 5 and Safari 1. ## ssl_cipher_list=ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA # Path to default Certificate Authority certificates. ## ssl_cacerts=/opt/mapr/hue/hue-3.10.0/cert.pem # Choose whether Hue should validate certificates received from the server. ## validate=true # Default LDAP/PAM/.. username and password of the hue user used for authentications with other services. # Inactive if password is empty. # e.g. LDAP pass-through authentication for HiveServer2 or Impala. Apps can override them individually. ## auth_username=hue ## auth_password= # Default encoding for site data ## default_site_encoding=utf-8 # Help improve Hue with anonymous usage analytics. # Use Google Analytics to see how many times an application or specific section of an application is used, nothing more. ## collect_usage=true # Tile layer server URL for the Leaflet map charts # Read more on http://leafletjs.com/reference.html#tilelayer ## leaflet_tile_layer=http://{s}.tile.osm.org/{z}/{x}/{y}.png # The copyright message for the specified Leaflet maps Tile Layer ## leaflet_tile_layer_attribution='© OpenStreetMap contributors' # X-Frame-Options HTTP header value. Use 'DENY' to deny framing completely ## http_x_frame_options=SAMEORIGIN # Enable X-Forwarded-Host header if the load balancer requires it. ## use_x_forwarded_host=false # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER. ## secure_proxy_ssl_header=false # Comma-separated list of Django middleware classes to use. # See https://docs.djangoproject.com/en/1.4/ref/middleware/ for more details on middlewares in Django. ## middleware=desktop.auth.backend.LdapSynchronizationBackend # Comma-separated list of regular expressions, which match the redirect URL. # For example, to restrict to your local domain and FQDN, the following value can be used: # ^\/.*$,^http:\/\/www.mydomain.com\/.*$ ## redirect_whitelist=^(\/[a-zA-Z0-9]+.*|\/)$ # Comma separated list of apps to not load at server startup. # e.g.: pig,zookeeper app_blacklist=search,zookeeper {{ ",security" if not sentry_host }}{{ ",impala" if not impala_host }}{{ ",hbase" if not hbase_host }}{{ ",sqoop" if not sqoop_host }}{{ ",spark" if not livy_host }}{{ ",beeswax" if not hive_host }} # Choose whether to show the new SQL editor. ## use_new_editor=true # The directory where to store the auditing logs. Auditing is disable if the value is empty. # e.g. /var/log/hue/audit.log ## audit_event_log_dir= # Size in KB/MB/GB for audit log to rollover. ## audit_log_max_file_size=100MB # A json file containing a list of log redaction rules for cleaning sensitive data # from log files. It is defined as: # # { # "version": 1, # "rules": [ # { # "description": "This is the first rule", # "trigger": "triggerstring 1", # "search": "regex 1", # "replace": "replace 1" # }, # { # "description": "This is the second rule", # "trigger": "triggerstring 2", # "search": "regex 2", # "replace": "replace 2" # } # ] # } # # Redaction works by searching a string for the [TRIGGER] string. If found, # the [REGEX] is used to replace sensitive information with the # [REDACTION_MASK]. If specified with `log_redaction_string`, the # `log_redaction_string` rules will be executed after the # `log_redaction_file` rules. # # For example, here is a file that would redact passwords and social security numbers: # { # "version": 1, # "rules": [ # { # "description": "Redact passwords", # "trigger": "password", # "search": "password=\".*\"", # "replace": "password=\"???\"" # }, # { # "description": "Redact social security numbers", # "trigger": "", # "search": "\d{3}-\d{2}-\d{4}", # "replace": "XXX-XX-XXXX" # } # ] # } ## log_redaction_file= # Comma separated list of strings representing the host/domain names that the Hue server can serve. # e.g.: localhost,domain1,* ## allowed_hosts=* # Administrators # ---------------- [[django_admins]] ## [[[admin1]]] ## name=john ## email=john@doe.com # UI customizations # ------------------- [[custom]] # Top banner HTML code # e.g.

Test Lab A2 Hue Services

## banner_top_html= # Login splash HTML code # e.g. WARNING: You are required to have authorization before you proceed ## login_splash_html=

GetHue.com



WARNING: You have accessed a computer managed by GetHue. You are required to have authorization from GetHue before you proceed. # Cache timeout in milliseconds for the assist, autocomplete, etc. # defaults to 86400000 (1 day), set to 0 to disable caching ## cacheable_ttl=86400000 # Configuration options for user authentication into the web application # ------------------------------------------------------------------------ [[auth]] # Authentication backend. Common settings are: # - django.contrib.auth.backends.ModelBackend (entirely Django backend) # - desktop.auth.backend.AllowAllBackend (allows everyone) # - desktop.auth.backend.AllowFirstUserDjangoBackend # (Default. Relies on Django and user manager, after the first login) # - desktop.auth.backend.LdapBackend # - desktop.auth.backend.PamBackend - WARNING: existing users in Hue may be unaccessible if they not exist in OS # - desktop.auth.backend.SpnegoDjangoBackend # - desktop.auth.backend.RemoteUserDjangoBackend # - libsaml.backend.SAML2Backend # - libopenid.backend.OpenIDBackend # - liboauth.backend.OAuthBackend # (New oauth, support Twitter, Facebook, Google+ and Linkedin # Multiple Authentication backends are supported by specifying a comma-separated list in order of priority. # However, in order to enable OAuthBackend, it must be the ONLY backend configured. backend=desktop.auth.backend.PamBackend # Class which defines extra accessor methods for User objects. ## user_aug=desktop.auth.backend.DefaultUserAugmentor # The service to use when querying PAM. pam_service=sudo sshd login # When using the desktop.auth.backend.RemoteUserDjangoBackend, this sets # the normalized name of the header that contains the remote user. # The HTTP header in the request is converted to a key by converting # all characters to uppercase, replacing any hyphens with underscores # and adding an HTTP_ prefix to the name. So, for example, if the header # is called Remote-User that would be configured as HTTP_REMOTE_USER # # Defaults to HTTP_REMOTE_USER ## remote_user_header=HTTP_REMOTE_USER # Ignore the case of usernames when searching for existing users. # Supported in remoteUserDjangoBackend and SpnegoDjangoBackend ## ignore_username_case=true # Forcibly cast usernames to lowercase, takes precedence over force_username_uppercase # Supported in remoteUserDjangoBackend and SpnegoDjangoBackend ## force_username_lowercase=true # Forcibly cast usernames to uppercase, cannot be combined with force_username_lowercase ## force_username_uppercase=false # Users will expire after they have not logged in for 'n' amount of seconds. # A negative number means that users will never expire. ## expires_after=-1 # Apply 'expires_after' to superusers. ## expire_superusers=true # Users will automatically be logged out after 'n' seconds of inactivity. # A negative number means that idle sessions will not be timed out. idle_session_timeout=-1 # Force users to change password on first login with desktop.auth.backend.AllowFirstUserDjangoBackend ## change_default_password=false # Number of login attempts allowed before a record is created for failed logins ## login_failure_limit=3 # After number of allowed login attempts are exceeded, do we lock out this IP and optionally user agent? ## login_lock_out_at_failure=false # If set, defines period of inactivity in seconds after which failed logins will be forgotten ## login_cooloff_time=60 # If True, lock out based on IP and browser user agent ## login_lock_out_by_combination_browser_user_agent_and_ip=false # If True, lock out based on IP and user ## login_lock_out_by_combination_user_and_ip=false # Configuration options for connecting to LDAP and Active Directory # ------------------------------------------------------------------- [[ldap]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # The NT domain used for LDAP authentication ## nt_domain=mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Execute this script to produce the bind user password. This will be used # when `bind_password` is not set. ## bind_password_script= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication # If nt_domain is specified, this config is completely ignored. # If nt_domain is not specified, this should take on the form "cn=,dc=example,dc=com", # where is replaced by whatever is provided at the login page. Depending on your ldap schema, # you can also specify additional/alternative comma-separated attributes like uid, ou, etc ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" # Create users in Hue when they try to login with their LDAP credentials # For use when using LdapBackend for Hue authentication ## create_users_on_login = true # Synchronize a users groups when they login ## sync_groups_on_login=false # Ignore the case of usernames when searching for existing users in Hue. ## ignore_username_case=true # Force usernames to lowercase when creating new users from LDAP. # Takes precedence over force_username_uppercase ## force_username_lowercase=true # Force usernames to uppercase, cannot be combined with force_username_lowercase ## force_username_uppercase=false # Use search bind authentication. # If set to true, hue will perform ldap search using bind credentials above (bind_dn, bind_password) # Hue will then search using the 'base_dn' for an entry with attr defined in 'user_name_attr', with the value # of short name provided on the login page. The search filter defined in 'user_filter' will also be used to limit # the search. Hue will search the entire subtree starting from base_dn. # If search_bind_authentication is set to false, Hue performs a direct bind to LDAP using the credentials provided # (not bind_dn and bind_password specified in hue.ini). There are 2 modes here - 'nt_domain' is specified or not. ## search_bind_authentication=true # Choose which kind of subgrouping to use: nested or suboordinate (deprecated). ## subgroups=suboordinate # Define the number of levels to search for nested members. ## nested_members_search_depth=10 # Whether or not to follow referrals ## follow_referrals=false # Enable python-ldap debugging. ## debug=false # Sets the debug level within the underlying LDAP C lib. ## debug_level=255 # Possible values for trace_level are 0 for no logging, 1 for only logging the method calls with arguments, # 2 for logging the method calls with arguments and the complete results and 9 for also logging the traceback of method calls. ## trace_level=0 [[[users]]] # Base filter for searching for users ## user_filter="objectclass=*" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName [[[groups]]] # Base filter for searching for groups ## group_filter="objectclass=*" # The group name attribute in the LDAP schema ## group_name_attr=cn # The attribute of the group object which identifies the members of the group ## group_member_attr=members [[[ldap_servers]]] ## [[[[mycompany]]]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # The NT domain used for LDAP authentication ## nt_domain=mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Execute this script to produce the bind user password. This will be used # when `bind_password` is not set. ## bind_password_script= # Pattern for searching for usernames -- Use for the parameter # For use when using LdapBackend for Hue authentication ## ldap_username_pattern="uid=,ou=People,dc=mycompany,dc=com" ## Use search bind authentication. ## search_bind_authentication=true # Whether or not to follow referrals ## follow_referrals=false # Enable python-ldap debugging. ## debug=false # Sets the debug level within the underlying LDAP C lib. ## debug_level=255 # Possible values for trace_level are 0 for no logging, 1 for only logging the method calls with arguments, # 2 for logging the method calls with arguments and the complete results and 9 for also logging the traceback of method calls. ## trace_level=0 ## [[[[[users]]]]] # Base filter for searching for users ## user_filter="objectclass=Person" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName ## [[[[[groups]]]]] # Base filter for searching for groups ## group_filter="objectclass=groupOfNames" # The username attribute in the LDAP schema ## group_name_attr=cn # Configuration options for specifying the Desktop Database. For more info, # see http://docs.djangoproject.com/en/1.4/ref/settings/#database-engine # ------------------------------------------------------------------------ [[database]] # Database engine is typically one of: # postgresql_psycopg2, mysql, sqlite3 or oracle. # # Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name # Note for Oracle, options={"threaded":true} must be set in order to avoid crashes. # Note for Oracle, you can use the Oracle Service Name by setting "port=0" and then "name=:/". # Note for MariaDB use the 'mysql' engine. engine=mysql host={{ db_host }} port=3306 user={{ hue_user }} password={{ hue_password }} name={{ hue_name }} # Execute this script to produce the database password. This will be used when `password` is not set. ## password_script=/path/script ## name=desktop/desktop.db ## options={} # Configuration options for specifying the Desktop session. # For more info, see https://docs.djangoproject.com/en/1.4/topics/http/sessions/ # ------------------------------------------------------------------------ [[session]] # The cookie containing the users' session ID will expire after this amount of time in seconds. # Default is 2 weeks. ## ttl=1209600 # The cookie containing the users' session ID will be secure. # Should only be enabled with HTTPS. ## secure=false # The cookie containing the users' session ID will use the HTTP only flag. ## http_only=true # Use session-length cookies. Logs out the user when she closes the browser window. ## expire_at_browser_close=false # Configuration options for connecting to an external SMTP server # ------------------------------------------------------------------------ [[smtp]] # The SMTP server information for email notification delivery host=localhost port=25 user= password= # Whether to use a TLS (secure) connection when talking to the SMTP server tls=no # Default email address to use for various automated notification from Hue ## default_from_email=hue@localhost # Configuration options for Kerberos integration for secured Hadoop clusters # ------------------------------------------------------------------------ [[kerberos]] # Path to Hue's Kerberos keytab file ## hue_keytab= # Kerberos principal name for Hue ## hue_principal=hue/hostname.foo.com # Path to kinit ## kinit_path=/path/to/kinit # Configuration options for using OAuthBackend (Core) login # ------------------------------------------------------------------------ [[oauth]] # The Consumer key of the application ## consumer_key=XXXXXXXXXXXXXXXXXXXXX # The Consumer secret of the application ## consumer_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # The Request token URL ## request_token_url=https://api.twitter.com/oauth/request_token # The Access token URL ## access_token_url=https://api.twitter.com/oauth/access_token # The Authorize URL ## authenticate_url=https://api.twitter.com/oauth/authorize # Configuration options for Metrics # ------------------------------------------------------------------------ [[metrics]] # Enable the metrics URL "/desktop/metrics" ## enable_web_metrics=True # If specified, Hue will write metrics to this file. ## location=/var/log/hue/metrics.json # Time in milliseconds on how frequently to collect metrics ## collection_interval=30000 ########################################################################### # Settings to configure the snippets available in the Notebook ########################################################################### [notebook] ## Show the notebook menu or not # show_notebooks=true ## Base URL to Remote GitHub Server # github_remote_url=https://github.com ## Base URL to GitHub API # github_api_url=https://api.github.com ## Client ID for Authorized GitHub Application # github_client_id= ## Client Secret for Authorized GitHub Application # github_client_secret= # One entry for each type of snippet. The first 5 will appear in the wheel. [[interpreters]] # Define the name and how to connect and execute the language. [[[hive]]] # The name of the snippet. name=Hive # The backend connection to use to communicate with the server. interface=hiveserver2 [[[impala]]] name=Impala interface=hiveserver2 # [[[sparksql]]] # name=SparkSql # interface=hiveserver2 [[[spark]]] name=Scala interface=livy [[[pyspark]]] name=PySpark interface=livy [[[r]]] name=R interface=livy [[[jar]]] name=Spark Submit Jar interface=livy-batch [[[py]]] name=Spark Submit Python interface=livy-batch [[[pig]]] name=Pig interface=pig [[[text]]] name=Text interface=text [[[markdown]]] name=Markdown interface=text [[[mysql]]] name = MySQL interface=rdbms [[[sqlite]]] name = SQLite interface=rdbms [[[postgresql]]] name = PostgreSQL interface=rdbms [[[oracle]]] name = Oracle interface=rdbms [[[solr]]] name = Solr SQL interface=solr ## Name of the collection handler # options='{"collection": "default"}' #[[[mysql]]] #name={{ rdbms_name }} #interface=jdbc ## Specific options for connecting to the server. ## The JDBC connectors, e.g. mysql.jar, need to be in the CLASSPATH environment variable. ## If 'user' and 'password' are omitted, they will be prompted in the UI. #options='{"url": "jdbc:mysql://{{ db_host }}:3306/hue", "driver": "com.mysql.jdbc.Driver", "user": "{{ rdbms_user }}", "password": "{{ rdbms_password }}"}' ## Main flag to override the automatic starting of the DBProxy server. # enable_dbproxy_server=true ## Flag to enable the SQL query builder of the table assist. # enable_query_builder=true ########################################################################### # Settings to configure your Hadoop cluster. ########################################################################### [hadoop] # Configuration for HDFS NameNode # ------------------------------------------------------------------------ [[hdfs_clusters]] # HA support by using HttpFs [[[default]]] # Enter the filesystem uri fs_defaultfs=maprfs:/// # NameNode logical name. ## logical_name= # Use WebHdfs/HttpFs as the communication mechanism. # Domain should be the NameNode or HttpFs host. # Default port is 14000 for HttpFs. webhdfs_url=http://{{ webhdfs_url | default("localhost", True) }}:14000/webhdfs/v1 # Change this if your HDFS cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} # Enable mutual ssl authentication # mutual_ssl_auth=False # ssl_cert=/opt/mapr/hue/hue-3.10.0/cert.pem # ssl_key=/opt/mapr/hue/hue-3.10.0/hue_private_keystore.pem # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs # have to be verified against certificate authority ## ssl_cert_ca_verify=True # File size restriction for viewing file (float) # '1.0' - default 1 GB file size restriction # '0' - no file size restrictions # >0 - set file size restriction in gigabytes, ex. 0.5, 1.0, 1.2... ## file_size=1.0 # Directory of the Hadoop configuration ## hadoop_conf_dir=$HADOOP_CONF_DIR when set or '/etc/hadoop/conf' {% if yarn_mode %} # Configuration for YARN (MR2) # ------------------------------------------------------------------------ [[yarn_clusters]] [[[default]]] # Enter the host on which you are running the ResourceManager resourcemanager_host={{ rm_host | default("localhost", True) }} # The port where the ResourceManager IPC listens on resourcemanager_port={{ rm_port | default("8032", True) }} # Whether to submit jobs to this cluster submit_to={{ "True" if yarn_mode else "False" }} # Change this if your YARN cluster is secured ## security_enabled=false # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY ## mechanism=MAPR-SECURITY # URL of the ResourceManager API resourcemanager_api_url=http://{{ rm_host | default("localhost", True) }}:{{ rm_api_port | default("8088", True)}} # URL of the ProxyServer API proxy_api_url=http://{{ rm_proxy_api | default(rm_host, True) }}:{{ rm_proxy_port | default("8088", True)}} # URL of the HistoryServer API history_server_api_url=http://{{ hs_host | default(rm_host, True) }}:{{ hs_api_port | default("19888", True)}} # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs # have to be verified against certificate authority ## ssl_cert_ca_verify=True # HA support by specifying multiple clusters # e.g. {% for rm in ha_rms %} [[[ha{{ loop.index }}]]] # Enter the host on which you are running the failover Resource Manager resourcemanager_api_url=http://{{ rm }}:8088 logical_name=ha-rm-{{ loop.index }} submit_to=True {% endfor %} {%- endif %} {%- if not yarn_mode %} # Configuration for MapReduce (MR1) # ------------------------------------------------------------------------ [[mapred_clusters]] [[[default]]] # Enter the host on which you are running the Hadoop JobTracker jobtracker_host={{ jt_host | default("localhost", True) }} # The port where the JobTracker IPC listens on jobtracker_port={{ jt_port | default("9001", True) }} # JobTracker logical name for HA ## logical_name= # Thrift plug-in port for the JobTracker thrift_port={{ jt_plugin_port | default("9290", True) }} # Whether to submit jobs to this cluster submit_to={{ "False" if yarn_mode else "True" }} # Change this if your MapReduce cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- if fo_jobtracker_host %} # HA support by specifying multiple clusters # e.g. # [[[ha]]] # Enter the logical name of the JobTrackers ## logical_name=my-jt-name {%- endif %} {%- endif %} ########################################################################### # Settings to configure the Filebrowser app ########################################################################### [filebrowser] # Location on local filesystem where the uploaded archives are temporary stored. ## archive_upload_tempdir=/tmp ########################################################################### # Settings to configure liboozie ########################################################################### [liboozie] # The URL where the Oozie service runs on. This is required in order for # users to submit jobs. oozie_url=http://{{ oozie_host | default("localhost", True) }}:{{ oozie_port | default("11000", True) }}/oozie # Requires FQDN in oozie_url if enabled ## security_enabled=false # Location on HDFS where the workflows/coordinator are deployed when submitted. ## remote_deployement_dir=/oozie/deployments # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} ########################################################################### # Settings to configure the Oozie app ########################################################################### [oozie] # Location on local FS where the examples are stored. ## local_data_dir=..../examples # Location on local FS where the data for the examples is stored. ## sample_data_dir=...thirdparty/sample_data # Location on HDFS where the oozie examples and workflows are stored. remote_data_dir=/oozie/workspaces # Maximum of Oozie workflows or coodinators to retrieve in one API call. ## oozie_jobs_count=50 # Use Cron format for defining the frequency of a Coordinator instead of the old frequency number/unit. ## enable_cron_scheduling=true ########################################################################### # Settings to configure Beeswax with Hive ########################################################################### {%- if hive_host %} [beeswax] # Host where HiveServer2 is running. # If Kerberos security is enabled, use fully-qualified domain name (FQDN). hive_server_host={{ hive_host | default("localhost", True) }} # Port where HiveServer2 Thrift server runs on. hive_server_port={{ hive_port | default("10000", True) }} # Hive configuration directory, where hive-site.xml is located hive_conf_dir={{ hive_conf_dir | default("/opt/mapr/hive/hive-1.0/conf", True) }} # Timeout in seconds for thrift calls to Hive service ## server_conn_timeout=120 # Change this if your Hive is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} # Path to HiveServer2 start script hive_server_bin=/opt/mapr/hive/hive-{{ hive_version | default("1.0", True) }}/bin/hiveserver2 # Choose whether to use the old GetLog() thrift call from before Hive 0.14 to retrieve the logs. # If false, use the FetchResults() thrift call from Hive 1.0 or more instead. ## use_get_log_api=false # Set a LIMIT clause when browsing a partitioned table. # A positive value will be set as the LIMIT. If 0 or negative, do not set any limit. ## browse_partitioned_table_limit=250 # A limit to the number of rows that can be downloaded from a query. # A value of -1 means there will be no limit. # A maximum of 65,000 is applied to XLS downloads. ## download_row_limit=1000000 # Hue will try to close the Hive query when the user leaves the editor page. # This will free all the query resources in HiveServer2, but also make its results inaccessible. ## close_queries=false # Thrift version to use when communicating with HiveServer2. # New column format is from version 7. thrift_version={{ thrift_version }} [[ssl]] # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Choose whether Hue should validate certificates received from the server. ## validate=true {%- endif %} ########################################################################### # Settings to configure Impala ########################################################################### {%- if impala_host %} [impala] # Host of the Impala Server (one of the Impalad) server_host={{ impala_host | default("localhost", True) }} # Port of the Impala Server server_port={{ impala_port | default("21050", True) }} # Kerberos principal ## impala_principal=impala/hostname.foo.com # Turn on/off impersonation mechanism when talking to Impala impersonation_enabled=False # Number of initial rows of a result set to ask Impala to cache in order # to support re-fetching them for downloading them. # Set to 0 for disabling the option and backward compatibility. querycache_rows=50000 # Timeout in seconds for thrift calls ## server_conn_timeout=120 # Hue will try to close the Impala query when the user leaves the editor page. # This will free all the query resources in Impala, but also make its results inaccessible. ## close_queries=true # If QUERY_TIMEOUT_S > 0, the query will be timed out (i.e. cancelled) if Impala does not do any work # (compute or send back results) for that query within QUERY_TIMEOUT_S seconds. ## query_timeout_s=0 [[ssl]] # SSL communication enabled for this server. ## enabled=false # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Choose whether Hue should validate certificates received from the server. ## validate=true {%- endif %} ########################################################################### # Settings to configure Pig ########################################################################### [pig] # Location of piggybank.jar on local filesystem. ## local_sample_dir=/opt/mapr/pig/pig-0.12/contrib/piggybank/java/ # Location piggybank.jar will be copied to in HDFS. ## remote_data_dir=/oozie/pig/examples ########################################################################### # Settings to configure Sqoop2 ########################################################################### {%- if sqoop_host %} [sqoop] # For autocompletion, fill out the librdbms section. # Sqoop server URL server_url=http://{{ sqoop_host | default("localhost", True) }}:{{ sqoop_port | default("12000", True) }}/sqoop # Change this if your cluster is secured security_enabled=${security_enabled} # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- endif %} ########################################################################### # Settings to configure Proxy ########################################################################### [proxy] # Comma-separated list of regular expressions, # which match 'host:port' of requested proxy target. ## whitelist=(localhost|127\.0\.0\.1):(50030|50070|50060|50075) # Comma-separated list of regular expressions, # which match any prefix of 'host:port/path' of requested proxy target. # This does not support matching GET parameters. ## blacklist= ########################################################################### # Settings to configure HBase Browser ########################################################################### {%- if hbase_host %} [hbase] # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'. # Use full hostname with security. hbase_clusters=(HBase|{{ hbase_host | default("localhost", True) }}:{{ hbase_port | default("9090", True) }}) # HBase configuration directory, where hbase-site.xml is located. hbase_conf_dir=/opt/mapr/hbase/hbase-{{ hbase_version | default("0.98.12", True) }}/conf # Hard limit of rows or columns per row fetched before truncating. ## truncate_limit = 500 # 'buffered' is the default of the HBase Thrift Server and supports security. # 'framed' can be used to chunk up responses, # which is useful when used in conjunction with the nonblocking server in Thrift. ## thrift_transport=buffered # Security mechanism of authentication none/GSSAPI/MAPR-SECURITY mechanism=${mechanism} {%- endif %} ########################################################################### # Settings to configure Solr Search ########################################################################### [search] # URL of the Solr Server ## solr_url=http://localhost:8983/solr/ # Requires FQDN in solr_url if enabled security_enabled=${security_enabled} ## Query sent when no term is entered ## empty_query=*:* # Use latest Solr 5.2+ features. ## latest=false ########################################################################### # Settings to configure Solr Indexer ########################################################################### [indexer] # Location of the solrctl binary. ## solrctl_path=/usr/bin/solrctl ########################################################################### # Settings to configure Job Designer ########################################################################### [jobsub] # Location on local FS where examples and template are stored. ## local_data_dir=..../data # Location on local FS where sample data is stored ## sample_data_dir=...thirdparty/sample_data ########################################################################### # Settings to configure Job Browser. ########################################################################### [jobbrowser] # Share submitted jobs information with all users. If set to false, # submitted jobs are visible only to the owner and administrators. ## share_jobs=true # Whether to disalbe the job kill button for all users in the jobbrowser ## disable_killing_jobs=false ########################################################################### # Settings to configure the Zookeeper application. ########################################################################### [zookeeper] [[clusters]] [[[default]]] # Zookeeper ensemble. Comma separated list of Host/Port. # e.g. localhost:5181,node2_ip@:5181,node3_ip@:5181 host_ports={{ zk_hosts_with_port | default("localhost:5181", True) }} # The URL of the REST contrib service (required for znode browsing). ## rest_url=http://localhost:9999 # Name of Kerberos principal when using security. ## principal_name=zookeeper ########################################################################### # Settings to configure the Spark application. ########################################################################### {%- if livy_host %} [spark] # Host address of the Livy Server. livy_server_host={{ livy_host }} # Port of the Livy Server. ## livy_server_port=8998 # Configure livy to start with 'process', 'thread', or 'yarn' workers. livy_server_session_kind=yarn # If livy should use proxy users when submitting a job. ## livy_impersonation_enabled=true # List of available types of snippets ## languages='[{"name": "Scala Shell", "type": "scala"},{"name": "PySpark Shell", "type": "python"},{"name": "R Shell", "type": "r"},{"name": "Jar", "type": "Jar"},{"name": "Python", "type": "py"},{"name": "Impala SQL", "type": "impala"},{"name": "Hive SQL", "type": "hive"},{"name": "Text", "type": "text"}]' {%- endif %} ########################################################################### # Settings for the User Admin application ########################################################################### [useradmin] # The name of the default user group that users will be a member of ## default_user_group=default [[password_policy]] # Set password policy to all users. The default policy requires password to be at least 8 characters long, # and contain both uppercase and lowercase letters, numbers, and special characters. ## is_enabled=false ## pwd_regex="^(?=.*?[A-Z])(?=(.*[a-z]){1,})(?=(.*[\d]){1,})(?=(.*[\W_]){1,}).{8,}$" ## pwd_hint="The password must be at least 8 characters long, and must contain both uppercase and lowercase letters, at least one number, and at least one special character." ## pwd_error_message="The password must be at least 8 characters long, and must contain both uppercase and lowercase letters, at least one number, and at least one special character." ########################################################################### # Settings for the Sentry lib ########################################################################### {%- if sentry_host %} [libsentry] # Hostname or IP of server. hostname={{ sentry_host }} # Port the sentry service is running on. port=8038 # Sentry configuration directory, where sentry-site.xml is located. sentry_conf_dir={{ sentry_conf }} {%- endif %} ########################################################################### # Settings to configure the ZooKeeper Lib ########################################################################### [libzookeeper] # ZooKeeper ensemble. Comma separated list of Host/Port. # e.g. localhost:2181,localhost:2182,localhost:2183 ## ensemble=localhost:2181 # Name of Kerberos principal when using security. ## principal_name=zookeeper ########################################################################### # Settings to configure SAML ########################################################################### ########################################################################### # Settings for the RDBMS application ########################################################################### [librdbms] # The RDBMS app can have any number of databases configured in the databases # section. A database is known by its section name # (IE sqlite, mysql, psql, and oracle in the list below). [[databases]] # sqlite configuration. ## [[[sqlite]]] # Name to show in the UI. ## nice_name=SQLite # For SQLite, name defines the path to the database. ## name=/opt/mapr/hue/hue-3.10.0/desktop/desktop.db # Database backend to use. ## engine=sqlite # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} # mysql, oracle, or postgresql configuration. {%- if db_host and rdbms_name %} [[[mysql]]] # Name to show in the UI. ## nice_name="My SQL DB" # For MySQL and PostgreSQL, name is the name of the database. # For Oracle, Name is instance of the Oracle server. For express edition # this is 'xe' by default. name={{ rdbms_name }} # Database backend to use. This can be: # 1. mysql # 2. postgresql # 3. oracle engine=mysql # IP or hostname of the database to connect to. host={{ db_host }} # Port the database server is listening to. Defaults are: # 1. MySQL: 3306 # 2. PostgreSQL: 5432 # 3. Oracle Express Edition: 1521 port=3306 # Username to authenticate with when connecting to the database. user={{ rdbms_user }} # Password matching the username to authenticate with when # connecting to the database. password= {{rdbms_password }} # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} {%- endif %} [libsaml] # Xmlsec1 binary path. This program should be executable by the user running Hue. ## xmlsec_binary=/usr/local/bin/xmlsec1 # Entity ID for Hue acting as service provider. # Can also accept a pattern where '' will be replaced with server URL base. ## entity_id="/saml2/metadata/" # Create users from SSO on login. ## create_users_on_login=true # Required attributes to ask for from IdP. # This requires a comma separated list. ## required_attributes=uid # Optional attributes to ask for from IdP. # This requires a comma separated list. ## optional_attributes= # IdP metadata in the form of a file. This is generally an XML file containing metadata that the Identity Provider generates. ## metadata_file= # Private key to encrypt metadata with. ## key_file= # Signed certificate to send along with encrypted metadata. ## cert_file= # Path to a file containing the password private key. ## key_file_password=/path/key # Execute this script to produce the private key password. This will be used when `key_file_password` is not set. ## key_file_password_script=/path/pwd.sh # A mapping from attributes in the response from the IdP to django user attributes. ## user_attribute_mapping={'uid': ('username', )} # Have Hue initiated authn requests be signed and provide a certificate. ## authn_requests_signed=false # Have Hue initiated logout requests be signed and provide a certificate. ## logout_requests_signed=false # Username can be sourced from 'attributes' or 'nameid'. ## username_source=attributes # Performs the logout or not. ## logout_enabled=true ########################################################################### # Settings to configure OpenID ########################################################################### [libopenid] # (Required) OpenId SSO endpoint url. ## server_endpoint_url=https://www.google.com/accounts/o8/id # OpenId 1.1 identity url prefix to be used instead of SSO endpoint url # This is only supported if you are using an OpenId 1.1 endpoint ## identity_url_prefix=https://app.onelogin.com/openid/your_company.com/ # Create users from OPENID on login. ## create_users_on_login=true # Use email for username ## use_email_for_username=true ########################################################################### # Settings to configure OAuth ########################################################################### [liboauth] # NOTE: # To work, each of the active (i.e. uncommented) service must have # applications created on the social network. # Then the "consumer key" and "consumer secret" must be provided here. # # The addresses where to do so are: # Twitter: https://dev.twitter.com/apps # Google+ : https://cloud.google.com/ # Facebook: https://developers.facebook.com/apps # Linkedin: https://www.linkedin.com/secure/developer # # Additionnaly, the following must be set in the application settings: # Twitter: Callback URL (aka Redirect URL) must be set to http://YOUR_HUE_IP_OR_DOMAIN_NAME/oauth/social_login/oauth_authenticated # Google+ : CONSENT SCREEN must have email address # Facebook: Sandbox Mode must be DISABLED # Linkedin: "In OAuth User Agreement", r_emailaddress is REQUIRED # The Consumer key of the application ## consumer_key_twitter= ## consumer_key_google= ## consumer_key_facebook= ## consumer_key_linkedin= # The Consumer secret of the application ## consumer_secret_twitter= ## consumer_secret_google= ## consumer_secret_facebook= ## consumer_secret_linkedin= # The Request token URL ## request_token_url_twitter=https://api.twitter.com/oauth/request_token ## request_token_url_google=https://accounts.google.com/o/oauth2/auth ## request_token_url_linkedin=https://www.linkedin.com/uas/oauth2/authorization ## request_token_url_facebook=https://graph.facebook.com/oauth/authorize # The Access token URL ## access_token_url_twitter=https://api.twitter.com/oauth/access_token ## access_token_url_google=https://accounts.google.com/o/oauth2/token ## access_token_url_facebook=https://graph.facebook.com/oauth/access_token ## access_token_url_linkedin=https://api.linkedin.com/uas/oauth2/accessToken # The Authenticate URL ## authenticate_url_twitter=https://api.twitter.com/oauth/authorize ## authenticate_url_google=https://www.googleapis.com/oauth2/v1/userinfo?access_token= ## authenticate_url_facebook=https://graph.facebook.com/me?access_token= ## authenticate_url_linkedin=https://api.linkedin.com/v1/people/~:(email-address)?format=json&oauth2_access_token= # Username Map. Json Hash format. # Replaces username parts in order to simplify usernames obtained # Example: {"@sub1.domain.com":"_S1", "@sub2.domain.com":"_S2"} # converts 'email@sub1.domain.com' to 'email_S1' ## username_map={} # Whitelisted domains (only applies to Google OAuth). CSV format. ## whitelisted_domains_google= ########################################################################### # Settings to configure Metadata ########################################################################### [metadata] # For metadata tagging and enhancement features [[optimizer]] # For SQL query and table analysis # Base URL to Optimizer API. ## api_url=https://alpha.optimizer.cloudera.com # The name of the product or group which will have API access to the emails associated with it. ## product_name=hue # A secret passphrase associated with the productName ## product_secret=hue # Execute this script to produce the product secret. This will be used when `product_secret` is not set. ## product_secret_script= # The email of the Optimizer account you want to associate with the Product. ## email=hue@gethue.com # The password associated with the Optimizer account you to associate with the Product. ## email_password=hue # Execute this script to produce the email password. This will be used when `email_password` is not set. ## password_script= # In secure mode (HTTPS), if Optimizer SSL certificates have to be verified against certificate authority. ## ssl_cert_ca_verify=True [[navigator]] # For tagging tables, files and getting lineage of data. # Navigator API URL (without version suffix) ## api_url=http://localhost:7187/api # Navigator API HTTP authentication username and password # Override the desktop default username and password of the hue user used for authentications with other services. # e.g. Used for LDAP/PAM pass-through authentication. ## auth_username=hue ## auth_password= sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/hue_sh_3.8.1.template0000666000175100017510000000561513245514472027056 0ustar zuulzuul00000000000000#!/bin/bash bin=`dirname "$0"` bin=`cd "$bin"; pwd` export HUE_HOME=${bin}/.. export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-{{ hadoop_version }}/etc/hadoop # # activate the Python virtual environment # source $HUE_HOME/build/env/bin/activate # Look for installed JDK if [ -z "$JAVA_HOME" ]; then sys_java="/usr/bin/java" if [ -e $sys_java ]; then jcmd=`readlink -f $sys_java` if [ -x ${jcmd%/jre/bin/java}/bin/javac ]; then JAVA_HOME=${jcmd%/jre/bin/java} elif [ -x ${jcmd%/java}/javac ]; then JAVA_HOME=${jcmd%/bin/java} fi [ -n "${JAVA_HOME}" ] && export JAVA_HOME fi fi export LD_LIBRARY_PATH=$HUE_HOME/build/env/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/mapr/lib:$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH # get arguments command=$1 shift startStop=$1 shift #get spark_home if [ $command == "livy_server" ]; then if [ -z "$SPARK_HOME" ]; then if [ -d /opt/mapr/spark/spark-* ]; then cd /opt/mapr/spark/spark-* SPARK_HOME=`pwd` export SPARK_HOME=`pwd` else echo "Cannot export spark home!" exit 0 fi fi export PATH="$PATH:$SPARK_HOME/bin" fi #file client impersonation MAPR_IMPERSONATION_ENABLED=" " export MAPR_IMPERSONATION_ENABLED HUE_PID_DIR=$HUE_HOME/pids if [ "$HUE_IDENT_STRING" = "" ]; then export HUE_IDENT_STRING=`id -nu` fi # get log directory if [ "$HUE_LOG_DIR" = "" ]; then export HUE_LOG_DIR="$HUE_HOME/logs" fi mkdir -p "$HUE_LOG_DIR" if [ "$HUE_PID_DIR" = "" ]; then export HUE_PID_DIR="$HUE_HOME/pids" fi mkdir -p "$HUE_PID_DIR" log=$HUE_LOG_DIR/hue-$HUE_IDENT_STRING-$command-$HOSTNAME.out pid=$HUE_PID_DIR/hue-$HUE_IDENT_STRING-$command.pid secure_sh_log=$HUE_LOG_DIR/secure-sh-log.out case $startStop in (start) if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo Hue web server running as process `cat $pid`. Stop it first. exit 1 fi fi cd $HUE_HOME # Generate secure keys nohup $HUE_HOME/bin/secure.sh >> "$secure_sh_log" 2>&1 < /dev/null & nohup $HUE_HOME/build/env/bin/hue $command >> "$log" 2>&1 < /dev/null & echo $! > $pid echo "`date` $command started, pid `cat $pid`" >> "$log" 2>&1 < /dev/null ;; (stop) if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo stopping $command kill `cat $pid` echo "`date` $command stopped, pid `cat $pid`" >> "$log" 2>&1 < /dev/null else echo no $command to stop fi else echo no $command to stop fi ;; (status) if [ -f $pid ]; then if kill -0 `cat $pid`; then echo $command running as process `cat $pid`. exit 0 fi echo $pid exists with pid `cat $pid` but no $command. exit 1 fi echo $command not running. exit 1 ;; (*) echo $usage exit 1 ;; esac # # deactivate the Python virtual environment # deactivate sahara-8.0.0/sahara/plugins/mapr/services/hue/resources/livy_sh_3.10.0.template0000666000175100017510000000366713245514472027335 0ustar zuulzuul00000000000000#!/usr/bin/env bash # # Licensed to Cloudera, Inc. under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. Cloudera, Inc. licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # LIVY ENVIRONMENT VARIABLES # # - JAVA_HOME Java runtime to use. By default use "java" from PATH. # - HADOOP_CONF_DIR Directory containing the Hadoop / YARN configuration to use. # - SPARK_HOME Spark which you would like to use in Livy. # - SPARK_CONF_DIR Optional directory where the Spark configuration lives. # (Default: $SPARK_HOME/conf) # - LIVY_LOG_DIR Where log files are stored. (Default: ${LIVY_HOME}/logs) # - LIVY_PID_DIR Where the pid file is stored. (Default: /tmp) # - LIVY_SERVER_JAVA_OPTS Java Opts for running livy server (You can set jvm related setting here, # like jvm memory/gc algorithm and etc.) # - LIVY_IDENT_STRING A name that identifies the Livy server instance, used to generate log file # names. (Default: name of the user starting Livy). # - LIVY_MAX_LOG_FILES Max number of log file to keep in the log directory. (Default: 5.) # - LIVY_NICENESS Niceness of the Livy server process when running in the background. (Default: 0.) export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-{{ hadoop_version }}/etc/hadoop export SPARK_HOME=/opt/mapr/spark/spark-{{ spark_version }} sahara-8.0.0/sahara/plugins/mapr/services/httpfs/0000775000175100017510000000000013245515027022020 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/httpfs/__init__.py0000666000175100017510000000000013245514472024124 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/httpfs/httpfs.py0000666000175100017510000000245613245514472023716 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu HTTP_FS = np.NodeProcess( name='httpfs', ui_name='HTTPFS', package='mapr-httpfs', open_ports=[14000] ) class HttpFS(s.Service): def __init__(self): super(HttpFS, self).__init__() self._name = 'httpfs' self._ui_name = 'HttpFS' self._version = '1.0' self._node_processes = [HTTP_FS] self._cluster_defaults = ['httpfs-default.json'] self._validation_rules = [vu.exactly(1, HTTP_FS)] def post_install(self, cluster_context, instances): self._set_service_dir_owner(cluster_context, instances) sahara-8.0.0/sahara/plugins/mapr/services/httpfs/resources/0000775000175100017510000000000013245515027024032 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/httpfs/resources/httpfs-default.json0000666000175100017510000000026313245514472027665 0ustar zuulzuul00000000000000[ { "value": "*", "priority": 1, "name": "httpfs.proxyuser.mapr.hosts" }, { "value": "*", "priority": 1, "name": "httpfs.proxyuser.mapr.groups" } ]sahara-8.0.0/sahara/plugins/mapr/services/__init__.py0000666000175100017510000000000013245514472022614 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/flume/0000775000175100017510000000000013245515027021620 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/flume/flume.py0000666000175100017510000000250113245514472023305 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu FLUME = np.NodeProcess( name='flume', ui_name='Flume', package='mapr-flume', open_ports=[] ) class Flume(s.Service): def __init__(self): super(Flume, self).__init__() self._name = 'flume' self._ui_name = 'Flume' self._node_processes = [FLUME] self._validation_rules = [vu.at_least(1, FLUME)] class FlumeV15(Flume): def __init__(self): super(FlumeV15, self).__init__() self._version = '1.5.0' class FlumeV16(Flume): def __init__(self): super(FlumeV16, self).__init__() self._version = '1.6.0' sahara-8.0.0/sahara/plugins/mapr/services/flume/__init__.py0000666000175100017510000000000013245514472023724 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/kafka/0000775000175100017510000000000013245515027021565 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/kafka/__init__.py0000666000175100017510000000000013245514472023671 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/kafka/kafka.py0000666000175100017510000000407313245514472023225 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu KAFKA = np.NodeProcess( name='kafka', ui_name='Kafka', package='mapr-kafka', open_ports=[9092] ) KAFKA_REST = np.NodeProcess( name='kafka', ui_name='Kafka Rest', package='mapr-kafka-rest', open_ports=[8082] ) KAFKA_CONNECT_HDFS = np.NodeProcess( name='kafka', ui_name='Kafka Connect HDFS', package='mapr-kafka-connect-hdfs' ) KAFKA_CONNECT_JDBC = np.NodeProcess( name='kafka', ui_name='Kafka Connect JDBC', package='mapr-kafka-connect-jdbc' ) class Kafka(s.Service): def __init__(self): super(Kafka, self).__init__() self._version = '0.9.0' self._name = 'kafka' self._ui_name = 'Kafka' self._node_processes = [KAFKA] class KafkaRest(s.Service): def __init__(self): super(KafkaRest, self).__init__() self._version = '2.0.1' self._name = 'kafka-eco' self._ui_name = 'Kafka Rest' self._node_processes = [KAFKA_REST] self._validation_rules = [vu.at_least(1, KAFKA)] class KafkaConnect(s.Service): def __init__(self): super(KafkaConnect, self).__init__() self._version = '2.0.1' self._name = 'kafka-connect' self._ui_name = 'Kafka Connect' self._node_processes = [KAFKA_CONNECT_HDFS, KAFKA_CONNECT_JDBC] self._validation_rules = [vu.at_least(1, KAFKA)] sahara-8.0.0/sahara/plugins/mapr/services/mapreduce/0000775000175100017510000000000013245515027022455 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mapreduce/mapreduce.py0000666000175100017510000001074013245514472025003 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu from sahara.swift import swift_helper from sahara.topology import topology_helper as topo JOB_TRACKER = np.NodeProcess( name='jobtracker', ui_name='JobTracker', package='mapr-jobtracker', open_ports=[9001, 50030] ) TASK_TRACKER = np.NodeProcess( name='tasktracker', ui_name='TaskTracker', package='mapr-tasktracker', open_ports=[50060] ) class MapReduce(s.Service): cluster_mode = 'classic' def __init__(self): super(MapReduce, self).__init__() self._ui_name = 'MapReduce' self._name = 'hadoop' self._version = '0.20.2' self._node_processes = [JOB_TRACKER, TASK_TRACKER] self._ui_info = [ ('JobTracker', JOB_TRACKER, {s.SERVICE_UI: 'http://%s:50030'}), ('TaskTracker', TASK_TRACKER, {s.SERVICE_UI: 'http://%s:50060'}), ] self._validation_rules = [ vu.at_least(1, JOB_TRACKER), vu.at_least(1, TASK_TRACKER), ] self._cluster_defaults = ['mapreduce-cluster.json'] def _get_packages(self, cluster_context, node_processes): result = [] result += self.dependencies result += [(np.package, None) for np in node_processes] return result # mapred-site.xml def get_config_files(self, cluster_context, configs, instance=None): core_site = bcf.HadoopXML("core-site.xml") core_site.remote_path = self.conf_dir(cluster_context) if instance: core_site.fetch(instance) core_site.add_properties(self._get_core_site_props(cluster_context)) mapred_site = bcf.HadoopXML("mapred-site.xml") mapred_site.remote_path = self.conf_dir(cluster_context) if instance: mapred_site.fetch(instance) mapred_site.load_properties(configs) mapred_site.add_properties( self._get_mapred_site_props(cluster_context)) return [core_site, mapred_site] def _get_core_site_props(self, cluster_context): result = {} if cluster_context.is_node_aware: result.update(self._get_core_site_node_aware_props()) for conf in swift_helper.get_swift_configs(): result[conf['name']] = conf['value'] for conf in self._get_impersonation_props(): result[conf['name']] = conf['value'] return result def _get_core_site_node_aware_props(self): result = topo.vm_awareness_core_config() result = {c['name']: c['value'] for c in result} result.update({ 'net.topology.script.file.name': '/opt/mapr/topology.sh', 'net.topology.script.number.args': '75', }) return result def _get_mapred_site_props(self, cluster_context): result = {} if cluster_context.is_node_aware: result.update(self._get_mapred_site_node_aware_props()) result.update({ 'jobtracker.thrift.address': '0.0.0.0:9290', 'mapred.jobtracker.plugins': 'org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin', }) return result def _get_mapred_site_node_aware_props(self): result = topo.vm_awareness_mapred_config() result = {c['name']: c['value'] for c in result} # This config causes failure result.pop('mapred.task.cache.levels') result['mapreduce.jobtracker.taskcache.levels'] = '3' return result def _get_impersonation_props(self): return [ {'name': 'hadoop.proxyuser.mapr.groups', 'value': '*'}, {'name': 'hadoop.proxyuser.mapr.hosts', 'value': '*'} ] def get_file_path(self, file_name): template = 'plugins/mapr/services/mapreduce/resources/%s' return template % file_name sahara-8.0.0/sahara/plugins/mapr/services/mapreduce/__init__.py0000666000175100017510000000000013245514472024561 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mapreduce/resources/0000775000175100017510000000000013245515027024467 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mapreduce/resources/mapreduce-cluster.json0000666000175100017510000000575613245514472031030 0ustar zuulzuul00000000000000[ { "name": "mapred.fairscheduler.assignmultiple", "value": true, "config_type": "bool", "description": "CoreDefaultProperties" }, { "name": "mapred.fairscheduler.eventlog.enabled", "value": false, "config_type": "bool", "description": "Enable scheduler logging in ${HADOOP_LOG_DIR}/fairscheduler/" }, { "name": "mapred.fairscheduler.smalljob.schedule.enable", "value": true, "config_type": "bool", "description": "Enable small job fast scheduling inside fair scheduler. TaskTrackers should reserve a slot called ephemeral slot which is used for smalljob if cluster is busy." }, { "name": "mapred.fairscheduler.smalljob.max.maps", "value": 10, "config_type": "int", "description": "Small job definition. Max number of maps allowed in small job." }, { "name": "mapred.fairscheduler.smalljob.max.reducers", "value": 10, "config_type": "int", "description": "Small job definition. Max number of reducers allowed in small job." }, { "name": "mapred.fairscheduler.smalljob.max.inputsize", "value": 10737418240, "config_type": "int", "description": "Small job definition. Max input size in bytes allowed for a small job. Default is 10 GB" }, { "name": "mapred.fairscheduler.smalljob.max.reducer.inputsize", "value": 1073741824, "config_type": "int", "description": "Small job definition. Max estimated input size for a reducer allowed in small job. Default is 1 GB per reducer." }, { "name": "mapred.cluster.ephemeral.tasks.memory.limit.mb", "value": 200, "config_type": "int", "description": "Small job definition. Max memory in mbytes reserved for an ephermal slot. Default is 200 mb. This value must be same on JobTracker and TaskTracker nodes." }, { "name": "mapreduce.jobtracker.node.labels.file", "value": "/", "description": "File on maprfs that has mapping of nodes and labels." }, { "name": "mapred.tasktracker.ephemeral.tasks.maximum", "value": 1, "config_type": "int", "description": "Reserved slot for small job scheduling" }, { "name": "mapred.tasktracker.ephemeral.tasks.timeout", "value": 10000, "config_type": "int", "description": "Maximum time in ms a task is allowed to occupy ephemeral slot" }, { "name": "mapred.tasktracker.ephemeral.tasks.ulimit", "value": 4294967296, "config_type": "int", "description": "Ulimit (bytes) on all tasks sheduled on an ephemeral slot" }, { "name": "mapreduce.tasktracker.group", "value": "root", "description": "Group to which TaskTracker belongs." }, { "name": "mapred.local.dir", "value": "/tmp/mapr-hadoop/mapred/local", "description": "The local directory where MapReduce stores job jar, xml files and creates work dirs for tasks. MapR hadoop uses a local volume map outputs." }, { "name": "mapred.map.child.java.opts", "value": "-Xmx1024m" }, { "name": "mapred.reduce.child.java.opts", "value": "-Xmx3072m" } ]sahara-8.0.0/sahara/plugins/mapr/services/oozie/0000775000175100017510000000000013245515027021635 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/oozie/oozie.py0000666000175100017510000001660313245514472023347 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslo_log import log as logging import sahara.context as context from sahara.i18n import _ import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.services.mysql.mysql as mysql import sahara.plugins.mapr.util.event_log as el import sahara.plugins.mapr.util.general as g import sahara.plugins.mapr.util.validation_utils as vu LOG = logging.getLogger(__name__) OOZIE_START_DELAY = 30 OOZIE = np.NodeProcess( name='oozie', ui_name='Oozie', package='mapr-oozie', open_ports=[11000] ) class Oozie(s.Service): def __init__(self): super(Oozie, self).__init__() self._name = 'oozie' self._ui_name = 'Oozie' self._node_processes = [OOZIE] self._cluster_defaults = ['oozie-default.json'] self._validation_rules = [vu.exactly(1, OOZIE)] self._ui_info = [('Oozie', OOZIE, {s.SERVICE_UI: 'http://%s:11000/oozie'})] def libext_path(self): return '/opt/mapr/oozie/oozie-%s/oozie-server/lib/' % self.version def get_config_files(self, cluster_context, configs, instance=None): oozie_site = bcf.HadoopXML("oozie-site.xml") oozie_site.remote_path = self.conf_dir(cluster_context) if instance: oozie_site.fetch(instance) oozie_site.load_properties(configs) oozie_site.add_properties(self._get_oozie_site_props(cluster_context)) return [oozie_site] def _get_oozie_site_props(self, cluster_context): oozie_specs = mysql.MySQL.OOZIE_SPECS return { 'oozie.db.schema.name': oozie_specs.db_name, 'oozie.service.JPAService.create.db.schema': True, 'oozie.service.JPAService.jdbc.driver': mysql.MySQL.DRIVER_CLASS, 'oozie.service.JPAService.jdbc.url': self._get_jdbc_uri( cluster_context), 'oozie.service.JPAService.jdbc.username': oozie_specs.user, 'oozie.service.JPAService.jdbc.password': oozie_specs.password, 'oozie.service.HadoopAccessorService.hadoop.configurations': '*=%s' % cluster_context.hadoop_conf } def _get_jdbc_uri(self, cluster_context): jdbc_uri = ('jdbc:mysql://%(db_host)s:%(db_port)s/%(db_name)s?' 'createDatabaseIfNotExist=true') jdbc_args = { 'db_host': mysql.MySQL.get_db_instance( cluster_context).internal_ip, 'db_port': mysql.MySQL.MYSQL_SERVER_PORT, 'db_name': mysql.MySQL.OOZIE_SPECS.db_name, } return jdbc_uri % jdbc_args def install(self, cluster_context, instances): # oozie requires executed configure.sh pass def post_configure(self, cluster_context, instances): super(Oozie, self).install(cluster_context, instances) oozie_instances = cluster_context.filter_instances(instances, service=self) for instance in oozie_instances: with instance.remote() as r: r.execute_command( 'sudo cp ' '/opt/mapr/oozie/oozie-%s/conf/warden.oozie.conf ' '/opt/mapr/conf/conf.d' % self.version) def post_install(self, cluster_context, instances): oozie_inst = cluster_context.get_instance(OOZIE) oozie_service = cluster_context.get_service(OOZIE) if oozie_service: symlink_cmd = ( 'cp /opt/mapr/lib/mysql-connector-*.jar %s' % self.libext_path()) with oozie_inst.remote() as r: LOG.debug('Installing MySQL connector for Oozie') r.execute_command(symlink_cmd, run_as_root=True, raise_when_error=False) self._set_service_dir_owner(cluster_context, instances) def post_start(self, cluster_context, instances): instances = cluster_context.filter_instances(instances, OOZIE) self._rebuild(cluster_context, instances) @el.provision_event(instance_reference=1) @g.remote_command(1) def _rebuild_oozie_war(self, remote, cluster_context): cmd = 'cp -f /opt/mapr-repository/ext-2.2.zip %(home)s/libext/ext-2.2.zip &&' \ ' %(home)s/bin/oozie-setup.sh -hadoop %(version)s' \ ' /opt/mapr/hadoop/hadoop-%(version)s' args = {'home': self.home_dir(cluster_context), 'version': cluster_context.hadoop_version} remote.execute_command(cmd % args, run_as_root=True) def update(self, cluster_context, instances=None): instances = instances or cluster_context.get_instances() instances = cluster_context.filter_instances(instances, OOZIE) self._rebuild(cluster_context, instances) @el.provision_step(_("Rebuilt Oozie war")) def _rebuild(self, cluster_context, instances): OOZIE.stop(filter(OOZIE.is_started, instances)) g.execute_on_instances( instances, self._rebuild_oozie_war, cluster_context) OOZIE.start(instances) context.sleep(OOZIE_START_DELAY) class OozieV401(Oozie): def __init__(self): super(OozieV401, self).__init__() self._version = '4.0.1' self._dependencies = [('mapr-oozie-internal', self.version)] class OozieV410(Oozie): def __init__(self): super(OozieV410, self).__init__() self._version = '4.1.0' self._dependencies = [('mapr-oozie-internal', self.version)] class OozieV420(Oozie): def __init__(self): super(OozieV420, self).__init__() self._version = '4.2.0' self._dependencies = [('mapr-oozie-internal', self.version)] def libext_path(self): return '/opt/mapr/oozie/oozie-%s/libext/' % self.version def post_install(self, cluster_context, instances): super(OozieV420, self).post_install(cluster_context, instances) self.fix_oozie_bug(cluster_context) def fix_oozie_bug(self, cluster_context): """Wrong maprfs jar bug On some environments Oozie installation process takes incorrect jar that causes failure to run jobs. This is a temporary bug in Oozie and is going to be fixed soon. """ if cluster_context.mapr_version != '5.1.0': return oozie_inst = cluster_context.get_instance(OOZIE) command = "sudo rm /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/kms/" \ "tomcat/webapps/kms/WEB-INF/lib/maprfs-5.1.0-mapr.jar" \ " && sudo ln -s /opt/mapr/lib/maprfs-5.1.0-mapr.jar" \ " /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/kms/" \ "tomcat/webapps/kms/WEB-INF/lib/maprfs-5.1.0-mapr.jar" with oozie_inst.remote() as r: r.execute_command(command, run_as_root=True) sahara-8.0.0/sahara/plugins/mapr/services/oozie/__init__.py0000666000175100017510000000000013245514472023741 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/oozie/resources/0000775000175100017510000000000013245515027023647 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/oozie/resources/oozie-default.json0000666000175100017510000000324013245514472027315 0ustar zuulzuul00000000000000[ { "value": "/oozie/share/lib", "name": "oozie.service.WorkflowAppService.system.libpath", "description": "System library path to use for workflow applications.This path is added to workflow application if their job properties setsthe property 'oozie.use.system.libpath' to true." }, { "value": "yarn-tez", "name": "mapreduce.framework.name", "description": "The runtime framework for executing MapReduce jobs." }, { "value": false, "name": "use.system.libpath.for.mapreduce.and.pig.jobs", "config_type": "bool", "description": "If set to true, submissions of MapReduce and Pig jobs will includeautomatically the system library path, thus not requiring users tospecify where the Pig JAR files are. Instead, the ones from the systemlibrary path are used." }, { "value": "*", "name": "oozie.service.ProxyUserService.proxyuser.mapr.hosts", "description": "List of hosts the '#USER#' user is allowed to perform 'doAs'operations.The '#USER#' must be replaced with the username o the user who isallowed to perform 'doAs' operations.The value can be the '*' wildcard or a list of hostnames.For multiple users copy this property and replace the user namein the property name." }, { "value": "*", "name": "oozie.service.ProxyUserService.proxyuser.mapr.groups", "description": "List of groups the '#USER#' user is allowed to impersonate usersfrom to perform 'doAs' operations.The '#USER#' must be replaced with the username o the user who isallowed to perform 'doAs' operations.The value can be the '*' wildcard or a list of groups.For multiple users copy this property and replace the user namein the property name." } ]sahara-8.0.0/sahara/plugins/mapr/services/maprfs/0000775000175100017510000000000013245515027022000 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/maprfs/__init__.py0000666000175100017510000000000013245514472024104 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/maprfs/maprfs.py0000666000175100017510000001260213245514472023650 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslo_log import log as logging from sahara import context from sahara.i18n import _ import sahara.plugins.mapr.domain.configuration_file as bcf import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.event_log as el import sahara.plugins.mapr.util.general as g import sahara.plugins.mapr.util.validation_utils as vu import sahara.plugins.provisioning as p from sahara.utils import cluster_progress_ops as cpo from sahara.utils import files LOG = logging.getLogger(__name__) CLDB = np.NodeProcess( name='cldb', ui_name='CLDB', package='mapr-cldb', open_ports=[7222, 7220, 7221] ) FILE_SERVER = np.NodeProcess( name='fileserver', ui_name='FileServer', package='mapr-fileserver', open_ports=[] ) NFS = np.NodeProcess( name='nfs', ui_name='NFS', package='mapr-nfs', open_ports=[2049, 9997, 9998] ) class MapRFS(s.Service): _CREATE_DISK_LIST = 'plugins/mapr/resources/create_disk_list_file.sh' _DISK_SETUP_CMD = '/opt/mapr/server/disksetup -F /tmp/disk.list' _DISK_SETUP_TIMEOUT = 600 ENABLE_MAPR_DB_NAME = 'Enable MapR-DB' HEAP_SIZE_PERCENT_NAME = 'MapR-FS heap size percent' ENABLE_MAPR_DB_CONFIG = p.Config( name=ENABLE_MAPR_DB_NAME, applicable_target='general', scope='cluster', config_type="bool", priority=1, default_value=True, description=_('Specifies that MapR-DB is in use.') ) HEAP_SIZE_PERCENT = p.Config( name=HEAP_SIZE_PERCENT_NAME, applicable_target='MapRFS', scope='cluster', config_type="int", priority=1, default_value=8, description=_( 'Specifies heap size for MapR-FS in percents of maximum value.') ) def __init__(self): super(MapRFS, self).__init__() self._ui_name = 'MapRFS' self._node_processes = [CLDB, FILE_SERVER, NFS] self._ui_info = [ ('Container Location Database (CLDB)', CLDB, {s.SERVICE_UI: 'http://%s:7221'}), ] self._validation_rules = [ vu.at_least(1, CLDB), vu.each_node_has(FILE_SERVER), vu.on_same_node(CLDB, FILE_SERVER), vu.has_volumes(), ] def service_dir(self, cluster_context): return def home_dir(self, cluster_context): return def conf_dir(self, cluster_context): return '%s/conf' % cluster_context.mapr_home def post_install(self, cluster_context, instances): LOG.debug('Initializing MapR FS') instances = instances or cluster_context.get_instances() file_servers = cluster_context.filter_instances(instances, FILE_SERVER) cpo.add_provisioning_step(cluster_context.cluster.id, _("Initializing MapR-FS"), len(file_servers)) with context.ThreadGroup() as tg: for instance in file_servers: tg.spawn('init-mfs-%s' % instance.id, self._init_mfs_instance, instance) LOG.info('MapR FS successfully initialized') @el.provision_event(instance_reference=1) def _init_mfs_instance(self, instance): self._generate_disk_list_file(instance, self._CREATE_DISK_LIST) self._execute_disksetup(instance) def _generate_disk_list_file(self, instance, path_to_disk_setup_script): LOG.debug('Creating disk list file') g.run_script(instance, path_to_disk_setup_script, 'root', *instance.storage_paths()) def _execute_disksetup(self, instance): with instance.remote() as rmt: rmt.execute_command( self._DISK_SETUP_CMD, run_as_root=True, timeout=self._DISK_SETUP_TIMEOUT) def get_configs(self): return [MapRFS.ENABLE_MAPR_DB_CONFIG, MapRFS.HEAP_SIZE_PERCENT] def get_config_files(self, cluster_context, configs, instance=None): default_path = 'plugins/mapr/services/maprfs/resources/cldb.conf' cldb_conf = bcf.PropertiesFile("cldb.conf") cldb_conf.remote_path = self.conf_dir(cluster_context) if instance: cldb_conf.fetch(instance) cldb_conf.parse(files.get_file_text(default_path)) cldb_conf.add_properties(self._get_cldb_conf_props(cluster_context)) warden_conf = bcf.PropertiesFile("warden.conf") warden_conf.remote_path = "/opt/mapr/conf/" if instance: warden_conf.fetch(instance) warden_conf.add_properties( {'service.command.mfs.heapsize.percent': configs[ self.HEAP_SIZE_PERCENT_NAME]}) return [cldb_conf, warden_conf] def _get_cldb_conf_props(self, cluster_context): zookeepers = cluster_context.get_zookeeper_nodes_ip_with_port() return {'cldb.zookeeper.servers': zookeepers} sahara-8.0.0/sahara/plugins/mapr/services/maprfs/resources/0000775000175100017510000000000013245515027024012 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/maprfs/resources/cldb.conf0000666000175100017510000000401413245514472025571 0ustar zuulzuul00000000000000# # CLDB Config file. # Properties defined in this file are loaded during startup # and are valid for only CLDB which loaded the config. # These parameters are not persisted anywhere else. # # Wait until minimum number of fileserver register with # CLDB before creating Root Volume cldb.min.fileservers=1 # CLDB listening port cldb.port=7222 # Number of worker threads cldb.numthreads=10 # CLDB webport cldb.web.port=7221 # CLDB https port cldb.web.https.port=7443 # Disable duplicate hostid detection cldb.detect.dup.hostid.enabled=false # Deprecated: This param is no longer supported. To configure # the container cache, use the param cldb.containers.cache.percent # Number of RW containers in cache #cldb.containers.cache.entries=1000000 # # Percentage (integer) of Xmx setting to be used for container cache #cldb.containers.cache.percent=20 # # Topology script to be used to determine # Rack topology of node # Script should take an IP address as input and print rack path # on STDOUT. eg # $>/home/mapr/topo.pl 10.10.10.10 # $>/mapr-rack1 # $>/home/mapr/topo.pl 10.10.10.20 # $>/mapr-rack2 #net.topology.script.file.name=/home/mapr/topo.pl # # Topology mapping file used to determine # Rack topology of node # File is of a 2 column format (space separated) # 1st column is an IP address or hostname # 2nd column is the rack path # Line starting with '#' is a comment # Example file contents # 10.10.10.10 /mapr-rack1 # 10.10.10.20 /mapr-rack2 # host.foo.com /mapr-rack3 #net.topology.table.file.name=/home/mapr/topo.txt # # ZooKeeper address #cldb.zookeeper.servers=10.250.1.91:5181 # Hadoop metrics jar version #hadoop.version=0.20.2 # CLDB JMX remote port cldb.jmxremote.port=7220 num.volmirror.threads=1 # Set this to set the default topology for all volumes and nodes # The default for all volumes is /data by default # UNCOMMENT the below to change the default topology. # For e.g., set cldb.default.topology=/mydata to create volumes # in /mydata topology and to place all nodes in /mydata topology # by default #cldb.default.topology=/mydata sahara-8.0.0/sahara/plugins/mapr/services/management/0000775000175100017510000000000013245515027022624 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/management/management.py0000666000175100017510000000364013245514472025322 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.password_utils as pu import sahara.plugins.mapr.util.validation_utils as vu ZK_CLIENT_PORT = 5181 ZOOKEEPER = np.NodeProcess( name='mapr-zookeeper', ui_name='ZooKeeper', package='mapr-zookeeper', open_ports=[ZK_CLIENT_PORT] ) WEB_SERVER = np.NodeProcess( name='webserver', ui_name='Webserver', package='mapr-webserver', open_ports=[8443] ) METRICS = np.NodeProcess( name='metrics', ui_name='Metrics', package='mapr-metrics', open_ports=[1111] ) class Management(s.Service): def __init__(self): super(Management, self).__init__() self._ui_name = 'Management' self._node_processes = [ZOOKEEPER, WEB_SERVER, METRICS] self._ui_info = None self._validation_rules = [ vu.at_least(1, ZOOKEEPER), vu.at_least(1, WEB_SERVER), vu.odd_count_of(ZOOKEEPER), ] def get_ui_info(self, cluster_context): # MCS uses credentials of the administrative user (PAM auth) return [('MapR Control System (MCS)', WEB_SERVER, {s.SERVICE_UI: 'https://%s:8443', 'Username': 'mapr', 'Password': pu.get_mapr_password(cluster_context.cluster)})] sahara-8.0.0/sahara/plugins/mapr/services/management/__init__.py0000666000175100017510000000000013245514472024730 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mahout/0000775000175100017510000000000013245515027022005 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mahout/__init__.py0000666000175100017510000000000013245514472024111 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/mahout/mahout.py0000666000175100017510000000312113245514472023656 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.validation_utils as vu MAHOUT = np.NodeProcess( name='mahout', ui_name='Mahout', package='mapr-mahout', open_ports=[] ) class Mahout(s.Service): def __init__(self): super(Mahout, self).__init__() self._name = 'mahout' self._ui_name = 'Mahout' self._node_processes = [MAHOUT] self._validation_rules = [vu.at_least(1, MAHOUT)] class MahoutV09(Mahout): def __init__(self): super(MahoutV09, self).__init__() self._version = '0.9' class MahoutV010(Mahout): def __init__(self): super(MahoutV010, self).__init__() self._version = '0.10.0' class MahoutV011(Mahout): def __init__(self): super(MahoutV011, self).__init__() self._version = '0.11.0' class MahoutV012(Mahout): def __init__(self): super(MahoutV012, self).__init__() self._version = '0.12.0' sahara-8.0.0/sahara/plugins/mapr/services/drill/0000775000175100017510000000000013245515027021616 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/services/drill/drill.py0000666000175100017510000000577213245514472023316 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.node_process as np import sahara.plugins.mapr.domain.service as s import sahara.plugins.mapr.util.commands as cmd import sahara.plugins.mapr.util.validation_utils as vu DRILL = np.NodeProcess( name='drill-bits', ui_name='Drill', package='mapr-drill', open_ports=[8047] ) DRILL_YARN = np.NodeProcess( name='drill-bits-yarn', ui_name='Drill on YARN', package='mapr-drill-yarn', open_ports=[8047] ) class Drill(s.Service): def __init__(self): super(Drill, self).__init__() self._name = 'drill' self._ui_name = 'Drill' self._node_processes = [DRILL] self._ui_info = [('Drill', DRILL, {s.SERVICE_UI: 'http://%s:8047'})] self._validation_rules = [ vu.node_client_package_conflict_vr([DRILL], DRILL_YARN) ] def install(self, cluster_context, instances): # Drill requires running cluster pass def post_start(self, cluster_context, instances): instances = instances or cluster_context.get_instances(DRILL) super(Drill, self).install(cluster_context, instances) self._set_service_dir_owner(cluster_context, instances) for instance in instances: cmd.re_configure_sh(instance, cluster_context) class DrillV07(Drill): def __init__(self): super(DrillV07, self).__init__() self._version = '0.7' class DrillV08(Drill): def __init__(self): super(DrillV08, self).__init__() self._version = '0.8' class DrillV09(Drill): def __init__(self): super(DrillV09, self).__init__() self._version = '0.9' class DrillV11(Drill): def __init__(self): super(DrillV11, self).__init__() self._version = "1.1" class DrillV12(Drill): def __init__(self): super(DrillV12, self).__init__() self._version = "1.2" class DrillV14(Drill): def __init__(self): super(DrillV14, self).__init__() self._version = "1.4" class DrillV16(Drill): def __init__(self): super(DrillV16, self).__init__() self._version = "1.6" class DrillV18(Drill): def __init__(self): super(DrillV18, self).__init__() self._version = "1.8" self._node_processes = [DRILL, DRILL_YARN] class DrillV19(Drill): def __init__(self): super(DrillV19, self).__init__() self._version = "1.8" self._node_processes = [DRILL, DRILL_YARN] sahara-8.0.0/sahara/plugins/mapr/services/drill/__init__.py0000666000175100017510000000000013245514472023722 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/domain/0000775000175100017510000000000013245515027020134 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/domain/node_process.py0000666000175100017510000001254313245514472023203 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import oslo_serialization.jsonutils as json from sahara.i18n import _ import sahara.plugins.mapr.util.general as util from sahara.utils import poll_utils as polls WARDEN_MANAGED_CMD = ('sudo -u mapr maprcli node services' ' -name %(service)s' ' -action %(action)s' ' -nodes %(nodes)s') class NodeProcess(object): def __init__(self, name, ui_name, package, open_ports=None): self._name = name self._ui_name = ui_name self._package = package self._open_ports = open_ports or [] @property def name(self): return self._name @property def ui_name(self): return self._ui_name @property def package(self): return self._package @property def open_ports(self): return self._open_ports def start(self, instances): self.execute_action(instances, Action.START) def restart(self, instances): self.execute_action(instances, Action.RESTART) def stop(self, instances): self.execute_action(instances, Action.STOP) def execute_action(self, instances, action): if len(instances) == 0: return nodes = ','.join(map(lambda i: i.internal_ip, instances)) args = {'service': self.name, 'action': action.name, 'nodes': nodes} command = WARDEN_MANAGED_CMD % args with instances[0].remote() as r: r.execute_command(command) self._wait_for_status(instances, action.status) def _wait_for_status(self, instances, status, sleep=3, timeout=60): def poll_status(instance): operation_name = _('Wait for {node_process} on {instance}' ' to change status to "{status}"') args = { 'node_process': self.ui_name, 'instance': instance.instance_name, 'status': status.name, } return polls.poll( get_status=lambda: self.status(instance) == status, operation_name=operation_name.format(**args), timeout=timeout, sleep=sleep, ) util.execute_on_instances(instances, poll_status) def status(self, instance): command = 'maprcli service list -node %s -json' % instance.internal_ip with instance.remote() as remote: ec, out = remote.execute_command(util._run_as('mapr', command)) node_processes = json.loads(out)['data'] for node_process in node_processes: if node_process['name'] == self.name: return Status.by_value(node_process['state']) return Status.NOT_CONFIGURED def is_started(self, instance): # At least tried to do it =) return self.status(instance) in [Status.RUNNING, Status.FAILED, Status.STAND_BY] class Status(object): class Item(object): def __init__(self, name, value): self._name = name self._value = value @property def name(self): return self._name @property def value(self): return self._value # The package for the service is not installed and/or # the service is not configured (configure.sh has not run) NOT_CONFIGURED = Item('Not Configured', 0) # The package for the service is installed and configured CONFIGURED = Item('Configured', 1) # The service is installed, started by the warden, and is currently running RUNNING = Item('Running', 2) # The service is installed and configure.sh has run, # but the service is not running STOPPED = Item('Stopped', 3) # The service is installed and configured, but not running FAILED = Item('Failed', 4) # The service is installed and is in standby mode, waiting to take over # in case of failure of another instance. # Mainly used for JobTracker warm standby STAND_BY = Item('Standby', 5) @staticmethod def items(): return [ Status.NOT_CONFIGURED, Status.CONFIGURED, Status.RUNNING, Status.STOPPED, Status.FAILED, Status.STAND_BY, ] @staticmethod def by_value(value): for v in Status.items(): if v.value == value: return v class Action(object): class Item(object): def __init__(self, name, status): self._name = name self._status = status @property def name(self): return self._name @property def status(self): return self._status START = Item('start', Status.RUNNING) STOP = Item('stop', Status.STOPPED) RESTART = Item('restart', Status.RUNNING) sahara-8.0.0/sahara/plugins/mapr/domain/distro.py0000666000175100017510000000473413245514472022027 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. class Distro(object): def __init__(self, name, internal_name, install_cmd, version_separator): self._name = name self._internal_name = internal_name self._install_command = install_cmd self._version_separator = version_separator @property def name(self): return self._name @property def internal_name(self): return self._internal_name @property def install_command(self): return self._install_command @property def version_separator(self): return self._version_separator def create_install_cmd(self, packages): s = self.version_separator def join_package_version(pv_item): p, v = pv_item if len(pv_item) > 1 else (pv_item[0], None) return p + s + v + '*' if v else p packages = ' '.join(map(join_package_version, packages)) command = '%(install_cmd)s %(packages)s' args = {'install_cmd': self.install_command, 'packages': packages} return command % args UBUNTU = Distro( name='Ubuntu', internal_name='Ubuntu', install_cmd='apt-get install --force-yes -y', version_separator='=', ) CENTOS = Distro( name='CentOS', internal_name='CentOS', install_cmd='yum install -y', version_separator='-', ) RHEL = Distro( name='RedHatEnterpriseServer', internal_name='RedHat', install_cmd='yum install -y', version_separator='-', ) SUSE = Distro( name='Suse', internal_name='Suse', install_cmd='zypper', version_separator=':', ) def get_all(): return [UBUNTU, CENTOS, RHEL, SUSE] def get(instance): with instance.remote() as r: name = r.get_os_distrib() for d in get_all(): if d.internal_name.lower() in name: return d def get_version(instance): with instance.remote() as r: version = r.get_os_version() return version sahara-8.0.0/sahara/plugins/mapr/domain/__init__.py0000666000175100017510000000000013245514472022240 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/domain/service.py0000666000175100017510000002017613245514472022161 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslo_log import log as logging from oslo_serialization import jsonutils as json import six import sahara.exceptions as e from sahara.i18n import _ import sahara.plugins.exceptions as ex from sahara.plugins.mapr.util import commands as cmd from sahara.plugins.mapr.util import event_log as el from sahara.plugins.mapr.util import general as g from sahara.plugins.mapr.util import service_utils as su import sahara.plugins.provisioning as p from sahara.utils import files LOG = logging.getLogger(__name__) SERVICE_UI = 'Web UI' _INSTALL_PACKAGES_TIMEOUT = 3600 @six.add_metaclass(g.Singleton) class Service(object): def __init__(self): self._name = None self._ui_name = None self._node_processes = [] self._version = None self._dependencies = [] self._ui_info = [] self._cluster_defaults = [] self._node_defaults = [] self._validation_rules = [] self._priority = 1 @property def name(self): return self._name @property def ui_name(self): return self._ui_name @property def version(self): return self._version @property def node_processes(self): return self._node_processes @property def dependencies(self): return self._dependencies @property def cluster_defaults(self): return self._cluster_defaults @property def node_defaults(self): return self._node_defaults @property def validation_rules(self): return self._validation_rules def get_ui_info(self, cluster_context): return self._ui_info def install(self, cluster_context, instances): service_instances = cluster_context.filter_instances(instances, service=self) @el.provision_step(_("Install %s service") % self.ui_name, cluster_context_reference=0, instances_reference=1) def _install(_context, _instances): g.execute_on_instances(_instances, self._install_packages_on_instance, _context) if service_instances: _install(cluster_context, service_instances) @el.provision_event(instance_reference=1) def _install_packages_on_instance(self, instance, cluster_context): processes = [p for p in self.node_processes if p.ui_name in instance.node_group.node_processes] if processes is not None and len(processes) > 0: packages = self._get_packages(cluster_context, processes) cmd = cluster_context.distro.create_install_cmd(packages) with instance.remote() as r: r.execute_command(cmd, run_as_root=True, timeout=_INSTALL_PACKAGES_TIMEOUT) def _get_packages(self, cluster_context, node_processes): result = [] result += self.dependencies result += [(np.package, self.version) for np in node_processes] return result def _set_service_dir_owner(self, cluster_context, instances): service_instances = cluster_context.filter_instances(instances, service=self) LOG.debug("Changing %s service dir owner", self.ui_name) for instance in service_instances: cmd.chown(instance, 'mapr:mapr', self.service_dir(cluster_context)) def post_install(self, cluster_context, instances): pass def post_start(self, cluster_context, instances): pass def configure(self, cluster_context, instances=None): pass def update(self, cluster_context, instances=None): pass def get_file_path(self, file_name): template = 'plugins/mapr/services/%(service)s/resources/%(file_name)s' args = {'service': self.name, 'file_name': file_name} return template % args def get_configs(self): result = [] for d_file in self.cluster_defaults: data = self._load_config_file(self.get_file_path(d_file)) result += [self._create_config_obj(c, self.ui_name) for c in data] for d_file in self.node_defaults: data = self._load_config_file(self.get_file_path(d_file)) result += [self._create_config_obj(c, self.ui_name, scope='node') for c in data] return result def get_configs_dict(self): result = dict() for conf_obj in self.get_configs(): result.update({conf_obj.name: conf_obj.default_value}) return {self.ui_name: result} def _load_config_file(self, file_path=None): return json.loads(files.get_file_text(file_path)) def get_config_files(self, cluster_context, configs, instance=None): return [] def _create_config_obj(self, item, target='general', scope='cluster', high_priority=False): def _prepare_value(value): if isinstance(value, str): return value.strip().lower() return value conf_name = _prepare_value(item.get('name', None)) conf_value = _prepare_value(item.get('value', None)) if not conf_name: raise ex.HadoopProvisionError(_("Config missing 'name'")) if conf_value is None: raise e.InvalidDataException( _("Config '%s' missing 'value'") % conf_name) if high_priority or item.get('priority', 2) == 1: priority = 1 else: priority = 2 return p.Config( name=conf_name, applicable_target=target, scope=scope, config_type=item.get('config_type', "string"), config_values=item.get('config_values', None), default_value=conf_value, is_optional=item.get('is_optional', True), description=item.get('description', None), priority=priority) def get_version_config(self, versions): return p.Config( name='%s Version' % self._ui_name, applicable_target=self.ui_name, scope='cluster', config_type='dropdown', config_values=[(v, v) for v in sorted(versions, reverse=True)], is_optional=False, description=_('Specify the version of the service'), priority=1) def __eq__(self, other): if isinstance(other, self.__class__): version_eq = self.version == other.version ui_name_eq = self.ui_name == other.ui_name return version_eq and ui_name_eq return NotImplemented def restart(self, instances): for node_process in self.node_processes: filtered_instances = su.filter_by_node_process(instances, node_process) if filtered_instances: node_process.restart(filtered_instances) def service_dir(self, cluster_context): args = {'mapr_home': cluster_context.mapr_home, 'name': self.name} return '%(mapr_home)s/%(name)s' % args def home_dir(self, cluster_context): args = { 'service_dir': self.service_dir(cluster_context), 'name': self.name, 'version': self.version, } return '%(service_dir)s/%(name)s-%(version)s' % args def conf_dir(self, cluster_context): return '%s/conf' % self.home_dir(cluster_context) def post_configure_sh(self, cluster_context, instances): pass def post_configure(self, cluster_context, instances): pass sahara-8.0.0/sahara/plugins/mapr/domain/configuration_file.py0000666000175100017510000001327313245514472024367 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import abc import os import re import jinja2 as j2 import six import sahara.exceptions as e from sahara.i18n import _ import sahara.utils.xmlutils as xml @six.add_metaclass(abc.ABCMeta) class FileAttr(object): def __init__(self, path, data, mode, owner): self.path = path self.data = data self.mode = mode self.owner = owner @six.add_metaclass(abc.ABCMeta) class BaseConfigurationFile(object): def __init__(self, file_name): self.f_name = file_name self._config_dict = dict() self._local_path = None self._remote_path = None self.mode = None self.owner = None @property def remote_path(self): return self._remote_path @remote_path.setter def remote_path(self, path): self._remote_path = os.path.join(path, self.f_name) @abc.abstractmethod def render(self): pass @abc.abstractmethod def parse(self, content): pass def fetch(self, instance): with instance.remote() as r: content = r.read_file_from(self.remote_path, run_as_root=True) self.parse(content) def load_properties(self, config_dict): for k, v in six.iteritems(config_dict): self.add_property(k, v) def add_property(self, name, value): self._config_dict[name] = value def add_properties(self, properties): for prop in six.iteritems(properties): self.add_property(*prop) def _get_config_value(self, name): return self._config_dict.get(name, None) def __repr__(self): return '' % self.f_name class HadoopXML(BaseConfigurationFile): def __init__(self, file_name): super(HadoopXML, self).__init__(file_name) def parse(self, content): configs = xml.parse_hadoop_xml_with_name_and_value(content) for cfg in configs: self.add_property(cfg["name"], cfg["value"]) def render(self): return xml.create_hadoop_xml(self._config_dict) class RawFile(BaseConfigurationFile): def __init__(self, file_name): super(RawFile, self).__init__(file_name) def render(self): return self._config_dict.get('content', '') def parse(self, content): self._config_dict.update({'content': content}) class PropertiesFile(BaseConfigurationFile): def __init__(self, file_name, separator='='): super(PropertiesFile, self).__init__(file_name) self.separator = separator def parse(self, content): for line in content.splitlines(): prop = line.strip() if len(prop) == 0: continue if prop[0] in ['#', '!']: continue name_value = prop.split(self.separator, 1) name = name_value[0] # check whether the value is empty value = name_value[1] if (len(name_value) == 2) else '' self.add_property(name.strip(), value.strip()) def render(self): lines = ['%s%s%s' % (k, self.separator, v) for k, v in six.iteritems(self._config_dict)] return "\n".join(lines) + '\n' class TemplateFile(BaseConfigurationFile): def __init__(self, file_name): super(TemplateFile, self).__init__(file_name) self._template = None @staticmethod def _j2_render(template, arg_dict): if template: return template.render(arg_dict) else: raise e.InvalidDataException(_('Template object must be defined')) def render(self): return self._j2_render(self._template, self._config_dict) def parse(self, content): self._template = j2.Template(content) class EnvironmentConfig(BaseConfigurationFile): def __init__(self, file_name): super(EnvironmentConfig, self).__init__(file_name) self._lines = [] self._regex = re.compile(r'export\s+(\w+)=(.+)') self._tmpl = 'export %s="%s"' def parse(self, content): for line in content.splitlines(): line = self._escape(line) match = self._regex.match(line) if match: name, value = match.groups() value = value.replace("\"", '') self._lines.append((name, value)) self.add_property(name, value) else: self._lines.append(line) @staticmethod def _escape(string): try: string = string.decode("utf-8") except AttributeError: pass string = six.text_type(string).strip() string = string.replace("\"", "") return string def render(self): result = [] for line in self._lines: if isinstance(line, tuple): name, value = line args = (name, self._config_dict.get(name) or value) result.append(self._tmpl % args) if name in self._config_dict: del self._config_dict[name] else: result.append(line) extra_ops = [self._tmpl % i for i in six.iteritems(self._config_dict)] return '\n'.join(result + extra_ops) + '\n' sahara-8.0.0/sahara/plugins/mapr/util/0000775000175100017510000000000013245515027017642 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/util/general.py0000666000175100017510000001510713245514472021642 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslo_utils import uuidutils from sahara.conductor import objects from sahara import context import sahara.utils.files as files def unique_list(iterable, mapper=lambda i: i): result = [] for item in iterable: value = mapper(item) if value not in result: result.append(value) return result def _run_as(user, command): if not user: return command return 'sudo -u %(user)s %(command)s' % {'user': user, 'command': command} def unique_file_name(base='/tmp'): return '%(base)s/%(uuid)s' % {'base': base, 'uuid': uuidutils.generate_uuid()} def remove(instance, path, recursive=True, run_as=None): with instance.remote() as r: args = {'recursive': '-r' if recursive else '', 'path': path} r.execute_command(_run_as(run_as, 'rm %(recursive)s %(path)s' % args)) def create_archive(instance, path, archive=None, run_as=None): if not archive: archive = unique_file_name('/tmp') args = {'archive': archive, 'path': path} tar = 'tar cf %(archive)s -C %(path)s .' % args with instance.remote() as r: r.execute_command(_run_as(run_as, tar)) return archive def unpack_archive(instance, src, dest, cleanup=False, run_as=None): with instance.remote() as r: r.execute_command(_run_as(run_as, 'mkdir -p %s' % dest)) untar = 'tar xf %(src)s -C %(dest)s' % {'src': src, 'dest': dest} r.execute_command(_run_as(run_as, untar)) if cleanup: r.execute_command(_run_as(run_as, 'rm -r %s' % src)) def copy_file(s_path, s_instance, d_path, d_instance, run_as=None, owner=None): with s_instance.remote() as sr: data = sr.read_file_from(s_path, run_as_root=(run_as == 'root')) with d_instance.remote() as dr: write_file(dr, d_path, data, owner=owner) def copy_dir(s_path, s_instance, d_path, d_instance, run_as=None): s_path = create_archive(s_instance, s_path, run_as=run_as) tmp_path = unique_file_name('/tmp') copy_file(s_path, s_instance, tmp_path, d_instance, run_as) unpack_archive(d_instance, tmp_path, d_path, True, run_as) remove(s_instance, s_path, True, run_as) def copy(s_path, s_instance, d_path, d_instance, run_as=None): if is_directory(s_instance, s_path): copy_dir(s_path, s_instance, d_path, d_instance, run_as) else: copy_file(s_path, s_instance, d_path, d_instance, run_as) def run_script(instance, script, run_as=None, *args, **kwargs): with instance.remote() as r: path = '/tmp/%s.sh' % uuidutils.generate_uuid() script = files.get_file_text(script) % kwargs r.write_file_to(path, script, run_as_root=(run_as == 'root')) r.execute_command(_run_as(run_as, 'chmod +x %s' % path)) r.execute_command(_run_as(run_as, '%s %s' % (path, ' '.join(args))), timeout=3600) # FIXME(aosadchyi): reuse existing remote remove(instance, path, run_as=run_as) def execute_on_instances(instances, function, *args, **kwargs): with context.ThreadGroup() as tg: for instance in instances: t_name = '%s-execution' % function.__name__ tg.spawn(t_name, function, instance, *args, **kwargs) def _replace(args, position, value): return args[:position] + (value,) + args[position + 1:] def remote_command(position): def wrap(func): def wrapped(*args, **kwargs): target = args[position] if isinstance(target, objects.Instance): with target.remote() as remote: return func(*_replace(args, position, remote), **kwargs) return func(*args, **kwargs) return wrapped return wrap def execute_command(instances, command, run_as=None): def _execute_command(instance): with instance.remote() as remote: remote.execute_command(_run_as(run_as, command), timeout=1800) execute_on_instances(instances, _execute_command) @remote_command(0) def is_directory(remote, path): command = '[ -d %s ]' % path ec = remote.execute_command(command, True, raise_when_error=False)[0] return not ec @remote_command(0) def chown(remote, owner, path): args = {'owner': owner, 'path': path} remote.execute_command('chown -R %(owner)s %(path)s' % args, True) @remote_command(0) def chmod(remote, mode, path): args = {'mode': mode, 'path': path} remote.execute_command('chmod -R %(mode)s %(path)s' % args, True) @remote_command(0) def mkdir(remote, path, mode=None, owner=''): args = {'mode': '-m %s' % mode if mode else '', 'path': path} remote.execute_command('mkdir -p %(mode)s %(path)s' % args, bool(owner)) if owner: chown(remote, owner, path) @remote_command(0) def write_file(remote, path, data, mode=None, owner=''): remote.write_file_to(path, data, run_as_root=bool(owner)) if mode: chmod(remote, mode, path) if owner: chown(remote, owner, path) @remote_command(0) def install_ssh_key(remote, user, private_key, public_key): ssh_dir = '/home/%s/.ssh' % user owner = '%s:%s' % (user, user) if not is_directory(remote, ssh_dir): mkdir(remote, ssh_dir, 700, owner) write_file(remote, '%s/id_rsa.pub' % ssh_dir, public_key, 644, owner) write_file(remote, '%s/id_rsa' % ssh_dir, private_key, 600, owner) @remote_command(0) def authorize_key(remote, user, public_key): authorized_keys = '/home/%s/.ssh/authorized_keys' % user remote.append_to_file(authorized_keys, public_key, run_as_root=True) @remote_command(0) def download(remote, url, output=None, run_as=None): args = {'url': url, 'output': '-O %s' % output if output else ''} command = _run_as(run_as, 'wget -q %(output)s %(url)s' % args) remote.execute_command(command, timeout=600) class Singleton(type): instances = {} def __call__(cls, *args, **kwargs): if cls not in cls.instances: new_instance = super(Singleton, cls).__call__(*args, **kwargs) cls.instances[cls] = new_instance return cls.instances[cls] sahara-8.0.0/sahara/plugins/mapr/util/password_utils.py0000666000175100017510000000443513245514472023311 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils from sahara import conductor from sahara import context from sahara.service.castellan import utils as key_manager MAPR_USER_PASSWORD = 'mapr_password' conductor = conductor.API def delete_password(cluster, pw_name): """delete the named password from the key manager This function will lookup the named password in the cluster entry and delete it from the key manager. :param cluster: The cluster record containing the password :param pw_name: The name associated with the password """ ctx = context.ctx() cluster = conductor.cluster_get(ctx, cluster.id) key_id = cluster.extra.get(pw_name) if cluster.extra else None if key_id is not None: key_manager.delete_key(key_id, ctx) def get_password(cluster, pw_name): """return a password for the named entry This function will return, or create and return, a password for the named entry. It will store the password in the key manager and use the ID in the database entry. :param cluster: The cluster record containing the password :param pw_name: The entry name associated with the password :returns: The cleartext password """ ctx = context.ctx() cluster = conductor.cluster_get(ctx, cluster.id) passwd = cluster.extra.get(pw_name) if cluster.extra else None if passwd: return key_manager.get_secret(passwd, ctx) passwd = uuidutils.generate_uuid() extra = cluster.extra.to_dict() if cluster.extra else {} extra[pw_name] = key_manager.store_secret(passwd, ctx) cluster = conductor.cluster_update(ctx, cluster, {'extra': extra}) return passwd def get_mapr_password(cluster): return get_password(cluster, MAPR_USER_PASSWORD) sahara-8.0.0/sahara/plugins/mapr/util/validation_utils.py0000666000175100017510000002314213245514472023575 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import functools as ft from sahara.conductor import resource as r import sahara.exceptions as ex from sahara.i18n import _ import sahara.plugins.exceptions as e from sahara.service.api import v10 as api import sahara.utils.openstack.nova as nova class LessThanCountException(e.InvalidComponentCountException): MESSAGE = _("Hadoop cluster should contain at least" " %(expected_count)d %(component)s component(s)." " Actual %(component)s count is %(actual_count)d") def __init__(self, component, expected_count, count): super(LessThanCountException, self).__init__( component, expected_count, count) args = { 'expected_count': expected_count, 'component': component, 'actual_count': count, } self.message = LessThanCountException.MESSAGE % args class MoreThanCountException(e.InvalidComponentCountException): MESSAGE = _("Hadoop cluster should contain at most" " %(expected_count)d %(component)s component(s)." " Actual %(component)s count is %(actual_count)d") def __init__(self, component, expected_count, count): super(MoreThanCountException, self).__init__( component, expected_count, count) args = { "expected_count": expected_count, "component": component, "actual_count": count, } self.message = MoreThanCountException.MESSAGE % args class EvenCountException(ex.SaharaException): MESSAGE = _("Hadoop cluster should contain odd number of %(component)s" " but %(actual_count)s found.") def __init__(self, component, count): super(EvenCountException, self).__init__() args = {'component': component, 'actual_count': count} self.message = EvenCountException.MESSAGE % args class NodeRequiredServiceMissingException(e.RequiredServiceMissingException): MISSING_MSG = _('Node "%(ng_name)s" is missing component %(component)s') REQUIRED_MSG = _('%(message)s, required by %(required_by)s') def __init__(self, service_name, ng_name, required_by=None): super(NodeRequiredServiceMissingException, self).__init__( service_name, required_by) args = {'ng_name': ng_name, 'component': service_name} self.message = ( NodeRequiredServiceMissingException.MISSING_MSG % args) if required_by: args = {'message': self.message, 'required_by': required_by} self.message = ( NodeRequiredServiceMissingException.REQUIRED_MSG % args) class NodeServiceConflictException(ex.SaharaException): MESSAGE = _('%(service)s service cannot be installed alongside' ' %(package)s package') ERROR_CODE = "NODE_PROCESS_CONFLICT" def __init__(self, service_name, conflicted_package): super(NodeServiceConflictException, self).__init__() args = { 'service': service_name, 'package': conflicted_package, } self.message = NodeServiceConflictException.MESSAGE % args self.code = NodeServiceConflictException.ERROR_CODE class NoVolumesException(ex.SaharaException): MESSAGE = _('%s must have at least 1 volume or ephemeral drive') ERROR_CODE = "NO_VOLUMES" def __init__(self, ng_name): super(NoVolumesException, self).__init__() self.message = NoVolumesException.MESSAGE % ng_name self.code = NoVolumesException.ERROR_CODE class NotRequiredImageException(ex.SaharaException): MESSAGE = _('Service %(service)s requires %(os)s OS.' ' Use %(os)s image and add "%(os)s" tag to it.') ERROR_CODE = "INVALID_IMAGE" def __init__(self, service, os): super(NotRequiredImageException, self).__init__() self.message = NotRequiredImageException.MESSAGE % {'service': service, 'os': os} self.code = NotRequiredImageException.ERROR_CODE def at_least(count, component): def validate(cluster_context, component, count): actual_count = cluster_context.get_instances_count(component) if not actual_count >= count: raise LessThanCountException( component.ui_name, count, actual_count) return ft.partial(validate, component=component, count=count) def at_most(count, component): def validate(cluster_context, component, count): actual_count = cluster_context.get_instances_count(component) if actual_count > count: raise MoreThanCountException( component.ui_name, count, actual_count) return ft.partial(validate, component=component, count=count) def exactly(count, component): def validate(cluster_context, component, count): actual_count = cluster_context.get_instances_count(component) if not actual_count == count: raise e.InvalidComponentCountException( component.ui_name, count, actual_count) return ft.partial(validate, component=component, count=count) def each_node_has(component): def validate(cluster_context, component): for node_group in cluster_context.cluster.node_groups: if component.ui_name not in node_group.node_processes: raise NodeRequiredServiceMissingException( component.ui_name, node_group.name) return ft.partial(validate, component=component) def odd_count_of(component): def validate(cluster_context, component): actual_count = cluster_context.get_instances_count(component) if actual_count > 1 and actual_count % 2 == 0: raise EvenCountException(component.ui_name, actual_count) return ft.partial(validate, component=component) def on_same_node(component, dependency): def validate(cluster_context, component, dependency): for ng in cluster_context.get_node_groups(component): if dependency.ui_name not in ng.node_processes: raise NodeRequiredServiceMissingException( dependency.ui_name, ng.name, component.ui_name) return ft.partial(validate, component=component, dependency=dependency) def depends_on(service, required_by): def validate(cluster_context, service, required_by): if not cluster_context.is_present(service): service_name = service.ui_name if service.version: service_name += " %s" % service.version raise e.RequiredServiceMissingException( service_name, required_by.ui_name) return ft.partial(validate, service=service, required_by=required_by) def node_client_package_conflict_vr(components, client_component): def validate(cluster_context, components): for ng in cluster_context.get_node_groups(): for c in components: nps = ng.node_processes if c in nps and client_component in nps: raise NodeServiceConflictException(c, client_component) return ft.partial(validate, components=components) def assert_present(service, cluster_context): if not cluster_context.is_present(service): raise e.RequiredServiceMissingException(service.ui_name) def required_os(os, required_by): def validate(cluster_context, os, required_by): for ng in cluster_context.get_node_groups(): nps = ng.node_processes for node_process in required_by.node_processes: if node_process.ui_name in nps: image_id = (ng.image_id or cluster_context.cluster.default_image_id) if not image_has_tag(image_id, os): raise NotRequiredImageException(required_by.ui_name, os) return ft.partial(validate, os=os, required_by=required_by) def create_fake_cluster(cluster, existing, additional): counts = existing.copy() counts.update(additional) def update_ng(node_group): ng_dict = node_group.to_dict() count = counts[node_group.id] ng_dict.update(dict(count=count)) return r.NodeGroupResource(ng_dict) def need_upd(node_group): return node_group.id in counts and counts[node_group.id] > 0 updated = map(update_ng, filter(need_upd, cluster.node_groups)) not_updated = filter(lambda ng: not need_upd(ng) and ng is not None, cluster.node_groups) cluster_dict = cluster.to_dict() cluster_dict.update({'node_groups': updated + not_updated}) fake = r.ClusterResource(cluster_dict) return fake def get_ephemeral(node_group): return nova.get_flavor(id=node_group.flavor_id).ephemeral def has_volumes(): def validate(cluster_context): for node_group in cluster_context.cluster.node_groups: if not (node_group.volumes_per_node or get_ephemeral(node_group)): raise NoVolumesException(node_group.name) return validate def image_has_tag(image_id, tag): image = api.get_registered_image(image_id) return tag in image.tags sahara-8.0.0/sahara/plugins/mapr/util/service_utils.py0000666000175100017510000000326613245514472023110 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import six from sahara.i18n import _ def get_node_process_name(node_process): # This import is placed here to avoid circular imports from sahara.plugins.mapr.domain import node_process as np # noqa if isinstance(node_process, np.NodeProcess): return node_process.ui_name if isinstance(node_process, six.string_types): return node_process raise TypeError(_("Invalid argument type %s") % type(node_process)) def has_node_process(instance, node_process): node_process_name = get_node_process_name(node_process) instance_node_processes = instance.node_group.node_processes return node_process_name in instance_node_processes def has_service(instance, service): return any(has_node_process(instance, node_process) for node_process in service.node_processes) def filter_by_node_process(instances, node_process): return [instance for instance in instances if has_node_process(instance, node_process)] def filter_by_service(instances, service): return [instance for instance in instances if has_service(instance, service)] sahara-8.0.0/sahara/plugins/mapr/util/maprfs_helper.py0000666000175100017510000000764613245514472023065 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import os from oslo_utils import uuidutils import sahara.plugins.mapr.util.general as g MV_TO_MAPRFS_CMD = ('sudo -u %(user)s' ' hadoop fs -copyFromLocal %(source)s %(target)s' ' && sudo rm -f %(source)s') CP_TO_MAPRFS_CMD = ('sudo -u %(user)s' ' hadoop fs -copyFromLocal %(source)s %(target)s') CP_FROM_MAPRFS_CMD = ('sudo hadoop fs -copyToLocal %(source)s %(target)s') RM_CMD = ('sudo -u %(user)s hadoop fs -rm %(path)s') MKDIR_CMD_MAPR4 = 'sudo -u %(user)s hadoop fs -mkdir -p %(path)s' MKDIR_CMD_MAPR3 = 'sudo -u %(user)s hadoop fs -mkdir %(path)s' def put_file_to_maprfs(r, content, file_name, path, hdfs_user): tmp_file_name = '/tmp/%s.%s' % (file_name, uuidutils.generate_uuid()) r.write_file_to(tmp_file_name, content) target = os.path.join(path, file_name) move_from_local(r, tmp_file_name, target, hdfs_user) def move_from_local(r, source, target, hdfs_user): args = {'user': hdfs_user, 'source': source, 'target': target} r.execute_command(MV_TO_MAPRFS_CMD % args) def copy_from_local(r, source, target, hdfs_user): args = {'user': hdfs_user, 'source': source, 'target': target} r.execute_command(CP_TO_MAPRFS_CMD % args) def copy_to_local(r, hdfs_source, target, hdfs_user, overwrite=True): args = {'source': hdfs_source, 'target': target} if overwrite: _rm_from_local(r, target) r.execute_command(CP_FROM_MAPRFS_CMD % args) def exchange(source, target, src_path, trg_path, hdfs_user, alias_dir="/"): # check if target path contains file name, if not set file same as source if trg_path[-1] == '/': trg_path = trg_path + src_path.rsplit('/', 1)[1] alias = alias_dir + _generate_file_name() copy_from_local(source, src_path, alias, hdfs_user) copy_to_local(target, alias, trg_path, hdfs_user) remove(source, alias, hdfs_user) def remove(r, path, hdfs_user): args = {'user': hdfs_user, 'path': path} r.execute_command(RM_CMD % args) def create_maprfs4_dir(remote, dir_name, hdfs_user): remote.execute_command(MKDIR_CMD_MAPR4 % {'user': hdfs_user, 'path': dir_name}) def create_maprfs3_dir(remote, dir_name, hdfs_user): remote.execute_command(MKDIR_CMD_MAPR3 % {'user': hdfs_user, 'path': dir_name}) def mkdir(remote, path, recursive=True, run_as=None): command = 'hadoop fs -mkdir %(recursive)s %(path)s' args = {'recursive': '-p' if recursive else '', 'path': path} remote.execute_command(g._run_as(run_as, command % args)) def chmod(remote, path, mode, recursive=True, run_as=None): command = 'hadoop fs -chmod %(recursive)s %(mode)s %(path)s' args = {'recursive': '-R' if recursive else '', 'path': path, 'mode': mode} remote.execute_command(g._run_as(run_as, command % args)) def _generate_file_name(length=10): return uuidutils.generate_uuid()[:length] def _rm_from_local(remote, file_name, exist=False): if exist: remote.execute_command("rm %s" % file_name, run_as_root=True) else: out, result = remote.execute_command("ls %s" % file_name, run_as_root=True, raise_when_error=False) if out == 0: remote.execute_command("rm %s" % file_name, run_as_root=True) sahara-8.0.0/sahara/plugins/mapr/util/__init__.py0000666000175100017510000000000013245514472021746 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/util/event_log.py0000666000175100017510000000421413245514472022204 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from sahara.utils import cluster_progress_ops as cpo def provision_step(name, cluster_context_reference=1, instances_reference=2): def wrapper(function): def wrapped(*args, **kwargs): cluster_context = _find_argument( cluster_context_reference, *args, **kwargs) instances = _find_argument(instances_reference, *args, **kwargs) cluster_id = cluster_context.cluster.id instance_count = len(instances) cpo.add_provisioning_step(cluster_id, name, instance_count) return function(*args, **kwargs) return wrapped return wrapper def provision_event(instance_reference=0, name=None, instance=None): def wrapper(function): def wrapped(*args, **kwargs): event_instance = instance or _find_argument(instance_reference, *args, **kwargs) if name: cpo.add_provisioning_step(event_instance.node_group.cluster.id, name, 1) try: result = function(*args, **kwargs) cpo.add_successful_event(event_instance) return result except Exception as exception: cpo.add_fail_event(event_instance, exception) raise exception return wrapped return wrapper def _find_argument(reference, *args, **kwargs): if isinstance(reference, int): return args[reference] if isinstance(reference, str): return kwargs[reference] sahara-8.0.0/sahara/plugins/mapr/util/commands.py0000666000175100017510000000175213245514472022027 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. def chown(instance, owner, path, run_as_root=True): cmd = 'chown -R %(owner)s %(path)s' % {'owner': owner, 'path': path} with instance.remote() as r: r.execute_command(cmd, run_as_root=run_as_root) def re_configure_sh(instance, cluster_context): with instance.remote() as r: command = '%s -R' % cluster_context.configure_sh_path r.execute_command(command, run_as_root=True) sahara-8.0.0/sahara/plugins/mapr/resources/0000775000175100017510000000000013245515027020677 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/resources/install_java.sh0000666000175100017510000000512413245514472023711 0ustar zuulzuul00000000000000#!/bin/bash set -e JAVA_TARGET_LOCATION="/usr/java" export JAVA_DOWNLOAD_URL=${JAVA_DOWNLOAD_URL:-"http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.tar.gz"} JAVA_HOME=$TARGET_ROOT$JAVA_TARGET_LOCATION mkdir -p $JAVA_HOME JAVA_FILE=$(basename $JAVA_DOWNLOAD_URL) wget --no-check-certificate --no-cookies -c \ --header "Cookie: gpw_e24=http://www.oracle.com/; \ oraclelicense=accept-securebackup-cookie" \ -O $JAVA_HOME/$JAVA_FILE $JAVA_DOWNLOAD_URL if [ $? -eq 0 ]; then echo "Java download successful" else echo "Error downloading $JAVA_DOWNLOAD_URL, exiting" exit 1 fi cd $JAVA_HOME if [[ $JAVA_FILE == *.tar.gz ]]; then echo -e "\n" | tar -zxf $JAVA_FILE JAVA_NAME=`ls -1 $JAVA_TARGET_LOCATION | grep -v tar.gz` chown -R root:root $JAVA_HOME cat >> /etc/profile.d/java.sh <<- EOF # Custom Java install export JAVA_HOME=$JAVA_TARGET_LOCATION/$JAVA_NAME export PATH=\$PATH:$JAVA_TARGET_LOCATION/$JAVA_NAME/bin EOF case "$1" in Ubuntu ) update-alternatives --install "/usr/bin/java" "java" \ "$JAVA_TARGET_LOCATION/$JAVA_NAME/bin/java" 1 update-alternatives --install "/usr/bin/javac" "javac" \ "$JAVA_TARGET_LOCATION/$JAVA_NAME/bin/javac" 1 update-alternatives --install "/usr/bin/javaws" "javaws" \ "$JAVA_TARGET_LOCATION/$JAVA_NAME/bin/javaws" 1 update-alternatives --set java \ $JAVA_TARGET_LOCATION/$JAVA_NAME/bin/java update-alternatives --set javac \ $JAVA_TARGET_LOCATION/$JAVA_NAME/bin/javac update-alternatives --set javaws \ $JAVA_TARGET_LOCATION/$JAVA_NAME/bin/javaws ;; Fedora | RedHatEnterpriseServer | CentOS ) alternatives --install /usr/bin/java java \ $JAVA_TARGET_LOCATION/$JAVA_NAME/bin/java 200000 alternatives --install /usr/bin/javaws javaws \ $JAVA_TARGET_LOCATION/$JAVA_NAME/bin/javaws 200000 alternatives --install /usr/bin/javac javac \ $JAVA_TARGET_LOCATION/$JAVA_NAME/bin/javac 200000 alternatives --install /usr/bin/jar jar \ $JAVA_TARGET_LOCATION/$JAVA_NAME/bin/jar 200000 ;; esac elif [[ $JAVA_FILE == *.bin ]]; then echo -e "\n" | sh $JAVA_FILE else echo "Unknown file type: $JAVA_FILE, exiting" exit 1 fi rm $JAVA_FILE sahara-8.0.0/sahara/plugins/mapr/resources/topology.sh0000666000175100017510000000060113245514472023111 0ustar zuulzuul00000000000000#!/bin/bash MAPR_HOME=/opt/mapr while [ $# -gt 0 ] ; do nodeArg=$1 exec< ${MAPR_HOME}/topology.data result="" while read line ; do ar=( $line ) if [ "${ar[0]}" = "$nodeArg" ]; then result="${ar[1]}" fi done shift if [ -z "$result" ]; then echo -n "/default/rack " else echo -n "$result " fi done sahara-8.0.0/sahara/plugins/mapr/resources/install_mysql_client.sh0000666000175100017510000000056613245514472025500 0ustar zuulzuul00000000000000#!/bin/bash if [[ $1 == *"Ubuntu"* ]]; then sudo apt-get install --force-yes -y mysql-client libmysql-java elif [[ $1 == *"CentOS"* ]] || [[ $1 == "RedHatEnterpriseServer" ]]; then sudo yum install -y mysql elif [[ $1 == *"SUSE"* ]]; then sudo zypper install mysql-community-server-client mysql-connector-java else echo "Unknown distribution" exit 1 fi sahara-8.0.0/sahara/plugins/mapr/resources/create_disk_list_file.sh0000666000175100017510000000035713245514472025554 0ustar zuulzuul00000000000000#!/bin/bash disk_list_file=/tmp/disk.list if [ -f ${disk_list_file} ]; then rm -f ${disk_list_file} fi for path in $*; do device=`findmnt ${path} -cno SOURCE` umount -f ${device} echo ${device} >> ${disk_list_file} done sahara-8.0.0/sahara/plugins/mapr/resources/install_scala.sh0000666000175100017510000000116013245514472024047 0ustar zuulzuul00000000000000#!/bin/bash #Current available version DEF_VERSION="2.11.5" VERSION=$(wget -qO- http://www.scala-lang.org |\ grep 'scala-version' | grep -Eo '([0-9]\.?)+') if [ $? != 0 -o -z ${VERSION} ]; then VERSION=${DEF_VERSION} fi PKG=scala-${VERSION} URL="http://downloads.typesafe.com/scala/${VERSION}" if [ "$1" = "Ubuntu" ]; then wget -N ${URL}/${PKG}.deb dpkg -i ${PKG}.deb rm ${PKG}.deb # install java if missing apt-get install -f -y --force-yes elif [ "$1" = 'CentOS' -o "$1" = 'RedHatEnterpriseServer' ]; then rpm -Uhv ${URL}/${PKG}.rpm else echo "Unknown distribution" exit 1 fi sahara-8.0.0/sahara/plugins/mapr/resources/add_mapr_repo.sh0000666000175100017510000000200313245514472024027 0ustar zuulzuul00000000000000#!/bin/sh if [ "$1" = "Ubuntu" ]; then sudo apt-get update cat >> /etc/apt/sources.list.d/maprtech.list <<- EOF deb %(ubuntu_mapr_base_repo)s deb %(ubuntu_mapr_ecosystem_repo)s EOF sudo apt-get install -y --force-yes wget wget -O - http://package.mapr.com/releases/pub/maprgpg.key | \ sudo apt-key add - sudo apt-get update elif [ "$1" = 'CentOS' -o "$1" = 'RedHatEnterpriseServer' ]; then cat >> /etc/yum.repos.d/maprtech.repo <<- EOF [maprtech] name=MapR Technologies baseurl=%(centos_mapr_base_repo)s enabled=1 gpgcheck=0 protect=1 [maprecosystem] name=MapR Technologies baseurl=%(centos_mapr_ecosystem_repo)s enabled=1 gpgcheck=0 protect=1 EOF rpm --import http://package.mapr.com/releases/pub/maprgpg.key yum install -y wget if [ ! -e /etc/yum.repos.d/epel.repo ]; then rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm fi else echo "Unknown distribution" exit 1 fi sahara-8.0.0/sahara/plugins/mapr/resources/images/0000775000175100017510000000000013245515027022144 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/resources/images/common/0000775000175100017510000000000013245515027023434 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/resources/images/common/oracle_java0000666000175100017510000000263013245514472025633 0ustar zuulzuul00000000000000#!/bin/bash # NOTE: $(dirname $0) is read-only, use space under $TARGET_ROOT JAVA_LOCATION=${JAVA_TARGET_LOCATION:-"/usr/java"} JAVA_NAME="oracle-jdk" JAVA_HOME=$JAVA_LOCATION/$JAVA_NAME JAVA_DOWNLOAD_URL=${JAVA_DOWNLOAD_URL:-"http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.tar.gz"} # FIXME: probably not idempotent, find a better condition if [ ! -d $JAVA_LOCATION ]; then if [ $test_only -eq 0 ]; then echo "Begin: installation of Java" mkdir -p $JAVA_LOCATION if [ -n "$JAVA_DOWNLOAD_URL" ]; then JAVA_FILE=$(basename $JAVA_DOWNLOAD_URL) wget --no-check-certificate --no-cookies -c \ --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \ -O $JAVA_LOCATION/$JAVA_FILE $JAVA_DOWNLOAD_URL elif [ -n "$JAVA_FILE" ]; then install -D -g root -o root -m 0755 $(dirname $0)/$JAVA_FILE $JAVA_LOCATION fi cd $JAVA_LOCATION echo "Decompressing Java archive" echo -e "\n" | tar -zxf $JAVA_FILE echo "Setting up $JAVA_NAME" chown -R root:root $JAVA_LOCATION JAVA_DIR=`ls -1 $JAVA_LOCATION | grep -v tar.gz` ln -s $JAVA_LOCATION/$JAVA_DIR $JAVA_HOME setup-java-home $JAVA_HOME $JAVA_HOME rm $JAVA_FILE echo "End: installation of Java" else exit 0 fi fi sahara-8.0.0/sahara/plugins/mapr/resources/images/common/resources/0000775000175100017510000000000013245515027025446 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/resources/images/common/resources/spec_5.1.0.json0000666000175100017510000000071413245514472030023 0ustar zuulzuul00000000000000{ "drill": [ "1.1.0", "1.2.0", "1.4.0" ], "flume": [ "1.5.0", "1.6.0" ], "hbase": [ "0.98.9", "0.98.12" ], "hive": [ "0.13", "1.0", "1.2" ], "httpfs": [ "1.0" ], "hue": [ "3.8.1", "3.9.0" ], "impala": [ "1.4.1" ], "mahout": [ "0.10.0" ], "oozie": [ "4.2.0" ], "pig": [ "0.14", "0.15" ], "sqoop": [ "2.0.0" ], "spark": [ "1.5.2" ] } sahara-8.0.0/sahara/plugins/mapr/resources/images/common/resources/versions.py0000666000175100017510000000444513245514472027704 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import argparse import sys from oslo_serialization import jsonutils as json _GROUP_VERSION_SEPARATOR = "," _ALL_GROUP_VERSION = "all" def _build_parser(): parser = argparse.ArgumentParser() parser.add_argument("packages", help="path to the packages.json") parser.add_argument("spec", help="path to the spec.json") parser.add_argument("--separator", default=":", help="separator between package name" " and version in output") return parser def _load_json(path): with open(path) as json_file: return json.load(json_file) def _version_matches(version, group_version): for gv in group_version.split(_GROUP_VERSION_SEPARATOR): if version.startswith(gv): return True return False def _get_packages(version, group_spec): for group_version in group_spec: if _version_matches(version, group_version): return group_spec[group_version] return group_spec[_ALL_GROUP_VERSION] def _get_package_versions(spec, package_groups): return [(package, version) for pg_name, versions in spec.items() for version in versions for package in _get_packages(version, package_groups[pg_name])] parser = _build_parser() def main(args=None): args = parser.parse_args(args or sys.argv[1:]) spec = _load_json(args.spec) package_groups = _load_json(args.packages) separator = args.separator package_versions = _get_package_versions(spec, package_groups) package_format = "%s" + separator + "%s\n" package_versions = map(lambda pv: package_format % pv, package_versions) sys.stdout.writelines(package_versions) if __name__ == "__main__": main() sahara-8.0.0/sahara/plugins/mapr/resources/images/common/resources/spec_5.2.0_ubuntu.json0000666000175100017510000000071513245514472031427 0ustar zuulzuul00000000000000{ "drill": [ "1.9.0" ], "flume": [ "1.6.0" ], "hbase": [ "1.1.1" ], "hive": [ "1.2" ], "httpfs": [ "1.0" ], "hue": [ "3.10.0" ], "mahout": [ "0.12.0" ], "oozie": [ "4.2.0" ], "pig": [ "0.16" ], "sqoop": [ "2.0.0" ], "spark": [ "2.0.1" ], "sentry": [ "1.6.0" ], "kafka": [ "0.9.0" ], "kafka-connect": [ "2.0.1" ], "kafka-rest": [ "2.0.1" ] } sahara-8.0.0/sahara/plugins/mapr/resources/images/common/resources/spec_5.2.0.json0000666000175100017510000000075413245514472030030 0ustar zuulzuul00000000000000{ "drill": [ "1.9.0" ], "flume": [ "1.6.0" ], "hbase": [ "1.1.1" ], "hive": [ "1.2" ], "httpfs": [ "1.0" ], "hue": [ "3.10.0" ], "impala": [ "2.5.0" ], "mahout": [ "0.12.0" ], "oozie": [ "4.2.0" ], "pig": [ "0.16" ], "sqoop": [ "2.0.0" ], "spark": [ "2.0.1" ], "sentry": [ "1.6.0" ], "kafka": [ "0.9.0" ], "kafka-connect": [ "2.0.1" ], "kafka-rest": [ "2.0.1" ] } sahara-8.0.0/sahara/plugins/mapr/resources/images/common/resources/package_utils.sh0000666000175100017510000001454513245514472030633 0ustar zuulzuul00000000000000# execute_in_directory execute_in_directory() { local directory="$(readlink -f "$1")"; shift local cmd="$*" pushd "$directory" && eval "$cmd" && popd } # get_distro get_distro() { echo "$DISTRO_NAME" } # download_apt_package [version] [directory] download_apt_package() { local package="$1" local version="${2:-}" local directory="${3:-$(pwd)}" local package_spec="$package${version:+=$version*}" execute_in_directory "$directory" apt-get --allow-unauthenticated download "$package_spec" } # download_yum_package [version] [directory] download_yum_package() { local package="$1" local version="${2:-}" local directory="${3:-$(pwd)}" local package_spec="$package${version:+-$version*}" yumdownloader --destdir "$directory" "$package_spec" } # download_package [version] [directory] [distro] download_package() { local package="$1" local version="${2:-}" local directory="${3:-$(pwd)}" local distro="${4:-$(get_distro)}" if [[ "$distro" == "ubuntu" ]]; then download_apt_package "$package" "$version" "$directory" elif [[ "$distro" == "centos" || "$distro" == "centos7" || "$distro" == "rhel" || "$distro" == "rhel7" ]]; then download_yum_package "$package" "$version" "$directory" fi } # get_packages [version_separator] get_packages() { local package_groups_file="$1" local spec_file="$2" local version_separator="${3:-:}" python "$VERSIONS_PY" --separator "$version_separator" "$package_groups_file" "$spec_file" } # download_packages [directory] [distro] download_packages() { local package_groups_file="$1" local spec_file="$2" local directory="${3:-$(pwd)}" local distro="${4:-$(get_distro)}" local version_separator=":" local packages="$(get_packages "$package_groups_file" "$spec_file" "$version_separator")" for package in $packages; do IFS="$version_separator" read -ra package_version <<< "$package" download_package "${package_version[@]}" "$directory" "$distro" done } # create_apt_repo create_apt_repo() { local directory="$(readlink -f "$1")" local binary_dir="$directory/binary" local packages_gz="$binary_dir/Packages.gz" mkdir -p "$binary_dir" execute_in_directory "$directory" "dpkg-scanpackages -m . /dev/null | gzip -9c > $packages_gz" } # create_yum_repo create_yum_repo() { local directory="$(readlink -f "$1")" createrepo "$directory" } # create_repo [distro] create_repo() { local directory="$(readlink -f "$1")" local distro="${2:-$(get_distro)}" if [[ "$distro" == "ubuntu" ]]; then create_apt_repo "$directory" elif [[ "$distro" == "centos" || "$distro" == "centos7" || "$distro" == "rhel" || "$distro" == "rhel7" ]]; then create_yum_repo "$directory" fi } # add_apt_repo add_apt_repo() { local repo_name="$1" local repo_url="$2" local repo="deb $repo_url" local repo_path="/etc/apt/sources.list.d/$repo_name.list" echo "$repo" > "$repo_path" && apt-get update } # add_yum_repo add_yum_repo() { local repo_name="$1" local repo_url="$2" local repo_path="/etc/yum.repos.d/$repo_name.repo" cat > "$repo_path" << EOF [$repo_name] name=$repo_name baseurl=$repo_url enabled=1 gpgcheck=0 protect=1 EOF yum clean all && rm -rf /var/cache/yum/* && yum check-update } # add_repo [distro] add_repo() { local repo_name="$1" local repo_url="$2" local distro="${3:-$(get_distro)}" if [[ "$distro" == "ubuntu" ]]; then add_apt_repo "$repo_name" "$repo_url" elif [[ "$distro" == "centos" || "$distro" == "centos7" || "$distro" == "rhel" || "$distro" == "rhel7" ]]; then add_yum_repo "$repo_name" "$repo_url" fi } # add_local_apt_repo add_local_apt_repo() { local repo_name="$1" local directory="$(readlink -f "$2")" local repo_url="file:$directory binary/" add_apt_repo "$repo_name" "$repo_url" } # add_local_yum_repo add_local_yum_repo() { local repo_name="$1" local directory="$(readlink -f "$2")" local repo_url="file://$directory" add_yum_repo "$repo_name" "$repo_url" } # add_local_repo [distro] add_local_repo() { local repo_name="$1" local directory="$(readlink -f "$2")" local distro="${3:-$(get_distro)}" if [[ "$distro" == "ubuntu" ]]; then add_local_apt_repo "$repo_name" "$directory" elif [[ "$distro" == "centos" || "$distro" == "centos7" || "$distro" == "rhel" || "$distro" == "rhel7" ]]; then add_local_yum_repo "$repo_name" "$directory" fi } # remove_apt_repo remove_apt_repo() { local repo_name="$1" local repo_path="/etc/apt/sources.list.d/$repo_name.list" rm "$repo_path" && apt-get update } # remove_yum_repo remove_yum_repo() { local repo_name="$1" local repo_path="/etc/yum.repos.d/$repo_name.repo" rm "$repo_path" } # remove_repo [distro] remove_repo() { local repo_name="$1" local distro="${2:-$(get_distro)}" if [[ "$distro" == "ubuntu" ]]; then remove_apt_repo "$repo_name" elif [[ "$distro" == "centos" || "$distro" == "centos7" || "$distro" == "rhel" || "$distro" == "rhel7" ]]; then remove_yum_repo "$repo_name" fi } # create_local_repo create_local_repo() { local repo_name="$1" local repo_url="$2" local package_groups_file="$3" local spec_file="$4" local directory="$5" add_repo "$repo_name" "$repo_url" mkdir -p "$directory" && directory="$(readlink -f "$directory")" download_packages "$package_groups_file" "$spec_file" "$directory" remove_repo "$repo_name" create_repo "$directory" } # localize_repo localize_repo() { local repo_name="$1" local repo_url="$2" local package_groups_file="$3" local spec_file="$4" local directory="$5" mkdir -p "$directory" && directory="$(readlink -f "$directory")" create_local_repo "$repo_name" "$repo_url" "$package_groups_file" "$spec_file" "$directory" add_local_repo "$repo_name" "$directory" } sahara-8.0.0/sahara/plugins/mapr/resources/images/common/resources/packages.json0000666000175100017510000000417113245514472030127 0ustar zuulzuul00000000000000{ "asynchbase": { "all": [ "mapr-asynchbase" ] }, "drill": { "all": [ "mapr-drill" ] }, "flume": { "all": [ "mapr-flume" ] }, "hbase": { "all": [ "mapr-hbase", "mapr-hbase-internal", "mapr-hbase-master", "mapr-hbase-regionserver", "mapr-hbasethrift", "mapr-hbase-rest" ], "0.98.12": [ "mapr-hbase", "mapr-hbase-internal", "mapr-hbase-master", "mapr-hbase-regionserver", "mapr-hbasethrift", "mapr-libhbase", "mapr-hbase-rest" ], "1.1.1": [ "mapr-hbase", "mapr-hbase-internal", "mapr-hbase-master", "mapr-hbase-regionserver", "mapr-hbasethrift", "mapr-libhbase", "mapr-hbase-rest" ] }, "hive": { "all": [ "mapr-hive", "mapr-hivemetastore", "mapr-hiveserver2" ] }, "httpfs": { "all": [ "mapr-httpfs" ] }, "hue": { "all": [ "mapr-hue", "mapr-hue-base", "mapr-hue-livy" ], "3.10.0": [ "mapr-hue", "mapr-hue-livy" ] }, "impala": { "all": [ "mapr-impala", "mapr-impala-catalog", "mapr-impala-server", "mapr-impala-statestore", "mapr-impala-udf" ] }, "mahout": { "all": [ "mapr-mahout" ] }, "oozie": { "all": [ "mapr-oozie", "mapr-oozie-internal" ] }, "pig": { "all": [ "mapr-pig" ] }, "sentry": { "all": [ "mapr-sentry" ] }, "spark": { "all": [ "mapr-spark", "mapr-spark-historyserver", "mapr-spark-master" ] }, "sqoop": { "all": [ "mapr-sqoop2-client", "mapr-sqoop2-server" ] }, "storm": { "all": [ "mapr-storm", "mapr-storm-ui", "mapr-storm-nimbus", "mapr-storm-supervisor" ] }, "tez": { "all": [ "mapr-tez" ] }, "kafka": { "all": [ "mapr-kafka" ] }, "kafka-connect": { "all": [ "mapr-kafka-connect-hdfs", "mapr-kafka-connect-jdbc" ] }, "kafka-rest": { "all": [ "mapr-kafka-rest" ] } } sahara-8.0.0/sahara/plugins/mapr/resources/images/common/configure_extjs0000666000175100017510000000122513245514476026566 0ustar zuulzuul00000000000000#!/bin/bash EXTJS_DESTINATION_DIR="/opt/mapr-repository" EXTJS_DOWNLOAD_URL="http://tarballs.openstack.org/sahara/dist/common-artifacts/ext-2.2.zip" EXTJS_NO_UNPACK=1 extjs_archive=/tmp/$(basename $EXTJS_DOWNLOAD_URL) if [ ! -f "${EXTJS_DESTINATION_DIR}/${extjs_archive}" ]; then if [ $test_only -eq 0 ]; then wget -O $extjs_archive $EXTJS_DOWNLOAD_URL mkdir -p $EXTJS_DESTINATION_DIR if [ -z "${EXTJS_NO_UNPACK:-}" ]; then unzip -d "$EXTJS_DESTINATION_DIR" $extjs_archive rm -f $extjs_archive else mv $extjs_archive $EXTJS_DESTINATION_DIR fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/mapr/resources/images/image.yaml0000666000175100017510000001330413245514472024120 0ustar zuulzuul00000000000000arguments: java_distro: default: openjdk description: The distribution of Java to install. Defaults to cloudera-jdk. choices: - openjdk - oracle-java plugin_version: default: 5.2.0 description: The distribution of MapR to install. Defaults to 5.2.0. hidden: True required: False scala_version: default: 2.11.6 description: The version of scala to install. Defaults to 2.11.6. hidden: True required: False hdfs_lib_dir: default: /usr/lib/hadoop-mapreduce description: The path to HDFS_LIB_DIR. Default to /usr/lib/hadoop-mapreduce required: False validators: - os_case: - ubuntu: - script: ubuntu/install_mapr_dependencies - package: - mtools - rpcbind - sdparm - syslinux - unzip - wget - zip - os_case: - centos: - package: - cups - cdparanoia-libs - cups-libs - createrepo - cvs - cyrus-sasl-gssapi - cyrus-sasl-plain - foomatic - foomatic-db - foomatic-db-filesystem - foomatic-db-ppds - gdbm-devel - gettext - ghostscript - ghostscript-fonts - glibc - glibc-common - glibc-devel - glibc-headers - gstreamer - gstreamer-plugins-base - gstreamer-tools - hdparm - irqbalance - iso-codes - kernel-headers - libXt - libXv - libXxf86vm - libgomp - libgudev1 - libicu - libmng - liboil - libtheora - libtirpc - libvisual - libxslt - mesa-dri-drivers - mesa-libGL - mesa-libGLU - mesa-private-llvm - nmap-ncat - numactl - openjpeg-libs - patch - pax - perl-CGI - perl-ExtUtils-MakeMaker - perl-ExtUtils-ParseXS - perl-Test-Harness - perl-Test-Simple - perl-devel - phonon-backend-gstreamer - poppler - poppler-data - poppler-utils - portreserve - qt - qt-x11 - qt3 - redhat-lsb - redhat-lsb-core - redhat-lsb-printing - urw-fonts - yum-utils - xml-common - ubuntu: - package: - binutils - daemon - dpkg-dev - dpkg-repack - gcc - gcc-4.8 - gcc-doc - gcc-multilib - iputils-arping - libasan0 - libatomic1 - libc-dev-bin - libc6 - libc6-dev - libcrypt-passwdmd5-perl - libgcc-4.8-dev - libgomp1 - libgssglue1 - libicu48 - libitm1 - libmysqlclient-dev - libmysqlclient16 - libmysqlclient18 - libnfsidmap2 - libquadmath0 - libsasl2-dev - libsasl2-modules-gssapi-mit - libssl0.9.8 - libtirpc1 - libtsan0 - libxslt1.1 - linux-libc-dev - manpages-dev - mysql-common - nfs-common - open-iscsi - openjdk-7-jre - syslinux-common - zlib1g-dev - script: common/configure_extjs - os_case: - centos: - copy_script: common/resources/package_utils.sh - copy_script: common/resources/packages.json - copy_script: common/resources/spec_5.1.0.json - copy_script: common/resources/spec_5.2.0.json - copy_script: common/resources/versions.py - script: centos/install_scala: env_vars: [scala_version] - script: centos/install_mapr_core_repository: env_vars: [plugin_version] - script: centos/install_mapr_eco_repository: env_vars: [plugin_version] - script: centos/selinux_permissive - argument_case: argument_name: java_distro cases: openjdk: - script: centos/install_openjdk oracle-java: - script: common/oracle_java - ubuntu: - copy_script: common/resources/package_utils.sh - copy_script: common/resources/packages.json - copy_script: common/resources/spec_5.1.0.json - copy_script: common/resources/spec_5.2.0.json - copy_script: common/resources/spec_5.2.0_ubuntu.json - copy_script: common/resources/versions.py - script: ubuntu/install_scala: env_vars: [scala_version] - script: ubuntu/install_mapr_core_repository: env_vars: [plugin_version] - script: ubuntu/install_mapr_eco_repository: env_vars: [plugin_version] - os_case: - ubuntu: - argument_case: argument_name: java_distro cases: openjdk: - script: ubuntu/install_openjdk oracle-java: - script: common/oracle_java sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/0000775000175100017510000000000013245515027023437 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/install_openjdk0000666000175100017510000000040313245514472026544 0ustar zuulzuul00000000000000#!/bin/bash echo "Installing OpenJDK" if [ $test_only -eq 0 ]; then yum install -y java-1.8.0-openjdk-devel JRE_HOME="/usr/lib/jvm/java-openjdk/jre" JDK_HOME="/usr/lib/jvm/java-openjdk" echo "OpenJDK has been installed" else exit 0 fi sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/update_repository0000666000175100017510000000012013245514472027141 0ustar zuulzuul00000000000000#!/bin/bash if [ $test_only -eq 0 ]; then yum clean all && yum repolist fi sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/install_scala0000666000175100017510000000147513245514472026207 0ustar zuulzuul00000000000000#!/bin/bash echo "START: installing Scala" sudo yum -y update exit 0 if [ $test_only -eq 0 ]; then RETURN_CODE="$(curl -s -o /dev/null -w "%{http_code}" http://www.scala-lang.org/)" if [ "$RETURN_CODE" != "200" ]; then echo "http://www.scala-lang.org is unreachable" && exit 1 fi if [ -n "${scala_version:-}" ]; then VERSION=$scala_version else VERSION="$(curl -s --fail http://www.scala-lang.org| tr -d '\n' | sed 's/^.*[^0-9]\+\([0-9\.\?]\+\)<.\+$/\1/')" if [ $? != 0 -o -z "${VERSION}" ]; then echo "Installing default version $scala_version" VERSION=$scala_version fi fi PKG=scala-${VERSION} URL="http://downloads.lightbend.com/scala/${VERSION}" rpm -Uhv ${URL}/${PKG}.rpm fi echo "END: installing Scala" sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/install_mapr_eco_repository0000666000175100017510000000136613245514472031207 0ustar zuulzuul00000000000000#!/bin/bash VERSIONS_PY="/tmp/versions.py" DISTRO_NAME=$distro source "/tmp/package_utils.sh" echo "START: installing MapR ecosystem repository" MAPR_REPO_URL="http://package.mapr.com/releases/MEP/MEP-2.0.0/redhat/" MAPR_REPO_DIR="/opt/mapr-repository/ecosystem" if [ ! -d "$MAPR_REPO_DIR" ] || [ -z "$(ls -A $MAPR_REPO_DIR)" ]; then if [ $test_only -eq 0 ]; then MAPR_REPO_NAME="mapr_ecosystem" MAPR_PKG_GROUPS="/tmp/packages.json" MAPR_SPEC="/tmp/spec_$plugin_version.json" echo "Creating local MapR ecosystem repository" localize_repo "$MAPR_REPO_NAME" "$MAPR_REPO_URL" "$MAPR_PKG_GROUPS" "$MAPR_SPEC" "$MAPR_REPO_DIR" echo $MAPR_SPEC fi fi echo "END: installing MapR ecosystem repository" sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/selinux_permissive0000666000175100017510000000044513245514472027327 0ustar zuulzuul00000000000000#!/bin/bash check=$(cat /etc/selinux/config | grep "SELINUX=permissive" | wc -l) if [ $check -eq 0 ]; then if [ $test_only -eq 0 ]; then echo "SELINUX=permissive" > /etc/selinux/config echo "SELINUXTYPE=targeted" >> /etc/selinux/config else exit 0 fi fi sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/install_mapr_core_repository0000666000175100017510000000141113245514472031360 0ustar zuulzuul00000000000000#!/bin/bash DISTRO_NAME=$distro source "/tmp/package_utils.sh" echo "START: installing MapR core repository" MAPR_REPO_URL="http://package.mapr.com/releases/v${plugin_version}/redhat/mapr-v${plugin_version}GA.rpm.tgz" MAPR_REPO_DIR="/opt/mapr-repository/core" if [ ! -d "$MAPR_REPO_DIR" ] || [ -z "$(ls -A $MAPR_REPO_DIR)" ]; then if [ $test_only -eq 0 ]; then MAPR_REPO_NAME="mapr_core" echo "Downloading MapR repository archive" mkdir -p "$MAPR_REPO_DIR" && curl "$MAPR_REPO_URL" | tar -xz -C "$MAPR_REPO_DIR" echo "Creating local repository" create_repo "$MAPR_REPO_DIR" echo "Adding MapR repository" add_local_repo "$MAPR_REPO_NAME" "$MAPR_REPO_DIR" fi fi echo "END: installing MapR core repository" sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/disable_firewall0000666000175100017510000000103513245514472026656 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files iptables.service | grep 'enabled' | wc -l) if [ $check -eq 1 ]; then if [ $test_only -eq 0 ]; then if type -p systemctl && [[ "$(systemctl --no-pager list-unit-files firewalld)" =~ 'enabled' ]]; then systemctl disable firewalld fi if type -p service; then service ip6tables save service iptables save chkconfig ip6tables off chkconfig iptables off fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/mapr/resources/images/centos/requiretty0000666000175100017510000000015313245514472025603 0ustar zuulzuul00000000000000#!/bin/bash if [ $test_only -eq 0 ]; then sed '/^Defaults requiretty*/ s/^/#/' -i /etc/sudoers fi sahara-8.0.0/sahara/plugins/mapr/resources/images/ubuntu/0000775000175100017510000000000013245515027023466 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/mapr/resources/images/ubuntu/install_openjdk0000666000175100017510000000051613245514472026600 0ustar zuulzuul00000000000000#!/bin/bash echo "Installing OpenJDK" if [ $test_only -eq 0 ]; then apt-get install -y openjdk-7-jdk JRE_PATH=$(update-alternatives --list java) JRE_HOME=${JRE_PATH%/bin/java} JDK_PATH=$(update-alternatives --list javac) JDK_HOME=${JDK_PATH%/bin/javac} echo "OpenJDK has been installed" else exit 0 fi sahara-8.0.0/sahara/plugins/mapr/resources/images/ubuntu/update_repository0000666000175100017510000000010113245514472027167 0ustar zuulzuul00000000000000#!/bin/bash if [ $test_only -eq 0 ]; then apt-get update fi sahara-8.0.0/sahara/plugins/mapr/resources/images/ubuntu/install_scala0000666000175100017510000000151213245514472026226 0ustar zuulzuul00000000000000#!/bin/bash echo "START: installing Scala" if [ $test_only -eq 0 ]; then RETURN_CODE="$(curl -s -o /dev/null -w "%{http_code}" http://www.scala-lang.org/)" if [ "$RETURN_CODE" != "200" ]; then echo "http://www.scala-lang.org is unreachable" && exit 1 fi if [ -n "${scala_version:-}" ]; then VERSION=$scala_version else VERSION="$(curl -s --fail http://www.scala-lang.org| tr -d '\n' | sed 's/^.*[^0-9]\+\([0-9\.\?]\+\)<.\+$/\1/')" if [ $? != 0 -o -z "${VERSION}" ]; then echo "Installing default version $scala_version" VERSION=$scala_version fi fi PKG=scala-${VERSION} URL="http://downloads.lightbend.com/scala/${VERSION}" wget -N ${URL}/${PKG}.deb dpkg -i ${PKG}.deb rm ${PKG}.deb fi echo "END: installing Scala" sahara-8.0.0/sahara/plugins/mapr/resources/images/ubuntu/install_mapr_eco_repository0000666000175100017510000000163513245514472031235 0ustar zuulzuul00000000000000#!/bin/bash VERSIONS_PY="/tmp/versions.py" DISTRO_NAME=$distro source "/tmp/package_utils.sh" echo "START: installing MapR ecosystem repository" MAPR_REPO_URL="http://package.mapr.com/releases/MEP/MEP-2.0.0/ubuntu/ binary trusty" MAPR_REPO_DIR="/opt/mapr-repository/ecosystem" if [ ! -d "$MAPR_REPO_DIR" ] || [ -z "$(ls -A $MAPR_REPO_DIR)" ]; then if [ $test_only -eq 0 ]; then MAPR_REPO_NAME="mapr_ecosystem" MAPR_PKG_GROUPS="/tmp/packages.json" if [ -f /tmp/spec_$plugin_version_ubuntu.json ]; then MAPR_SPEC="/tmp/spec_$plugin_version_ubuntu.json" else MAPR_SPEC="/tmp/spec_$plugin_version.json" fi echo "Creating local MapR ecosystem repository" localize_repo "$MAPR_REPO_NAME" "$MAPR_REPO_URL" "$MAPR_PKG_GROUPS" "$MAPR_SPEC" "$MAPR_REPO_DIR" echo $MAPR_SPEC fi fi echo "END: installing MapR ecosystem repository" sahara-8.0.0/sahara/plugins/mapr/resources/images/ubuntu/install_mapr_dependencies0000666000175100017510000000111213245514472030604 0ustar zuulzuul00000000000000#!/bin/bash echo "START: installing MapR core dependencies" if [ ! -f /etc/apt/sources.list.d/security_repo.list ]; then if [ $test_only -eq 0 ]; then # Required for libicu48 cat >> /etc/apt/sources.list.d/security_repo.list << EOF deb http://security.ubuntu.com/ubuntu precise-security main EOF # Required for libmysqlclient16 cat >> /etc/apt/sources.list.d/security_repo.list << EOF deb http://old-releases.ubuntu.com/ubuntu lucid-security main EOF else exit 0 fi fi apt-get update echo "END: installing MapR core dependencies" sahara-8.0.0/sahara/plugins/mapr/resources/images/ubuntu/install_mapr_core_repository0000666000175100017510000000141113245514472031407 0ustar zuulzuul00000000000000#!/bin/bash DISTRO_NAME=$distro source "/tmp/package_utils.sh" echo "START: installing MapR core repository" MAPR_REPO_URL="http://package.mapr.com/releases/v${plugin_version}/ubuntu/mapr-v${plugin_version}GA.deb.tgz" MAPR_REPO_DIR="/opt/mapr-repository/core" if [ ! -d "$MAPR_REPO_DIR" ] || [ -z "$(ls -A $MAPR_REPO_DIR)" ]; then if [ $test_only -eq 0 ]; then MAPR_REPO_NAME="mapr_core" echo "Downloading MapR repository archive" mkdir -p "$MAPR_REPO_DIR" && curl "$MAPR_REPO_URL" | tar -xz -C "$MAPR_REPO_DIR" echo "Creating local repository" create_repo "$MAPR_REPO_DIR" echo "Adding MapR repository" add_local_repo "$MAPR_REPO_NAME" "$MAPR_REPO_DIR" fi fi echo "END: installing MapR core repository" sahara-8.0.0/sahara/plugins/mapr/resources/install_mysql.sh0000666000175100017510000000230413245514472024132 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -f /etc/init.d/mysql* ]; then if [[ $1 == *"Ubuntu"* ]]; then sudo debconf-set-selections <<< \ 'mysql-server mysql-server/root_password password root' sudo debconf-set-selections <<< \ 'mysql-server mysql-server/root_password_again password root' sudo apt-get install --force-yes -y mysql-server sudo apt-get install --force-yes -y libmysqlclient16 mysql -uroot -proot mysql -e "UPDATE user SET Password=PASSWORD('') \ WHERE User='root'; FLUSH PRIVILEGES;" sudo sed -i "s/^\(bind-address\s*=\s*\).*\$/\10.0.0.0/" \ /etc/mysql/my.cnf sudo service mysql restart elif [[ $1 == *"CentOS"* ]] || \ [[ $1 == "RedHatEnterpriseServer" ]]; then if [[ $2 == "7" ]]; then sudo yum install -y mariadb-server else sudo yum install -y mysql-server fi elif [[ $1 == *"SUSE"* ]]; then sudo zypper mysql-server else echo "Unknown distribution" exit 1 fi else echo "Mysql server already installed" fi sahara-8.0.0/sahara/plugins/mapr/resources/add_security_repos.sh0000666000175100017510000000022413245514472025125 0ustar zuulzuul00000000000000#!/bin/sh cat >> /etc/apt/sources.list.d/security_repo.list <<- EOF deb http://security.ubuntu.com/ubuntu precise-security main EOF apt-get update sahara-8.0.0/sahara/plugins/spark/0000775000175100017510000000000013245515027017046 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/spark/plugin.py0000666000175100017510000005533513245514472020736 0ustar zuulzuul00000000000000# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import os from oslo_config import cfg from oslo_log import log as logging from sahara import conductor from sahara import context from sahara.i18n import _ from sahara.plugins import exceptions as ex from sahara.plugins import provisioning as p from sahara.plugins import recommendations_utils as ru from sahara.plugins.spark import config_helper as c_helper from sahara.plugins.spark import edp_engine from sahara.plugins.spark import run_scripts as run from sahara.plugins.spark import scaling as sc from sahara.plugins.spark import shell_engine from sahara.plugins import utils from sahara.swift import swift_helper from sahara.topology import topology_helper as th from sahara.utils import cluster_progress_ops as cpo from sahara.utils import files as f from sahara.utils import general as ug from sahara.utils import remote conductor = conductor.API LOG = logging.getLogger(__name__) CONF = cfg.CONF class SparkProvider(p.ProvisioningPluginBase): def __init__(self): self.processes = { "HDFS": ["namenode", "datanode"], "Spark": ["master", "slave"] } def get_title(self): return "Apache Spark" def get_description(self): return _("This plugin provides an ability to launch Spark on Hadoop " "CDH cluster without any management consoles.") def get_labels(self): default = {'enabled': {'status': True}, 'stable': {'status': True}} deprecated = {'enabled': {'status': True}, 'deprecated': {'status': True}} result = {'plugin_labels': copy.deepcopy(default)} stable_versions = ['2.2', '2.1.0', '1.6.0'] result['version_labels'] = { version: copy.deepcopy( default if version in stable_versions else deprecated ) for version in self.get_versions() } return result def get_versions(self): return ['2.2', '2.1.0', '1.6.0', '1.3.1'] def get_configs(self, hadoop_version): return c_helper.get_plugin_configs() def get_node_processes(self, hadoop_version): return self.processes def validate(self, cluster): nn_count = sum([ng.count for ng in utils.get_node_groups(cluster, "namenode")]) if nn_count != 1: raise ex.InvalidComponentCountException("namenode", 1, nn_count) dn_count = sum([ng.count for ng in utils.get_node_groups(cluster, "datanode")]) if dn_count < 1: raise ex.InvalidComponentCountException("datanode", _("1 or more"), nn_count) rep_factor = utils.get_config_value_or_default('HDFS', "dfs.replication", cluster) if dn_count < rep_factor: raise ex.InvalidComponentCountException( 'datanode', _('%s or more') % rep_factor, dn_count, _('Number of %(dn)s instances should not be less ' 'than %(replication)s') % {'dn': 'datanode', 'replication': 'dfs.replication'}) # validate Spark Master Node and Spark Slaves sm_count = sum([ng.count for ng in utils.get_node_groups(cluster, "master")]) if sm_count < 1: raise ex.RequiredServiceMissingException("Spark master") if sm_count >= 2: raise ex.InvalidComponentCountException("Spark master", "1", sm_count) sl_count = sum([ng.count for ng in utils.get_node_groups(cluster, "slave")]) if sl_count < 1: raise ex.InvalidComponentCountException("Spark slave", _("1 or more"), sl_count) def update_infra(self, cluster): pass def configure_cluster(self, cluster): self._setup_instances(cluster) @cpo.event_wrapper( True, step=utils.start_process_event_message("NameNode")) def _start_namenode(self, nn_instance): with remote.get_remote(nn_instance) as r: run.format_namenode(r) run.start_processes(r, "namenode") def start_spark(self, cluster): sm_instance = utils.get_instance(cluster, "master") if sm_instance: self._start_spark(cluster, sm_instance) @cpo.event_wrapper( True, step=utils.start_process_event_message("SparkMasterNode")) def _start_spark(self, cluster, sm_instance): with remote.get_remote(sm_instance) as r: run.start_spark_master(r, self._spark_home(cluster)) LOG.info("Spark service has been started") def start_cluster(self, cluster): nn_instance = utils.get_instance(cluster, "namenode") dn_instances = utils.get_instances(cluster, "datanode") # Start the name node self._start_namenode(nn_instance) # start the data nodes self._start_datanode_processes(dn_instances) run.await_datanodes(cluster) LOG.info("Hadoop services have been started") with remote.get_remote(nn_instance) as r: r.execute_command("sudo -u hdfs hdfs dfs -mkdir -p /user/$USER/") r.execute_command("sudo -u hdfs hdfs dfs -chown $USER " "/user/$USER/") # start spark nodes self.start_spark(cluster) swift_helper.install_ssl_certs(utils.get_instances(cluster)) LOG.info('Cluster has been started successfully') self._set_cluster_info(cluster) def _spark_home(self, cluster): return utils.get_config_value_or_default("Spark", "Spark home", cluster) def _extract_configs_to_extra(self, cluster): sp_master = utils.get_instance(cluster, "master") sp_slaves = utils.get_instances(cluster, "slave") extra = dict() config_master = config_slaves = '' if sp_master is not None: config_master = c_helper.generate_spark_env_configs(cluster) if sp_slaves is not None: slavenames = [] for slave in sp_slaves: slavenames.append(slave.hostname()) config_slaves = c_helper.generate_spark_slaves_configs(slavenames) else: config_slaves = "\n" # Any node that might be used to run spark-submit will need # these libs for swift integration config_defaults = c_helper.generate_spark_executor_classpath(cluster) extra['job_cleanup'] = c_helper.generate_job_cleanup_config(cluster) extra['sp_master'] = config_master extra['sp_slaves'] = config_slaves extra['sp_defaults'] = config_defaults if c_helper.is_data_locality_enabled(cluster): topology_data = th.generate_topology_map( cluster, CONF.enable_hypervisor_awareness) extra['topology_data'] = "\n".join( [k + " " + v for k, v in topology_data.items()]) + "\n" return extra def _add_instance_ng_related_to_extra(self, cluster, instance, extra): extra = extra.copy() ng = instance.node_group nn = utils.get_instance(cluster, "namenode") extra['xml'] = c_helper.generate_xml_configs( ng.configuration(), instance.storage_paths(), nn.hostname(), None) extra['setup_script'] = c_helper.generate_hadoop_setup_script( instance.storage_paths(), c_helper.extract_hadoop_environment_confs(ng.configuration())) return extra def _start_datanode_processes(self, dn_instances): if len(dn_instances) == 0: return cpo.add_provisioning_step( dn_instances[0].cluster_id, utils.start_process_event_message("DataNodes"), len(dn_instances)) with context.ThreadGroup() as tg: for i in dn_instances: tg.spawn('spark-start-dn-%s' % i.instance_name, self._start_datanode, i) @cpo.event_wrapper(mark_successful_on_exit=True) def _start_datanode(self, instance): with instance.remote() as r: run.start_processes(r, "datanode") def _setup_instances(self, cluster, instances=None): extra = self._extract_configs_to_extra(cluster) if instances is None: instances = utils.get_instances(cluster) self._push_configs_to_nodes(cluster, extra, instances) def _push_configs_to_nodes(self, cluster, extra, new_instances): all_instances = utils.get_instances(cluster) cpo.add_provisioning_step( cluster.id, _("Push configs to nodes"), len(all_instances)) with context.ThreadGroup() as tg: for instance in all_instances: extra = self._add_instance_ng_related_to_extra( cluster, instance, extra) if instance in new_instances: tg.spawn('spark-configure-%s' % instance.instance_name, self._push_configs_to_new_node, cluster, extra, instance) else: tg.spawn('spark-reconfigure-%s' % instance.instance_name, self._push_configs_to_existing_node, cluster, extra, instance) @cpo.event_wrapper(mark_successful_on_exit=True) def _push_configs_to_new_node(self, cluster, extra, instance): files_hadoop = { os.path.join(c_helper.HADOOP_CONF_DIR, "core-site.xml"): extra['xml']['core-site'], os.path.join(c_helper.HADOOP_CONF_DIR, "hdfs-site.xml"): extra['xml']['hdfs-site'], } sp_home = self._spark_home(cluster) files_spark = { os.path.join(sp_home, 'conf/spark-env.sh'): extra['sp_master'], os.path.join(sp_home, 'conf/slaves'): extra['sp_slaves'], os.path.join(sp_home, 'conf/spark-defaults.conf'): extra['sp_defaults'] } files_init = { '/tmp/sahara-hadoop-init.sh': extra['setup_script'], 'id_rsa': cluster.management_private_key, 'authorized_keys': cluster.management_public_key } # pietro: This is required because the (secret) key is not stored in # .ssh which hinders password-less ssh required by spark scripts key_cmd = ('sudo cp $HOME/id_rsa $HOME/.ssh/; ' 'sudo chown $USER $HOME/.ssh/id_rsa; ' 'sudo chmod 600 $HOME/.ssh/id_rsa') storage_paths = instance.storage_paths() dn_path = ' '.join(c_helper.make_hadoop_path(storage_paths, '/dfs/dn')) nn_path = ' '.join(c_helper.make_hadoop_path(storage_paths, '/dfs/nn')) hdfs_dir_cmd = ('sudo mkdir -p %(nn_path)s %(dn_path)s &&' 'sudo chown -R hdfs:hadoop %(nn_path)s %(dn_path)s &&' 'sudo chmod 755 %(nn_path)s %(dn_path)s' % {"nn_path": nn_path, "dn_path": dn_path}) with remote.get_remote(instance) as r: r.execute_command( 'sudo chown -R $USER:$USER /etc/hadoop' ) r.execute_command( 'sudo chown -R $USER:$USER %s' % sp_home ) r.write_files_to(files_hadoop) r.write_files_to(files_spark) r.write_files_to(files_init) r.execute_command( 'sudo chmod 0500 /tmp/sahara-hadoop-init.sh' ) r.execute_command( 'sudo /tmp/sahara-hadoop-init.sh ' '>> /tmp/sahara-hadoop-init.log 2>&1') r.execute_command(hdfs_dir_cmd) r.execute_command(key_cmd) if c_helper.is_data_locality_enabled(cluster): r.write_file_to( '/etc/hadoop/topology.sh', f.get_file_text( 'plugins/spark/resources/topology.sh')) r.execute_command( 'sudo chmod +x /etc/hadoop/topology.sh' ) self._write_topology_data(r, cluster, extra) self._push_master_configs(r, cluster, extra, instance) self._push_cleanup_job(r, cluster, extra, instance) @cpo.event_wrapper(mark_successful_on_exit=True) def _push_configs_to_existing_node(self, cluster, extra, instance): node_processes = instance.node_group.node_processes need_update_hadoop = (c_helper.is_data_locality_enabled(cluster) or 'namenode' in node_processes) need_update_spark = ('master' in node_processes or 'slave' in node_processes) if need_update_spark: sp_home = self._spark_home(cluster) files = { os.path.join(sp_home, 'conf/spark-env.sh'): extra['sp_master'], os.path.join(sp_home, 'conf/slaves'): extra['sp_slaves'], os.path.join( sp_home, 'conf/spark-defaults.conf'): extra['sp_defaults'] } r = remote.get_remote(instance) r.write_files_to(files) self._push_cleanup_job(r, cluster, extra, instance) if need_update_hadoop: with remote.get_remote(instance) as r: self._write_topology_data(r, cluster, extra) self._push_master_configs(r, cluster, extra, instance) def _write_topology_data(self, r, cluster, extra): if c_helper.is_data_locality_enabled(cluster): topology_data = extra['topology_data'] r.write_file_to('/etc/hadoop/topology.data', topology_data) def _push_master_configs(self, r, cluster, extra, instance): node_processes = instance.node_group.node_processes if 'namenode' in node_processes: self._push_namenode_configs(cluster, r) def _push_cleanup_job(self, r, cluster, extra, instance): node_processes = instance.node_group.node_processes if 'master' in node_processes: if extra['job_cleanup']['valid']: r.write_file_to('/etc/hadoop/tmp-cleanup.sh', extra['job_cleanup']['script']) r.execute_command("chmod 755 /etc/hadoop/tmp-cleanup.sh") cmd = 'sudo sh -c \'echo "%s" > /etc/cron.d/spark-cleanup\'' r.execute_command(cmd % extra['job_cleanup']['cron']) else: r.execute_command("sudo rm -f /etc/hadoop/tmp-cleanup.sh") r.execute_command("sudo rm -f /etc/crond.d/spark-cleanup") def _push_namenode_configs(self, cluster, r): r.write_file_to('/etc/hadoop/dn.incl', utils.generate_fqdn_host_names( utils.get_instances(cluster, "datanode"))) r.write_file_to('/etc/hadoop/dn.excl', '') def _set_cluster_info(self, cluster): nn = utils.get_instance(cluster, "namenode") sp_master = utils.get_instance(cluster, "master") info = {} if nn: address = utils.get_config_value_or_default( 'HDFS', 'dfs.http.address', cluster) port = address[address.rfind(':') + 1:] info['HDFS'] = { 'Web UI': 'http://%s:%s' % (nn.get_ip_or_dns_name(), port) } info['HDFS']['NameNode'] = 'hdfs://%s:8020' % nn.hostname() if sp_master: port = utils.get_config_value_or_default( 'Spark', 'Master webui port', cluster) if port is not None: info['Spark'] = { 'Web UI': 'http://%s:%s' % ( sp_master.get_ip_or_dns_name(), port) } ctx = context.ctx() conductor.cluster_update(ctx, cluster, {'info': info}) # Scaling def validate_scaling(self, cluster, existing, additional): self._validate_existing_ng_scaling(cluster, existing) self._validate_additional_ng_scaling(cluster, additional) def decommission_nodes(self, cluster, instances): sls = utils.get_instances(cluster, "slave") dns = utils.get_instances(cluster, "datanode") decommission_dns = False decommission_sls = False for i in instances: if 'datanode' in i.node_group.node_processes: dns.remove(i) decommission_dns = True if 'slave' in i.node_group.node_processes: sls.remove(i) decommission_sls = True nn = utils.get_instance(cluster, "namenode") spark_master = utils.get_instance(cluster, "master") if decommission_sls: sc.decommission_sl(spark_master, instances, sls) if decommission_dns: sc.decommission_dn(nn, instances, dns) def scale_cluster(self, cluster, instances): master = utils.get_instance(cluster, "master") r_master = remote.get_remote(master) run.stop_spark(r_master, self._spark_home(cluster)) self._setup_instances(cluster, instances) nn = utils.get_instance(cluster, "namenode") run.refresh_nodes(remote.get_remote(nn), "dfsadmin") dn_instances = [instance for instance in instances if 'datanode' in instance.node_group.node_processes] self._start_datanode_processes(dn_instances) swift_helper.install_ssl_certs(instances) run.start_spark_master(r_master, self._spark_home(cluster)) LOG.info("Spark master service has been restarted") def _get_scalable_processes(self): return ["datanode", "slave"] def _validate_additional_ng_scaling(self, cluster, additional): scalable_processes = self._get_scalable_processes() for ng_id in additional: ng = ug.get_by_id(cluster.node_groups, ng_id) if not set(ng.node_processes).issubset(scalable_processes): raise ex.NodeGroupCannotBeScaled( ng.name, _("Spark plugin cannot scale nodegroup" " with processes: %s") % ' '.join(ng.node_processes)) def _validate_existing_ng_scaling(self, cluster, existing): scalable_processes = self._get_scalable_processes() dn_to_delete = 0 for ng in cluster.node_groups: if ng.id in existing: if ng.count > existing[ng.id] and ("datanode" in ng.node_processes): dn_to_delete += ng.count - existing[ng.id] if not set(ng.node_processes).issubset(scalable_processes): raise ex.NodeGroupCannotBeScaled( ng.name, _("Spark plugin cannot scale nodegroup" " with processes: %s") % ' '.join(ng.node_processes)) dn_amount = len(utils.get_instances(cluster, "datanode")) rep_factor = utils.get_config_value_or_default('HDFS', "dfs.replication", cluster) if dn_to_delete > 0 and dn_amount - dn_to_delete < rep_factor: raise ex.ClusterCannotBeScaled( cluster.name, _("Spark plugin cannot shrink cluster because " "there would be not enough nodes for HDFS " "replicas (replication factor is %s)") % rep_factor) def get_edp_engine(self, cluster, job_type): if edp_engine.EdpEngine.job_type_supported(job_type): return edp_engine.EdpEngine(cluster) if shell_engine.ShellEngine.job_type_supported(job_type): return shell_engine.ShellEngine(cluster) return None def get_edp_job_types(self, versions=None): res = {} for vers in self.get_versions(): if not versions or vers in versions: res[vers] = shell_engine.ShellEngine.get_supported_job_types() if edp_engine.EdpEngine.edp_supported(vers): res[vers].extend( edp_engine.EdpEngine.get_supported_job_types()) return res def get_edp_config_hints(self, job_type, version): if (edp_engine.EdpEngine.edp_supported(version) and edp_engine.EdpEngine.job_type_supported(job_type)): return edp_engine.EdpEngine.get_possible_job_config(job_type) if shell_engine.ShellEngine.job_type_supported(job_type): return shell_engine.ShellEngine.get_possible_job_config(job_type) return {} def get_open_ports(self, node_group): cluster = node_group.cluster ports_map = { 'namenode': [8020, 50070, 50470], 'datanode': [50010, 1004, 50075, 1006, 50020], 'master': [ int(utils.get_config_value_or_default("Spark", "Master port", cluster)), int(utils.get_config_value_or_default("Spark", "Master webui port", cluster)), ], 'slave': [ int(utils.get_config_value_or_default("Spark", "Worker webui port", cluster)) ] } ports = [] for process in node_group.node_processes: if process in ports_map: ports.extend(ports_map[process]) return ports def recommend_configs(self, cluster, scaling=False): want_to_configure = { 'cluster_configs': { 'dfs.replication': ('HDFS', 'dfs.replication') } } provider = ru.HadoopAutoConfigsProvider( want_to_configure, self.get_configs( cluster.hadoop_version), cluster, scaling) provider.apply_recommended_configs() sahara-8.0.0/sahara/plugins/spark/config_helper.py0000666000175100017510000004645613245514472022250 0ustar zuulzuul00000000000000# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging import six from sahara import conductor as c from sahara.plugins import provisioning as p from sahara.plugins import utils from sahara.swift import swift_helper as swift from sahara.topology import topology_helper as topology from sahara.utils import files as f from sahara.utils import types from sahara.utils import xmlutils as x conductor = c.API LOG = logging.getLogger(__name__) CONF = cfg.CONF CORE_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/spark/resources/core-default.xml') HDFS_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/spark/resources/hdfs-default.xml') SWIFT_DEFAULTS = swift.read_default_swift_configs() XML_CONFS = { "HDFS": [CORE_DEFAULT, HDFS_DEFAULT, SWIFT_DEFAULTS] } _default_executor_classpath = ":".join( ['/usr/lib/hadoop-mapreduce/hadoop-openstack.jar']) SPARK_CONFS = { 'Spark': { "OPTIONS": [ { 'name': 'Executor extra classpath', 'description': 'Value for spark.executor.extraClassPath' ' in spark-defaults.conf' ' (default: %s)' % _default_executor_classpath, 'default': '%s' % _default_executor_classpath, 'priority': 2, }, { 'name': 'Master port', 'description': 'Start the master on a different port' ' (default: 7077)', 'default': '7077', 'priority': 2, }, { 'name': 'Worker port', 'description': 'Start the Spark worker on a specific port' ' (default: random)', 'default': 'random', 'priority': 2, }, { 'name': 'Master webui port', 'description': 'Port for the master web UI (default: 8080)', 'default': '8080', 'priority': 1, }, { 'name': 'Worker webui port', 'description': 'Port for the worker web UI (default: 8081)', 'default': '8081', 'priority': 1, }, { 'name': 'Worker cores', 'description': 'Total number of cores to allow Spark' ' applications to use on the machine' ' (default: all available cores)', 'default': 'all', 'priority': 2, }, { 'name': 'Worker memory', 'description': 'Total amount of memory to allow Spark' ' applications to use on the machine, e.g. 1000m,' ' 2g (default: total memory minus 1 GB)', 'default': 'all', 'priority': 1, }, { 'name': 'Worker instances', 'description': 'Number of worker instances to run on each' ' machine (default: 1)', 'default': '1', 'priority': 2, }, { 'name': 'Spark home', 'description': 'The location of the spark installation' ' (default: /opt/spark)', 'default': '/opt/spark', 'priority': 2, }, { 'name': 'Minimum cleanup seconds', 'description': 'Job data will never be purged before this' ' amount of time elapses (default: 86400 = 1 day)', 'default': '86400', 'priority': 2, }, { 'name': 'Maximum cleanup seconds', 'description': 'Job data will always be purged after this' ' amount of time elapses (default: 1209600 = 14 days)', 'default': '1209600', 'priority': 2, }, { 'name': 'Minimum cleanup megabytes', 'description': 'No job data will be purged unless the total' ' job data exceeds this size (default: 4096 = 4GB)', 'default': '4096', 'priority': 2, }, ] } } HADOOP_CONF_DIR = "/etc/hadoop/conf" ENV_CONFS = { "HDFS": { 'Name Node Heap Size': 'HADOOP_NAMENODE_OPTS=\\"-Xmx%sm\\"', 'Data Node Heap Size': 'HADOOP_DATANODE_OPTS=\\"-Xmx%sm\\"' } } ENABLE_DATA_LOCALITY = p.Config('Enable Data Locality', 'general', 'cluster', config_type="bool", priority=1, default_value=True, is_optional=True) ENABLE_SWIFT = p.Config('Enable Swift', 'general', 'cluster', config_type="bool", priority=1, default_value=True, is_optional=False) DATANODES_STARTUP_TIMEOUT = p.Config( 'DataNodes startup timeout', 'general', 'cluster', config_type='int', priority=1, default_value=10800, is_optional=True, description='Timeout for DataNodes startup, in seconds') # Default set to 1 day, which is the default Keystone token # expiration time. After the token is expired we can't continue # scaling anyway. DECOMMISSIONING_TIMEOUT = p.Config('Decommissioning Timeout', 'general', 'cluster', config_type='int', priority=1, default_value=86400, is_optional=True, description='Timeout for datanode' ' decommissioning operation' ' during scaling, in seconds') HIDDEN_CONFS = ['fs.defaultFS', 'dfs.namenode.name.dir', 'dfs.datanode.data.dir'] CLUSTER_WIDE_CONFS = ['dfs.block.size', 'dfs.permissions', 'dfs.replication', 'dfs.replication.min', 'dfs.replication.max', 'io.file.buffer.size'] PRIORITY_1_CONFS = ['dfs.datanode.du.reserved', 'dfs.datanode.failed.volumes.tolerated', 'dfs.datanode.max.xcievers', 'dfs.datanode.handler.count', 'dfs.namenode.handler.count'] # for now we have not so many cluster-wide configs # lets consider all of them having high priority PRIORITY_1_CONFS += CLUSTER_WIDE_CONFS def _initialise_configs(): configs = [] for service, config_lists in six.iteritems(XML_CONFS): for config_list in config_lists: for config in config_list: if config['name'] not in HIDDEN_CONFS: cfg = p.Config(config['name'], service, "node", is_optional=True, config_type="string", default_value=str(config['value']), description=config['description']) if cfg.default_value in ["true", "false"]: cfg.config_type = "bool" cfg.default_value = (cfg.default_value == 'true') elif types.is_int(cfg.default_value): cfg.config_type = "int" cfg.default_value = int(cfg.default_value) if config['name'] in CLUSTER_WIDE_CONFS: cfg.scope = 'cluster' if config['name'] in PRIORITY_1_CONFS: cfg.priority = 1 configs.append(cfg) for service, config_items in six.iteritems(ENV_CONFS): for name, param_format_str in six.iteritems(config_items): configs.append(p.Config(name, service, "node", default_value=1024, priority=1, config_type="int")) for service, config_items in six.iteritems(SPARK_CONFS): for item in config_items['OPTIONS']: cfg = p.Config(name=item["name"], description=item["description"], default_value=item["default"], applicable_target=service, scope="cluster", is_optional=True, priority=item["priority"]) configs.append(cfg) configs.append(DECOMMISSIONING_TIMEOUT) configs.append(ENABLE_SWIFT) configs.append(DATANODES_STARTUP_TIMEOUT) if CONF.enable_data_locality: configs.append(ENABLE_DATA_LOCALITY) return configs # Initialise plugin Hadoop configurations PLUGIN_CONFIGS = _initialise_configs() def get_plugin_configs(): return PLUGIN_CONFIGS def generate_cfg_from_general(cfg, configs, general_config, rest_excluded=False): if 'general' in configs: for nm in general_config: if nm not in configs['general'] and not rest_excluded: configs['general'][nm] = general_config[nm]['default_value'] for name, value in configs['general'].items(): if value: cfg = _set_config(cfg, general_config, name) LOG.debug("Applying config: {name}".format(name=name)) else: cfg = _set_config(cfg, general_config) return cfg def _get_hostname(service): return service.hostname() if service else None def generate_xml_configs(configs, storage_path, nn_hostname, hadoop_port): if hadoop_port is None: hadoop_port = 8020 cfg = { 'fs.defaultFS': 'hdfs://%s:%s' % (nn_hostname, str(hadoop_port)), 'dfs.namenode.name.dir': extract_hadoop_path(storage_path, '/dfs/nn'), 'dfs.datanode.data.dir': extract_hadoop_path(storage_path, '/dfs/dn'), 'dfs.hosts': '/etc/hadoop/dn.incl', 'dfs.hosts.exclude': '/etc/hadoop/dn.excl' } # inserting user-defined configs for key, value in extract_hadoop_xml_confs(configs): cfg[key] = value # Add the swift defaults if they have not been set by the user swft_def = [] if is_swift_enabled(configs): swft_def = SWIFT_DEFAULTS swift_configs = extract_name_values(swift.get_swift_configs()) for key, value in six.iteritems(swift_configs): if key not in cfg: cfg[key] = value # invoking applied configs to appropriate xml files core_all = CORE_DEFAULT + swft_def if CONF.enable_data_locality: cfg.update(topology.TOPOLOGY_CONFIG) # applying vm awareness configs core_all += topology.vm_awareness_core_config() xml_configs = { 'core-site': x.create_hadoop_xml(cfg, core_all), 'hdfs-site': x.create_hadoop_xml(cfg, HDFS_DEFAULT) } return xml_configs def _get_spark_opt_default(opt_name): for opt in SPARK_CONFS["Spark"]["OPTIONS"]: if opt_name == opt["name"]: return opt["default"] return None def generate_spark_env_configs(cluster): configs = [] # master configuration sp_master = utils.get_instance(cluster, "master") configs.append('SPARK_MASTER_IP=' + sp_master.hostname()) # point to the hadoop conf dir so that Spark can read things # like the swift configuration without having to copy core-site # to /opt/spark/conf configs.append('HADOOP_CONF_DIR=' + HADOOP_CONF_DIR) masterport = utils.get_config_value_or_default("Spark", "Master port", cluster) if masterport and masterport != _get_spark_opt_default("Master port"): configs.append('SPARK_MASTER_PORT=' + str(masterport)) masterwebport = utils.get_config_value_or_default("Spark", "Master webui port", cluster) if (masterwebport and masterwebport != _get_spark_opt_default("Master webui port")): configs.append('SPARK_MASTER_WEBUI_PORT=' + str(masterwebport)) # configuration for workers workercores = utils.get_config_value_or_default("Spark", "Worker cores", cluster) if workercores and workercores != _get_spark_opt_default("Worker cores"): configs.append('SPARK_WORKER_CORES=' + str(workercores)) workermemory = utils.get_config_value_or_default("Spark", "Worker memory", cluster) if (workermemory and workermemory != _get_spark_opt_default("Worker memory")): configs.append('SPARK_WORKER_MEMORY=' + str(workermemory)) workerport = utils.get_config_value_or_default("Spark", "Worker port", cluster) if workerport and workerport != _get_spark_opt_default("Worker port"): configs.append('SPARK_WORKER_PORT=' + str(workerport)) workerwebport = utils.get_config_value_or_default("Spark", "Worker webui port", cluster) if (workerwebport and workerwebport != _get_spark_opt_default("Worker webui port")): configs.append('SPARK_WORKER_WEBUI_PORT=' + str(workerwebport)) workerinstances = utils.get_config_value_or_default("Spark", "Worker instances", cluster) if (workerinstances and workerinstances != _get_spark_opt_default("Worker instances")): configs.append('SPARK_WORKER_INSTANCES=' + str(workerinstances)) return '\n'.join(configs) # workernames need to be a list of worker names def generate_spark_slaves_configs(workernames): return '\n'.join(workernames) def generate_spark_executor_classpath(cluster): cp = utils.get_config_value_or_default("Spark", "Executor extra classpath", cluster) if cp: return "spark.executor.extraClassPath " + cp return "\n" def extract_hadoop_environment_confs(configs): """Returns environment specific Hadoop configurations. :returns: list of Hadoop parameters which should be passed via environment """ lst = [] for service, srv_confs in configs.items(): if ENV_CONFS.get(service): for param_name, param_value in srv_confs.items(): for cfg_name, cfg_format_str in ENV_CONFS[service].items(): if param_name == cfg_name and param_value is not None: lst.append(cfg_format_str % param_value) return lst def extract_hadoop_xml_confs(configs): """Returns xml specific Hadoop configurations. :returns: list of Hadoop parameters which should be passed into general configs like core-site.xml """ lst = [] for service, srv_confs in configs.items(): if XML_CONFS.get(service): for param_name, param_value in srv_confs.items(): for cfg_list in XML_CONFS[service]: names = [cfg['name'] for cfg in cfg_list] if param_name in names and param_value is not None: lst.append((param_name, param_value)) return lst def generate_hadoop_setup_script(storage_paths, env_configs): script_lines = ["#!/bin/bash -x"] script_lines.append("echo -n > /tmp/hadoop-env.sh") for line in env_configs: if 'HADOOP' in line: script_lines.append('echo "%s" >> /tmp/hadoop-env.sh' % line) script_lines.append("cat /etc/hadoop/hadoop-env.sh >> /tmp/hadoop-env.sh") script_lines.append("cp /tmp/hadoop-env.sh /etc/hadoop/hadoop-env.sh") hadoop_log = storage_paths[0] + "/log/hadoop/\$USER/" script_lines.append('sed -i "s,export HADOOP_LOG_DIR=.*,' 'export HADOOP_LOG_DIR=%s," /etc/hadoop/hadoop-env.sh' % hadoop_log) hadoop_log = storage_paths[0] + "/log/hadoop/hdfs" script_lines.append('sed -i "s,export HADOOP_SECURE_DN_LOG_DIR=.*,' 'export HADOOP_SECURE_DN_LOG_DIR=%s," ' '/etc/hadoop/hadoop-env.sh' % hadoop_log) for path in storage_paths: script_lines.append("chown -R hadoop:hadoop %s" % path) script_lines.append("chmod -f -R 755 %s ||" "echo 'Permissions unchanged'" % path) return "\n".join(script_lines) def generate_job_cleanup_config(cluster): args = { 'minimum_cleanup_megabytes': utils.get_config_value_or_default( "Spark", "Minimum cleanup megabytes", cluster), 'minimum_cleanup_seconds': utils.get_config_value_or_default( "Spark", "Minimum cleanup seconds", cluster), 'maximum_cleanup_seconds': utils.get_config_value_or_default( "Spark", "Maximum cleanup seconds", cluster) } job_conf = {'valid': (args['maximum_cleanup_seconds'] > 0 and (args['minimum_cleanup_megabytes'] > 0 and args['minimum_cleanup_seconds'] > 0))} if job_conf['valid']: job_conf['cron'] = f.get_file_text( 'plugins/spark/resources/spark-cleanup.cron'), job_cleanup_script = f.get_file_text( 'plugins/spark/resources/tmp-cleanup.sh.template') job_conf['script'] = job_cleanup_script.format(**args) return job_conf def extract_name_values(configs): return {cfg['name']: cfg['value'] for cfg in configs} def make_hadoop_path(base_dirs, suffix): return [base_dir + suffix for base_dir in base_dirs] def extract_hadoop_path(lst, hadoop_dir): if lst: return ",".join(make_hadoop_path(lst, hadoop_dir)) def _set_config(cfg, gen_cfg, name=None): if name in gen_cfg: cfg.update(gen_cfg[name]['conf']) if name is None: for name in gen_cfg: cfg.update(gen_cfg[name]['conf']) return cfg def _get_general_config_value(conf, option): if 'general' in conf and option.name in conf['general']: return conf['general'][option.name] return option.default_value def _get_general_cluster_config_value(cluster, option): return _get_general_config_value(cluster.cluster_configs, option) def is_data_locality_enabled(cluster): if not CONF.enable_data_locality: return False return _get_general_cluster_config_value(cluster, ENABLE_DATA_LOCALITY) def is_swift_enabled(configs): return _get_general_config_value(configs, ENABLE_SWIFT) def get_decommissioning_timeout(cluster): return _get_general_cluster_config_value(cluster, DECOMMISSIONING_TIMEOUT) def get_port_from_config(service, name, cluster=None): address = utils.get_config_value_or_default(service, name, cluster) return utils.get_port_from_address(address) sahara-8.0.0/sahara/plugins/spark/shell_engine.py0000666000175100017510000000205313245514472022061 0ustar zuulzuul00000000000000# Copyright (c) 2015 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import utils as plugin_utils from sahara.service.edp.spark import engine as shell_engine class ShellEngine(shell_engine.SparkShellJobEngine): def __init__(self, cluster): super(ShellEngine, self).__init__(cluster) self.master = plugin_utils.get_instance(cluster, "master") @staticmethod def job_type_supported(job_type): return (job_type in shell_engine.SparkShellJobEngine. get_supported_job_types()) sahara-8.0.0/sahara/plugins/spark/run_scripts.py0000666000175100017510000000613513245514472022005 0ustar zuulzuul00000000000000# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from oslo_log import log as logging from sahara.i18n import _ from sahara.plugins.spark import config_helper as c_helper from sahara.plugins import utils from sahara.utils import cluster_progress_ops from sahara.utils import poll_utils LOG = logging.getLogger(__name__) def start_processes(remote, *processes): for proc in processes: if proc == "namenode": remote.execute_command("sudo service hadoop-hdfs-namenode start") elif proc == "datanode": remote.execute_command("sudo service hadoop-hdfs-datanode start") else: remote.execute_command("screen -d -m sudo hadoop %s" % proc) def refresh_nodes(remote, service): remote.execute_command("sudo -u hdfs hadoop %s -refreshNodes" % service) def format_namenode(nn_remote): nn_remote.execute_command("sudo -u hdfs hadoop namenode -format") def clean_port_hadoop(nn_remote): nn_remote.execute_command(("sudo netstat -tlnp" "| awk '/:8020 */" "{split($NF,a,\"/\"); print a[1]}'" "| xargs sudo kill -9")) def start_spark_master(nn_remote, sp_home): nn_remote.execute_command("bash " + os.path.join(sp_home, "sbin/start-all.sh")) def stop_spark(nn_remote, sp_home): nn_remote.execute_command("bash " + os.path.join(sp_home, "sbin/stop-all.sh")) @cluster_progress_ops.event_wrapper( True, step=_("Await DataNodes start up"), param=("cluster", 0)) def await_datanodes(cluster): datanodes_count = len(utils.get_instances(cluster, "datanode")) if datanodes_count < 1: return log_msg = _("Waiting on %d DataNodes to start up") % datanodes_count with utils.get_instance(cluster, "namenode").remote() as r: poll_utils.plugin_option_poll( cluster, _check_datanodes_count, c_helper.DATANODES_STARTUP_TIMEOUT, log_msg, 1, {"remote": r, "count": datanodes_count}) def _check_datanodes_count(remote, count): if count < 1: return True LOG.debug("Checking DataNodes count") ex_code, stdout = remote.execute_command( 'sudo su -lc "hdfs dfsadmin -report" hdfs | ' 'grep \'Live datanodes\|Datanodes available:\' | ' 'grep -o \'[0-9]\+\' | head -n 1') LOG.debug("DataNodes count='{count}'".format(count=stdout.strip())) return stdout and int(stdout) == count sahara-8.0.0/sahara/plugins/spark/__init__.py0000666000175100017510000000000013245514472021152 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/spark/scaling.py0000666000175100017510000000735613245514472021060 0ustar zuulzuul00000000000000# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import six from sahara import context from sahara.i18n import _ from sahara.plugins.spark import config_helper as c_helper from sahara.plugins.spark import run_scripts as run from sahara.plugins import utils from sahara.utils import cluster_progress_ops as cpo from sahara.utils import poll_utils from sahara.utils import remote @cpo.event_wrapper(True, step=_("Decommission %s") % "Slaves") def decommission_sl(master, inst_to_be_deleted, survived_inst): if survived_inst is not None: slavenames = [] for slave in survived_inst: slavenames.append(slave.hostname()) slaves_content = c_helper.generate_spark_slaves_configs(slavenames) else: slaves_content = "\n" cluster = master.cluster sp_home = utils.get_config_value_or_default("Spark", "Spark home", cluster) r_master = remote.get_remote(master) run.stop_spark(r_master, sp_home) # write new slave file to master files = {os.path.join(sp_home, 'conf/slaves'): slaves_content} r_master.write_files_to(files) # write new slaves file to each survived slave as well for i in survived_inst: with remote.get_remote(i) as r: r.write_files_to(files) run.start_spark_master(r_master, sp_home) def _is_decommissioned(r, inst_to_be_deleted): cmd = r.execute_command("sudo -u hdfs hadoop dfsadmin -report") datanodes_info = parse_dfs_report(cmd[1]) for i in inst_to_be_deleted: for dn in datanodes_info: if (dn["Name"].startswith(i.internal_ip)) and ( dn["Decommission Status"] != "Decommissioned"): return False return True @cpo.event_wrapper(True, step=_("Decommission %s") % "DataNodes") def decommission_dn(nn, inst_to_be_deleted, survived_inst): with remote.get_remote(nn) as r: r.write_file_to('/etc/hadoop/dn.excl', utils.generate_fqdn_host_names( inst_to_be_deleted)) run.refresh_nodes(remote.get_remote(nn), "dfsadmin") context.sleep(3) poll_utils.plugin_option_poll( nn.cluster, _is_decommissioned, c_helper.DECOMMISSIONING_TIMEOUT, _("Decommission %s") % "DataNodes", 3, { 'r': r, 'inst_to_be_deleted': inst_to_be_deleted}) r.write_files_to({ '/etc/hadoop/dn.incl': utils. generate_fqdn_host_names(survived_inst), '/etc/hadoop/dn.excl': ""}) def parse_dfs_report(cmd_output): report = cmd_output.rstrip().split(os.linesep) array = [] started = False for line in report: if started: array.append(line) if line.startswith("Datanodes available"): started = True res = [] datanode_info = {} for i in six.moves.xrange(0, len(array)): if array[i]: idx = str.find(array[i], ':') name = array[i][0:idx] value = array[i][idx + 2:] datanode_info[name.strip()] = value.strip() if not array[i] and datanode_info: res.append(datanode_info) datanode_info = {} if datanode_info: res.append(datanode_info) return res sahara-8.0.0/sahara/plugins/spark/edp_engine.py0000666000175100017510000000445613245514472021533 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import six from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import utils as plugin_utils from sahara.service.edp.spark import engine as edp_engine class EdpEngine(edp_engine.SparkJobEngine): edp_base_version = "1.3.1" def __init__(self, cluster): super(EdpEngine, self).__init__(cluster) self.master = plugin_utils.get_instance(cluster, "master") self.plugin_params["spark-user"] = "" self.plugin_params["spark-submit"] = os.path.join( plugin_utils. get_config_value_or_default("Spark", "Spark home", self.cluster), "bin/spark-submit") self.plugin_params["deploy-mode"] = "client" port_str = six.text_type( plugin_utils.get_config_value_or_default( "Spark", "Master port", self.cluster)) self.plugin_params["master"] = ('spark://%(host)s:' + port_str) driver_cp = plugin_utils.get_config_value_or_default( "Spark", "Executor extra classpath", self.cluster) self.plugin_params["driver-class-path"] = driver_cp @staticmethod def edp_supported(version): return version >= EdpEngine.edp_base_version @staticmethod def job_type_supported(job_type): return job_type in edp_engine.SparkJobEngine.get_supported_job_types() def validate_job_execution(self, cluster, job, data): if not self.edp_supported(cluster.hadoop_version): raise ex.InvalidDataException( _('Spark {base} or higher required to run {type} jobs').format( base=EdpEngine.edp_base_version, type=job.type)) super(EdpEngine, self).validate_job_execution(cluster, job, data) sahara-8.0.0/sahara/plugins/spark/resources/0000775000175100017510000000000013245515027021060 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/spark/resources/topology.sh0000666000175100017510000000061213245514472023274 0ustar zuulzuul00000000000000#!/bin/bash HADOOP_CONF=/etc/hadoop while [ $# -gt 0 ] ; do nodeArg=$1 exec< ${HADOOP_CONF}/topology.data result="" while read line ; do ar=( $line ) if [ "${ar[0]}" = "$nodeArg" ] ; then result="${ar[1]}" fi done shift if [ -z "$result" ] ; then echo -n "/default/rack " else echo -n "$result " fi done sahara-8.0.0/sahara/plugins/spark/resources/spark-cleanup.cron0000666000175100017510000000013613245514472024515 0ustar zuulzuul00000000000000# Cleans up old Spark job directories once per hour. 0 * * * * root /etc/hadoop/tmp-cleanup.shsahara-8.0.0/sahara/plugins/spark/resources/core-default.xml0000666000175100017510000004734313245514472024174 0ustar zuulzuul00000000000000 hadoop.tmp.dir /tmp/hadoop-${user.name} A base for other temporary directories. hadoop.native.lib true Should native hadoop libraries, if present, be used. hadoop.http.filter.initializers A comma separated list of class names. Each class in the list must extend org.apache.hadoop.http.FilterInitializer. The corresponding Filter will be initialized. Then, the Filter will be applied to all user facing jsp and servlet web pages. The ordering of the list defines the ordering of the filters. hadoop.security.group.mapping org.apache.hadoop.security.ShellBasedUnixGroupsMapping Class for user to group mapping (get groups for a given user) hadoop.security.authorization false Is service-level authorization enabled? hadoop.security.instrumentation.requires.admin false Indicates if administrator ACLs are required to access instrumentation servlets (JMX, METRICS, CONF, STACKS). hadoop.security.authentication simple Possible values are simple (no authentication), and kerberos hadoop.security.token.service.use_ip true Controls whether tokens always use IP addresses. DNS changes will not be detected if this option is enabled. Existing client connections that break will always reconnect to the IP of the original host. New clients will connect to the host's new IP but fail to locate a token. Disabling this option will allow existing and new clients to detect an IP change and continue to locate the new host's token. hadoop.security.use-weak-http-crypto false If enabled, use KSSL to authenticate HTTP connections to the NameNode. Due to a bug in JDK6, using KSSL requires one to configure Kerberos tickets to use encryption types that are known to be cryptographically weak. If disabled, SPNEGO will be used for HTTP authentication, which supports stronger encryption types. hadoop.logfile.size 10000000 The max size of each log file hadoop.logfile.count 10 The max number of log files io.file.buffer.size 4096 The size of buffer for use in sequence files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. io.bytes.per.checksum 512 The number of bytes per checksum. Must not be larger than io.file.buffer.size. io.skip.checksum.errors false If true, when a checksum error is encountered while reading a sequence file, entries are skipped, instead of throwing an exception. io.compression.codecs org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec A list of the compression codec classes that can be used for compression/decompression. io.serializations org.apache.hadoop.io.serializer.WritableSerialization A list of serialization classes that can be used for obtaining serializers and deserializers. fs.defaultFS file:/// The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem. fs.trash.interval 0 Number of minutes between trash checkpoints. If zero, the trash feature is disabled. fs.file.impl org.apache.hadoop.fs.LocalFileSystem The FileSystem for file: uris. fs.hdfs.impl org.apache.hadoop.hdfs.DistributedFileSystem The FileSystem for hdfs: uris. fs.s3.impl org.apache.hadoop.fs.s3.S3FileSystem The FileSystem for s3: uris. fs.s3n.impl org.apache.hadoop.fs.s3native.NativeS3FileSystem The FileSystem for s3n: (Native S3) uris. fs.kfs.impl org.apache.hadoop.fs.kfs.KosmosFileSystem The FileSystem for kfs: uris. fs.hftp.impl org.apache.hadoop.hdfs.HftpFileSystem fs.hsftp.impl org.apache.hadoop.hdfs.HsftpFileSystem fs.webhdfs.impl org.apache.hadoop.hdfs.web.WebHdfsFileSystem fs.ftp.impl org.apache.hadoop.fs.ftp.FTPFileSystem The FileSystem for ftp: uris. fs.ramfs.impl org.apache.hadoop.fs.InMemoryFileSystem The FileSystem for ramfs: uris. fs.har.impl org.apache.hadoop.fs.HarFileSystem The filesystem for Hadoop archives. fs.har.impl.disable.cache true Don't cache 'har' filesystem instances. fs.checkpoint.dir ${hadoop.tmp.dir}/dfs/namesecondary Determines where on the local filesystem the DFS secondary name node should store the temporary images to merge. If this is a comma-delimited list of directories then the image is replicated in all of the directories for redundancy. fs.checkpoint.edits.dir ${fs.checkpoint.dir} Determines where on the local filesystem the DFS secondary name node should store the temporary edits to merge. If this is a comma-delimited list of directoires then teh edits is replicated in all of the directoires for redundancy. Default value is same as fs.checkpoint.dir fs.checkpoint.period 3600 The number of seconds between two periodic checkpoints. fs.checkpoint.size 67108864 The size of the current edit log (in bytes) that triggers a periodic checkpoint even if the fs.checkpoint.period hasn't expired. fs.s3.block.size 67108864 Block size to use when writing files to S3. fs.s3.buffer.dir ${hadoop.tmp.dir}/s3 Determines where on the local filesystem the S3 filesystem should store files before sending them to S3 (or after retrieving them from S3). fs.s3.maxRetries 4 The maximum number of retries for reading or writing files to S3, before we signal failure to the application. fs.s3.sleepTimeSeconds 10 The number of seconds to sleep between each S3 retry. local.cache.size 10737418240 The limit on the size of cache you want to keep, set by default to 10GB. This will act as a soft limit on the cache directory for out of band data. io.seqfile.compress.blocksize 1000000 The minimum block size for compression in block compressed SequenceFiles. io.seqfile.lazydecompress true Should values of block-compressed SequenceFiles be decompressed only when necessary. io.seqfile.sorter.recordlimit 1000000 The limit on number of records to be kept in memory in a spill in SequenceFiles.Sorter io.mapfile.bloom.size 1048576 The size of BloomFilter-s used in BloomMapFile. Each time this many keys is appended the next BloomFilter will be created (inside a DynamicBloomFilter). Larger values minimize the number of filters, which slightly increases the performance, but may waste too much space if the total number of keys is usually much smaller than this number. io.mapfile.bloom.error.rate 0.005 The rate of false positives in BloomFilter-s used in BloomMapFile. As this value decreases, the size of BloomFilter-s increases exponentially. This value is the probability of encountering false positives (default is 0.5%). hadoop.util.hash.type murmur The default implementation of Hash. Currently this can take one of the two values: 'murmur' to select MurmurHash and 'jenkins' to select JenkinsHash. ipc.client.idlethreshold 4000 Defines the threshold number of connections after which connections will be inspected for idleness. ipc.client.kill.max 10 Defines the maximum number of clients to disconnect in one go. ipc.client.connection.maxidletime 10000 The maximum time in msec after which a client will bring down the connection to the server. ipc.client.connect.max.retries 10 Indicates the number of retries a client will make to establish a server connection. ipc.server.listen.queue.size 128 Indicates the length of the listen queue for servers accepting client connections. ipc.server.tcpnodelay false Turn on/off Nagle's algorithm for the TCP socket connection on the server. Setting to true disables the algorithm and may decrease latency with a cost of more/smaller packets. ipc.client.tcpnodelay false Turn on/off Nagle's algorithm for the TCP socket connection on the client. Setting to true disables the algorithm and may decrease latency with a cost of more/smaller packets. webinterface.private.actions false If set to true, the web interfaces of JT and NN may contain actions, such as kill job, delete file, etc., that should not be exposed to public. Enable this option if the interfaces are only reachable by those who have the right authorization. hadoop.rpc.socket.factory.class.default org.apache.hadoop.net.StandardSocketFactory Default SocketFactory to use. This parameter is expected to be formatted as "package.FactoryClassName". hadoop.rpc.socket.factory.class.ClientProtocol SocketFactory to use to connect to a DFS. If null or empty, use hadoop.rpc.socket.class.default. This socket factory is also used by DFSClient to create sockets to DataNodes. hadoop.socks.server Address (host:port) of the SOCKS server to be used by the SocksSocketFactory. topology.node.switch.mapping.impl org.apache.hadoop.net.ScriptBasedMapping The default implementation of the DNSToSwitchMapping. It invokes a script specified in topology.script.file.name to resolve node names. If the value for topology.script.file.name is not set, the default value of DEFAULT_RACK is returned for all node names. net.topology.impl org.apache.hadoop.net.NetworkTopology The default implementation of NetworkTopology which is classic three layer one. topology.script.file.name The script name that should be invoked to resolve DNS names to NetworkTopology names. Example: the script would take host.foo.bar as an argument, and return /rack1 as the output. topology.script.number.args 100 The max number of args that the script configured with topology.script.file.name should be run with. Each arg is an IP address. hadoop.security.uid.cache.secs 14400 NativeIO maintains a cache from UID to UserName. This is the timeout for an entry in that cache. hadoop.http.authentication.type simple Defines authentication used for Oozie HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# hadoop.http.authentication.token.validity 36000 Indicates how long (in seconds) an authentication token is valid before it has to be renewed. hadoop.http.authentication.signature.secret.file ${user.home}/hadoop-http-auth-signature-secret The signature secret for signing the authentication tokens. If not set a random secret is generated at startup time. The same secret should be used for JT/NN/DN/TT configurations. hadoop.http.authentication.cookie.domain The domain to use for the HTTP cookie that stores the authentication token. In order to authentiation to work correctly across all Hadoop nodes web-consoles the domain must be correctly set. IMPORTANT: when using IP addresses, browsers ignore cookies with domain settings. For this setting to work properly all nodes in the cluster must be configured to generate URLs with hostname.domain names on it. hadoop.http.authentication.simple.anonymous.allowed true Indicates if anonymous requests are allowed when using 'simple' authentication. hadoop.http.authentication.kerberos.principal HTTP/localhost@LOCALHOST Indicates the Kerberos principal to be used for HTTP endpoint. The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification. hadoop.http.authentication.kerberos.keytab ${user.home}/hadoop.keytab Location of the keytab file with the credentials for the principal. Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop. hadoop.relaxed.worker.version.check false By default datanodes refuse to connect to namenodes if their build revision (svn revision) do not match, and tasktrackers refuse to connect to jobtrackers if their build version (version, revision, user, and source checksum) do not match. This option changes the behavior of hadoop workers to only check for a version match (eg "1.0.2") but ignore the other build fields (revision, user, and source checksum). hadoop.skip.worker.version.check false By default datanodes refuse to connect to namenodes if their build revision (svn revision) do not match, and tasktrackers refuse to connect to jobtrackers if their build version (version, revision, user, and source checksum) do not match. This option changes the behavior of hadoop workers to skip doing a version check at all. This option supersedes the 'hadoop.relaxed.worker.version.check' option. hadoop.jetty.logs.serve.aliases true Enable/Disable aliases serving from jetty ipc.client.fallback-to-simple-auth-allowed false When a client is configured to attempt a secure connection, but attempts to connect to an insecure server, that server may instruct the client to switch to SASL SIMPLE (unsecure) authentication. This setting controls whether or not the client will accept this instruction from the server. When false (the default), the client will not allow the fallback to SIMPLE authentication, and will abort the connection. sahara-8.0.0/sahara/plugins/spark/resources/README.rst0000666000175100017510000000155313245514472022560 0ustar zuulzuul00000000000000Apache Spark and HDFS Configurations for Sahara =============================================== This directory contains default XML configuration files and Spark scripts: * core-default.xml, * hdfs-default.xml, * spark-env.sh.template, * topology.sh These files are used by Sahara's plugin for Apache Spark and Cloudera HDFS. XML config files were taken from here: * https://github.com/apache/hadoop-common/blob/release-1.2.1/src/core/core-default.xml * https://github.com/apache/hadoop-common/blob/release-1.2.1/src/hdfs/hdfs-default.xml Cloudera packages use the same configuration files as standard Apache Hadoop. XML configs are used to expose default Hadoop configurations to the users through Sahara's REST API. It allows users to override some config values which will be pushed to the provisioned VMs running Hadoop services as part of appropriate xml config. sahara-8.0.0/sahara/plugins/spark/resources/tmp-cleanup.sh.template0000666000175100017510000000230513245514472025460 0ustar zuulzuul00000000000000#!/bin/sh MINIMUM_CLEANUP_MEGABYTES={minimum_cleanup_megabytes} MINIMUM_CLEANUP_SECONDS={minimum_cleanup_seconds} MAXIMUM_CLEANUP_SECONDS={maximum_cleanup_seconds} CURRENT_TIMESTAMP=`date +%s` POSSIBLE_CLEANUP_THRESHOLD=$(($CURRENT_TIMESTAMP - $MINIMUM_CLEANUP_SECONDS)) DEFINITE_CLEANUP_THRESHOLD=$(($CURRENT_TIMESTAMP - $MAXIMUM_CLEANUP_SECONDS)) unset MAY_DELETE unset WILL_DELETE if [ ! -d /tmp/spark-edp ] then exit 0 fi cd /tmp/spark-edp for JOB in $(find . -maxdepth 1 -mindepth 1 -type d -printf '%f\n') do for EXECUTION in $(find $JOB -maxdepth 1 -mindepth 1 -type d -printf '%f\n') do TIMESTAMP=`stat $JOB/$EXECUTION --printf '%Y'` if [[ $TIMESTAMP -lt $DEFINITE_CLEANUP_THRESHOLD ]] then WILL_DELETE="$WILL_DELETE $JOB/$EXECUTION" else if [[ $TIMESTAMP -lt $POSSIBLE_CLEANUP_THRESHOLD ]] then MAY_DELETE="$MAY_DELETE $JOB/$EXECUTION" fi fi done done for EXECUTION in $WILL_DELETE do rm -Rf $EXECUTION done for EXECUTION in $(ls $MAY_DELETE -trd) do if [[ `du -s -BM | grep -o '[0-9]\+'` -le $MINIMUM_CLEANUP_MEGABYTES ]]; then break fi rm -Rf $EXECUTION done sahara-8.0.0/sahara/plugins/spark/resources/spark-env.sh.template0000666000175100017510000000216513245514472025145 0ustar zuulzuul00000000000000#!/usr/bin/env bash # This file contains environment variables required to run Spark. Copy it as # spark-env.sh and edit that to configure Spark for your site. # # The following variables can be set in this file: # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos # - SPARK_JAVA_OPTS, to set node-specific JVM options for Spark. Note that # we recommend setting app-wide options in the application's driver program. # Examples of node-specific options : -Dspark.local.dir, GC options # Examples of app-wide options : -Dspark.serializer # # If using the standalone deploy mode, you can also set variables for it here: # - SPARK_MASTER_IP, to bind the master to a different IP address or hostname # - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports # - SPARK_WORKER_CORES, to set the number of cores to use on this machine # - SPARK_WORKER_MEMORY, to set how much memory to use (e.g. 1000m, 2g) # - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT # - SPARK_WORKER_INSTANCES, to set the number of worker processes per node sahara-8.0.0/sahara/plugins/spark/resources/hdfs-default.xml0000666000175100017510000005467213245514472024173 0ustar zuulzuul00000000000000 dfs.namenode.logging.level info The logging level for dfs namenode. Other values are "dir"(trac e namespace mutations), "block"(trace block under/over replications and block creations/deletions), or "all". dfs.namenode.rpc-address RPC address that handles all clients requests. If empty then we'll get the value from fs.default.name. The value of this property will take the form of hdfs://nn-host1:rpc-port. dfs.secondary.http.address 0.0.0.0:50090 The secondary namenode http server address and port. If the port is 0 then the server will start on a free port. dfs.datanode.address 0.0.0.0:50010 The datanode server address and port for data transfer. If the port is 0 then the server will start on a free port. dfs.datanode.http.address 0.0.0.0:50075 The datanode http server address and port. If the port is 0 then the server will start on a free port. dfs.datanode.ipc.address 0.0.0.0:50020 The datanode ipc server address and port. If the port is 0 then the server will start on a free port. dfs.datanode.handler.count 3 The number of server threads for the datanode. dfs.http.address 0.0.0.0:50070 The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port. dfs.https.enable false Decide if HTTPS(SSL) is supported on HDFS dfs.https.need.client.auth false Whether SSL client certificate authentication is required dfs.https.server.keystore.resource ssl-server.xml Resource file from which ssl server keystore information will be extracted dfs.https.client.keystore.resource ssl-client.xml Resource file from which ssl client keystore information will be extracted dfs.datanode.https.address 0.0.0.0:50475 dfs.https.address 0.0.0.0:50470 dfs.datanode.dns.interface default The name of the Network Interface from which a data node should report its IP address. dfs.datanode.dns.nameserver default The host name or IP address of the name server (DNS) which a DataNode should use to determine the host name used by the NameNode for communication and display purposes. dfs.replication.considerLoad true Decide if chooseTarget considers the target's load or not dfs.default.chunk.view.size 32768 The number of bytes to view for a file on the browser. dfs.datanode.du.reserved 0 Reserved space in bytes per volume. Always leave this much space free for non dfs use. dfs.namenode.name.dir ${hadoop.tmp.dir}/dfs/name Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. dfs.name.edits.dir ${dfs.name.dir} Determines where on the local filesystem the DFS name node should store the transaction (edits) file. If this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. Default value is same as dfs.name.dir dfs.namenode.edits.toleration.length 0 The length in bytes that namenode is willing to tolerate when the edit log is corrupted. The edit log toleration feature checks the entire edit log. It computes read length (the length of valid data), corruption length and padding length. In case that corruption length is non-zero, the corruption will be tolerated only if the corruption length is less than or equal to the toleration length. For disabling edit log toleration feature, set this property to -1. When the feature is disabled, the end of edit log will not be checked. In this case, namenode will startup normally even if the end of edit log is corrupted. dfs.web.ugi webuser,webgroup The user account used by the web interface. Syntax: USERNAME,GROUP1,GROUP2, ... dfs.permissions true If "true", enable permission checking in HDFS. If "false", permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories. dfs.permissions.supergroup supergroup The name of the group of super-users. dfs.block.access.token.enable false If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanodes. dfs.block.access.key.update.interval 600 Interval in minutes at which namenode updates its access keys. dfs.block.access.token.lifetime 600 The lifetime of access tokens in minutes. dfs.datanode.data.dir ${hadoop.tmp.dir}/dfs/data Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. dfs.datanode.data.dir.perm 755 Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic. dfs.replication 3 Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. dfs.replication.max 512 Maximal block replication. dfs.replication.min 1 Minimal block replication. dfs.block.size 67108864 The default block size for new files. dfs.df.interval 60000 Disk usage statistics refresh interval in msec. dfs.client.block.write.retries 3 The number of retries for writing blocks to the data nodes, before we signal failure to the application. dfs.blockreport.intervalMsec 3600000 Determines block reporting interval in milliseconds. dfs.blockreport.initialDelay 0 Delay for first block report in seconds. dfs.heartbeat.interval 3 Determines datanode heartbeat interval in seconds. dfs.namenode.handler.count 10 The number of server threads for the namenode. dfs.safemode.threshold.pct 0.999f Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.replication.min. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent. dfs.namenode.safemode.min.datanodes 0 Specifies the number of datanodes that must be considered alive before the name node exits safemode. Values less than or equal to 0 mean not to take the number of live datanodes into account when deciding whether to remain in safe mode during startup. Values greater than the number of datanodes in the cluster will make safe mode permanent. dfs.safemode.extension 30000 Determines extension of safe mode in milliseconds after the threshold level is reached. dfs.balance.bandwidthPerSec 1048576 Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. dfs.hosts Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted. dfs.hosts.exclude Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded. dfs.max.objects 0 The maximum number of files, directories and blocks dfs supports. A value of zero indicates no limit to the number of objects that dfs supports. dfs.namenode.decommission.interval 30 Namenode periodicity in seconds to check if decommission is complete. dfs.namenode.decommission.nodes.per.interval 5 The number of nodes namenode checks if decommission is complete in each dfs.namenode.decommission.interval. dfs.replication.interval 3 The periodicity in seconds with which the namenode computes repliaction work for datanodes. dfs.access.time.precision 3600000 The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS. dfs.support.append This option is no longer supported. HBase no longer requires that this option be enabled as sync is now enabled by default. See HADOOP-8230 for additional information. dfs.namenode.delegation.key.update-interval 86400000 The update interval for master key for delegation tokens in the namenode in milliseconds. dfs.namenode.delegation.token.max-lifetime 604800000 The maximum lifetime in milliseconds for which a delegation token is valid. dfs.namenode.delegation.token.renew-interval 86400000 The renewal interval for delegation token in milliseconds. dfs.datanode.failed.volumes.tolerated 0 The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown. dfs.datanode.max.xcievers 4096 Specifies the maximum number of threads to use for transferring data in and out of the DN. dfs.datanode.readahead.bytes 4193404 While reading block files, if the Hadoop native libraries are available, the datanode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader's position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the datanode will attempt to read ahead. This feature may be disabled by configuring this property to 0. If the native libraries are not available, this configuration has no effect. dfs.datanode.drop.cache.behind.reads false In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This behavior is automatically disabled for workloads which read only short sections of a block (e.g HBase random-IO workloads). This may improve performance for some workloads by freeing buffer cache spage usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect. dfs.datanode.drop.cache.behind.writes false In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spage usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect. dfs.datanode.sync.behind.writes false If this configuration is enabled, the datanode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. If the Hadoop native libraries are not available, this configuration has no effect. dfs.client.use.datanode.hostname false Whether clients should use datanode hostnames when connecting to datanodes. dfs.datanode.use.datanode.hostname false Whether datanodes should use datanode hostnames when connecting to other datanodes for data transfer. dfs.client.local.interfaces A comma separated list of network interface names to use for data transfer between the client and datanodes. When creating a connection to read from or write to a datanode, the client chooses one of the specified interfaces at random and binds its socket to the IP of that interface. Individual names may be specified as either an interface name (eg "eth0"), a subinterface name (eg "eth0:0"), or an IP address (which may be specified using CIDR notation to match a range of IPs). dfs.image.transfer.bandwidthPerSec 0 Specifies the maximum amount of bandwidth that can be utilized for image transfer in term of the number of bytes per second. A default value of 0 indicates that throttling is disabled. dfs.webhdfs.enabled false Enable WebHDFS (REST API) in Namenodes and Datanodes. dfs.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal} dfs.secondary.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal} dfs.namenode.invalidate.work.pct.per.iteration 0.32f *Note*: Advanced property. Change with caution. This determines the percentage amount of block invalidations (deletes) to do over a single DN heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DN. Value should be a positive, non-zero percentage in float notation (X.Yf), with 1.0f meaning 100%. dfs.namenode.replication.work.multiplier.per.iteration 2 *Note*: Advanced property. Change with caution. This determines the total amount of block transfers to begin in parallel at a DN, for replication, when such a command list is being sent over a DN heartbeat by the NN. The actual number is obtained by multiplying this multiplier with the total number of live nodes in the cluster. The result number is the number of blocks to begin transfers immediately for, per DN heartbeat. This number can be any positive, non-zero integer. dfs.namenode.avoid.read.stale.datanode false Indicate whether or not to avoid reading from "stale" datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Stale datanodes will be moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes. dfs.namenode.avoid.write.stale.datanode false Indicate whether or not to avoid writing to "stale" datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Writes will avoid using stale datanodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads. dfs.namenode.stale.datanode.interval 30000 Default time interval for marking a datanode as "stale", i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as "stale" by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. dfs.namenode.write.stale.datanode.ratio 0.5f When the ratio of number stale datanodes to total datanodes marked is greater than this ratio, stop avoiding writing to stale nodes so as to prevent causing hotspots. dfs.datanode.plugins Comma-separated list of datanode plug-ins to be activated. dfs.namenode.plugins Comma-separated list of namenode plug-ins to be activated. sahara-8.0.0/sahara/plugins/kerberos.py0000666000175100017510000003221613245514476020131 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging from oslo_utils import uuidutils from sahara import conductor as cond from sahara import context from sahara import exceptions as exc from sahara.i18n import _ from sahara.plugins import provisioning as base from sahara.plugins import utils as pl_utils from sahara.service.castellan import utils as key_manager from sahara.utils import cluster as cl_utils from sahara.utils import cluster_progress_ops as cpo from sahara.utils import files conductor = cond.API LOG = logging.getLogger(__name__) CONF = cfg.CONF POLICY_FILES_DIR = '/tmp/UnlimitedPolicy' class KDCInstallationFailed(exc.SaharaException): code = 'KDC_INSTALL_FAILED' message_template = _('KDC installation failed by reason: {reason}') def __init__(self, reason): message = self.message_template.format(reason=reason) super(KDCInstallationFailed, self).__init__(message) def _config(**kwargs): return base.Config( applicable_target='Kerberos', priority=1, is_optional=True, scope='cluster', **kwargs) enable_kerberos = _config( name='Enable Kerberos Security', config_type='bool', default_value=False) use_existing_kdc = _config( name='Existing KDC', config_type='bool', default_value=False) kdc_server_ip = _config( name='Server IP of KDC', config_type='string', default_value='192.168.0.1', description=_('Server IP of KDC server when using existing KDC')) realm_name = _config( name='Realm Name', config_type='string', default_value='SAHARA-KDC', description=_('The name of realm to be used')) admin_principal = _config( name='Admin principal', config_type='string', default_value='sahara/admin', description=_('Admin principal for existing KDC server')) admin_password = _config( name='Admin password', config_type='string', default_value='') policy_url = _config( name="JCE libraries", config_type='string', default_value=('http://tarballs.openstack.org/sahara/dist/' 'common-artifacts/'), description=_('Java Cryptography Extension (JCE) ' 'Unlimited Strength Jurisdiction Policy Files location') ) def get_config_list(): return [ enable_kerberos, use_existing_kdc, kdc_server_ip, realm_name, admin_principal, admin_password, policy_url, ] def get_kdc_host(cluster, server): if using_existing_kdc(cluster): return "server.%s" % CONF.node_domain return server.fqdn() def is_kerberos_security_enabled(cluster): return pl_utils.get_config_value_or_default( cluster=cluster, config=enable_kerberos) def using_existing_kdc(cluster): return pl_utils.get_config_value_or_default( cluster=cluster, config=use_existing_kdc) def get_kdc_server_ip(cluster): return pl_utils.get_config_value_or_default( cluster=cluster, config=kdc_server_ip) def get_realm_name(cluster): return pl_utils.get_config_value_or_default( cluster=cluster, config=realm_name) def get_admin_principal(cluster): return pl_utils.get_config_value_or_default( cluster=cluster, config=admin_principal) def get_admin_password(cluster): # TODO(vgridnev): support in follow-up improved secret storage for # configs return pl_utils.get_config_value_or_default( cluster=cluster, config=admin_password) def get_policy_url(cluster): return pl_utils.get_config_value_or_default( cluster=cluster, config=policy_url) def setup_clients(cluster, server=None, instances=None): if not instances: instances = cl_utils.get_instances(cluster) server_ip = None cpo.add_provisioning_step( cluster.id, _("Setting Up Kerberos clients"), len(instances)) if not server: server_ip = get_kdc_server_ip(cluster) with context.ThreadGroup() as tg: for instance in instances: tg.spawn('setup-client-%s' % instance.instance_name, _setup_client_node, cluster, instance, server, server_ip) def prepare_policy_files(cluster, instances=None): if instances is None: instances = pl_utils.get_instances(cluster) remote_url = get_policy_url(cluster) cpo.add_provisioning_step( cluster.id, _("Preparing policy files"), len(instances)) with context.ThreadGroup() as tg: for inst in instances: tg.spawn( 'policy-files', _prepare_policy_files, inst, remote_url) def deploy_infrastructure(cluster, server=None): if not is_kerberos_security_enabled(cluster): LOG.debug("Kerberos security disabled for cluster") return if not using_existing_kdc(cluster): deploy_kdc_server(cluster, server) setup_clients(cluster, server) def _execute_script(client, script): with client.remote() as remote: script_path = '/tmp/%s' % uuidutils.generate_uuid()[:8] remote.write_file_to(script_path, script) remote.execute_command('chmod +x %s' % script_path) remote.execute_command('bash %s' % script_path) remote.execute_command('rm -rf %s' % script_path) def _get_kdc_config(cluster, os): if os == "ubuntu": data = files.get_file_text('plugins/resources/kdc_conf') else: data = files.get_file_text('plugins/resources/kdc_conf_redhat') return data % { 'realm_name': get_realm_name(cluster) } def _get_krb5_config(cluster, server_fqdn): data = files.get_file_text('plugins/resources/krb5_config') return data % { 'realm_name': get_realm_name(cluster), 'server': server_fqdn, 'node_domain': CONF.node_domain, } def _get_short_uuid(): return "%s%s" % (uuidutils.generate_uuid()[:8], uuidutils.generate_uuid()[:8]) def get_server_password(cluster): if using_existing_kdc(cluster): return get_admin_password(cluster) ctx = context.ctx() cluster = conductor.cluster_get(ctx, cluster) extra = cluster.extra.to_dict() if cluster.extra else {} passwd_key = 'admin-passwd-kdc' if passwd_key not in extra: passwd = _get_short_uuid() key_id = key_manager.store_secret(passwd, ctx) extra[passwd_key] = key_id cluster = conductor.cluster_update(ctx, cluster, {'extra': extra}) passwd = key_manager.get_secret(extra.get(passwd_key), ctx) return passwd def _get_configs_dir(os): if os == "ubuntu": return "/etc/krb5kdc" return "/var/kerberos/krb5kdc" def _get_kdc_conf_path(os): return "%s/kdc.conf" % _get_configs_dir(os) def _get_realm_create_command(os): if os == 'ubuntu': return "krb5_newrealm" return "kdb5_util create -s" def _get_acl_config_path(os): return "%s/kadm5.acl" % _get_configs_dir(os) def _get_acl_config(): return "*/admin * " def _get_start_command(os, version): if os == "ubuntu": return ("sudo service krb5-kdc restart && " "sudo service krb5-admin-server restart") if version.startswith('6'): return ("sudo /etc/rc.d/init.d/krb5kdc start " "&& sudo /etc/rc.d/init.d/kadmin start") if version.startswith('7'): return ("sudo systemctl start krb5kdc &&" "sudo systemctl start kadmin") raise ValueError( _("Unable to get kdc server start command")) def _get_server_installation_script(cluster, server_fqdn, os, version): data = files.get_file_text( 'plugins/resources/mit-kdc-server-init.sh.template') return data % { 'kdc_conf': _get_kdc_config(cluster, os), 'kdc_conf_path': _get_kdc_conf_path(os), 'acl_conf': _get_acl_config(), 'acl_conf_path': _get_acl_config_path(os), 'realm_create': _get_realm_create_command(os), 'krb5_conf': _get_krb5_config(cluster, server_fqdn), 'admin_principal': get_admin_principal(cluster), 'password': get_server_password(cluster), 'os': os, 'start_command': _get_start_command(os, version), } @cpo.event_wrapper(True, step=_("Deploy KDC server"), param=('cluster', 0)) def deploy_kdc_server(cluster, server): with server.remote() as r: os = r.get_os_distrib() version = r.get_os_version() script = _get_server_installation_script( cluster, server.fqdn(), os, version) _execute_script(server, script) def _push_etc_hosts_entry(client, entry): with client.remote() as r: r.execute_command('echo %s | sudo tee -a /etc/hosts' % entry) def _get_client_installation_script(cluster, server_fqdn, os): data = files.get_file_text('plugins/resources/krb-client-init.sh.template') return data % { 'os': os, 'krb5_conf': _get_krb5_config(cluster, server_fqdn), } @cpo.event_wrapper(True, param=('client', 1)) def _setup_client_node(cluster, client, server=None, server_ip=None): if server: server_fqdn = server.fqdn() elif server_ip: server_fqdn = "server." % CONF.node_domain _push_etc_hosts_entry( client, "%s %s %s" % (server_ip, server_fqdn, server)) else: raise KDCInstallationFailed(_('Server or server ip are not provided')) with client.remote() as r: os = r.get_os_distrib() script = _get_client_installation_script(cluster, server_fqdn, os) _execute_script(client, script) @cpo.event_wrapper(True) def _prepare_policy_files(instance, remote_url): with instance.remote() as r: cmd = 'cut -f2 -d \"=\" /etc/profile.d/99-java.sh | head -1' exit_code, java_home = r.execute_command(cmd) java_home = java_home.strip() results = [ r.execute_command( "ls %s/local_policy.jar" % POLICY_FILES_DIR, raise_when_error=False)[0] != 0, r.execute_command( "ls %s/US_export_policy.jar" % POLICY_FILES_DIR, raise_when_error=False)[0] != 0 ] # a least one exit code is not zero if any(results): r.execute_command('mkdir %s' % POLICY_FILES_DIR) r.execute_command( "sudo curl %s/local_policy.jar -o %s/local_policy.jar" % ( remote_url, POLICY_FILES_DIR)) r.execute_command( "sudo curl %s/US_export_policy.jar -o " "%s/US_export_policy.jar" % ( remote_url, POLICY_FILES_DIR)) r.execute_command( 'sudo cp %s/*.jar %s/lib/security/' % (POLICY_FILES_DIR, java_home)) def _get_script_for_user_creation(cluster, instance, user): data = files.get_file_text( 'plugins/resources/create-principal-keytab') cron_file = files.get_file_text('plugins/resources/cron-file') cron_script = files.get_file_text('plugins/resources/cron-script') data = data % { 'user': user, 'admin_principal': get_admin_principal(cluster), 'admin_password': get_server_password(cluster), 'principal': "%s/sahara-%s@%s" % ( user, instance.fqdn(), get_realm_name(cluster)), 'keytab': '%s-sahara-%s.keytab' % (user, instance.fqdn()) } cron_script_location = '/tmp/sahara-kerberos/%s.sh' % _get_short_uuid() cron_file = cron_file % {'refresher': cron_script_location, 'user': user} cron_script = cron_script % { 'principal': "%s/sahara-%s@%s" % ( user, instance.fqdn(), get_realm_name(cluster)), 'keytab': '%s-sahara-%s.keytab' % (user, instance.fqdn()), 'user': user, } return data, cron_file, cron_script, cron_script_location def _create_keytabs_for_user(instance, user): script, cron, cron_script, cs_location = _get_script_for_user_creation( instance.cluster, instance, user) _execute_script(instance, script) # setting up refresher with instance.remote() as r: tmp_location = '/tmp/%s' % _get_short_uuid() r.write_file_to(tmp_location, cron_script, run_as_root=True) r.execute_command( "cat {0} | sudo tee {1} " "&& rm -rf {0} && sudo chmod +x {1}".format( tmp_location, cs_location)) r.execute_command( 'echo "%s" | sudo tee /etc/cron.d/%s.cron' % ( cron, _get_short_uuid())) # executing script r.execute_command('sudo bash %s' % cs_location) @cpo.event_wrapper( True, step=_('Setting up keytabs for users'), param=('cluster', 0)) def create_keytabs_for_map(cluster, mapper): # cluster parameter is used by event log feature with context.ThreadGroup() as tg: for user, instances in mapper.items(): for instance in instances: tg.spawn( 'create-keytabs', _create_keytabs_for_user, instance, user) sahara-8.0.0/sahara/plugins/cdh/0000775000175100017510000000000013245515026016463 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/deploy.py0000666000175100017510000001012413245514472020335 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import kerberos PACKAGES = [ 'cloudera-manager-agent', 'cloudera-manager-daemons', 'cloudera-manager-server', 'cloudera-manager-server-db-2', 'flume-ng', 'hadoop-hdfs-datanode', 'hadoop-hdfs-namenode', 'hadoop-hdfs-secondarynamenode', 'hadoop-kms' 'hadoop-mapreduce', 'hadoop-mapreduce-historyserver', 'hadoop-yarn-nodemanager', 'hadoop-yarn-resourcemanager', 'hbase', 'hbase-solr', 'hive-hcatalog', 'hive-metastore', 'hive-server2', 'hive-webhcat-server', 'hue', 'impala', 'impala-server', 'impala-state-store', 'impala-catalog', 'impala-shell', 'kafka', 'kafka-server' 'keytrustee-keyprovider', 'oozie', 'oracle-j2sdk1.7', 'sentry', 'solr-server', 'solr-doc', 'search', 'spark-history-server', 'sqoop2', 'unzip', 'zookeeper' ] def setup_kerberos_for_cluster(cluster, cloudera_utils): if kerberos.is_kerberos_security_enabled(cluster): manager = cloudera_utils.pu.get_manager(cluster) kerberos.deploy_infrastructure(cluster, manager) cloudera_utils.full_cluster_stop(cluster) kerberos.prepare_policy_files(cluster) cloudera_utils.push_kerberos_configs(cluster) cloudera_utils.full_cluster_start(cluster) kerberos.create_keytabs_for_map( cluster, {'hdfs': cloudera_utils.pu.get_hdfs_nodes(cluster), 'spark': [cloudera_utils.pu.get_spark_historyserver(cluster)]}) def prepare_scaling_kerberized_cluster(cluster, cloudera_utils, instances): if kerberos.is_kerberos_security_enabled(cluster): server = None if not kerberos.using_existing_kdc(cluster): server = cloudera_utils.pu.get_manager(cluster) kerberos.setup_clients(cluster, server) kerberos.prepare_policy_files(cluster) # manager can correctly handle updating configs cloudera_utils.push_kerberos_configs(cluster) kerberos.create_keytabs_for_map( cluster, {'hdfs': cloudera_utils.pu.get_hdfs_nodes(cluster, instances)}) def get_open_ports(node_group): ports = [9000] # for CM agent ports_map = { 'CLOUDERA_MANAGER': [7180, 7182, 7183, 7432, 7184, 8084, 8086, 10101, 9997, 9996, 8087, 9998, 9999, 8085, 9995, 9994], 'HDFS_NAMENODE': [8020, 8022, 50070, 50470], 'HDFS_SECONDARYNAMENODE': [50090, 50495], 'HDFS_DATANODE': [50010, 1004, 50075, 1006, 50020], 'YARN_RESOURCEMANAGER': [8030, 8031, 8032, 8033, 8088], 'YARN_STANDBYRM': [8030, 8031, 8032, 8033, 8088], 'YARN_NODEMANAGER': [8040, 8041, 8042], 'YARN_JOBHISTORY': [10020, 19888], 'HIVE_METASTORE': [9083], 'HIVE_SERVER2': [10000], 'HUE_SERVER': [8888], 'OOZIE_SERVER': [11000, 11001], 'SPARK_YARN_HISTORY_SERVER': [18088], 'ZOOKEEPER_SERVER': [2181, 3181, 4181, 9010], 'HBASE_MASTER': [60000], 'HBASE_REGIONSERVER': [60020], 'FLUME_AGENT': [41414], 'SENTRY_SERVER': [8038], 'SOLR_SERVER': [8983, 8984], 'SQOOP_SERVER': [8005, 12000], 'KEY_VALUE_STORE_INDEXER': [], 'IMPALA_CATALOGSERVER': [25020, 26000], 'IMPALA_STATESTORE': [25010, 24000], 'IMPALAD': [21050, 21000, 23000, 25000, 28000, 22000], 'KMS': [16000, 16001], 'JOURNALNODE': [8480, 8481, 8485] } for process in node_group.node_processes: if process in ports_map: ports.extend(ports_map[process]) return ports sahara-8.0.0/sahara/plugins/cdh/db_helper.py0000666000175100017510000001010713245514472020766 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils from sahara import conductor from sahara import context from sahara.service.castellan import utils as key_manager from sahara.utils import files CM_PASSWORD = 'cm_password' HIVE_DB_PASSWORD = 'hive_db_password' SENTRY_DB_PASSWORD = 'sentry_db_password' conductor = conductor.API def delete_password_from_keymanager(cluster, pwname): """delete the named password from the key manager This function will lookup the named password in the cluster entry and delete it from the key manager. :param cluster: The cluster record containing the password :param pwname: The name associated with the password """ ctx = context.ctx() cluster = conductor.cluster_get(ctx, cluster.id) key_id = cluster.extra.get(pwname) if cluster.extra else None if key_id is not None: key_manager.delete_key(key_id, ctx) def delete_passwords_from_keymanager(cluster): """delete all passwords associated with a cluster This function will remove all passwords stored in a cluster database entry from the key manager. :param cluster: The cluster record containing the passwords """ delete_password_from_keymanager(cluster, CM_PASSWORD) delete_password_from_keymanager(cluster, HIVE_DB_PASSWORD) delete_password_from_keymanager(cluster, SENTRY_DB_PASSWORD) def get_password_from_db(cluster, pwname): """return a password for the named entry This function will return, or create and return, a password for the named entry. It will store the password in the key manager and use the ID in the database entry. :param cluster: The cluster record containing the password :param pwname: The entry name associated with the password :returns: The cleartext password """ ctx = context.ctx() cluster = conductor.cluster_get(ctx, cluster.id) passwd = cluster.extra.get(pwname) if cluster.extra else None if passwd: return key_manager.get_secret(passwd, ctx) passwd = uuidutils.generate_uuid() extra = cluster.extra.to_dict() if cluster.extra else {} extra[pwname] = key_manager.store_secret(passwd, ctx) conductor.cluster_update(ctx, cluster, {'extra': extra}) return passwd def get_cm_password(cluster): return get_password_from_db(cluster, CM_PASSWORD) def remote_execute_db_script(remote, script_content): script_name = 'script_to_exec.sql' remote.write_file_to(script_name, script_content) psql_cmd = ('PGPASSWORD=$(sudo head -1 /var/lib/cloudera-scm-server-db' '/data/generated_password.txt) psql -U cloudera-scm ' '-h localhost -p 7432 -d scm -f %s') % script_name remote.execute_command(psql_cmd) remote.execute_command('rm %s' % script_name) def get_hive_db_password(cluster): return get_password_from_db(cluster, 'hive_db_password') def get_sentry_db_password(cluster): return get_password_from_db(cluster, 'sentry_db_password') def create_hive_database(cluster, remote): db_password = get_hive_db_password(cluster) create_db_script = files.get_file_text( 'plugins/cdh/db_resources/create_hive_db.sql') create_db_script = create_db_script % db_password remote_execute_db_script(remote, create_db_script) def create_sentry_database(cluster, remote): db_password = get_sentry_db_password(cluster) create_db_script = files.get_file_text( 'plugins/cdh/db_resources/create_sentry_db.sql') create_db_script = create_db_script % db_password remote_execute_db_script(remote, create_db_script) sahara-8.0.0/sahara/plugins/cdh/validation.py0000666000175100017510000004460713245514472021210 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ from sahara.plugins import exceptions as ex from sahara.plugins import utils as u from sahara.utils import general as gu class Validator(object): PU = None @classmethod def validate_cluster_creating(cls, cluster): cls._basic_validation(cluster) cls._oozie_validation(cluster) cls._hive_validation(cluster) cls._hue_validation(cluster) cls._hbase_validation(cluster) cls._flume_validation(cluster) cls._sentry_validation(cluster) cls._solr_validation(cluster) cls._sqoop_validation(cluster) cls._hbase_indexer_validation(cluster) cls._impala_validation(cluster) cls._kms_validation(cluster) cls._hdfs_ha_validation(cluster) cls._yarn_ha_validation(cluster) @classmethod def _basic_validation(cls, cluster): mng_count = cls.get_inst_count(cluster, 'CLOUDERA_MANAGER') if mng_count != 1: raise ex.InvalidComponentCountException('CLOUDERA_MANAGER', 1, mng_count) nn_count = cls.get_inst_count(cluster, 'HDFS_NAMENODE') if nn_count != 1: raise ex.InvalidComponentCountException( 'HDFS_NAMENODE', 1, nn_count) snn_count = cls.get_inst_count(cluster, 'HDFS_SECONDARYNAMENODE') if snn_count != 1: raise ex.InvalidComponentCountException('HDFS_SECONDARYNAMENODE', 1, snn_count) dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') replicas = cls.PU.get_config_value('HDFS', 'dfs_replication', cluster) if dn_count < replicas: raise ex.InvalidComponentCountException( 'HDFS_DATANODE', replicas, dn_count, _('Number of datanodes must be not' ' less than dfs_replication.')) du_reserved = cls.PU.get_config_value( 'DATANODE', 'dfs_datanode_du_reserved', cluster) du_reserved = du_reserved/1073741824. for node_group in cluster.node_groups: volume_size = node_group.volumes_size if volume_size and volume_size < du_reserved: raise ex.InvalidVolumeSizeException(volume_size, du_reserved) rm_count = cls.get_inst_count(cluster, 'YARN_RESOURCEMANAGER') if rm_count > 1: raise ex.InvalidComponentCountException('YARN_RESOURCEMANAGER', _('0 or 1'), rm_count) hs_count = cls.get_inst_count(cluster, 'YARN_JOBHISTORY') if hs_count > 1: raise ex.InvalidComponentCountException('YARN_JOBHISTORY', _('0 or 1'), hs_count) if rm_count > 0 and hs_count < 1: raise ex.RequiredServiceMissingException( 'YARN_JOBHISTORY', required_by='YARN_RESOURCEMANAGER') nm_count = cls.get_inst_count(cluster, 'YARN_NODEMANAGER') if rm_count == 0: if nm_count > 0: raise ex.RequiredServiceMissingException( 'YARN_RESOURCEMANAGER', required_by='YARN_NODEMANAGER') @classmethod def _oozie_validation(cls, cluster): oo_count = cls.get_inst_count(cluster, 'OOZIE_SERVER') dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') nm_count = cls.get_inst_count(cluster, 'YARN_NODEMANAGER') hs_count = cls.get_inst_count(cluster, 'YARN_JOBHISTORY') if oo_count > 1: raise ex.InvalidComponentCountException( 'OOZIE_SERVER', _('0 or 1'), oo_count) if oo_count == 1: if dn_count < 1: raise ex.RequiredServiceMissingException( 'HDFS_DATANODE', required_by='OOZIE_SERVER') if nm_count < 1: raise ex.RequiredServiceMissingException( 'YARN_NODEMANAGER', required_by='OOZIE_SERVER') if hs_count != 1: raise ex.RequiredServiceMissingException( 'YARN_JOBHISTORY', required_by='OOZIE_SERVER') @classmethod def _hive_validation(cls, cluster): hms_count = cls.get_inst_count(cluster, 'HIVE_METASTORE') hvs_count = cls.get_inst_count(cluster, 'HIVE_SERVER2') whc_count = cls.get_inst_count(cluster, 'HIVE_WEBHCAT') rm_count = cls.get_inst_count(cluster, 'YARN_RESOURCEMANAGER') if hms_count and rm_count < 1: raise ex.RequiredServiceMissingException( 'YARN_RESOURCEMANAGER', required_by='HIVE_METASTORE') if hms_count and not hvs_count: raise ex.RequiredServiceMissingException( 'HIVE_SERVER2', required_by='HIVE_METASTORE') if hvs_count and not hms_count: raise ex.RequiredServiceMissingException( 'HIVE_METASTORE', required_by='HIVE_SERVER2') if whc_count and not hms_count: raise ex.RequiredServiceMissingException( 'HIVE_METASTORE', required_by='WEBHCAT') @classmethod def _hue_validation(cls, cluster): hue_count = cls.get_inst_count(cluster, 'HUE_SERVER') if hue_count > 1: raise ex.InvalidComponentCountException( 'HUE_SERVER', _('0 or 1'), hue_count) shs_count = cls.get_inst_count(cluster, 'SPARK_YARN_HISTORY_SERVER') hms_count = cls.get_inst_count(cluster, 'HIVE_METASTORE') oo_count = cls.get_inst_count(cluster, 'OOZIE_SERVER') rm_count = cls.get_inst_count(cluster, 'YARN_RESOURCEMANAGER') if shs_count > 1: raise ex.InvalidComponentCountException( 'SPARK_YARN_HISTORY_SERVER', _('0 or 1'), shs_count) if shs_count and not rm_count: raise ex.RequiredServiceMissingException( 'YARN_RESOURCEMANAGER', required_by='SPARK_YARN_HISTORY_SERVER') if oo_count < 1 and hue_count: raise ex.RequiredServiceMissingException( 'OOZIE_SERVER', required_by='HUE_SERVER') if hms_count < 1 and hue_count: raise ex.RequiredServiceMissingException( 'HIVE_METASTORE', required_by='HUE_SERVER') @classmethod def _hbase_validation(cls, cluster): hbm_count = cls.get_inst_count(cluster, 'HBASE_MASTER') hbr_count = cls.get_inst_count(cluster, 'HBASE_REGIONSERVER') zk_count = cls.get_inst_count(cluster, 'ZOOKEEPER_SERVER') if hbm_count == 1: if zk_count < 1: raise ex.RequiredServiceMissingException( 'ZOOKEEPER', required_by='HBASE') if hbr_count < 1: raise ex.InvalidComponentCountException( 'HBASE_REGIONSERVER', _('at least 1'), hbr_count) elif hbm_count > 1: raise ex.InvalidComponentCountException('HBASE_MASTER', _('0 or 1'), hbm_count) elif hbr_count >= 1: raise ex.InvalidComponentCountException('HBASE_MASTER', _('at least 1'), hbm_count) @classmethod def validate_additional_ng_scaling(cls, cluster, additional): rm = cls.PU.get_resourcemanager(cluster) scalable_processes = cls._get_scalable_processes() for ng_id in additional: ng = gu.get_by_id(cluster.node_groups, ng_id) if not set(ng.node_processes).issubset(scalable_processes): msg = _("CDH plugin cannot scale nodegroup with processes: " "%(processes)s") raise ex.NodeGroupCannotBeScaled( ng.name, msg % {'processes': ' '.join(ng.node_processes)}) if not rm and 'YARN_NODEMANAGER' in ng.node_processes: msg = _("CDH plugin cannot scale node group with processes " "which have no master-processes run in cluster") raise ex.NodeGroupCannotBeScaled(ng.name, msg) @classmethod def validate_existing_ng_scaling(cls, cluster, existing): scalable_processes = cls._get_scalable_processes() dn_to_delete = 0 for ng in cluster.node_groups: if ng.id in existing: if (ng.count > existing[ng.id] and "HDFS_DATANODE" in ng.node_processes): dn_to_delete += ng.count - existing[ng.id] if not set(ng.node_processes).issubset(scalable_processes): msg = _("CDH plugin cannot scale nodegroup" " with processes: %(processes)s") raise ex.NodeGroupCannotBeScaled( ng.name, msg % {'processes': ' '.join(ng.node_processes)}) dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') - dn_to_delete replicas = cls.PU.get_config_value('HDFS', 'dfs_replication', cluster) if dn_count < replicas: raise ex.ClusterCannotBeScaled( cluster, _('Number of datanodes must be not' ' less than dfs_replication.')) @classmethod def _get_scalable_processes(cls): return ['HDFS_DATANODE', 'YARN_NODEMANAGER'] @classmethod def _flume_validation(cls, cluster): a_count = cls.get_inst_count(cluster, 'FLUME_AGENT') dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') if a_count >= 1: if dn_count < 1: raise ex.RequiredServiceMissingException( 'HDFS_DATANODE', required_by='FLUME_AGENT') @classmethod def _sentry_validation(cls, cluster): snt_count = cls.get_inst_count(cluster, 'SENTRY_SERVER') dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') zk_count = cls.get_inst_count(cluster, 'ZOOKEEPER_SERVER') if snt_count > 1: raise ex.InvalidComponentCountException( 'SENTRY_SERVER', _('0 or 1'), snt_count) if snt_count == 1: if dn_count < 1: raise ex.RequiredServiceMissingException( 'HDFS_DATANODE', required_by='SENTRY_SERVER') if zk_count < 1: raise ex.RequiredServiceMissingException( 'ZOOKEEPER', required_by='SENTRY_SERVER') @classmethod def _solr_validation(cls, cluster): slr_count = cls.get_inst_count(cluster, 'SOLR_SERVER') zk_count = cls.get_inst_count(cluster, 'ZOOKEEPER_SERVER') dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') if slr_count >= 1: if dn_count < 1: raise ex.RequiredServiceMissingException( 'HDFS_DATANODE', required_by='SOLR_SERVER') if zk_count < 1: raise ex.RequiredServiceMissingException( 'ZOOKEEPER', required_by='SOLR_SERVER') @classmethod def _sqoop_validation(cls, cluster): s2s_count = cls.get_inst_count(cluster, 'SQOOP_SERVER') dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') hs_count = cls.get_inst_count(cluster, 'YARN_JOBHISTORY') nm_count = cls.get_inst_count(cluster, 'YARN_NODEMANAGER') if s2s_count > 1: raise ex.InvalidComponentCountException( 'SQOOP_SERVER', _('0 or 1'), s2s_count) if s2s_count == 1: if dn_count < 1: raise ex.RequiredServiceMissingException( 'HDFS_DATANODE', required_by='SQOOP_SERVER') if nm_count < 1: raise ex.RequiredServiceMissingException( 'YARN_NODEMANAGER', required_by='SQOOP_SERVER') if hs_count != 1: raise ex.RequiredServiceMissingException( 'YARN_JOBHISTORY', required_by='SQOOP_SERVER') @classmethod def _hbase_indexer_validation(cls, cluster): lhbi_count = cls.get_inst_count(cluster, 'HBASE_INDEXER') zk_count = cls.get_inst_count(cluster, 'ZOOKEEPER_SERVER') dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') slr_count = cls.get_inst_count(cluster, 'SOLR_SERVER') hbm_count = cls.get_inst_count(cluster, 'HBASE_MASTER') if lhbi_count >= 1: if dn_count < 1: raise ex.RequiredServiceMissingException( 'HDFS_DATANODE', required_by='HBASE_INDEXER') if zk_count < 1: raise ex.RequiredServiceMissingException( 'ZOOKEEPER', required_by='HBASE_INDEXER') if slr_count < 1: raise ex.RequiredServiceMissingException( 'SOLR_SERVER', required_by='HBASE_INDEXER') if hbm_count < 1: raise ex.RequiredServiceMissingException( 'HBASE_MASTER', required_by='HBASE_INDEXER') @classmethod def _impala_validation(cls, cluster): ics_count = cls.get_inst_count(cluster, 'IMPALA_CATALOGSERVER') iss_count = cls.get_inst_count(cluster, 'IMPALA_STATESTORE') id_count = cls.get_inst_count(cluster, 'IMPALAD') dn_count = cls.get_inst_count(cluster, 'HDFS_DATANODE') hms_count = cls.get_inst_count(cluster, 'HIVE_METASTORE') if ics_count > 1: raise ex.InvalidComponentCountException('IMPALA_CATALOGSERVER', _('0 or 1'), ics_count) if iss_count > 1: raise ex.InvalidComponentCountException('IMPALA_STATESTORE', _('0 or 1'), iss_count) if ics_count == 1: datanode_ng = u.get_node_groups(cluster, "HDFS_DATANODE") impalad_ng = u.get_node_groups(cluster, "IMPALAD") datanodes = set(ng.id for ng in datanode_ng) impalads = set(ng.id for ng in impalad_ng) if datanodes != impalads: raise ex.InvalidClusterTopology( _("IMPALAD must be installed on every HDFS_DATANODE")) if iss_count != 1: raise ex.RequiredServiceMissingException( 'IMPALA_STATESTORE', required_by='IMPALA') if id_count < 1: raise ex.RequiredServiceMissingException( 'IMPALAD', required_by='IMPALA') if dn_count < 1: raise ex.RequiredServiceMissingException( 'HDFS_DATANODE', required_by='IMPALA') if hms_count < 1: raise ex.RequiredServiceMissingException( 'HIVE_METASTORE', required_by='IMPALA') @classmethod def _kms_validation(cls, cluster): kms_count = cls.get_inst_count(cluster, 'KMS') if kms_count > 1: raise ex.InvalidComponentCountException('KMS', _('0 or 1'), kms_count) @classmethod def _hdfs_ha_validation(cls, cluster): jn_count = cls.get_inst_count(cluster, 'HDFS_JOURNALNODE') zk_count = cls.get_inst_count(cluster, 'ZOOKEEPER_SERVER') require_anti_affinity = cls.PU.c_helper.get_required_anti_affinity( cluster) if jn_count > 0: if jn_count < 3: raise ex.InvalidComponentCountException('HDFS_JOURNALNODE', _('not less than 3'), jn_count) if not jn_count % 2: raise ex.InvalidComponentCountException('HDFS_JOURNALNODE', _('be odd'), jn_count) if zk_count < 1: raise ex.RequiredServiceMissingException('ZOOKEEPER', required_by='HDFS HA') if require_anti_affinity: if 'HDFS_SECONDARYNAMENODE' not in \ cls._get_anti_affinity(cluster): raise ex.NameNodeHAConfigurationError( _('HDFS_SECONDARYNAMENODE should be enabled ' 'in anti_affinity.')) if 'HDFS_NAMENODE' not in cls._get_anti_affinity(cluster): raise ex.NameNodeHAConfigurationError( _('HDFS_NAMENODE should be enabled in anti_affinity.')) @classmethod def _yarn_ha_validation(cls, cluster): rm_count = cls.get_inst_count(cluster, 'YARN_RESOURCEMANAGER') zk_count = cls.get_inst_count(cluster, 'ZOOKEEPER_SERVER') stdb_rm_count = cls.get_inst_count(cluster, 'YARN_STANDBYRM') require_anti_affinity = cls.PU.c_helper.get_required_anti_affinity( cluster) if stdb_rm_count > 1: raise ex.InvalidComponentCountException( 'YARN_STANDBYRM', _('0 or 1'), stdb_rm_count) if stdb_rm_count > 0: if rm_count < 1: raise ex.RequiredServiceMissingException( 'YARN_RESOURCEMANAGER', required_by='RM HA') if zk_count < 1: raise ex.RequiredServiceMissingException( 'ZOOKEEPER', required_by='RM HA') if require_anti_affinity: if 'YARN_RESOURCEMANAGER' not in \ cls._get_anti_affinity(cluster): raise ex.ResourceManagerHAConfigurationError( _('YARN_RESOURCEMANAGER should be enabled in ' 'anti_affinity.')) if 'YARN_STANDBYRM' not in cls._get_anti_affinity(cluster): raise ex.ResourceManagerHAConfigurationError( _('YARN_STANDBYRM should be' ' enabled in anti_affinity.')) @classmethod def _get_anti_affinity(cls, cluster): return cluster.anti_affinity @classmethod def get_inst_count(cls, cluster, process): return sum([ng.count for ng in u.get_node_groups(cluster, process)]) sahara-8.0.0/sahara/plugins/cdh/v5_11_0/0000775000175100017510000000000013245515026017535 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_11_0/deploy.py0000666000175100017510000001260413245514472021414 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ from sahara.plugins.cdh import commands as cmd from sahara.plugins.cdh import deploy as common_deploy from sahara.plugins.cdh.v5_11_0 import cloudera_utils as cu from sahara.plugins import utils as gu from sahara.service.edp import hdfs_helper as h from sahara.utils import cluster_progress_ops as cpo CU = cu.ClouderaUtilsV5110() PACKAGES = common_deploy.PACKAGES def configure_cluster(cluster): instances = gu.get_instances(cluster) if not cmd.is_pre_installed_cdh(CU.pu.get_manager(cluster).remote()): CU.pu.configure_os(instances) CU.pu.install_packages(instances, PACKAGES) CU.pu.start_cloudera_agents(instances) CU.pu.start_cloudera_manager(cluster) CU.update_cloudera_password(cluster) CU.configure_rack_awareness(cluster) CU.await_agents(cluster, instances) CU.create_mgmt_service(cluster) CU.create_services(cluster) CU.configure_services(cluster) CU.configure_instances(instances, cluster) CU.deploy_configs(cluster) @cpo.event_wrapper( True, step=_("Start roles: NODEMANAGER, DATANODE"), param=('cluster', 0)) def _start_roles(cluster, instances): for instance in instances: if 'HDFS_DATANODE' in instance.node_group.node_processes: hdfs = CU.get_service_by_role('DATANODE', instance=instance) CU.start_roles(hdfs, CU.pu.get_role_name(instance, 'DATANODE')) if 'YARN_NODEMANAGER' in instance.node_group.node_processes: yarn = CU.get_service_by_role('NODEMANAGER', instance=instance) CU.start_roles(yarn, CU.pu.get_role_name(instance, 'NODEMANAGER')) def scale_cluster(cluster, instances): if not instances: return if not cmd.is_pre_installed_cdh(instances[0].remote()): CU.pu.configure_os(instances) CU.pu.install_packages(instances, PACKAGES) CU.pu.start_cloudera_agents(instances) CU.await_agents(cluster, instances) CU.configure_rack_awareness(cluster) CU.configure_instances(instances, cluster) CU.update_configs(instances) common_deploy.prepare_scaling_kerberized_cluster( cluster, CU, instances) CU.pu.configure_swift(cluster, instances) _start_roles(cluster, instances) CU.refresh_datanodes(cluster) CU.refresh_yarn_nodes(cluster) CU.restart_stale_services(cluster) def decommission_cluster(cluster, instances): dns = [] dns_to_delete = [] nms = [] nms_to_delete = [] for i in instances: if 'HDFS_DATANODE' in i.node_group.node_processes: dns.append(CU.pu.get_role_name(i, 'DATANODE')) dns_to_delete.append( CU.pu.get_role_name(i, 'HDFS_GATEWAY')) if 'YARN_NODEMANAGER' in i.node_group.node_processes: nms.append(CU.pu.get_role_name(i, 'NODEMANAGER')) nms_to_delete.append( CU.pu.get_role_name(i, 'YARN_GATEWAY')) if dns: CU.decommission_nodes( cluster, 'DATANODE', dns, dns_to_delete) if nms: CU.decommission_nodes( cluster, 'NODEMANAGER', nms, nms_to_delete) CU.delete_instances(cluster, instances) CU.refresh_datanodes(cluster) CU.refresh_yarn_nodes(cluster) CU.restart_stale_services(cluster) @cpo.event_wrapper(True, step=_("Prepare cluster"), param=('cluster', 0)) def _prepare_cluster(cluster): if CU.pu.get_oozie(cluster): CU.pu.install_extjs(cluster) if CU.pu.get_hive_metastore(cluster): CU.pu.configure_hive(cluster) if CU.pu.get_sentry(cluster): CU.pu.configure_sentry(cluster) @cpo.event_wrapper( True, step=_("Finish cluster starting"), param=('cluster', 0)) def _finish_cluster_starting(cluster): if CU.pu.get_hive_metastore(cluster): CU.pu.put_hive_hdfs_xml(cluster) server = CU.pu.get_hbase_master(cluster) if CU.pu.c_helper.is_hbase_common_lib_enabled(cluster) and server: with server.remote() as r: h.create_hbase_common_lib(r) if CU.pu.get_flumes(cluster): flume = CU.get_service_by_role('AGENT', cluster) CU.start_service(flume) def start_cluster(cluster): _prepare_cluster(cluster) CU.first_run(cluster) CU.pu.configure_swift(cluster) if len(CU.pu.get_jns(cluster)) > 0: CU.enable_namenode_ha(cluster) # updating configs for NameNode role on needed nodes CU.update_role_config(CU.pu.get_secondarynamenode(cluster), 'HDFS_NAMENODE') if CU.pu.get_stdb_rm(cluster): CU.enable_resourcemanager_ha(cluster) # updating configs for ResourceManager on needed nodes CU.update_role_config(CU.pu.get_stdb_rm(cluster), 'YARN_STANDBYRM') _finish_cluster_starting(cluster) common_deploy.setup_kerberos_for_cluster(cluster, CU) def get_open_ports(node_group): ports = common_deploy.get_open_ports(node_group) return ports sahara-8.0.0/sahara/plugins/cdh/v5_11_0/validation.py0000666000175100017510000000137013245514472022250 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_11_0 import plugin_utils as pu from sahara.plugins.cdh import validation class ValidatorV5110(validation.Validator): PU = pu.PluginUtilsV5110() sahara-8.0.0/sahara/plugins/cdh/v5_11_0/versionhandler.py0000666000175100017510000000271513245514472023145 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import abstractversionhandler as avm from sahara.plugins.cdh.v5_11_0 import cloudera_utils from sahara.plugins.cdh.v5_11_0 import config_helper from sahara.plugins.cdh.v5_11_0 import deploy from sahara.plugins.cdh.v5_11_0 import edp_engine from sahara.plugins.cdh.v5_11_0 import images from sahara.plugins.cdh.v5_11_0 import plugin_utils from sahara.plugins.cdh.v5_11_0 import validation class VersionHandler(avm.BaseVersionHandler): def __init__(self): super(VersionHandler, self).__init__() self.config_helper = config_helper.ConfigHelperV5110() self.cloudera_utils = cloudera_utils.ClouderaUtilsV5110() self.plugin_utils = plugin_utils.PluginUtilsV5110() self.deploy = deploy self.edp_engine = edp_engine self.images = images self.validation = validation.ValidatorV5110() sahara-8.0.0/sahara/plugins/cdh/v5_11_0/config_helper.py0000666000175100017510000000740313245514472022725 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import config_helper as c_h from sahara.plugins import provisioning as p from sahara.utils import files as f class ConfigHelperV5110(c_h.ConfigHelper): path_to_config = 'plugins/cdh/v5_11_0/resources/' CDH5_UBUNTU_REPO = ( 'deb [arch=amd64] http://archive.cloudera.com/cdh5' '/ubuntu/xenial/amd64/cdh trusty-cdh5.11.0 contrib' '\ndeb-src http://archive.cloudera.com/cdh5/ubuntu' '/xenial/amd64/cdh trusty-cdh5.11.0 contrib') DEFAULT_CDH5_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/cdh5/ubuntu' '/xenial/amd64/cdh/archive.key') CM5_UBUNTU_REPO = ( 'deb [arch=amd64] http://archive.cloudera.com/cm5' '/ubuntu/xenial/amd64/cm trusty-cm5.11.0 contrib' '\ndeb-src http://archive.cloudera.com/cm5/ubuntu' '/xenial/amd64/cm trusty-cm5.11.0 contrib') DEFAULT_CM5_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/cm5/ubuntu' '/xenial/amd64/cm/archive.key') CDH5_CENTOS_REPO = ( '[cloudera-cdh5]' '\nname=Cloudera\'s Distribution for Hadoop, Version 5' '\nbaseurl=http://archive.cloudera.com/cdh5/redhat/6' '/x86_64/cdh/5.11.0/' '\ngpgkey = http://archive.cloudera.com/cdh5/redhat/6' '/x86_64/cdh/RPM-GPG-KEY-cloudera' '\ngpgcheck = 1') CM5_CENTOS_REPO = ( '[cloudera-manager]' '\nname=Cloudera Manager' '\nbaseurl=http://archive.cloudera.com/cm5/redhat/6' '/x86_64/cm/5.11.0/' '\ngpgkey = http://archive.cloudera.com/cm5/redhat/6' '/x86_64/cm/RPM-GPG-KEY-cloudera' '\ngpgcheck = 1') KEY_TRUSTEE_UBUNTU_REPO_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/ubuntu/xenial/amd64/navigator-' 'keytrustee/cloudera.list') DEFAULT_KEY_TRUSTEE_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/ubuntu/xenial/amd64/navigator-' 'keytrustee/archive.key') KEY_TRUSTEE_CENTOS_REPO_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/redhat/6/x86_64/navigator-' 'keytrustee/navigator-keytrustee5.repo') DEFAULT_SWIFT_LIB_URL = ( 'https://repository.cloudera.com/artifactory/repo/org' '/apache/hadoop/hadoop-openstack/2.6.0-cdh5.11.0' '/hadoop-openstack-2.6.0-cdh5.11.0.jar') SWIFT_LIB_URL = p.Config( 'Hadoop OpenStack library URL', 'general', 'cluster', priority=1, default_value=DEFAULT_SWIFT_LIB_URL, description=("Library that adds Swift support to CDH. The file" " will be downloaded by VMs.")) HIVE_SERVER2_SENTRY_SAFETY_VALVE = f.get_file_text( path_to_config + 'hive-server2-sentry-safety.xml') HIVE_METASTORE_SENTRY_SAFETY_VALVE = f.get_file_text( path_to_config + 'hive-metastore-sentry-safety.xml') SENTRY_IMPALA_CLIENT_SAFETY_VALVE = f.get_file_text( path_to_config + 'sentry-impala-client-safety.xml') def __init__(self): super(ConfigHelperV5110, self).__init__() self.priority_one_confs = self._load_json( self.path_to_config + 'priority-one-confs.json') self._init_all_ng_plugin_configs() sahara-8.0.0/sahara/plugins/cdh/v5_11_0/images.py0000666000175100017510000000277513245514472021375 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import images from sahara.plugins import utils as plugin_utils _validator = images.SaharaImageValidator.from_yaml( 'plugins/cdh/v5_11_0/resources/images/image.yaml', resource_roots=['plugins/cdh/v5_11_0/resources/images']) def get_image_arguments(): return _validator.get_argument_list() def pack_image(remote, test_only=False, image_arguments=None): _validator.validate(remote, test_only=test_only, image_arguments=image_arguments) def validate_images(cluster, test_only=False, image_arguments=None): image_arguments = get_image_arguments() if not test_only: instances = plugin_utils.get_instances(cluster) else: instances = plugin_utils.get_instances(cluster)[0] for instance in instances: with instance.remote() as r: _validator.validate(r, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/cdh/v5_11_0/cloudera_utils.py0000666000175100017510000000204013245514472023127 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import cloudera_utils as cu from sahara.plugins.cdh.v5_11_0 import config_helper from sahara.plugins.cdh.v5_11_0 import plugin_utils as pu from sahara.plugins.cdh.v5_11_0 import validation class ClouderaUtilsV5110(cu.ClouderaUtils): def __init__(self): cu.ClouderaUtils.__init__(self) self.pu = pu.PluginUtilsV5110() self.validator = validation.ValidatorV5110 self.c_helper = config_helper.ConfigHelperV5110() sahara-8.0.0/sahara/plugins/cdh/v5_11_0/__init__.py0000666000175100017510000000000013245514472021642 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_11_0/plugin_utils.py0000666000175100017510000000146313245514472022637 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import plugin_utils as pu from sahara.plugins.cdh.v5_11_0 import config_helper class PluginUtilsV5110(pu.AbstractPluginUtils): def __init__(self): self.c_helper = config_helper.ConfigHelperV5110() sahara-8.0.0/sahara/plugins/cdh/v5_11_0/edp_engine.py0000666000175100017510000000367213245514472022222 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import confighints_helper as ch_helper from sahara.plugins.cdh import edp_engine from sahara.plugins.cdh.v5_11_0 import cloudera_utils as cu from sahara.service.edp.oozie import engine as oozie_engine from sahara.utils import edp class EdpOozieEngine(edp_engine.EdpOozieEngine): def __init__(self, cluster): super(EdpOozieEngine, self).__init__(cluster) self.cloudera_utils = cu.ClouderaUtilsV5110() @staticmethod def get_possible_job_config(job_type): if edp.compare_job_type(job_type, edp.JOB_TYPE_HIVE): return {'job_config': ch_helper.get_possible_hive_config_from( 'plugins/cdh/v5_11_0/resources/hive-site.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_MAPREDUCE_STREAMING): return {'job_config': ch_helper.get_possible_mapreduce_config_from( 'plugins/cdh/v5_11_0/resources/mapred-site.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_PIG): return {'job_config': ch_helper.get_possible_pig_config_from( 'plugins/cdh/v5_11_0/resources/mapred-site.xml')} return oozie_engine.OozieJobEngine.get_possible_job_config(job_type) class EdpSparkEngine(edp_engine.EdpSparkEngine): edp_base_version = "5.11.0" sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/0000775000175100017510000000000013245515026021547 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/sentry-sentry_server.json0000666000175100017510000004776013245514472026722 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When computing the overall Sentry Server health, consider the host's health.", "display_name": "Sentry Server Host Health Test", "name": "sentry_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_sentry_server_audit_health", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_sentry_server_unexpected_exits", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Sentry Server in Bytes", "name": "sentry_server_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Sentry Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Sentry Server", "name": "role_config_suppression_sentry_server_java_opts", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Sentry Server logs. Typically used by log4j or logback.", "display_name": "Sentry Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Log Directory", "name": "role_config_suppression_sentry_server_log_dir", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Sentry Server", "name": "sentry_server_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_sentry_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Directory where Sentry Server will place its log files.", "display_name": "Sentry Server Log Directory", "name": "sentry_server_log_dir", "value": "/var/log/sentry" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Sentry Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_sentry_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_sentry_server_host_health", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_sentry_server_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "RPC port number of Sentry Server.", "display_name": "Sentry Server RPC Port", "name": "sentry_service_server_rpc_port", "value": "8038" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_sentry_server_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Sentry Server logs. Typically used by log4j or logback.", "display_name": "Sentry Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Enables the health test that the Sentry Server's process state is consistent with the role configuration", "display_name": "Sentry Server Process Health Test", "name": "sentry_server_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "sentry_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Sentry Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "sentry_env_safety_valve", "value": null }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for Sentry Server logs", "display_name": "Sentry Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_sentry_server_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_sentry_server_swap_memory_usage", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/kafka-gateway.json0000666000175100017510000000615613245514472025174 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into the client configuration for kafka-conf/kafka-client.conf.", "display_name": "Kafka Client Advanced Configuration Snippet (Safety Valve) for kafka-conf/kafka-client.conf", "name": "kafka-conf/kafka-client.conf_client_config_safety_valve", "value": null }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/kafka" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "50" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Client Advanced Configuration Snippet (Safety Valve) for kafka-conf/kafka-client.conf parameter.", "display_name": "Suppress Parameter Validation: Kafka Client Advanced Configuration Snippet (Safety Valve) for kafka-conf/kafka-client.conf", "name": "role_config_suppression_kafka-conf/kafka-client.conf_client_config_safety_valve", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/solr-service.json0000666000175100017510000005457513245514472025105 0ustar zuulzuul00000000000000[ { "desc": "If Solr does not respond on its web URL within this time interval, the Catalina process is killed.", "display_name": "Solrd Watchdog Timeout", "name": "solrd_watchdog_timeout", "value": "70" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_solr_sentry_safety_valve", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Solr might connect to. This is used when Solr is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Solr TLS/SSL Certificate Trust Store File", "name": "solr_https_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Solr Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Solr Server Count Validator", "name": "service_config_suppression_solr_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP Secure URI and Start TLS Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP Secure URI and Start TLS Validator", "name": "service_config_suppression_solr_ldaps_or_tls_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_solr_core_site_safety_valve", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "solr" }, { "desc": "Name of the Sentry service that this Solr service instance depends on. If selected, Solr uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_solr_ldap_uri", "value": "false" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "ldap_domain", "value": null }, { "desc": "The password for the Solr TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Solr TLS/SSL Certificate Trust Store Password", "name": "solr_https_truststore_password", "value": null }, { "desc": "The password for the Solr JKS keystore file.", "display_name": "Solr TLS/SSL Server JKS Keystore File Password", "name": "solr_https_keystore_password", "value": null }, { "desc": "Name of the HDFS service that this Search service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "solr_ldap_enable_starttls", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_solr_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Data Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Data Directory", "name": "service_config_suppression_hdfs_data_dir", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Search service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_solr_https_truststore_file", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "solr_env_safety_valve", "value": null }, { "desc": "HDFS directory used for storage by this Solr service.", "display_name": "HDFS Data Directory", "name": "hdfs_data_dir", "value": "/solr" }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "sentry_solr_provider_resource", "value": "/user/solr/sentry/sentry-provider.ini" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "solr" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "solr_hdfs_site_safety_valve", "value": null }, { "desc": "The health test thresholds of the overall Solr Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Solr Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Solr Servers falls below the critical threshold.", "display_name": "Healthy Solr Server Monitoring Thresholds", "name": "solr_solr_servers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_sentry_solr_provider_resource", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solrd Watchdog Timeout parameter.", "display_name": "Suppress Parameter Validation: Solrd Watchdog Timeout", "name": "service_config_suppression_solrd_watchdog_timeout", "value": "false" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "solr_enable_ldap_auth", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "solr" }, { "desc": "Choose the authentication mechanism used by Solr.", "display_name": "Solr Secure Authentication", "name": "solr_security_authentication", "value": "simple" }, { "desc": "Encrypt communication between clients and Solr using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)). Additional manual steps must be performed; see Enabling TLS/SSL for Solr.", "display_name": "Enable TLS/SSL for Solr", "name": "solr_use_ssl", "value": "false" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access.", "display_name": "Sentry User to Group Mapping Class", "name": "solr_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "solr_core_site_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP TLS Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP TLS Validator", "name": "service_config_suppression_solr_ldap_tls_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using Sentry service instead (supported in CDH 5.8 and later), add Sentry service as a dependency to Solr service. Sentry service provides concurrent and secure access to authorization policy metadata and is recommended for enabling Sentry. Sentry with policy files is supported only on CDH 5 or later deployments. Sentry requires authentication to be turned on for Solr.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "solr_sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_solr_https_keystore_password", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "solr_sentry_safety_valve", "value": null }, { "desc": "Enable the background watchdog thread that can kill Catalina process if Solr is not responsive.", "display_name": "Enable Solrd Watchdog", "name": "solrd_enable_watchdog", "value": "true" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "service_config_suppression_solr_hdfs_site_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "solr_ldap_uri", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode", "name": "service_config_suppression_zookeeper_znode", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Solr is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Solr TLS/SSL Server JKS Keystore File Location", "name": "solr_https_keystore_file", "value": null }, { "desc": "ZooKeeper znode used to store information about this Solr service.", "display_name": "ZooKeeper Znode", "name": "zookeeper_znode", "value": "/solr" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_solr_ldap_basedn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_solr_https_keystore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_solr_https_truststore_password", "value": "false" }, { "desc": "Whether to suppress the results of the Solr Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Solr Server Health", "name": "service_health_suppression_solr_solr_servers_healthy", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "This parameter is useful when authenticating against a non-Active Directory server, such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,<this parameter>. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "solr_ldap_basedn", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_ldap_domain", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/flume-agent.json0000666000175100017510000012444113245514472024662 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Used to select an agent configuration to use from flume.conf. Multiple agents can share the same agent name, in which case they are assigned the same agent configuration.", "display_name": "Agent Name", "name": "agent_name", "value": "tier1" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Text that goes into morphlines.conf file used by the Flume-NG Solr sink. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service.", "display_name": "Morphlines File", "name": "agent_morphlines_conf_file", "value": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Application configuration file in HOCON format (Human-Optimized Config Object Notation). \n# HOCON syntax is defined at http://github.com/typesafehub/config/blob/master/HOCON.md\n# and also used by Akka (http://www.akka.io) and Play (http://www.playframework.org/).\n# For more examples see http://doc.akka.io/docs/akka/2.1.2/general/configuration.html\n\n# morphline.conf example file\n# this is a comment\n\n# Specify server locations in a SOLR_LOCATOR variable; used later in variable substitutions:\nSOLR_LOCATOR : {\n # Name of solr collection\n collection : collection1\n \n # ZooKeeper ensemble\n zkHost : \"$ZK_HOST\"\n \n # Relative or absolute path to a directory containing conf/solrconfig.xml and conf/schema.xml\n # If this path is uncommented it takes precedence over the configuration stored in ZooKeeper. \n # solrHomeDir : \"example/solr/collection1\"\n \n # The maximum number of documents to send to Solr per network batch (throughput knob)\n # batchSize : 100\n}\n\n# Specify an array of one or more morphlines, each of which defines an ETL \n# transformation chain. A morphline consists of one or more (potentially \n# nested) commands. A morphline is a way to consume records (e.g. Flume events, \n# HDFS files or blocks), turn them into a stream of records, and pipe the stream \n# of records through a set of easily configurable transformations on it's way to \n# Solr (or a MapReduceIndexerTool RecordWriter that feeds via a Reducer into Solr).\nmorphlines : [\n {\n # Name used to identify a morphline. E.g. used if there are multiple morphlines in a \n # morphline config file\n id : morphline1 \n \n # Import all morphline commands in these java packages and their subpackages.\n # Other commands that may be present on the classpath are not visible to this morphline.\n importCommands : [\"org.kitesdk.**\", \"org.apache.solr.**\"]\n \n commands : [ \n { \n # Parse Avro container file and emit a record for each avro object\n readAvroContainer {\n # Optionally, require the input record to match one of these MIME types:\n # supportedMimeTypes : [avro/binary]\n \n # Optionally, use a custom Avro schema in JSON format inline:\n # schemaString : \"\"\"\"\"\"\n \n # Optionally, use a custom Avro schema file in JSON format:\n # schemaFile : /path/to/syslog.avsc\n }\n } \n \n { \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # extractAvroPaths is a command that uses zero or more avro path expressions to extract \n # values from an Avro object. Each expression consists of a record output field name (on \n # the left side of the colon ':') as well as zero or more path steps (on the right hand \n # side), each path step separated by a '/' slash. Avro arrays are traversed with the '[]'\n # notation.\n #\n # The result of a path expression is a list of objects, each of which is added to the \n # given record output field.\n # \n # The path language supports all Avro concepts, including nested structures, records, \n # arrays, maps, unions, etc, as well as a flatten option that collects the primitives in \n # a subtree into a flat list.\n extractAvroPaths {\n flatten : false\n paths : { \n id : /id \n text : /text \n user_friends_count : /user_friends_count\n user_location : /user_location\n user_description : /user_description\n user_statuses_count : /user_statuses_count\n user_followers_count : /user_followers_count\n user_name : /user_name\n user_screen_name : /user_screen_name\n created_at : /created_at\n retweet_count : /retweet_count\n retweeted : /retweeted\n in_reply_to_user_id : /in_reply_to_user_id\n source : /source\n in_reply_to_status_id : /in_reply_to_status_id\n media_url_https : /media_url_https\n expanded_url : /expanded_url\n }\n }\n }\n \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # convert timestamp field to native Solr timestamp format\n # e.g. 2012-09-06T07:14:34Z to 2012-09-06T07:14:34.000Z\n {\n convertTimestamp {\n field : created_at\n inputFormats : [\"yyyy-MM-dd'T'HH:mm:ss'Z'\", \"yyyy-MM-dd\"]\n inputTimezone : America/Los_Angeles\n# outputFormat : \"yyyy-MM-dd'T'HH:mm:ss.SSSZ\" \n outputTimezone : UTC\n }\n }\n \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # Command that sanitizes record fields that are unknown to Solr schema.xml by either \n # deleting them (renameToPrefix is absent or a zero length string), or by moving them to a\n # field prefixed with the given renameToPrefix (e.g. renameToPrefix = \"ignored_\" to use \n # typical dynamic Solr fields).\n #\n # Recall that Solr throws an exception on any attempt to load a document that contains a \n # field that isn't specified in schema.xml.\n {\n sanitizeUnknownSolrFields {\n # Location from which to fetch Solr schema\n solrLocator : ${SOLR_LOCATOR}\n \n # renameToPrefix : \"ignored_\"\n }\n } \n \n # log the record at DEBUG level to SLF4J\n { logDebug { format : \"output record: {}\", args : [\"@{}\"] } } \n \n # load the record into a SolrServer or MapReduce SolrOutputFormat.\n { \n loadSolr {\n solrLocator : ${SOLR_LOCATOR}\n }\n }\n ]\n }\n]\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Plugin directories parameter.", "display_name": "Suppress Parameter Validation: Plugin directories", "name": "role_config_suppression_agent_plugin_dirs", "value": "false" }, { "desc": "When computing the overall Agent health, consider the host's health.", "display_name": "Agent Host Health Test", "name": "flume_agent_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "agent_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The maximum size, in megabytes, per log file for Agent logs. Typically used by log4j or logback.", "display_name": "Agent Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_flume_agent_swap_memory_usage", "value": "false" }, { "desc": "Text that goes verbatim into grok-dictionary.conf file used by the Flume-NG Solr sink.", "display_name": "Grok Dictionary File", "name": "agent_grok_dictionary_conf_file", "value": "USERNAME [a-zA-Z0-9._-]+\nUSER %{USERNAME}\nINT (?:[+-]?(?:[0-9]+))\nBASE10NUM (?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))\nNUMBER (?:%{BASE10NUM})\nBASE16NUM (?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))\nUUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}\n\n# Networking\nMAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})\nCISCOMAC (?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})\nWINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})\nCOMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})\nIP (?/(?>[\\w_%!$@:.,-]+|\\\\.)*)+\n#UNIXPATH (?/dev/pts/%{NONNEGINT})\nBSDTTY (?>/dev/tty[pq][a-z0-9])\nTTY (?:%{BSDTTY}|%{LINUXTTY})\nWINPATH (?>[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+\nURIPROTO [A-Za-z]+(\\+[A-Za-z+]+)?\nURIHOST %{IPORHOST}(?::%{POSINT:port})?\n# uripath comes loosely from RFC1738, but mostly from what Firefox\n# doesn't turn into %XX\nURIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=#%_\\-]*)+\n#URIPARAM \\?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?\nURIPARAM \\?[A-Za-z0-9$.+!*'|(){},~#%&/=:;_?\\-\\[\\]]*\nURIPATHPARAM %{URIPATH}(?:%{URIPARAM})?\nURI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?\n\n# Months: January, Feb, 3, 03, 12, December\nMONTH \\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b\nMONTHNUM (?:0?[1-9]|1[0-2])\nMONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\n\n# Days: Monday, Tue, Thu, etc...\nDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)\n\n# Years?\nYEAR (?>\\d\\d){1,2}\n# Time: HH:MM:SS\n#TIME \\d{2}:\\d{2}(?::\\d{2}(?:\\.\\d+)?)?\n# I'm still on the fence about using grok to perform the time match,\n# since it's probably slower.\n# TIME %{POSINT<24}:%{POSINT<60}(?::%{POSINT<60}(?:\\.%{POSINT})?)?\nHOUR (?:2[0123]|[01]?[0-9])\nMINUTE (?:[0-5][0-9])\n# '60' is a leap second in most time standards and thus is valid.\nSECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\nTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])\n# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)\nDATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}\nDATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}\nISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))\nISO8601_SECOND (?:%{SECOND}|60)\nTIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?\nDATE %{DATE_US}|%{DATE_EU}\nDATESTAMP %{DATE}[- ]%{TIME}\nTZ (?:[PMCE][SD]T)\nDATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}\nDATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}\n\n# Syslog Dates: Month Day HH:MM:SS\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}\nPROG (?:[\\w._/%-]+)\nSYSLOGPROG %{PROG:program}(?:\\[%{POSINT:pid}\\])?\nSYSLOGHOST %{IPORHOST}\nSYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>\nHTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}\n\n# Shortcuts\nQS %{QUOTEDSTRING}\n\n# Log formats\nSYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:\nCOMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}\n\n# Log Levels\nLOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Name parameter.", "display_name": "Suppress Parameter Validation: Agent Name", "name": "role_config_suppression_agent_name", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_flume_agent_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Morphlines File parameter.", "display_name": "Suppress Parameter Validation: Morphlines File", "name": "role_config_suppression_agent_morphlines_conf_file", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "agent_web_metric_collection_enabled", "value": "true" }, { "desc": "

Verbatim contents of flume.conf. Multiple agents may be configured from the same configuration file; the Agent Name setting can be overridden to select which agent configuration to use for each agent.

To integrate with a secured cluster, you can use the substitution strings \"$KERBEROS_PRINCIPAL\" and \"$KERBEROS_KEYTAB\", which will be replaced by the principal name and the keytab path respectively.

", "display_name": "Configuration File", "name": "agent_config_file", "value": "# Please paste flume.conf here. Example:\n\n# Sources, channels, and sinks are defined per\n# agent name, in this case 'tier1'.\ntier1.sources = source1\ntier1.channels = channel1\ntier1.sinks = sink1\n\n# For each source, channel, and sink, set\n# standard properties.\ntier1.sources.source1.type = netcat\ntier1.sources.source1.bind = 127.0.0.1\ntier1.sources.source1.port = 9999\ntier1.sources.source1.channels = channel1\ntier1.channels.channel1.type = memory\ntier1.sinks.sink1.type = logger\ntier1.sinks.sink1.channel = channel1\n\n# Other properties are specific to each type of\n# source, channel, or sink. In this case, we\n# specify the capacity of the memory channel.\ntier1.channels.channel1.capacity = 100\n" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_flume_agent_file_descriptor", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Agent logs. Typically used by log4j or logback.", "display_name": "Agent Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Agent Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Custom Mime-types File parameter.", "display_name": "Suppress Parameter Validation: Custom Mime-types File", "name": "role_config_suppression_agent_custom_mimetypes_file", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Agent Log Directory parameter.", "display_name": "Suppress Parameter Validation: Flume Agent Log Directory", "name": "role_config_suppression_flume_agent_log_dir", "value": "false" }, { "desc": "Enables the health test that the Agent's process state is consistent with the role configuration", "display_name": "Agent Process Health Test", "name": "flume_agent_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_flume_agent_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_flume_agent_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "flume_agent_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The port on which the Flume web server listens for requests.", "display_name": "HTTP Port", "name": "agent_http_port", "value": "41414" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Agent Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Agent in Bytes", "name": "agent_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Configuration File parameter.", "display_name": "Suppress Parameter Validation: Configuration File", "name": "role_config_suppression_agent_config_file", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Disables import of ZooKeeper configuration from the HBase classpath. This prevents zoo.cfg from overriding hbase-site.xml for Zookeeper quorum information. This option is only supported on CDH 4.4 or later deployments.", "display_name": "HBase sink prefer hbase-site.xml over Zookeeper config", "name": "agent_disable_zoo_cfg", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Agent Environment Advanced Configuration Snippet (Safety Valve)", "name": "AGENT_role_env_safety_valve", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Directory where Flume Agent will place its log files.", "display_name": "Flume Agent Log Directory", "name": "flume_agent_log_dir", "value": "/var/log/flume-ng" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Flume Agent parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Flume Agent", "name": "role_config_suppression_flume_agent_java_opts", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Grok Dictionary File parameter.", "display_name": "Suppress Parameter Validation: Grok Dictionary File", "name": "role_config_suppression_agent_grok_dictionary_conf_file", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_flume_agent_log_directory_free_space", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Text that goes verbatim into custom-mimetypes.xml file used by the Flume-NG Solr sink.", "display_name": "Custom Mime-types File", "name": "agent_custom_mimetypes_file", "value": "\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Home Directory parameter.", "display_name": "Suppress Parameter Validation: Flume Home Directory", "name": "role_config_suppression_agent_home_dir", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here. Note that Flume agent only uses options that start with -D and -X (including -XX).", "display_name": "Java Configuration Options for Flume Agent", "name": "flume_agent_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_flume_agent_scm_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_flume_agent_host_health", "value": "false" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_flume_agent_audit_health", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Agent Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_agent_role_env_safety_valve", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "List of Flume plugin directories. This overrides the default Flume plugin directory.", "display_name": "Plugin directories", "name": "agent_plugin_dirs", "value": "/usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d" }, { "desc": "Home directory for Flume user. The File Channel uses paths for checkpoint and data directories that are within the user home.", "display_name": "Flume Home Directory", "name": "agent_home_dir", "value": "/var/lib/flume-ng" }, { "desc": "The minimum log level for Agent logs", "display_name": "Agent Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/sentry-impala-client-safety.xml0000666000175100017510000000063613245514472027636 0ustar zuulzuul00000000000000 sentry.service.client.server.rpc-port 3893 sentry.service.client.server.rpc-address hostname sentry.service.client.server.rpc-connection-timeout 200000 sentry.service.security.mode none sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-failovercontroller.json0000666000175100017510000006006013245514472027307 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where Failover Controller will place its log files.", "display_name": "Failover Controller Log Directory", "name": "failover_controller_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hdfs_failovercontroller_swap_memory_usage", "value": "false" }, { "desc": "The RPC timeout for the HA health monitor.", "display_name": "HA Health Monitor RPC Timeout", "name": "ha_health_monitor_rpc_timeout_ms", "value": "45000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "failovercontroller_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hdfs_failovercontroller_unexpected_exits", "value": "false" }, { "desc": "When computing the overall Failover Controller health, consider the host's health.", "display_name": "Failover Controller Host Health Test", "name": "failovercontroller_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_failovercontroller_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hdfs_failovercontroller_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Log Directory parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Log Directory", "name": "role_config_suppression_failover_controller_log_dir", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Failover Controller logs. Typically used by log4j or logback.", "display_name": "Failover Controller Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_hdfs_failovercontroller_audit_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_fc_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Failover Controller in Bytes", "name": "failover_controller_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hdfs_failovercontroller_scm_health", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hdfs_failovercontroller_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Failover Controller Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Failover Controller Environment Advanced Configuration Snippet (Safety Valve)", "name": "FAILOVERCONTROLLER_role_env_safety_valve", "value": null }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hdfs_failovercontroller_file_descriptor", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Failover Controller logs. Typically used by log4j or logback.", "display_name": "Failover Controller Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "fc_config_safety_valve", "value": null }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Enables the health test that the Failover Controller's process state is consistent with the role configuration", "display_name": "Failover Controller Process Health Test", "name": "failovercontroller_scm_health_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Failover Controller", "name": "failover_controller_java_opts", "value": "" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Failover Controller parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Failover Controller", "name": "role_config_suppression_failover_controller_java_opts", "value": "false" }, { "desc": "The minimum log level for Failover Controller logs", "display_name": "Failover Controller Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hdfs_failovercontroller_log_directory_free_space", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/oozie-oozie_server.json0000666000175100017510000011562713245514472026322 0ustar zuulzuul00000000000000[ { "desc": "The period over which to compute the moving average of the callable queue size.", "display_name": "Oozie Server Callable Queue Monitoring Period", "name": "oozie_server_callable_queue_window", "value": "5" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "oozie_server_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Comma-separated list of ActionService executor extension classes. Only action types with associated executors can be used in workflows. For CDH 5.4 and higher, this parameter is used only to specify additional classes for workflows. All executor extension classes included in that release will be added automatically and do not need to be specified.", "display_name": "Oozie ActionService Executor Extension Classes", "name": "oozie_executor_extension_classes", "value": "" }, { "desc": "Comma-separated list of Oozie plug-ins to be activated. If one plugin cannot be loaded, all the plugins are ignored.", "display_name": "Oozie Server Plugins", "name": "oozie_plugins_list", "value": "" }, { "desc": "Enables the health test that the Oozie Server's process state is consistent with the role configuration", "display_name": "Oozie Server Process Health Test", "name": "oozie_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Password parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Password", "name": "role_config_suppression_oozie_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Plugins parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Plugins", "name": "role_config_suppression_oozie_plugins_list", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Oozie Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_oozie_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_oozie_https_keystore_password", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "oozie_server_web_metric_collection_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Oozie Server", "name": "oozie_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Host parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Host", "name": "role_config_suppression_oozie_email_smtp_host", "value": "false" }, { "desc": "Completed workflow jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Workflow Jobs", "name": "purgeservice_older_than", "value": "30" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_oozie_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie SchemaService Workflow Extension Schemas parameter.", "display_name": "Suppress Parameter Validation: Oozie SchemaService Workflow Extension Schemas", "name": "role_config_suppression_oozie_workflow_extension_schemas", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The SMTP server port to use for Oozie email action", "display_name": "Oozie Email Action SMTP Port", "name": "oozie_email_smtp_prt", "value": "25" }, { "desc": "Whether to suppress the results of the Callable Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Callable Queue Size", "name": "role_health_suppression_oozie_server_callablequeue_size_health", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Data Directory parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Data Directory", "name": "role_config_suppression_oozie_data_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_oozie_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_oozie_server_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action From Address parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action From Address", "name": "role_config_suppression_oozie_email_from_address", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Log Directory", "name": "role_config_suppression_oozie_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Enable SMTP authentication for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Enabled", "name": "oozie_email_smtp_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml", "name": "role_config_suppression_oozie_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "SMTP password for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Password", "name": "oozie_email_smtp_password", "value": null }, { "desc": "Maximum concurrency for a given callable type. Each command is a callable type: submit, start, run, etc. Each action type is a callable type: MapReduce, SSH, sub-workflow, etc. All commands that use action executors (action-start, action-end. etc.) use the action type as the callable type.", "display_name": "Maximum concurrency for a given callable type", "name": "oozie_service_callablequeueservice_callable_concurrency", "value": "10" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "If true, enables the Oozie Server web console. ExtJS 2.2 zip archive must be extracted to /var/lib/oozie on the same host as the Oozie Server.", "display_name": "Enable Oozie Server Web Console", "name": "oozie_web_console", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "oozie_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Username for connecting to the database used by Oozie Server. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database User", "name": "oozie_database_user", "value": "sa" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_oozie_server_audit_health", "value": "false" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_oozie_server_pause_duration", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_oozie_server_file_descriptor", "value": "false" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "oozie_server_pause_duration_window", "value": "5" }, { "desc": "The from address to be used for mailing all emails for Oozie email action", "display_name": "Oozie Email Action From Address", "name": "oozie_email_from_address", "value": "oozie@localhost" }, { "desc": "Port of Oozie Server", "display_name": "Oozie HTTP Port", "name": "oozie_http_port", "value": "11000" }, { "desc": "Whether to suppress the results of the Oozie Server Shared Library Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Oozie Server Shared Library Check", "name": "role_health_suppression_oozie_server_shared_lib_version_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where Oozie Server will place its log files.", "display_name": "Oozie Server Log Directory", "name": "oozie_log_dir", "value": "/var/log/oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Oozie Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Oozie Server", "name": "role_config_suppression_oozie_java_opts", "value": "false" }, { "desc": "Completed bundle jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Bundle Jobs", "name": "purgeservice_bundle_older_than", "value": "7" }, { "desc": "Maximum callable queue size", "display_name": "Maximum Callable Queue Size", "name": "oozie_service_callablequeueservice_queue_size", "value": "10000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The password for the Oozie JKS keystore file.", "display_name": "Oozie TLS/SSL Server JKS Keystore File Password", "name": "oozie_https_keystore_password", "value": null }, { "desc": "Completed coordinator jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Coordinator Jobs", "name": "purgeservice_coord_older_than", "value": "7" }, { "desc": "Workflow Status metrics collection interval.", "display_name": "Workflow Status Metrics Collection Interval", "name": "oozie_job_metric_collection_interval", "value": "1" }, { "desc": "The minimum log level for Oozie Server logs", "display_name": "Oozie Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_oozie_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_oozie_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Name parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Name", "name": "role_config_suppression_oozie_database_name", "value": "false" }, { "desc": "When computing the overall Oozie Server health, consider the host's health.", "display_name": "Oozie Server Host Health Test", "name": "oozie_server_host_health_enabled", "value": "true" }, { "desc": "Password for connecting to the database used by Oozie Server. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database Password", "name": "oozie_database_password", "value": "" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie ActionService Executor Extension Classes parameter.", "display_name": "Suppress Parameter Validation: Oozie ActionService Executor Extension Classes", "name": "role_config_suppression_oozie_executor_extension_classes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds of the weighted average size of the Oozie Server callable queue over a recent period. See also Oozie Server Callable Queue Monitoring Period.", "display_name": "Oozie Server Callable Queue Monitoring Threshold", "name": "oozie_server_callable_queue_threshold", "value": "{\"critical\":\"95.0\",\"warning\":\"80.0\"}" }, { "desc": "Directory where the Oozie Server places its data. Only applicable when using Derby as the database type.", "display_name": "Oozie Server Data Directory", "name": "oozie_data_dir", "value": "/var/lib/oozie/data" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_oozie_server_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Oozie is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Oozie TLS/SSL Server JKS Keystore File Location", "name": "oozie_https_keystore_file", "value": "/var/lib/oozie/.keystore" }, { "desc": "Comma-separated list of SchemaService workflow extension schemas for additional action types. From CDH 5.4 and higher, this parameter is used only to specify additional schemas for workflows. All schemas included in that release will be added automatically and do not need to be specified.", "display_name": "Oozie SchemaService Workflow Extension Schemas", "name": "oozie_workflow_extension_schemas", "value": "" }, { "desc": "Port of the Oozie Server when using TLS/SSL.", "display_name": "Oozie HTTPS Port", "name": "oozie_https_port", "value": "11443" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to use the Codehale-based metrics for instrumentation. Enabling this disables the 'instrumentation' REST endpoint and enables the 'metrics' REST endpoint (<hostname:port>/v2/admin/metrics).", "display_name": "Enable The Metrics Instrumentation Service", "name": "oozie_use_metric_instrumentation", "value": "true" }, { "desc": "Type of the database used by Oozie Server.", "display_name": "Oozie Server Database Type", "name": "oozie_database_type", "value": "derby" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Host parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Host", "name": "role_config_suppression_oozie_database_host", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_oozie_https_keystore_file", "value": "false" }, { "desc": "The SMTP server host to use for Oozie email action", "display_name": "Oozie Email Action SMTP Host", "name": "oozie_email_smtp_host", "value": "localhost" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_oozie_server_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_oozie_https_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Authentication Password parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Authentication Password", "name": "role_config_suppression_oozie_email_smtp_password", "value": "false" }, { "desc": "SMTP username for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Username", "name": "oozie_email_smtp_username", "value": null }, { "desc": "The password for the Oozie TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Oozie TLS/SSL Certificate Trust Store Password", "name": "oozie_https_truststore_password", "value": null }, { "desc": "Number of threads used for executing callables", "display_name": "Number Threads For Executing Callables", "name": "oozie_service_callablequeueservice_threads", "value": "50" }, { "desc": "The maximum number of rolled log files to keep for Oozie Server logs. Typically used by log4j or logback.", "display_name": "Oozie Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "720" }, { "desc": "The admin port Oozie server runs.", "display_name": "Oozie Admin Port", "name": "oozie_admin_port", "value": "11001" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database User parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database User", "name": "role_config_suppression_oozie_database_user", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Oozie might connect to. This is used when Oozie is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Oozie TLS/SSL Certificate Trust Store File", "name": "oozie_https_truststore_file", "value": null }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "If true, enables version check for Oozie Server and installed shared libraries.", "display_name": "Enable Oozie Server Shared Libraries Version Check", "name": "oozie_server_shared_lib_version_check_enabled", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "oozie_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Name of the database used by Oozie Server.", "display_name": "Oozie Server Database Name", "name": "oozie_database_name", "value": "oozie" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to purge completed workflows and their corresponding coordinator actions of long-running coordinator jobs if the completed workflow jobs are older than the value specified in oozie.service.PurgeService.older.than.", "display_name": "Enable Purge for Long-Running Coordinator Jobs", "name": "purgeservice_purge_old_coord_action", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_oozie_https_truststore_password", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "For advanced use only. A string to be inserted into oozie-site.xml for this role only.", "display_name": "Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml", "name": "oozie_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Authentication Username parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Authentication Username", "name": "role_config_suppression_oozie_email_smtp_username", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Oozie Server in Bytes", "name": "oozie_java_heapsize", "value": "1073741824" }, { "desc": "Hostname of the database used by Oozie Server. If the port is non-default for your database type, use host:port notation. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database Host", "name": "oozie_database_host", "value": "localhost" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Oozie Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "OOZIE_SERVER_role_env_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/cdh_config.py0000666000175100017510000000370313245514472024215 0ustar zuulzuul00000000000000# Copyright (c) 2017 Massachusetts Open Cloud # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from cm_api.api_client import ApiResource cm_host = "localhost" api = ApiResource(cm_host, username="admin", password="admin") # nosec c = api.get_all_clusters()[0] services = c.get_all_services() def process_service(service): service_name = service.name if service_name == "spark_on_yarn": service_name = "spark" for role_cfgs in service.get_all_role_config_groups(): role_cm_cfg = role_cfgs.get_config(view='full') role_cfg = parse_config(role_cm_cfg) role_name = role_cfgs.roleType.lower() write_cfg(role_cfg, '%s-%s.json' % (service_name, role_name)) service_cm_cfg = service.get_config(view='full')[0] service_cfg = parse_config(service_cm_cfg) write_cfg(service_cfg, '%s-service.json' % service_name) def parse_config(config): cfg = [] for name, value in config.items(): p = { 'name': value.name, 'value': value.default, 'display_name': value.displayName, 'desc': value.description } cfg.append(p) return cfg def write_cfg(cfg, file_name): to_write = __import__('json').dumps(cfg, sort_keys=True, indent=4, separators=(',', ': ')) with open(file_name, 'w') as f: f.write(to_write) if __name__ == '__main__': for service in services: process_service(service) sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/impala-statestore.json0000666000175100017510000006052013245514472026111 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "statestore_web_metric_collection_enabled", "value": "true" }, { "desc": "The amount of time allowed after this role is started that failures of health tests that rely on communication with this role will be tolerated.", "display_name": "Health Test Startup Tolerance", "name": "statestore_startup_tolerance", "value": "5" }, { "desc": "Username for Statestore webserver authentication.", "display_name": "Statestore Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_statestore_cmd_args_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "statestore_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_statestore_web_metric_collection", "value": "false" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "StateStore Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Port where StateStoreService is exported.", "display_name": "StateStore Service Port", "name": "state_store_port", "value": "24000" }, { "desc": "When computing the overall Impala StateStore health, consider the host's health.", "display_name": "Impala StateStore Host Health Test", "name": "statestore_host_health_enabled", "value": "true" }, { "desc": "Directory where a StateStore core dump is placed.", "display_name": "StateStore Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/statestore" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_statestore_scm_health", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Breakpad Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Statestore Breakpad Dump Directory", "name": "role_config_suppression_minidump_path", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_statestore_audit_health", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when StateStore Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "StateStore Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "Maximum number of tasks allowed to be pending at the thread manager underlying the StateStore Thrift server (0 allows infinitely many pending tasks)", "display_name": "Maximum StateStore Pending Tasks", "name": "state_store_pending_task_count_max", "value": "0" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_statestore_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Directory where StateStore will place its log files.", "display_name": "StateStore Log Directory", "name": "log_dir", "value": "/var/log/statestore" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Port where StateStore debug web server runs.", "display_name": "StateStore HTTP Server Port", "name": "statestore_webserver_port", "value": "25010" }, { "desc": "Maximum number of Breakpad dump files stored by Statestore Role.", "display_name": "Statestore Max Breakpad Dump Files", "name": "max_minidumps", "value": "9" }, { "desc": "Enables the health test that the Impala StateStore's process state is consistent with the role configuration", "display_name": "Impala StateStore Process Health Test", "name": "statestore_scm_health_enabled", "value": "true" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Statestore Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when StateStore Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "StateStore Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_statestore_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_statestore_host_health", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to StateStore command line flags.", "display_name": "Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "statestore_cmd_args_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Number of worker threads for the thread manager underlying the StateStore Thrift server.", "display_name": "StateStore Worker Threads", "name": "state_store_num_server_worker_threads", "value": "4" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Log Directory parameter.", "display_name": "Suppress Parameter Validation: StateStore Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_statestore_memory_rss_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala StateStore Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala StateStore Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_statestore_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Statestore Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "The password for the private key in the StateStore Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "StateStore Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_statestore_file_descriptor", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above. ", "display_name": "StateStore Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The maximum size, in megabytes, per log file for Impala StateStore logs. Typically used by log4j or logback.", "display_name": "Impala StateStore Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_statestore_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "statestore_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: StateStore Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "Enable/Disable StateStore web server. This web server contains useful information about StateStore daemon.", "display_name": "Enable StateStore Web Server", "name": "statestore_enable_webserver", "value": "true" }, { "desc": "Directory for storing Statestore Breakpad dumps.", "display_name": "Statestore Breakpad Dump Directory", "name": "minidump_path", "value": "/var/log/impala-minidumps" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala StateStore Environment Advanced Configuration Snippet (Safety Valve)", "name": "STATESTORE_role_env_safety_valve", "value": null }, { "desc": "Password for Statestore webserver authentication.", "display_name": "Statestore Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "The minimum log level for Impala StateStore logs", "display_name": "Impala StateStore Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "StateStore Maximum Log Files", "name": "max_log_files", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_statestore_swap_memory_usage", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/kafka-kafka_mirror_maker.json0000666000175100017510000011176013245514472027357 0ustar zuulzuul00000000000000[ { "desc": "A MirrorMaker message handler of type MirrorMakerMessageHandler that will process every record in-between producer and consumer.", "display_name": "MirrorMaker Message Handler", "name": "message.handler", "value": "" }, { "desc": "For advanced use only. A string to be inserted into mirror_maker_producers.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties", "name": "mirror_maker_producers.properties_role_safety_valve", "value": null }, { "desc": "Regular expression that represents a set of topics to mirror. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used.", "display_name": "Topic Whitelist", "name": "whitelist", "value": "" }, { "desc": "For advanced use only. A string to be inserted into mirror_maker_consumers.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties", "name": "mirror_maker_consumers.properties_role_safety_valve", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Consumer Rebalance Listener parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Consumer Rebalance Listener", "name": "role_config_suppression_consumer.rebalance.listener", "value": "false" }, { "desc": "The timeout used to detect failures when using Kafka's group management facilities. When a consumer's heartbeat is not received within the session timeout, the broker will mark the consumer as failed and rebalance the group. Note that the value must be in the allowable range as configured in the broker by group.min.session.timeout.ms and group.max.session.timeout.ms. Only respected by Kafka 2.0 or later.", "display_name": "Consumer Session Timeout", "name": "session.timeout.ms", "value": "30000" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kafka_kafka_mirror_maker_swap_memory_usage", "value": "false" }, { "desc": "Arguments used by MirrorMaker consumer rebalance listener.", "display_name": "MirrorMaker Consumer Rebalance Listener Arguments", "name": "consumer.rebalance.listener.args", "value": "" }, { "desc": "The maximum amount of time the consumer will wait for the response of a request. If the response is not received before the timeout elapses the consumer will resend the request if necessary or fail the request if retries are exhausted. Only respected by Kafka 2.0 or later.", "display_name": "Consumer Request Timeout", "name": "consumer.request.timeout.ms", "value": "40000" }, { "desc": "For advanced use only. A string to be inserted into ssl_client.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties", "name": "ssl_client.properties_role_safety_valve", "value": null }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Number of messages that are buffered between producer and consumer. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Message Queue Size", "name": "queue.size", "value": "10000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topic Blacklist parameter.", "display_name": "Suppress Parameter Validation: Topic Blacklist", "name": "role_config_suppression_blacklist", "value": "false" }, { "desc": "The total bytes of memory the producer can use to buffer records waiting to be sent to the server. Only respected by Kafka 2.0 or later.", "display_name": "Producer Buffer Memory", "name": "buffer.memory", "value": "33554432" }, { "desc": "The password for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Kafka MirrorMaker TLS/SSL Certificate Trust Store Password", "name": "ssl_client_truststore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kafka_mirror_maker_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Offset commit interval in milliseconds.", "display_name": "Offset Commit Interval", "name": "offset.commit.interval.ms", "value": "60000" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka MirrorMaker is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "When computing the overall Kafka MirrorMaker health, consider the host's health.", "display_name": "Kafka MirrorMaker Host Health Test", "name": "kafka_mirror_maker_host_health_enabled", "value": "true" }, { "desc": "Only required if source Kafka cluster requires client authentication.", "display_name": "Source Kafka Cluster's Client Auth", "name": "source.ssl.client.auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties", "name": "role_config_suppression_mirror_maker_consumers.properties_role_safety_valve", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "These arguments are passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags are passed here.", "display_name": "Additional MirrorMaker Java Options", "name": "mirror_maker_java_opts", "value": "-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" }, { "desc": "Arguments used by MirrorMaker message handler.", "display_name": "MirrorMaker Message Handler Arguments", "name": "message.handler.args", "value": "" }, { "desc": "The password that protects the private key contained in the JKS keystore used when Kafka MirrorMaker is acting as a TLS/SSL server.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Destination Broker List parameter.", "display_name": "Suppress Parameter Validation: Destination Broker List", "name": "role_config_suppression_bootstrap.servers", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Kafka MirrorMaker logs. Typically used by log4j or logback.", "display_name": "Kafka MirrorMaker Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The log directory for log files of the role Kafka MirrorMaker.", "display_name": "Kafka MirrorMaker Log Directory", "name": "log_dir", "value": "/var/log/kafka" }, { "desc": "Stop the entire mirror maker when a send failure occurs.", "display_name": "Abort on Send Failure", "name": "abort.on.send.failure", "value": "true" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka MirrorMaker might connect to. This is used when Kafka MirrorMaker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Kafka MirrorMaker TLS/SSL Certificate Trust Store File", "name": "ssl_client_truststore_location", "value": "" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Number of producer instances. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Number of Producers", "name": "num.producers", "value": "1" }, { "desc": "Port for JMX.", "display_name": "JMX Port", "name": "jmx_port", "value": "9394" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Enables the health test that the Kafka MirrorMaker's process state is consistent with the role configuration", "display_name": "Kafka MirrorMaker Process Health Test", "name": "kafka_mirror_maker_scm_health_enabled", "value": "true" }, { "desc": "The compression type for all data generated by the producer. Only respected by Kafka 2.0 or later.", "display_name": "Producer Compression Type", "name": "compression.type", "value": "none" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kafka_kafka_mirror_maker_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties", "name": "role_config_suppression_ssl_client.properties_role_safety_valve", "value": "false" }, { "desc": "The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.", "display_name": "Consumer Minimum Fetch Size", "name": "fetch.min.bytes", "value": "1" }, { "desc": "Protocol to be used for communication with destination kafka cluster.", "display_name": "Destination Kafka Cluster's Security Protocol", "name": "destination.security.protocol", "value": "PLAINTEXT" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topic Whitelist parameter.", "display_name": "Suppress Parameter Validation: Topic Whitelist", "name": "role_config_suppression_whitelist", "value": "false" }, { "desc": "The minimum log level for Kafka MirrorMaker logs", "display_name": "Kafka MirrorMaker Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties", "name": "role_config_suppression_ssl_server.properties_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kafka_mirror_maker_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of MirrorMaker parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of MirrorMaker", "name": "role_config_suppression_mirror_maker_max_heap_size", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "Encrypt communication between clients and Kafka MirrorMaker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Kafka MirrorMaker", "name": "ssl_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Consumer Rebalance Listener Arguments parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Consumer Rebalance Listener Arguments", "name": "role_config_suppression_consumer.rebalance.listener.args", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kafka_kafka_mirror_maker_file_descriptor", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Name of the consumer group used by MirrorMaker. When multiple role instances are configured with the same topics and same group ID, the role instances load-balance replication for the topics. When multiple role instances are configured with the same topics but different group ID, each role instance replicates all the events for those topics - this can be used to replicate the source cluster into multiple destination clusters.", "display_name": "Consumer Group ID", "name": "group.id", "value": "cloudera_mirrormaker" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Message Handler Arguments parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Message Handler Arguments", "name": "role_config_suppression_message.handler.args", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Regular expression that represents a set of topics to avoid mirroring. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Topic Blacklist", "name": "blacklist", "value": "" }, { "desc": "Maximum number of bytes that can be buffered between producer and consumer. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Queue Size", "name": "queue.byte.size", "value": "100000000" }, { "desc": "The maximum amount of time the producer will wait for the response of a request. If the response is not received before the timeout elapses the producer will resend the request if necessary or fail the request if retries are exhausted. Only respected by Kafka 2.0 or later.", "display_name": "Producer Request Timeout", "name": "producer.request.timeout.ms", "value": "30000" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kafka_kafka_mirror_maker_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Message Handler parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Message Handler", "name": "role_config_suppression_message.handler", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_MIRROR_MAKER_role_env_safety_valve", "value": null }, { "desc": "Only required if destination Kafka cluster requires client authentication.", "display_name": "Destination Kafka Cluster's Client Auth", "name": "destination.ssl.client.auth", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional MirrorMaker Java Options parameter.", "display_name": "Suppress Parameter Validation: Additional MirrorMaker Java Options", "name": "role_config_suppression_mirror_maker_java_opts", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into ssl_server.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties", "name": "ssl_server.properties_role_safety_valve", "value": null }, { "desc": "A consumer rebalance listener of type ConsumerRebalanceListener to be invoked when MirrorMaker's consumer rebalances.", "display_name": "MirrorMaker Consumer Rebalance Listener", "name": "consumer.rebalance.listener", "value": "" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kafka_kafka_mirror_maker_host_health", "value": "false" }, { "desc": "This configuration controls the batch size in bytes. The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. Only respected by Kafka 2.0 or later.", "display_name": "Producer Batch Size", "name": "batch.size", "value": "16384" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Number of consumer threads.", "display_name": "Number of Consumer Threads", "name": "num.streams", "value": "1" }, { "desc": "The maximum number of rolled log files to keep for Kafka MirrorMaker logs. Typically used by log4j or logback.", "display_name": "Kafka MirrorMaker Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The upper bound on the delay for batching. Once the producer gets a batch.size worth of records for a partition it will be sent immediately regardless of this setting. However if fewer than this many bytes accumulated for this partition the producer will 'linger' for the specified time waiting for more records to show up. Only respected by Kafka 2.0 or later.", "display_name": "Producer Linger Time", "name": "linger.ms", "value": "0" }, { "desc": "Protocol to be used for communication with source kafka cluster.", "display_name": "Source Kafka Cluster's Security Protocol", "name": "source.security.protocol", "value": "PLAINTEXT" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties", "name": "role_config_suppression_mirror_maker_producers.properties_role_safety_valve", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Source Broker List parameter.", "display_name": "Suppress Parameter Validation: Source Broker List", "name": "role_config_suppression_source.bootstrap.servers", "value": "false" }, { "desc": "Run with MirrorMaker settings that eliminate potential loss of data. This impacts performance, but is highly recommended. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Avoid Data Loss", "name": "no.data.loss", "value": "true" }, { "desc": "The password for the Kafka MirrorMaker JKS keystore file.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Consumer Group ID parameter.", "display_name": "Suppress Parameter Validation: Consumer Group ID", "name": "role_config_suppression_group.id", "value": "false" }, { "desc": "List of brokers on destination cluster. This should be more than one, for high availability, but there's no need to list all brokers.", "display_name": "Destination Broker List", "name": "bootstrap.servers", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_kafka_kafka_mirror_maker_audit_health", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "List of brokers on source cluster. This should be more than one, for high availability, but there's no need to list all brokers.", "display_name": "Source Broker List", "name": "source.bootstrap.servers", "value": null }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in megabytes.", "display_name": "Java Heap Size of MirrorMaker", "name": "mirror_maker_max_heap_size", "value": "1024" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hue-kt_renewer.json0000666000175100017510000003502713245514472025403 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_kt_renewer_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kt_renewer_swap_memory_usage", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Ticket Renewer Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kerberos Ticket Renewer Log Directory", "name": "role_config_suppression_kt_renewer_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_kt_renewer_audit_health", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kt_renewer_unexpected_exits", "value": "false" }, { "desc": "Interval in seconds with which Hue's Kerberos ticket will get renewed.", "display_name": "Hue Keytab Renewal Interval", "name": "keytab_reinit_frequency", "value": "3600" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Enables the health test that the Kerberos Ticket Renewer's process state is consistent with the role configuration", "display_name": "Kerberos Ticket Renewer Process Health Test", "name": "kt_renewer_scm_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve)", "name": "KT_RENEWER_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When computing the overall Kerberos Ticket Renewer health, consider the host's health.", "display_name": "Kerberos Ticket Renewer Host Health Test", "name": "kt_renewer_host_health_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kt_renewer_host_health", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kt_renewer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Directory where Kerberos Ticket Renewer will place its log files.", "display_name": "Kerberos Ticket Renewer Log Directory", "name": "kt_renewer_log_dir", "value": "/var/log/hue" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kt_renewer_role_env_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kt_renewer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_kt_renewer_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kt_renewer_scm_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/solr-solr_server.json0000666000175100017510000006627613245514472026013 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Solr Load Balancer", "name": "role_config_suppression_solr_load_balancer", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Number of blocks per cache slab. The size of the cache is 8 KB (the block size) times the number of blocks per slab times the number of slabs.", "display_name": "HDFS Block Cache Blocks per Slab", "name": "solr_hdfs_blockcache_blocksperbank", "value": "16384" }, { "desc": "When computing the overall Solr Server health, consider the host's health.", "display_name": "Solr Server Host Health Test", "name": "solr_server_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_solr_server_audit_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "solr_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Solr Server in Bytes", "name": "solr_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Solr Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Solr Server", "name": "role_config_suppression_solr_java_opts", "value": "false" }, { "desc": "Enables the health test that the Solr Server's process state is consistent with the role configuration", "display_name": "Solr Server Process Health Test", "name": "solr_server_scm_health_enabled", "value": "true" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Solr Server logs. Typically used by log4j or logback.", "display_name": "Solr Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Solr Server API Liveness heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Solr Server API Liveness", "name": "role_health_suppression_solr_core_status_collection_health", "value": "false" }, { "desc": "The maximum number of request processing threads to be created by Solr server, which determines the maximum number of simultaneous requests that can be handled.", "display_name": "Solr Max Connector Threads", "name": "solr_max_connector_thread", "value": "10000" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_solr_server_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds on the duration of the Solr Server API request.", "display_name": "Solr Server API Liveness Request Duration", "name": "solr_core_status_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_solr_server_host_health", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_solr_server_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Solr Server Log Directory", "name": "role_config_suppression_solr_log_dir", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of Solr Server in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of Solr Server in Bytes", "name": "role_config_suppression_solr_java_heapsize", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_solr_server_file_descriptor", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Solr Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The time in milliseconds a client is allowed to not talk to ZooKeeper before its session expires.", "display_name": "ZooKeeper Client Timeout", "name": "zookeeper_client_timeout", "value": "15000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "solr_server_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Use off-heap memory when caching HDFS blocks in Solr.", "display_name": "HDFS Block Cache Off-Heap Memory", "name": "solr_hdfs_blockcache_direct_memory_allocation", "value": "true" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather status of Solr Cores from the Solr Server with a simple API request.", "display_name": "Solr Server API Liveness", "name": "solr_core_status_collection_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Solr Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SOLR_SERVER_role_env_safety_valve", "value": null }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "HTTPS port of Solr Server.", "display_name": "Solr HTTPS port", "name": "solr_https_port", "value": "8985" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Timeout (in seconds) for graceful shutdown of this Solr server. Once this timeout is reached, Solr server is abruptly shutdown. A value of 0 means no timeout.", "display_name": "Graceful Shutdown Timeout", "name": "solr_graceful_stop_timeout", "value": "180" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_solr_server_gc_duration", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Admin port of the Solr Server.", "display_name": "Solr Admin Port", "name": "solr_admin_port", "value": "8984" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Data Directory parameter.", "display_name": "Suppress Parameter Validation: Solr Data Directory", "name": "role_config_suppression_solr_data_dir", "value": "false" }, { "desc": "Maximum amount of off-heap memory in bytes that may be allocated by the Java process. Passed to Java -XX:MaxDirectMemorySize. If unset, defaults to the size of the heap.", "display_name": "Java Direct Memory Size of Solr Server in Bytes", "name": "solr_java_direct_memory_size", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_solr_server_role_env_safety_valve", "value": "false" }, { "desc": "Enable caching of HDFS blocks in Solr. There is one block cache per Solr collection. configured to use off-heap memory, Maximum Off-Heap Memory must be set high enough to account for all block caches.", "display_name": "HDFS Block Cache", "name": "solr_hdfs_blockcache_enabled", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_solr_server_unexpected_exits", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Solr Server logs. Typically used by log4j or logback.", "display_name": "Solr Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Address of the load balancer, specified in host:port format.", "display_name": "Solr Load Balancer", "name": "solr_load_balancer", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "solr_server_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_solr_server_log_directory_free_space", "value": "false" }, { "desc": "Directory on local file system where Solr Server keeps the configurations for collections.", "display_name": "Solr Data Directory", "name": "solr_data_dir", "value": "/var/lib/solr" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "solr_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Number of slabs per block cache. The size of the cache is 8 KB (the block size) times the number of blocks per slab times the number of slabs.", "display_name": "HDFS Block Cache Number of Slabs", "name": "solr_hdfs_blockcache_slab_count", "value": "1" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_solr_server_web_metric_collection", "value": "false" }, { "desc": "Directory where Solr Server will place its log files.", "display_name": "Solr Server Log Directory", "name": "solr_log_dir", "value": "/var/log/solr" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "solr_server_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Solr Server", "name": "solr_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The minimum log level for Solr Server logs", "display_name": "Solr Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_solr_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "HTTP port of Solr Server.", "display_name": "Solr HTTP Port", "name": "solr_http_port", "value": "8983" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hbase-hbasethriftserver.json0000666000175100017510000007500713245514472027273 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbasethriftserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase Thrift Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase Thrift Server", "name": "role_config_suppression_hbase_thriftserver_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_thrift_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server DNS Network Interface", "name": "role_config_suppression_hbase_thriftserver_dns_interface", "value": "false" }, { "desc": "The \"core size\" of the thread pool. New threads are created on every connection until this many threads are created.", "display_name": "HBase Thrift Server Min Worker Threads", "name": "hbase_thriftserver_min_worker_threads", "value": "200" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_thrift_server_host_health", "value": "false" }, { "desc": "The port that HBase Thrift Server binds to.", "display_name": "HBase Thrift Server Port", "name": "hbase_thriftserver_port", "value": "9090" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Bind Address parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Bind Address", "name": "role_config_suppression_hbase_thriftserver_bindaddress", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_thrift_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server DNS Name Server", "name": "role_config_suppression_hbase_thriftserver_dns_nameserver", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Log Directory", "name": "role_config_suppression_hbase_thriftserver_log_dir", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HBase Thrift Server logs. Typically used by log4j or logback.", "display_name": "HBase Thrift Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_thrift_server_log_directory_free_space", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase Thrift Server over HTTP is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location", "name": "hbase_thriftserver_http_keystore_file", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase Thrift Server in Bytes", "name": "hbase_thriftserver_java_heapsize", "value": "1073741824" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The password for the HBase Thrift Server over HTTP JKS keystore file.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password", "name": "hbase_thriftserver_http_keystore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "If true, HBase Thrift Server Web UI will bind to a wildcard address (0.0.0.0). Otherwise it will bind to a host name. Only available in CDH 4.3 and later.", "display_name": "HBase Thrift Server Web UI Bind to Wildcard Address", "name": "hbase_thriftserver_info_bind_to_wildcard", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_hbase_thriftserver_http_keystore_file", "value": "false" }, { "desc": "Address to bind the HBase Thrift Server to. When using the THsHaServer or the TNonblockingServer, always binds to 0.0.0.0 irrespective of this configuration value.", "display_name": "HBase Thrift Server Bind Address", "name": "hbase_thriftserver_bindaddress", "value": "0.0.0.0" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Type of HBase Thrift Server.", "display_name": "HBase Thrift Server Type", "name": "hbase_thriftserver_type", "value": "threadpool" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_hbase_thriftserver_http_keystore_password", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASETHRIFTSERVER_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When computing the overall HBase Thrift Server health, consider the host's health.", "display_name": "HBase Thrift Server Host Health Test", "name": "hbasethriftserver_host_health_enabled", "value": "true" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase Thrift Server over HTTP is acting as a TLS/SSL server.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password", "name": "hbase_thriftserver_http_keystore_keypassword", "value": null }, { "desc": "Enables the health test that the HBase Thrift Server's process state is consistent with the role configuration", "display_name": "HBase Thrift Server Process Health Test", "name": "hbasethriftserver_scm_health_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase Thrift Server", "name": "hbase_thriftserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Use framed transport. When using the THsHaServer or TNonblockingServer, framed transport is always used irrespective of this configuration value.", "display_name": "Enable HBase Thrift Server Framed Transport", "name": "hbase_thriftserver_framed", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for HBase Thrift Server logs. Typically used by log4j or logback.", "display_name": "HBase Thrift Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The port that HBase Thrift Server Web UI binds to.", "display_name": "HBase Thrift Server Web UI Port", "name": "hbase_thriftserver_info_port", "value": "9095" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_hbase_thriftserver_http_keystore_keypassword", "value": "false" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_hbase_thrift_server_audit_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_thrift_server_swap_memory_usage", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The host name or IP address of the DNS name server which an HBase Thrift Server should use to determine the host name used for communication and display purposes.", "display_name": "HBase Thrift Server DNS Name Server", "name": "hbase_thriftserver_dns_nameserver", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_thrift_server_scm_health", "value": "false" }, { "desc": "Use the TCompactProtocol instead of the default TBinaryProtocol. TCompactProtocol is a binary protocol that is more compact than the default and typically more efficient.", "display_name": "Enable HBase Thrift Server Compact Protocol", "name": "hbase_thriftserver_compact", "value": "false" }, { "desc": "The name of the DNS network interface from which an HBase Thrift Server should report its IP address.", "display_name": "HBase Thrift Server DNS Network Interface", "name": "hbase_thriftserver_dns_interface", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Directory where HBase Thrift Server will place its log files.", "display_name": "HBase Thrift Server Log Directory", "name": "hbase_thriftserver_log_dir", "value": "/var/log/hbase" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_thriftserver_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_thrift_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbasethriftserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_thriftserver_config_safety_valve", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for HBase Thrift Server logs", "display_name": "HBase Thrift Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Encrypt communication between clients and HBase Thrift Server over HTTP using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HBase Thrift Server over HTTP", "name": "hbase_thriftserver_http_use_ssl", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-journalnode.json0000666000175100017510000007440213245514472025721 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_journal_node_audit_health", "value": "false" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "journalnode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_journal_node_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Fsync Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Fsync Latency", "name": "role_health_suppression_journal_node_fsync_latency", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: JournalNode Log Directory", "name": "role_config_suppression_journalnode_log_dir", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of JournalNode in Bytes", "name": "journalNode_java_heapsize", "value": "536870912" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "journalnode_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_journal_node_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_journal_node_host_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Enables the health test that the JournalNode's process state is consistent with the role configuration", "display_name": "JournalNode Process Health Test", "name": "journalnode_scm_health_enabled", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The maximum number of rolled log files to keep for JournalNode logs. Typically used by log4j or logback.", "display_name": "JournalNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JournalNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a JournalNode Edits Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "JournalNode Edits Directory Free Space Monitoring Percentage Thresholds", "name": "journalnode_edits_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_jn_config_safety_valve", "value": "false" }, { "desc": "Directory on the local file system where NameNode edits are written.", "display_name": "JournalNode Edits Directory", "name": "dfs_journalnode_edits_dir", "value": null }, { "desc": "Whether to suppress the results of the Sync Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sync Status", "name": "role_health_suppression_journal_node_sync_status", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_journal_node_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The base port where the secure JournalNode web UI listens. Combined with the JournalNode's hostname to build its secure web UI address.", "display_name": "Secure JournalNode Web UI Port (TLS/SSL)", "name": "dfs_journalnode_https_port", "value": "8481" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_journal_node_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "jn_config_safety_valve", "value": null }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "When computing the overall JournalNode health, consider the host's health.", "display_name": "JournalNode Host Health Test", "name": "journalnode_host_health_enabled", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JournalNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for JournalNode", "name": "role_config_suppression_journalnode_java_opts", "value": "false" }, { "desc": "The amount of time at JournalNode startup allowed for the active NameNode to get in sync with the JournalNode.", "display_name": "Active NameNode Sync Status Startup Tolerance", "name": "journalnode_sync_status_startup_tolerance", "value": "180" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The health test thresholds for JournalNode fsync latency.", "display_name": "JournalNode Fsync Latency Thresholds", "name": "journalnode_fsync_latency_thresholds", "value": "{\"critical\":\"3000.0\",\"warning\":\"1000.0\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "journalnode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_journal_node_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JournalNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_journalnode_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "JournalNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enables the health check that verifies the active NameNode's sync status to the JournalNode", "display_name": "Active NameNode Sync Status Health Check", "name": "journalnode_sync_status_enabled", "value": "true" }, { "desc": "Port for the JournalNode's RPC. Combined with the JournalNode's hostname to build its RPC address.", "display_name": "JournalNode RPC Port", "name": "dfs_journalnode_rpc_port", "value": "8485" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "journalnode_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Port for the JournalNode HTTP web UI. Combined with the JournalNode hostname to build its HTTP address.", "display_name": "JournalNode HTTP Port", "name": "dfs_journalnode_http_port", "value": "8480" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Edits Directory parameter.", "display_name": "Suppress Parameter Validation: JournalNode Edits Directory", "name": "role_config_suppression_dfs_journalnode_edits_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "If enabled, the JournalNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind JournalNode to Wildcard Address", "name": "journalnode_bind_wildcard", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_journal_node_web_metric_collection", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "journalnode_gc_duration_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory.", "display_name": "JournalNode Edits Directory Free Space Monitoring Absolute Thresholds", "name": "journalnode_edits_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The maximum size, in megabytes, per log file for JournalNode logs. Typically used by log4j or logback.", "display_name": "JournalNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_journal_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "Directory where JournalNode will place its log files.", "display_name": "JournalNode Log Directory", "name": "journalnode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "JournalNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "JOURNALNODE_role_env_safety_valve", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for JournalNode", "name": "journalNode_java_opts", "value": "" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the JournalNode Edits Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JournalNode Edits Directory Free Space", "name": "role_health_suppression_journal_node_edits_directory_free_space", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for JournalNode logs", "display_name": "JournalNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_journal_node_gc_duration", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/yarn-service.json0000666000175100017510000011161013245514472025057 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds of the overall NodeManager health. The check returns \"Concerning\" health if the percentage of \"Healthy\" NodeManagers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" NodeManagers falls below the critical threshold.", "display_name": "Healthy NodeManager Monitoring Thresholds", "name": "yarn_nodemanagers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Entries to add to the classpaths of YARN applications.", "display_name": "YARN Application Classpath", "name": "yarn_application_classpath", "value": "$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "service_config_suppression_yarn_service_replication_config_safety_valve", "value": "false" }, { "desc": "The user that this service's processes should run as. (Except the Job History Server, which has its own user)", "display_name": "System User", "name": "process_username", "value": "yarn" }, { "desc": "Name of the HDFS service that this YARN service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Whether to suppress the results of the ResourceManager Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ResourceManager Health", "name": "service_health_suppression_yarn_resourcemanagers_health", "value": "false" }, { "desc": "The tolerance window used in YARN service tests that depend on detection of the active ResourceManager.", "display_name": "Active ResourceManager Detection Window", "name": "yarn_active_resourcemanager_detecton_window", "value": "3" }, { "desc": "The amount of time after ResourceManager(s) start that the lack of an active ResourceManager will be tolerated. This is an advanced option that does not often need to be changed.", "display_name": "ResourceManager Activation Startup Tolerance", "name": "yarn_resourcemanager_activation_startup_tolerance", "value": "180" }, { "desc": "When computing the overall YARN service health, whether to consider the health of the standby ResourceManager.", "display_name": "Standby ResourceManager Health Check", "name": "yarn_standby_resourcemanager_health_enabled", "value": "true" }, { "desc": "Whether users and groups specified in Admin ACL should be checked for authorization to perform admin operations.", "display_name": "Enable ResourceManager ACLs", "name": "yarn_acl_enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "yarn_hadoop_policy_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Application Classpath parameter.", "display_name": "Suppress Parameter Validation: YARN Application Classpath", "name": "service_config_suppression_yarn_application_classpath", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the NodeManager Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NodeManager Count Validator", "name": "service_config_suppression_nodemanager_count_validator", "value": "false" }, { "desc": "Password for the TLS/SSL client truststore. Overrides the cluster-wide default truststore password set in HDFS.", "display_name": "TLS/SSL Client Truststore File Password", "name": "ssl_client_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "yarn_core_site_safety_valve", "value": null }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "service_config_suppression_mapreduce_service_replication_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_yarn_service_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules.", "display_name": "Fair Scheduler Allocations (Deployed)", "name": "yarn_fs_scheduled_allocations", "value": "{\"defaultFairSharePreemptionThreshold\":null,\"defaultFairSharePreemptionTimeout\":null,\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":null,\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":[{\"create\":true,\"name\":\"specified\",\"queue\":null,\"rules\":null},{\"create\":null,\"name\":\"nestedUserQueue\",\"queue\":null,\"rules\":[{\"create\":null,\"name\":\"default\",\"queue\":\"users\",\"rules\":null}]},{\"create\":true,\"name\":\"default\",\"queue\":null,\"rules\":null}],\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":\"drf\",\"type\":null},{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"users\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":\"drf\",\"type\":\"parent\"}],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":\"drf\",\"type\":null}],\"userMaxAppsDefault\":null,\"users\":[]}" }, { "desc": "Controls which applications non-admin users can see in the applications list view", "display_name": "Non-Admin Users Applications List Visibility Settings", "name": "user_application_list_settings", "value": "ALL" }, { "desc": "When computing the overall YARN health, consider JobHistory Server's health", "display_name": "JobHistory Server Role Health Test", "name": "yarn_jobhistoryserver_health_enabled", "value": "true" }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted shuffle and encrypted web UIs. Applies to all configurations of daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "service_config_suppression_yarn_hadoop_policy_config_safety_valve", "value": "false" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. Typically edited using the Rules configuration UI.", "display_name": "Fair Scheduler Configuration Rules (Staged)", "name": "yarn_fs_schedule_rules_draft", "value": null }, { "desc": "Password for the server keystore file used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Whether to suppress the results of the JobHistory Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JobHistory Server Health", "name": "service_health_suppression_yarn_jobhistory_health", "value": "false" }, { "desc": "Enables Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Note: This is effective only if Kerberos is enabled for the HDFS service.", "display_name": "Enable Kerberos Authentication for HTTP Web-Consoles", "name": "hadoop_secure_web_ui", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Secure Web UI Validator configuration validator.", "display_name": "Suppress Configuration Validator: Secure Web UI Validator", "name": "service_config_suppression_hadoop_secure_web_ui", "value": "false" }, { "desc": "Path to the truststore file used when roles of this service act as TLS/SSL clients. Overrides the cluster-wide default truststore location set in HDFS. This truststore must be in JKS format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. The contents of the truststore can be modified without restarting any roles. By default, changes to its contents are picked up within ten seconds. If not set, the default Java truststore is used to verify certificates.", "display_name": "TLS/SSL Client Truststore File Location", "name": "ssl_client_truststore_location", "value": null }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "mapreduce.output.fileoutputformat.compressfalsemapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.DefaultCodecio.compression.codecsorg.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec" }, { "desc": "For advanced use only, a string to be inserted into yarn-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_service_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress the results of the YARN Container Usage Aggregation heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: YARN Container Usage Aggregation", "name": "service_health_suppression_yarn_usage_aggregation_health", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into yarn-site.xml. Applies to all HDFS Replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_service_replication_config_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "yarn_service_env_safety_valve", "value": null }, { "desc": "Whether to enable log aggregation", "display_name": "Enable Log Aggregation", "name": "yarn_log_aggregation_enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "service_config_suppression_yarn_ssl_client_safety_valve", "value": "false" }, { "desc": "ACL that determines which users and groups can submit and kill applications in any pool, and can issue commands on ResourceManager roles.", "display_name": "Admin ACL", "name": "yarn_admin_acl", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Client Truststore File Password parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Client Truststore File Password", "name": "service_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "Cluster ID used when ResourceManager is Highly Available.", "display_name": "RM-HA Cluster ID", "name": "yarn_rm_ha_cluster_id", "value": "yarnRM" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "This determines the user Linux container executor should run as on a non-secure cluster. If this value is set to true, then all containers will be launched as the user specified in yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If this value is set to false, then containers will run as the user who submitted the application.", "display_name": "Limit Nonsecure Container Executor Users", "name": "yarn_nodemanager_linux_container_executor_nonsecure_mode_limit_users", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "yarn_ssl_server_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Llama Proxy User Hosts", "name": "service_config_suppression_llama_proxy_user_hosts_list", "value": "false" }, { "desc": "When set to true, pools specified in applications but not explicitly configured, are created at runtime with default settings. When set to false, applications specifying pools not explicitly configured run in a pool named default. This setting applies when an application explicitly specifies a pool and when the application runs in a pool named with the username associated with the application.", "display_name": "Allow Undeclared Pools", "name": "yarn_scheduler_fair_allow_undeclared_pools", "value": "true" }, { "desc": "How long to keep aggregation logs before deleting them.", "display_name": "Log Aggregation Retention Period", "name": "yarn_log_aggregation_retain_seconds", "value": "604800" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service MapReduce Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_yarn_service_mapred_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_yarn_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Llama Proxy User Groups", "name": "service_config_suppression_llama_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether YARN uses the Linux Container Executor both in secure (Kerberos) and insecure (not Kerberos) environments. Cgroups enforcement only works when the Linux Container Executor is used.", "display_name": "Always Use Linux Container Executor", "name": "yarn_service_lce_always", "value": "false" }, { "desc": "Whether to serve logs over HTTP from YARN web servers. This includes listing the logs directory at the /logs endpoint, which may be a security concern.", "display_name": "Serve logs over HTTP", "name": "http_logs_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User's Home Directory parameter.", "display_name": "Suppress Parameter Validation: System User's Home Directory", "name": "service_config_suppression_hdfs_user_home_dir", "value": "false" }, { "desc": "UNIX user that containers run as when Linux-container-executor is used in nonsecure mode.", "display_name": "UNIX User for Nonsecure Mode with Linux Container Executor", "name": "yarn_nodemanager_linux_container_executor_nonsecure_mode_local_user", "value": "nobody" }, { "desc": "For advanced use only, a string to be inserted into mapred-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)", "name": "yarn_service_mapred_safety_valve", "value": null }, { "desc": "Comma-delimited list of hosts where you want to allow the Llama (AM for Impala) user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Llama Proxy User Hosts", "name": "llama_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the ResourceManager Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: ResourceManager Count Validator", "name": "service_config_suppression_resourcemanager_count_validator", "value": "false" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use.", "display_name": "Fair Scheduler Configuration Rules (Deployed)", "name": "yarn_fs_schedule_rules", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the JobHistory Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: JobHistory Server Count Validator", "name": "service_config_suppression_jobhistory_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_yarn_core_site_safety_valve", "value": "false" }, { "desc": "The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory.", "display_name": "System User's Home Directory", "name": "hdfs_user_home_dir", "value": "/var/lib/hadoop-yarn" }, { "desc": "Whether YARN creates a cgroup per container, thereby isolating the CPU usage of containers. When set, yarn.nodemanager.linux-container-executor.resources-handler.class is configured to org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler. The host (in Cloudera Manager) must have cgroups enabled. The number of shares allocated to all YARN containers is configured by adjusting the CPU shares value of the Node Manager in the Resource Management configuration group.", "display_name": "Use CGroups for Resource Management", "name": "yarn_service_cgroups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the UNIX User for Nonsecure Mode with Linux Container Executor parameter.", "display_name": "Suppress Parameter Validation: UNIX User for Nonsecure Mode with Linux Container Executor", "name": "service_config_suppression_yarn_nodemanager_linux_container_executor_nonsecure_mode_local_user", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. Typically edited using the Pools configuration UI.", "display_name": "Fair Scheduler Allocations (Staged)", "name": "yarn_fs_scheduled_allocations_draft", "value": null }, { "desc": "Name of the ZooKeeper service that this YARN service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "For advanced use only, a string to be inserted into ssl-client.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "yarn_ssl_client_safety_valve", "value": null }, { "desc": "When computing the overall YARN service health, whether to consider the active ResourceManager's health.", "display_name": "Active ResourceManager Role Health Check", "name": "yarn_resourcemanagers_health_enabled", "value": "true" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "yarn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RM-HA Cluster ID parameter.", "display_name": "Suppress Parameter Validation: RM-HA Cluster ID", "name": "service_config_suppression_yarn_rm_ha_cluster_id", "value": "false" }, { "desc": "The group that this service's processes should run as. (Except the Job History Server, which has its own group.)", "display_name": "System Group", "name": "process_groupname", "value": "hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin ACL parameter.", "display_name": "Suppress Parameter Validation: Admin ACL", "name": "service_config_suppression_yarn_admin_acl", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Controls which applications an admin user can see in the applications list view", "display_name": "Admin Users Applications List Visibility Settings", "name": "admin_application_list_settings", "value": "ALL" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Comma-delimited list of groups that you want to allow the Llama (AM for Impala) user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Llama Proxy User Groups", "name": "llama_proxy_user_groups_list", "value": "*" }, { "desc": "For advanced use only, a string to be inserted into mapred-site.xml. Applies to all HDFS Replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "mapreduce_service_replication_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the NodeManager Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Health", "name": "service_health_suppression_yarn_node_managers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Client Truststore File Location parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Client Truststore File Location", "name": "service_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "service_config_suppression_yarn_service_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "This is the maximum time to wait for running containers and applications to complete before transition a DECOMMISSIONING node into DECOMMISSIONED. (-1 indicates infinite timeout, 0 indicates non-graceful)", "display_name": "Node Manager Graceful Decommission Timeout", "name": "yarn_resourcemanager_nodemanager_graceful_decommission_timeout_secs", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hive-hiveserver2.json0000666000175100017510000014367013245514472025660 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Downloaded Resources Directory.", "display_name": "Hive Downloaded Resources Directory Free Space Monitoring Absolute Thresholds", "name": "hiveserver2_downloaded_resources_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Local Directory where Hive stores jars and data when performing a MapJoin optimization. If not specified, Hive uses a default location.", "display_name": "Hive Local Scratch Directory", "name": "hiveserver2_exec_local_scratchdir", "value": null }, { "desc": "Local directory where Hive stores jars downloaded for remote file systems (HDFS). If not specified, Hive uses a default location.", "display_name": "Hive Downloaded Resources Directory", "name": "hiveserver2_downloaded_resources_dir", "value": null }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "hiveserver2_pause_duration_window", "value": "5" }, { "desc": "Minimum number of worker threads in HiveServer2's thread pool", "display_name": "Min HiveServer2 Threads", "name": "hiveserver2_min_threads", "value": "5" }, { "desc": "When enabled, Spark will add and remove executors dynamically to Hive jobs. This is done based on the workload.", "display_name": "Enable Dynamic Executor Allocation", "name": "hiveserver2_spark_dynamic_allocation_enabled", "value": "true" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The check interval for session/operation timeout, in milliseconds, which can be disabled by setting to zero or a negative value. ", "display_name": "Session Check Interval", "name": "hiveserver2_session_check_interval", "value": "900000" }, { "desc": "The maximum percentage of heap to be used for hash in ReduceSink operator for Top-K selection. 0 means the optimization is disabled. Accepted values are between 0 and 1.", "display_name": "Maximum ReduceSink Top-K Memory Usage", "name": "hiveserver2_limit_pushdown_memory_usage", "value": "0.1" }, { "desc": "Whether to try sorted merge bucket (SMB) join.", "display_name": "Hive Optimize Sorted Merge Bucket Join", "name": "hiveserver2_optimize_bucketmapjoin_sortedmerge", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Number of Spark executors assigned to each application. This should not be set when Dynamic Executor Allocation is enabled.", "display_name": "Spark Executors Per Application", "name": "hiveserver2_spark_executor_instances", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_hs2_config_safety_valve", "value": "false" }, { "desc": "Some select queries can be converted to a single FETCH task instead of a MapReduce task, minimizing latency. A value of none disables all conversion, minimal converts simple queries such as SELECT * and filter on partition columns, and more converts SELECT queries including FILTERS.", "display_name": "Fetch Task Query Conversion", "name": "hiveserver2_fetch_task_conversion", "value": "minimal" }, { "desc": "Portion of total memory used in map-side partial aggregation. When exceeded, the partially aggregated results will be flushed from the map task to the reducers.", "display_name": "Ratio of Memory Usage for Map-Side Aggregation", "name": "hiveserver2_map_aggr_hash_memory_ratio", "value": "0.5" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hiveserver2_host_health", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When enabled, HiveServer2 will temporarily save logs associated with ongoing operations. This enables clients like beeline and Hue to request and display logs for a particular ongoing operation. Logs are removed upon completion of operation.", "display_name": "Enable HiveServer2 Operations Logging", "name": "hive_server2_logging_operation_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_hiveserver2_audit_health", "value": "false" }, { "desc": "The number of threads used by HiveServer2 to move data from the staging directory to another location (typically to the final table location). A separate thread pool of workers of this size is used for each query, which means this configuration can be set on a per-query basis too.", "display_name": "Move Files Thread Count", "name": "hiveserver2_mv_files_thread", "value": "15" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Number of threads used to load dynamically generated partitions. Loading requires renaming the file its final location, and updating some metadata about the new partition. Increasing this can improve performance when there are a lot of partitions dynamically generated.", "display_name": "Load Dynamic Partitions Thread Count", "name": "hiveserver2_load_dynamic_partitions_thread_count", "value": "15" }, { "desc": "Upper bound on the number of executors used by the application at any given time. This is used by dynamic executor allocation.", "display_name": "Upper Bound on Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_max_executors", "value": "2147483647" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HiveServer2 WebUI is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HiveServer2 WebUI TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "Whether column stats for a table are fetched during explain.", "display_name": "Load Column Statistics", "name": "hiveserver2_stats_fetch_column_stats", "value": "true" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HiveServer2 in Bytes", "name": "hiveserver2_java_heapsize", "value": "4294967296" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When the number of ReduceSink operators after merging is less than this number, the ReduceDeDuplication optimization will be disabled.", "display_name": "Mininum Reducers for ReduceDeDuplication Optimization", "name": "hiveserver2_optimize_reducededuplication_min_reducer", "value": "4" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hiveserver2_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The desired file size after merging. This should be larger than hive.merge.smallfiles.avgsize.", "display_name": "Desired File Size After Merging", "name": "hiveserver2_merge_size_per_task", "value": "268435456" }, { "desc": "Whether to use the indexing optimization for all queries.", "display_name": "Enable Automatic Use of Indexes", "name": "hiveserver2_optimize_index_filter", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hiveserver2_log_directory_free_space", "value": "false" }, { "desc": "HiveServer2 will impersonate the beeline client user when talking to other services such as MapReduce and HDFS.", "display_name": "HiveServer2 Enable Impersonation", "name": "hiveserver2_enable_impersonation", "value": "true" }, { "desc": "Enable optimization that checks if a query can be answered using statistics. If so, answers the query using only statistics stored in metastore.", "display_name": "Enable Stats Optimization", "name": "hiveserver2_compute_query_using_stats", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hive_metrics_sample_file_location", "value": "false" }, { "desc": "Enable optimization that converts common join into MapJoin based on input file size.", "display_name": "Enable MapJoin Optimization", "name": "hiveserver2_enable_mapjoin", "value": "true" }, { "desc": "An XML string that will be inserted verbatim into the Fair Scheduler allocations file. This configuration only has effect in CDH 5.8 or later.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "hiveserver2_fair_scheduler_safety_valve", "value": null }, { "desc": "Batch size for the msck repair command (recover partitions command). If the value is greater than zero, new partition information will be sent from HiveServer2 to the Metastore in batches, which can potentially improve memory usage in the Metastore and avoid client read timeout exceptions. If this value is 0, all partition information will sent in a single Thrift call.", "display_name": "MSCK Repair Batch Size", "name": "hiveserver2_msck_repair_batch_size", "value": "0" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hiveserver2_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Local Scratch Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Hive Local Scratch Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Hive Local Scratch Directory Free Space Monitoring Percentage Thresholds", "name": "hiveserver2_exec_local_scratch_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Local Scratch Directory.", "display_name": "Hive Local Scratch Directory Free Space Monitoring Absolute Thresholds", "name": "hiveserver2_exec_local_scratch_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_hs2_config_safety_valve", "value": null }, { "desc": "Ratio between 0.0 and 1.0 of entries in the vectorized group-by aggregation hash that is flushed when the memory threshold is exceeded.", "display_name": "Vectorized GroupBy Flush Ratio", "name": "hiveserver2_vectorized_groupby_flush_ratio", "value": "0.1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 WebUI TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 WebUI TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HiveServer2 parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HiveServer2", "name": "role_config_suppression_hiveserver2_java_opts", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hiveserver2_heap_dump_directory_free_space", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "When computing the overall HiveServer2 health, consider the host's health.", "display_name": "HiveServer2 Host Health Test", "name": "hiveserver2_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hiveserver2_scm_health", "value": "false" }, { "desc": "In vectorized group-by, the number of row entries added to the hash table before re-checking average variable size for memory usage estimation.", "display_name": "Vectorized GroupBy Check Interval", "name": "hiveserver2_vectorized_groupby_checkinterval", "value": "4096" }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hive/metrics-hivemetastore/metrics.log. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Location", "name": "hive_metrics_sample_file_location", "value": null }, { "desc": "Initial number of executors used by the application at any given time. This is required if the dynamic executor allocation feature is enabled.", "display_name": "Initial Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_initial_executors", "value": "1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Merge small files at the end of a map-only job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Map-Only Job", "name": "hiveserver2_merge_mapfiles", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 WebUI TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 WebUI TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Lower bound on the number of executors used by the application at any given time. This is used by dynamic executor allocation.", "display_name": "Lower Bound on Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_min_executors", "value": "1" }, { "desc": "The number of rows with the same key value to be cached in memory per SMB-joined table.", "display_name": "Hive SMB Join Cache Rows", "name": "hiveserver2_smbjoin_cache_rows", "value": "10000" }, { "desc": "Maximum number of worker threads in HiveServer2's thread pool", "display_name": "Max HiveServer2 Threads", "name": "hiveserver2_max_threads", "value": "100" }, { "desc": "The minimum log level for HiveServer2 logs", "display_name": "HiveServer2 Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "This is the amount of extra off-heap memory that can be requested from YARN, per executor process. This, together with spark.executor.memory, is the total memory that YARN can use to create JVM for an executor process.", "display_name": "Spark Executor Memory Overhead", "name": "hiveserver2_spark_yarn_executor_memory_overhead", "value": "26" }, { "desc": "Enable optimization that vectorizes query execution by streamlining operations by processing a block of 1024 rows at a time.", "display_name": "Enable Vectorization Optimization", "name": "hiveserver2_vectorized_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_hs2_env_safety_valve", "value": "false" }, { "desc": "Number of cores per Spark executor.", "display_name": "Spark Executor Cores", "name": "hiveserver2_spark_executor_cores", "value": "1" }, { "desc": "The default execution engine for running hive queries. Can be set to mr for MapReduce or spark for Spark.", "display_name": "Default Execution Engine", "name": "hs2_execution_engine", "value": "mr" }, { "desc": "Address of the load balancer used for HiveServer2 roles, specified in host:port format. If port is not specified, the port used by HiveServer2 is used. Note: Changing this property regenerates Kerberos keytabs for all HiveServer2 roles.", "display_name": "HiveServer2 Load Balancer", "name": "hiverserver2_load_balancer", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Port on which HiveServer2 will listen for connections.", "display_name": "HiveServer2 Port", "name": "hs2_thrift_address_port", "value": "10000" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HiveServer2", "name": "hiveserver2_java_opts", "value": "-XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The server address of HiverServer2 host to be used for communication between Hive client and remote Spark driver. Default is empty, which means the address will be determined in the same way as for hive.server2.thrift.bind.host. This is only necessary if the host has mutiple network addresses and if a different network address other than hive.server2.thrift.bind.host is to be used.", "display_name": "Spark RPC Server Address", "name": "hive_spark_client_rpc_server_address", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HiveServer2 Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_hs2_env_safety_valve", "value": null }, { "desc": "Maximum size of each Spark executor's Java heap memory when Hive is running on Spark.", "display_name": "Spark Executor Maximum Java Heap Size", "name": "hiveserver2_spark_executor_memory", "value": "268435456" }, { "desc": "Maximum size of each Spark driver's Java heap memory when Hive is running on Spark.", "display_name": "Spark Driver Maximum Java Heap Size", "name": "hiveserver2_spark_driver_memory", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Maximum number of threads that Hive uses to list input files. Increasing this value can improve performance when there are a lot of partitions being read, or when running on blobstores.", "display_name": "Input Listing Max Threads", "name": "hiveserver2_input_listing_max_threads", "value": "15" }, { "desc": "Encrypt communication between clients and HiveServer2 WebUI using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HiveServer2 WebUI", "name": "ssl_enabled", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Remove extra map-reduce jobs if the data is already clustered by the same key, eliminating the need to repartition the dataset again.", "display_name": "Enable ReduceDeDuplication Optimization", "name": "hiveserver2_optimize_reducededuplication", "value": "true" }, { "desc": "Merge small files at the end of a Spark job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Spark Job", "name": "hiveserver2_merge_sparkfiles", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Operations Log Directory parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Operations Log Directory", "name": "role_config_suppression_hive_server2_logging_operation_log_location", "value": "false" }, { "desc": "Whether to suppress the results of the Hive Local Scratch Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Local Scratch Directory Free Space", "name": "role_health_suppression_hiveserver2_exec_local_scratch_directory_free_space", "value": "false" }, { "desc": "Whether to vectorize the reduce side of query execution.", "display_name": "Enable Reduce-Side Vectorization", "name": "hiveserver2_vectorized_reduce_enabled", "value": "false" }, { "desc": "Merge small files at the end of a map-reduce job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Map-Reduce Job", "name": "hiveserver2_merge_mapredfiles", "value": "false" }, { "desc": "Session will be considered to be idle only if there is no activity, and there is no pending operation. This setting takes effect only if session idle timeout (hive.server2.idle.session.timeout) and checking (hive.server2.session.check.interval) are enabled.", "display_name": "Exclude Live Operations From Session Idle Time", "name": "hiveserver2_idle_session_timeout_check_operation", "value": "true" }, { "desc": "The frequency at which the metrics are logged to the sample file. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Logging Frequency", "name": "hive_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "The port the HiveServer2 WebUI will listen on. This can be set to 0 to disable the WebUI.", "display_name": "HiveServer2 WebUI Port", "name": "hiveserver2_webui_port", "value": "10002" }, { "desc": "Enables the health test that the HiveServer2's process state is consistent with the role configuration", "display_name": "HiveServer2 Process Health Test", "name": "hiveserver2_scm_health_enabled", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Directory where HiveServer2 will place its log files.", "display_name": "HiveServer2 Log Directory", "name": "hive_log_dir", "value": "/var/log/hive" }, { "desc": "If enabled, the HiveServer2 binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind HiveServer2 to Wildcard Address", "name": "hiveserver2_webui_bind_wildcard", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hiveserver2_fair_scheduler_safety_valve", "value": "false" }, { "desc": "Operation will be closed when not accessed for this duration of time, in milliseconds; disable by setting to zero. For a positive value, checked for operations in terminal state only (FINISHED, CANCELED, CLOSED, ERROR). For a negative value, checked for all of the operations regardless of state.", "display_name": "Idle Operation Timeout", "name": "hiveserver2_idle_operation_timeout", "value": "21600000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Downloaded Resources Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Downloaded Resources Directory", "name": "role_config_suppression_hiveserver2_downloaded_resources_dir", "value": "false" }, { "desc": "When dynamic partition is enabled, reducers keep only one record writer at all times, which lowers the memory pressure on reducers.", "display_name": "Enable Sorted Dynamic Partition Optimizer", "name": "hiveserver2_optimize_sort_dynamic_partition", "value": "false" }, { "desc": "Controls whether the Hive metrics subsystem is enabled for the role.", "display_name": "Enable Metrics Subsystem", "name": "hive_metrics_enabled", "value": "true" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "hiveserver2_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hiveserver2_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark RPC Server Address parameter.", "display_name": "Suppress Parameter Validation: Spark RPC Server Address", "name": "role_config_suppression_hive_spark_client_rpc_server_address", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Executor Cores parameter.", "display_name": "Suppress Parameter Validation: Spark Executor Cores", "name": "role_config_suppression_hiveserver2_spark_executor_cores", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HiveServer2 logs. Typically used by log4j or logback.", "display_name": "HiveServer2 Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "If Hive auto convert join is on, and the sum of the size for n-1 of the tables/partitions for a n-way join is smaller than the specified size, the join is directly converted to a MapJoin (there is no conditional task).", "display_name": "Hive Auto Convert Join Noconditional Size", "name": "hiveserver2_auto_convert_join_noconditionaltask_size", "value": "20971520" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_hiveserver2_pause_duration", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Log Directory parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Log Directory", "name": "role_config_suppression_hive_log_dir", "value": "false" }, { "desc": "When enabled, HiveServer2 logs EXPLAIN EXTENDED output for every query at INFO log4j level.", "display_name": "Enable Explain Logging", "name": "hiveserver2_enable_explain_output", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress the results of the Hive Downloaded Resources Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Downloaded Resources Directory Free Space", "name": "role_health_suppression_hiveserver2_downloaded_resources_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Directory in HDFS where Hive writes intermediate data between MapReduce jobs. If not specified, Hive uses a default location.", "display_name": "Hive HDFS Scratch Directory", "name": "hiveserver2_exec_scratchdir", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Load Balancer parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Load Balancer", "name": "role_config_suppression_hiverserver2_load_balancer", "value": "false" }, { "desc": "Session will be closed when not accessed for this duration of time, in milliseconds; disable by setting to zero or a negative value.", "display_name": "Idle Session Timeout", "name": "hiveserver2_idle_session_timeout", "value": "43200000" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The maximum size, in megabytes, per log file for HiveServer2 logs. Typically used by log4j or logback.", "display_name": "HiveServer2 Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Enabled the Calcite-based Cost-Based Optimizer for HiveServer2.", "display_name": "Enable Cost-Based Optimizer for Hive", "name": "hiveserver2_enable_cbo", "value": "false" }, { "desc": "The password for the HiveServer2 WebUI JKS keystore file.", "display_name": "HiveServer2 WebUI TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Enable map-side partial aggregation, which cause the mapper to generate fewer rows. This reduces the data to be sorted and distributed to reducers.", "display_name": "Enable Map-Side Aggregation", "name": "hiveserver2_map_aggr", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The max threads for the HiveServer2 WebUI.", "display_name": "HiveServer2 WebUI Max Threads", "name": "hiveserver2_webui_max_threads", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Local Scratch Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Local Scratch Directory", "name": "role_config_suppression_hiveserver2_exec_local_scratchdir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive HDFS Scratch Directory parameter.", "display_name": "Suppress Parameter Validation: Hive HDFS Scratch Directory", "name": "role_config_suppression_hiveserver2_exec_scratchdir", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Downloaded Resources Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Hive Downloaded Resources Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Hive Downloaded Resources Directory Free Space Monitoring Percentage Thresholds", "name": "hiveserver2_downloaded_resources_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When writing data to a table on a blobstore (such as S3), whether or not the blobstore should be used to store intermediate data during Hive query execution. Setting this to true can degrade performance for queries that spawn multiple MR / Spark jobs, but is useful for queries whose intermediate data cannot fit in the allocated HDFS cluster.", "display_name": "Store Intermediate Data on Blobstore", "name": "hiveserver2_blobstore_use_blobstore_as_scratchdir", "value": "false" }, { "desc": "When the average output file size of a job is less than the value of this property, Hive will start an additional map-only job to merge the output files into bigger files. This is only done for map-only jobs if hive.merge.mapfiles is true, for map-reduce jobs if hive.merge.mapredfiles is true, and for Spark jobs if hive.merge.sparkfiles is true.", "display_name": "Small File Average Size Merge Threshold", "name": "hiveserver2_merge_smallfiles_avgsize", "value": "16777216" }, { "desc": "Above this size, queries are converted to fetch tasks.", "display_name": "Fetch Task Query Conversion Threshold", "name": "hiveserver2_fetch_task_conversion_threshold", "value": "268435456" }, { "desc": "This is the amount of extra off-heap memory that can be requested from YARN, per driver. This, together with spark.driver.memory, is the total memory that YARN can use to create JVM for a driver process.", "display_name": "Spark Driver Memory Overhead", "name": "hiveserver2_spark_yarn_driver_memory_overhead", "value": "26" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hiveserver2_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Top level directory where operation logs are temporarily stored if Enable HiveServer2 Operations Logging is true. Logs are stored in session and operation level subdirectories under this location and are removed on completion of operation.", "display_name": "HiveServer2 Operations Log Directory", "name": "hive_server2_logging_operation_log_location", "value": "/var/log/hive/operation_logs" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hue-service.json0000666000175100017510000011235013245514472024671 0ustar zuulzuul00000000000000[ { "desc": "Create users in Hue when they try to login with their LDAP credentials. For use when using LdapBackend for Hue authentication.", "display_name": "Create LDAP users on login", "name": "create_users_on_login", "value": "true" }, { "desc": "Number of threads used by the Hue web server.", "display_name": "Hue Web Server Threads", "name": "cherrypy_server_threads", "value": "50" }, { "desc": "Comma-separated list of regular expressions, which match 'host:port' of requested proxy target.", "display_name": "Whitelist", "name": "whitelist", "value": "(localhost|127\\.0\\.0\\.1):(50030|50070|50060|50075)" }, { "desc": "The health test thresholds of the overall Load Balancer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Load Balancers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Load Balancers falls below the critical threshold.", "display_name": "Healthy Load Balancer Monitoring Thresholds", "name": "hue_load_balancer_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Thrift Usage Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Thrift Usage Validator", "name": "service_config_suppression_hue_hbase_thrift_server_validator", "value": "false" }, { "desc": "The health test thresholds of the overall Kerberos Ticket Renewer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Kerberos Ticket Renewers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Kerberos Ticket Renewers falls below the critical threshold.", "display_name": "Healthy Kerberos Ticket Renewer Monitoring Thresholds", "name": "hue_kt_renewers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hue" }, { "desc": "The distinguished name to use as a search base for finding users and groups. This should be similar to 'dc=hadoop,dc=mycompany,dc=com'.", "display_name": "LDAP Search Base", "name": "base_dn", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Temporary Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Temporary Directory", "name": "service_config_suppression_hdfs_tmp_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Password parameter.", "display_name": "Suppress Parameter Validation: Hue Database Password", "name": "service_config_suppression_database_password", "value": "false" }, { "desc": "Whether to suppress the results of the Hue Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hue Server Health", "name": "service_health_suppression_hue_hue_servers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Name Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Name Attribute", "name": "service_config_suppression_group_name_attr", "value": "false" }, { "desc": "The name of a default group that users will be added to at creation time.", "display_name": "Default User Group", "name": "default_user_group", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Kerberos Ticket Renewer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kerberos Ticket Renewer Count Validator", "name": "service_config_suppression_kt_renewer_count_validator", "value": "false" }, { "desc": "Distinguished name of the user to bind as. This is used to connect to LDAP/AD for searching user and group information. This may be left blank if the LDAP server supports anonymous binds.", "display_name": "LDAP Bind User Distinguished Name", "name": "bind_dn", "value": null }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "use_start_tls", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_nt_domain", "value": "false" }, { "desc": "The password of the bind user.", "display_name": "LDAP Bind Password", "name": "bind_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default User Group parameter.", "display_name": "Suppress Parameter Validation: Default User Group", "name": "service_config_suppression_default_user_group", "value": "false" }, { "desc": "Type of database used for Hue", "display_name": "Hue Database Type", "name": "database_type", "value": "sqlite3" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the PAM Backend Service Name parameter.", "display_name": "Suppress Parameter Validation: PAM Backend Service Name", "name": "service_config_suppression_pam_auth_service", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Web Interface Role parameter.", "display_name": "Suppress Parameter Validation: HDFS Web Interface Role", "name": "service_config_suppression_hue_webhdfs", "value": "false" }, { "desc": "The SAML password used to log in to the Navigator Metadata Server. Only used when the Navigator Metadata Service Auth uses SAML.", "display_name": "Navigator Metadata Server SAML Password", "name": "navmetadataserver_saml_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Navigator Metadata Server SAML Password parameter.", "display_name": "Suppress Parameter Validation: Navigator Metadata Server SAML Password", "name": "service_config_suppression_navmetadataserver_saml_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Bind User Distinguished Name parameter.", "display_name": "Suppress Parameter Validation: LDAP Bind User Distinguished Name", "name": "service_config_suppression_bind_dn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Username parameter.", "display_name": "Suppress Parameter Validation: Hue Database Username", "name": "service_config_suppression_database_user", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Name of host where the Hue database is running. Not necessary for SQLite3.", "display_name": "Hue Database Hostname", "name": "database_host", "value": "localhost" }, { "desc": "Name of the Sentry service that this Hue service instance depends on", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "The LDAP password used to log in to the Navigator Metadata Server. Only used when the Navigator Metadata Service Auth uses LDAP.", "display_name": "Navigator Metadata Server LDAP Password", "name": "navmetadataserver_ldap_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_ldap_url", "value": "false" }, { "desc": "The SAML username used to log in to the Navigator Metadata Server. Only used when the Navigator Metadata Service Auth uses SAML.", "display_name": "Navigator Metadata Server SAML User", "name": "navmetadataserver_saml_user", "value": null }, { "desc": "The location on disk of the certificate, in .pem format, used to confirm the authenticity of the LDAP server certificate. This is the Certificate Authority (CA) certificate, and it was used to sign the LDAP server certificate. If not set, all certificates are trusted, which means that an attacker could potentially intercept otherwise encrypted usernames and passwords.", "display_name": "LDAP Server CA Certificate", "name": "ldap_cert", "value": null }, { "desc": "HTTPFS role or Namenode (if webhdfs is enabled) that hue can use to communicate with HDFS.", "display_name": "HDFS Web Interface Role", "name": "hue_webhdfs", "value": null }, { "desc": "Whether to suppress the results of the Load Balancer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Load Balancer Health", "name": "service_health_suppression_hue_load_balancer_healthy", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Navigator Metadata Server LDAP Password parameter.", "display_name": "Suppress Parameter Validation: Navigator Metadata Server LDAP Password", "name": "service_config_suppression_navmetadataserver_ldap_password", "value": "false" }, { "desc": "Name of Hue database.", "display_name": "Hue Database Name", "name": "database_name", "value": "hue" }, { "desc": "The attribute of the group object that identifies the members of the group. For Active Directory, this is typically 'member'.", "display_name": "LDAP Group Membership Attribute", "name": "group_member_attr", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Blacklist parameter.", "display_name": "Suppress Parameter Validation: Blacklist", "name": "service_config_suppression_blacklist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Whitelist parameter.", "display_name": "Suppress Parameter Validation: Whitelist", "name": "service_config_suppression_whitelist", "value": "false" }, { "desc": "The base filter for searching for users. For Active Directory, this is typically '(objectClass=user)'.", "display_name": "LDAP User Filter", "name": "user_filter", "value": null }, { "desc": "Name of the Impala service that this Hue service instance depends on", "display_name": "Impala Service", "name": "impala_service", "value": null }, { "desc": "Name of the Oozie service that this Hue service instance depends on", "display_name": "Oozie Service", "name": "oozie_service", "value": null }, { "desc": "Thrift server to use for HBase app.", "display_name": "HBase Thrift Server", "name": "hue_hbase_thrift", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Membership Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Membership Attribute", "name": "service_config_suppression_group_member_attr", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Hostname parameter.", "display_name": "Suppress Parameter Validation: Hue Database Hostname", "name": "service_config_suppression_database_host", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini parameter.", "display_name": "Suppress Parameter Validation: Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", "name": "service_config_suppression_hue_service_safety_valve", "value": "false" }, { "desc": "Name of the Solr service that this Hue service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "Name of the Sqoop service that this Hue service instance depends on", "display_name": "Sqoop Service", "name": "sqoop_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Navigator Metadata Server LDAP User parameter.", "display_name": "Suppress Parameter Validation: Navigator Metadata Server LDAP User", "name": "service_config_suppression_navmetadataserver_ldap_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Server CA Certificate parameter.", "display_name": "Suppress Parameter Validation: LDAP Server CA Certificate", "name": "service_config_suppression_ldap_cert", "value": "false" }, { "desc": "The LDAP username used to log in to the Navigator Metadata Server. Only used when the Navigator Metadata Service Auth uses LDAP.", "display_name": "Navigator Metadata Server LDAP User", "name": "navmetadataserver_ldap_user", "value": null }, { "desc": "In debug mode, Django displays a detailed traceback when an exception occurs. Debugging information may contain sensitive data. Django remembers every SQL query it executes in debug mode, which will rapidly consume memory.", "display_name": "Enable Django Debug Mode", "name": "django_debug_enable", "value": "false" }, { "desc": "Comma-separated list of regular expressions, which match any prefix of 'host:port/path' of requested proxy target. This does not support matching GET parameters.", "display_name": "Blacklist", "name": "blacklist", "value": "()" }, { "desc": "Search Bind Authentication connects to the LDAP server using credentials provided in the 'bind_dn' and 'bind_password' configurations. If these configurations are not set, then an anonymous search is performed.", "display_name": "Use Search Bind Authentication", "name": "search_bind_authentication", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Username Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Username Attribute", "name": "service_config_suppression_user_name_attr", "value": "false" }, { "desc": "Mode of authenticating login credentials. Select desktop.auth.backend.LdapBackend to use LDAP to authenticate login credentials. LDAP requires you to also set the LDAP URL, Active Directory Domain, and optionally LDAP certificate if you are using secure LDAP. Select desktop.auth.backend.PamBackend to use PAM to authenticate login credentials.", "display_name": "Authentication Backend", "name": "auth_backend", "value": "desktop.auth.backend.AllowFirstUserDjangoBackend" }, { "desc": "Whether to suppress configuration warnings produced by the Load Balancer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Load Balancer Count Validator", "name": "service_config_suppression_hue_load_balancer_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Filter parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Filter", "name": "service_config_suppression_group_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Username Pattern parameter.", "display_name": "Suppress Parameter Validation: LDAP Username Pattern", "name": "service_config_suppression_ldap_username_pattern", "value": "false" }, { "desc": "Enable debug output in HTTP Internal Server Error (status 500) responses. Debugging information may contain sensitive data. If Enable Django Debug Mode is set, this is automatically enabled.", "display_name": "Enable Debugging of Internal Server Error Responses", "name": "http_500_debug_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Database Directory", "name": "service_config_suppression_database_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "hue_sentry_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hue_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the User Augmentor parameter.", "display_name": "Suppress Parameter Validation: User Augmentor", "name": "service_config_suppression_user_augmentor", "value": "false" }, { "desc": "The username attribute in the LDAP schema. For Active Directory, this is typically 'sAMAccountName'.", "display_name": "LDAP Username Attribute", "name": "user_name_attr", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Search Base parameter.", "display_name": "Suppress Parameter Validation: LDAP Search Base", "name": "service_config_suppression_base_dn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Navigator Metadata Server SAML User parameter.", "display_name": "Suppress Parameter Validation: Navigator Metadata Server SAML User", "name": "service_config_suppression_navmetadataserver_saml_user", "value": "false" }, { "desc": "Password for Hue database. Not necessary for SQLite3.", "display_name": "Hue Database Password", "name": "database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS HTTPFS Usage Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS HTTPFS Usage Validator", "name": "service_config_suppression_hdfs_httpfs_present_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_hue_sentry_safety_valve", "value": "false" }, { "desc": "When enabled, Hue can talk to the Navigator Metadata Server to provide tagging of tables, columns, and other type of objects and metadata. You must enable Navigator Metadata Server Integration, enable Navigator Audit Collection, and make sure there is a Navigator Metadata Server in the Management Service.", "display_name": "Enable Navigator Metadata Server Integration", "name": "enable_navmetadataserver", "value": "false" }, { "desc": "Name of the HBase service that this Hue service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Base filter for searching for groups. For Active Directory, this is typically '(objectClass=group)'.", "display_name": "LDAP Group Filter", "name": "group_filter", "value": null }, { "desc": "Time zone name.", "display_name": "Time Zone", "name": "time_zone", "value": "America/Los_Angeles" }, { "desc": "Whether to suppress the results of the Kerberos Ticket Renewer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Kerberos Ticket Renewer Health", "name": "service_health_suppression_hue_kt_renewers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Database Dump File parameter.", "display_name": "Suppress Parameter Validation: Database Dump File", "name": "service_config_suppression_database_dump_file", "value": "false" }, { "desc": "Whether to suppress the results of the Beeswax Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Beeswax Server Health", "name": "service_health_suppression_hue_beeswax_server_health", "value": "false" }, { "desc": "The health test thresholds of the overall Hue Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Hue Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Hue Servers falls below the critical threshold.", "display_name": "Healthy Hue Server Monitoring Thresholds", "name": "hue_hue_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The group name attribute in the LDAP schema. For Active Directory, this is typically 'cn'.", "display_name": "LDAP Group Name Attribute", "name": "group_name_attr", "value": null }, { "desc": "HDFS directory used for storing temporary files.", "display_name": "HDFS Temporary Directory", "name": "hdfs_tmp_dir", "value": "/tmp" }, { "desc": "Default encoding for site data.", "display_name": "Default Site Encoding", "name": "default_site_encoding", "value": "utf" }, { "desc": "If the database is SQLite3, this is the filename of the database to use, and the directory of this file must be writable by the 'hue' user.", "display_name": "Hue Database Directory", "name": "database_dir", "value": "/var/lib/hue/desktop.db" }, { "desc": "When you enable anonymous usage data collection Hue tracks anonymised pages and application versions in order to gather information about each application's usage levels. The data collected does not include any hostnames or IDs. Data collection option is available on CDH 4.4 and later deployments.", "display_name": "Enable Usage Data Collection", "name": "usage_data_collection_enable", "value": "true" }, { "desc": "Name of the ZooKeeper service that this Hue service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "ldap_url", "value": null }, { "desc": "The PAM service name to use when authenticating over desktop.auth.backend.PamBackend. This is typically the name of a file under /etc/pam.d/ on the Hue host.", "display_name": "PAM Backend Service Name", "name": "pam_auth_service", "value": "login" }, { "desc": "Whether to suppress configuration warnings produced by the Hue Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hue Server Count Validator", "name": "service_config_suppression_hue_server_count_validator", "value": "false" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "nt_domain", "value": null }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hue" }, { "desc": "Class that defines extra accessor methods for user objects.", "display_name": "User Augmentor", "name": "user_augmentor", "value": "desktop.auth.backend.DefaultUserAugmentor" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hue Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hue_service_env_safety_valve", "value": "false" }, { "desc": "File where the database gets dumped to or loaded from.", "display_name": "Database Dump File", "name": "database_dump_file", "value": "/tmp/hue_database_dump.json" }, { "desc": "LDAP Username Pattern for use with non-Active Directory LDAP implementations. Must contain the special '<username>' string for replacement during authentication.", "display_name": "LDAP Username Pattern", "name": "ldap_username_pattern", "value": null }, { "desc": "Port on host where the Hue database is running. Not necessary for SQLite3.", "display_name": "Hue Database Port", "name": "database_port", "value": "3306" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hue" }, { "desc": "When computing the overall HUE health, consider Beeswax Server's health", "display_name": "Beeswax Server Role Health Test", "name": "hue_beeswax_server_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP User Filter parameter.", "display_name": "Suppress Parameter Validation: LDAP User Filter", "name": "service_config_suppression_user_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Bind Password parameter.", "display_name": "Suppress Parameter Validation: LDAP Bind Password", "name": "service_config_suppression_bind_password", "value": "false" }, { "desc": "The username to use to log into the Hue database. Not necessary for SQLite3.", "display_name": "Hue Database Username", "name": "database_user", "value": "hue" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Time Zone parameter.", "display_name": "Suppress Parameter Validation: Time Zone", "name": "service_config_suppression_time_zone", "value": "false" }, { "desc": "Name of the Hive service that this Hue service instance depends on", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Name parameter.", "display_name": "Suppress Parameter Validation: Hue Database Name", "name": "service_config_suppression_database_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Site Encoding parameter.", "display_name": "Suppress Parameter Validation: Default Site Encoding", "name": "service_config_suppression_default_site_encoding", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hue_safety_valve.ini. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", "name": "hue_service_safety_valve", "value": null }, { "desc": "The selection of external Navigator Metadata Server Authentication backend. Hue server will forward the authentication credentials to the Navigator Metadata Server. This User must have 'Navigator Administrator' privilege.", "display_name": "Navigator Metadata Server Auth", "name": "navmetadataserver_auth_type", "value": "CMDB" }, { "desc": "Auto logout/idle session timeout in seconds, -1 second is equivalent to no automatic logout.", "display_name": "Auto Logout Timeout", "name": "idle_session_timeout", "value": "-1" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/spark-spark_yarn_history_server.json0000666000175100017510000005700313245514472031113 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "History Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SPARK_YARN_HISTORY_SERVER_role_env_safety_valve", "value": null }, { "desc": "A comma-separated list of algorithm names to enable when TLS/SSL is enabled. By default, all algorithms supported by the JRE are enabled.", "display_name": "Enabled SSL/TLS Algorithms", "name": "ssl_server_algorithms", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The password for the History Server JKS keystore file.", "display_name": "History Server TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Protocol parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Protocol", "name": "role_config_suppression_ssl_server_protocol", "value": "false" }, { "desc": "The port of the history server WebUI", "display_name": "History Server WebUI Port", "name": "history_server_web_port", "value": "18088" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Enabled SSL/TLS Algorithms parameter.", "display_name": "Suppress Parameter Validation: Enabled SSL/TLS Algorithms", "name": "role_config_suppression_ssl_server_algorithms", "value": "false" }, { "desc": "How often to poll HDFS for new applications.", "display_name": "HDFS Polling Interval", "name": "history_server_fs_poll_interval", "value": "10" }, { "desc": "The maximum size, in megabytes, per log file for History Server logs. Typically used by log4j or logback.", "display_name": "History Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: History Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "How often the History Server will clean up event log files.", "display_name": "Event Log Cleaner Interval", "name": "event_log_cleaner_interval", "value": "86400" }, { "desc": "The version of the TLS/SSL protocol to use when TLS/SSL is enabled.", "display_name": "TLS/SSL Protocol", "name": "ssl_server_protocol", "value": "TLSv1.2" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "For advanced use only. A string to be inserted into spark-conf/spark-history-server.conf for this role only.", "display_name": "History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-history-server.conf", "name": "spark-conf/spark-history-server.conf_role_safety_valve", "value": null }, { "desc": "Max number of application UIs to keep in the History Server's memory. All applications will still be available, but may take longer to load if they're not in memory.", "display_name": "Retained App Count", "name": "history_server_retained_apps", "value": "50" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_swap_memory_usage", "value": "false" }, { "desc": "When computing the overall History Server health, consider the host's health.", "display_name": "History Server Host Health Test", "name": "spark_yarn_history_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The log directory for log files of the role History Server.", "display_name": "History Server Log Directory", "name": "log_dir", "value": "/var/log/spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: History Server TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Encrypt communication between clients and History Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for History Server", "name": "ssl_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-history-server.conf parameter.", "display_name": "Suppress Parameter Validation: History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-history-server.conf", "name": "role_config_suppression_spark-conf/spark-history-server.conf_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "spark_yarn_history_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Comma-separated list of users who can view all applications when authentication is enabled.", "display_name": "Admin Users", "name": "history_server_admin_users", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "History Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes.", "display_name": "Java Heap Size of History Server in Bytes", "name": "history_server_max_heapsize", "value": "536870912" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_host_health", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_unexpected_exits", "value": "false" }, { "desc": "Specifies the maximum age of the event logs.", "display_name": "Maximum Event Log Age", "name": "event_log_cleaner_max_age", "value": "604800" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: History Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Port where to listen for TLS/SSL connections. HTTP connections will be redirected to this port when TLS/SSL is enabled.", "display_name": "TLS/SSL Port Number", "name": "ssl_server_port", "value": "18488" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when History Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "History Server TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "role_config_suppression_spark-conf/spark-env.sh_role_safety_valve", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The maximum number of rolled log files to keep for History Server logs. Typically used by log4j or logback.", "display_name": "History Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Enables the health test that the History Server's process state is consistent with the role configuration", "display_name": "History Server Process Health Test", "name": "spark_yarn_history_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_audit_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin Users parameter.", "display_name": "Suppress Parameter Validation: Admin Users", "name": "role_config_suppression_history_server_admin_users", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: History Server TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: History Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_spark_yarn_history_server_role_env_safety_valve", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only. A string to be inserted into spark-conf/spark-env.sh for this role only.", "display_name": "History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_role_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for History Server logs", "display_name": "History Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Enables user authentication using SPNEGO (requires Kerberos), and enables access control to application history data.", "display_name": "Enable User Authentication", "name": "history_server_spnego_enabled", "value": "false" }, { "desc": "Specifies whether the History Server should periodically clean up event logs from storage.", "display_name": "Enable Event Log Cleaner", "name": "event_log_cleaner_enabled", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/yarn-nodemanager.json0000666000175100017510000014256313245514472025712 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Containers Environment Variable parameter.", "display_name": "Suppress Parameter Validation: Containers Environment Variable", "name": "role_config_suppression_yarn_nodemanager_admin_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NodeManager parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NodeManager", "name": "role_config_suppression_node_manager_java_opts", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed System Users parameter.", "display_name": "Suppress Parameter Validation: Allowed System Users", "name": "role_config_suppression_container_executor_allowed_system_users", "value": "false" }, { "desc": "When computing the overall NodeManager health, consider the host's health.", "display_name": "NodeManager Host Health Test", "name": "nodemanager_host_health_enabled", "value": "true" }, { "desc": "Enables the health test that the NodeManager's process state is consistent with the role configuration", "display_name": "NodeManager Process Health Test", "name": "nodemanager_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the NodeManager Health Checker heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Health Checker", "name": "role_health_suppression_node_manager_health_checker", "value": "false" }, { "desc": "The minimum Linux user ID allowed. Used to prevent other super users.", "display_name": "Minimum User ID", "name": "container_executor_min_user_id", "value": "1000" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_node_manager_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Local Directories parameter.", "display_name": "Suppress Parameter Validation: NodeManager Local Directories", "name": "role_config_suppression_yarn_nodemanager_local_dirs", "value": "false" }, { "desc": "Whether to suppress the results of the ResourceManager Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ResourceManager Connectivity", "name": "role_health_suppression_node_manager_connectivity", "value": "false" }, { "desc": "The minimum space that must be available on a disk for it to be used. This applies to local directories and log directories.", "display_name": "Disk Health Checker Minimum Health Disk Space", "name": "yarn_nodemanager_disk_health_checker_min_free_space_per_disk_mb", "value": "0" }, { "desc": "Number of threads to use for localization fetching.", "display_name": "Localizer Fetch Thread Count", "name": "yarn_nodemanager_localizer_fetch_thread_count", "value": "4" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_node_manager_audit_health", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "nodemanager_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Recovery Directory.", "display_name": "NodeManager Recovery Directory Free Space Monitoring Absolute Thresholds", "name": "nodemanager_recovery_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_node_manager_log_directory_free_space", "value": "false" }, { "desc": "Number of threads container manager uses.", "display_name": "Container Manager Thread Count", "name": "yarn_nodemanager_container_manager_thread_count", "value": "20" }, { "desc": "Number of virtual CPU cores that can be allocated for containers. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores", "name": "yarn_nodemanager_resource_cpu_vcores", "value": "8" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Address where the localizer IPC is.", "display_name": "Localizer Cache Cleanup Interval", "name": "yarn_nodemanager_localizer_cache_cleanup_interval_ms", "value": "600000" }, { "desc": "List of users banned from running containers.", "display_name": "Banned System Users", "name": "container_executor_banned_users", "value": "hdfs,yarn,mapred,bin" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Container Log Directories parameter.", "display_name": "Suppress Parameter Validation: NodeManager Container Log Directories", "name": "role_config_suppression_yarn_nodemanager_log_dirs", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "nodemanager_mapred_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Local Directories.", "display_name": "NodeManager Local Directories Free Space Monitoring Absolute Thresholds", "name": "nodemanager_local_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Banned System Users parameter.", "display_name": "Suppress Parameter Validation: Banned System Users", "name": "role_config_suppression_container_executor_banned_users", "value": "false" }, { "desc": "Amount of CPU reserved for all the containers on each node.", "display_name": "Containers CPU Limit Percentage", "name": "yarn_nodemanager_resource_percentage_physical_cpu_limit", "value": "100" }, { "desc": "Amount of physical memory, in MiB, that can be allocated for containers.", "display_name": "Container Memory", "name": "yarn_nodemanager_resource_memory_mb", "value": "8192" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NodeManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_nodemanager_role_env_safety_valve", "value": "false" }, { "desc": "Environment variables that containers may override rather than use NodeManager's default.", "display_name": "Containers Environment Variables Whitelist ", "name": "yarn_nodemanager_env_whitelist", "value": "JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME" }, { "desc": "Comma-separated list of arguments which are to be passed to node health script when it is being launched.", "display_name": "Healthchecker Script Arguments", "name": "mapred_healthchecker_script_args", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_node_manager_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_node_manager_unexpected_exits", "value": "false" }, { "desc": "CGroups allows cpu usage limits to be hard or soft. When this setting is true, containers cannot use more CPU usage than allocated even if spare CPU is available. This ensures that containers can only use CPU that they were allocated. When set to false, containers can use spare CPU if available. It should be noted that irrespective of whether set to true or false, at no time can the combined CPU usage of all containers exceed the value specified in Containers CPU Limit Percentage.", "display_name": "Strict CGroup Resource Usage", "name": "yarn_nodemanager_linux_container_executor_cgroups_strict_resource_usage", "value": "false" }, { "desc": "If enabled, adds 'org.apache.hadoop.mapred.ShuffleHandler' to the NodeManager auxiliary services. This is required for MapReduce applications.", "display_name": "Enable Shuffle Auxiliary Service", "name": "mapreduce_aux_service", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Container Log Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Container Log Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Container Log Directories Free Space Monitoring Percentage Thresholds", "name": "nodemanager_log_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Log Directory parameter.", "display_name": "Suppress Parameter Validation: NodeManager Log Directory", "name": "role_config_suppression_node_manager_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The maximum size, in megabytes, per log file for NodeManager logs. Typically used by log4j or logback.", "display_name": "NodeManager Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "List of users explicitly whitelisted to be allowed to run containers. Users with IDs lower than the \"Minimum User Id\" setting may be whitelisted by using this setting.", "display_name": "Allowed System Users", "name": "container_executor_allowed_system_users", "value": "nobody,impala,hive,llama,hbase" }, { "desc": "Directory where NodeManager will place its log files.", "display_name": "NodeManager Log Directory", "name": "node_manager_log_dir", "value": "/var/log/hadoop-yarn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Recovery Directory parameter.", "display_name": "Suppress Parameter Validation: NodeManager Recovery Directory", "name": "role_config_suppression_yarn_nodemanager_recovery_dir", "value": "false" }, { "desc": "Enables the health check that verifies the NodeManager is seen as healthy by the ResourceManager.", "display_name": "NodeManager Health Checker Health Check", "name": "nodemanager_health_checker_health_enabled", "value": "true" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "nodemanager_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The address of the NodeManager IPC.", "display_name": "NodeManager IPC Address", "name": "yarn_nodemanager_address", "value": "8041" }, { "desc": "Environment variables that should be forwarded from the NodeManager's environment to the container's.", "display_name": "Containers Environment Variable", "name": "yarn_nodemanager_admin_env", "value": "MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the CGroups Hierarchy parameter.", "display_name": "Suppress Parameter Validation: CGroups Hierarchy", "name": "role_config_suppression_linux_container_executor_cgroups_hierarchy", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_node_manager_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Remote App Log Directory Suffix parameter.", "display_name": "Suppress Parameter Validation: Remote App Log Directory Suffix", "name": "role_config_suppression_yarn_nodemanager_remote_app_log_dir_suffix", "value": "false" }, { "desc": "Number of threads used in cleanup.", "display_name": "Cleanup Thread Count", "name": "yarn_nodemanager_delete_thread_count", "value": "4" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NodeManager", "name": "node_manager_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dlibrary.leveldbjni.path={{CONF_DIR}}" }, { "desc": "Target size of localizer cache in MB, per local directory.", "display_name": "Localizer Cache Target Size", "name": "yarn_nodemanager_localizer_cache_target_size_mb", "value": "10240" }, { "desc": "Absolute path to the script which is periodically run by the node health monitoring service to determine if the node is healthy or not. If the value of this key is empty or the file does not exist in the location configured here, the node health monitoring service is not started.", "display_name": "Healthchecker Script Path", "name": "mapred_healthchecker_script_path", "value": "" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Recovery Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Recovery Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Recovery Directory Free Space Monitoring Percentage Thresholds", "name": "nodemanager_recovery_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_nodemanager_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "nodemanager_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Healthchecker Script Path parameter.", "display_name": "Suppress Parameter Validation: Healthchecker Script Path", "name": "role_config_suppression_mapred_healthchecker_script_path", "value": "false" }, { "desc": "The HTTPS port of the NodeManager web application.", "display_name": "NodeManager Web Application HTTPS Port (TLS/SSL)", "name": "nodemanager_webserver_https_port", "value": "8044" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "nodemanager_web_metric_collection_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "nodemanager_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_node_manager_heap_dump_directory_free_space", "value": "false" }, { "desc": "The local filesystem directory in which the NodeManager stores state when recovery is enabled. Recovery is enabled by default.", "display_name": "NodeManager Recovery Directory", "name": "yarn_nodemanager_recovery_dir", "value": "/var/lib/hadoop-yarn/yarn-nm-recovery" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NodeManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_node_manager_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the NodeManager Local Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Local Directories Free Space", "name": "role_health_suppression_nodemanager_local_data_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "HDFS directory where application logs are stored when an application completes.", "display_name": "Remote App Log Directory", "name": "yarn_nodemanager_remote_app_log_dir", "value": "/tmp/logs" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Frequency, in milliseconds, of running disk health checker.", "display_name": "Disk Health Checker Frequency", "name": "yarn_nodemanager_disk_health_checker_interval_ms", "value": "120000" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_node_manager_gc_duration", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "nodemanager_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Healthchecker Script Arguments parameter.", "display_name": "Suppress Parameter Validation: Healthchecker Script Arguments", "name": "role_config_suppression_mapred_healthchecker_script_args", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Path (rooted in the cgroups hierarchy on the machine) where to place YARN-managed cgroups.", "display_name": "CGroups Hierarchy", "name": "linux_container_executor_cgroups_hierarchy", "value": "/hadoop-yarn" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Number of threads to handle localization requests.", "display_name": "Localizer Client Thread Count", "name": "yarn_nodemanager_localizer_client_thread_count", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Local Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Local Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Local Directories Free Space Monitoring Percentage Thresholds", "name": "nodemanager_local_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Single User Mode Overrides Validator configuration validator.", "display_name": "Suppress Configuration Validator: Single User Mode Overrides Validator", "name": "role_config_suppression_single_user_mode_override_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_nodemanager_mapred_safety_valve", "value": "false" }, { "desc": "The HTTP Port of the NodeManager web application.", "display_name": "NodeManager Web Application HTTP Port", "name": "nodemanager_webserver_port", "value": "8042" }, { "desc": "Heartbeat interval to ResourceManager", "display_name": "Heartbeat Interval", "name": "yarn_nodemanager_heartbeat_interval_ms", "value": "1000" }, { "desc": "The amount of time to wait for the NodeManager to fully start up and connect to the ResourceManager before enforcing the connectivity check.", "display_name": "NodeManager Connectivity Tolerance at Startup", "name": "nodemanager_connectivity_tolerance_seconds", "value": "180" }, { "desc": "List of directories on the local filesystem where a NodeManager stores intermediate data files.", "display_name": "NodeManager Local Directories", "name": "yarn_nodemanager_local_dirs", "value": null }, { "desc": "Enables the health check that verifies the NodeManager is connected to the ResourceManager.", "display_name": "NodeManager Connectivity Health Check", "name": "nodemanager_connectivity_health_enabled", "value": "true" }, { "desc": "Number of seconds after an application finishes before the NodeManager's DeletionService will delete the application's localized file and log directory. To diagnose YARN application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories.", "display_name": "Localized Dir Deletion Delay", "name": "yarn_nodemanager_delete_debug_delay_sec", "value": "0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NodeManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The maximum number of rolled log files to keep for NodeManager logs. Typically used by log4j or logback.", "display_name": "NodeManager Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the NodeManager Container Log Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Container Log Directories Free Space", "name": "role_health_suppression_nodemanager_log_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Remote App Log Directory parameter.", "display_name": "Suppress Parameter Validation: Remote App Log Directory", "name": "role_config_suppression_yarn_nodemanager_remote_app_log_dir", "value": "false" }, { "desc": "List of directories on the local filesystem where a NodeManager stores container log files.", "display_name": "NodeManager Container Log Directories", "name": "yarn_nodemanager_log_dirs", "value": "/var/log/hadoop-yarn/container" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NodeManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "NODEMANAGER_role_env_safety_valve", "value": null }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress the results of the NodeManager Recovery Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Recovery Directory Free Space", "name": "role_health_suppression_nodemanager_recovery_directory_free_space", "value": "false" }, { "desc": "Time in seconds to retain user logs. Only applicable if log aggregation is disabled.", "display_name": "Log Retain Duration", "name": "yarn_nodemanager_log_retain_seconds", "value": "10800" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The minimum fraction of number of disks to be healthy for the NodeManager to launch new containers. This correspond to both local directories and log directories; that is, if there are fewer healthy local directories (or log directories) available, then new containers will not be launched on this node.", "display_name": "Disk Health Checker Minimum Health Disks Fraction", "name": "yarn_nodemanager_disk_health_checker_min_healthy_disks", "value": "0.25" }, { "desc": "Address where the localizer IPC is.", "display_name": "Localizer Port", "name": "yarn_nodemanager_localizer_address", "value": "8040" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Container Executor Group parameter.", "display_name": "Suppress Parameter Validation: Container Executor Group", "name": "role_config_suppression_container_executor_group", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Containers Environment Variables Whitelist parameter.", "display_name": "Suppress Parameter Validation: Containers Environment Variables Whitelist ", "name": "role_config_suppression_yarn_nodemanager_env_whitelist", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The minimum log level for NodeManager logs", "display_name": "NodeManager Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}", "display_name": "Remote App Log Directory Suffix", "name": "yarn_nodemanager_remote_app_log_dir_suffix", "value": "logs" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum allowed connections for the shuffle. Set to 0 (zero) to indicate no limit on the number of connections.", "display_name": "Max Shuffle Connections", "name": "mapreduce_shuffle_max_connections", "value": "0" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the NodeManager will check for full disk. This applies to local directories and log directories.", "display_name": "Dish Health Checker Max Disk Utilization Percent", "name": "yarn_nodemanager_disk_health_checker_max_disk_utilization_per_disk_percentage", "value": "90.0" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_node_manager_host_health", "value": "false" }, { "desc": "The system group that owns the container-executor binary. This does not need to be changed unless the ownership of the binary is explicitly changed.", "display_name": "Container Executor Group", "name": "container_executor_group", "value": "yarn" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Container Log Directories.", "display_name": "NodeManager Container Log Directories Free Space Monitoring Absolute Thresholds", "name": "nodemanager_log_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum allowed threads for serving shuffle connections. Set to zero to indicate the default of 2 times the number of available processors.", "display_name": "Max Shuffle Threads", "name": "mapreduce_shuffle_max_threads", "value": "80" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NodeManager in Bytes", "name": "node_manager_java_heapsize", "value": "1073741824" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/sentry-gateway.json0000666000175100017510000000602213245514472025433 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/sentry" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Client Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Sentry Client Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "role_config_suppression_sentry_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for sentry-site.xml.", "display_name": "Sentry Client Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "sentry_client_config_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-balancer.json0000666000175100017510000002264713245514472025154 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Balancer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Balancer", "name": "role_config_suppression_balancer_java_opts", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "balancer_config_safety_valve", "value": null }, { "desc": "Thread pool size for executing block moves.", "display_name": "Mover Threads", "name": "dfs_balancer_mover_threads", "value": "1000" }, { "desc": "The policy that should be used to rebalance HDFS storage. The default DataNode policy balances the storage at the DataNode level. This is similar to the balancing policy from prior releases. The BlockPool policy balances the storage at the block pool level as well as at the DataNode level. The BlockPool policy is relevant only to a Federated HDFS service.", "display_name": "Rebalancing Policy", "name": "rebalancing_policy", "value": "DataNode" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Included Hosts parameter.", "display_name": "Suppress Parameter Validation: Included Hosts", "name": "role_config_suppression_rebalancer_include_hosts", "value": "false" }, { "desc": "Maximum amount of data to move per node in each iteration of the balancer.", "display_name": "Maximum Iteration Size", "name": "dfs_balancer_max_size_to_move", "value": "10737418240" }, { "desc": "Number of block moves to permit in parallel.", "display_name": "Maximum Concurrent Moves", "name": "dfs_balancer_max_concurrent_moves", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_balancer_config_safety_valve", "value": "false" }, { "desc": "The percentage deviation from average utilization, after which a node will be rebalanced. (for example, '10.0' for 10%).", "display_name": "Rebalancing Threshold", "name": "rebalancer_threshold", "value": "10.0" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Balancer", "name": "balancer_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Excluded Hosts parameter.", "display_name": "Suppress Parameter Validation: Excluded Hosts", "name": "role_config_suppression_rebalancer_exclude_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Balancer in Bytes", "name": "balancer_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hosts to include in the balancing process (uses all, if none specified).", "display_name": "Included Hosts", "name": "rebalancer_include_hosts", "value": null }, { "desc": "Manual override to specify which DataNodes should be used to off-load data to less full nodes.", "display_name": "Source Hosts", "name": "rebalancer_source_hosts", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Source Hosts parameter.", "display_name": "Suppress Parameter Validation: Source Hosts", "name": "role_config_suppression_rebalancer_source_hosts", "value": "false" }, { "desc": "Hosts to exclude from the balancing process.", "display_name": "Excluded Hosts", "name": "rebalancer_exclude_hosts", "value": null }, { "desc": "Thread pool size for dispatching block moves.", "display_name": "Dispatcher Threads", "name": "dfs_balancer_dispatcher_threads", "value": "200" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/sentry-service.json0000666000175100017510000003160613245514472025440 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the Sentry Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Sentry Server Count Validator", "name": "service_config_suppression_sentry_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Password parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Password", "name": "service_config_suppression_sentry_server_database_password", "value": "false" }, { "desc": "Whether to suppress the results of the Sentry Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sentry Server Health", "name": "service_health_suppression_sentry_sentry_servers_healthy", "value": "false" }, { "desc": "Password for Sentry Server database.", "display_name": "Sentry Server Database Password", "name": "sentry_server_database_password", "value": "" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "sentry" }, { "desc": "User for Sentry Server database.", "display_name": "Sentry Server Database User", "name": "sentry_server_database_user", "value": "sentry" }, { "desc": "Type of Sentry Server database.", "display_name": "Sentry Server Database Type", "name": "sentry_server_database_type", "value": "mysql" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Host parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Host", "name": "service_config_suppression_sentry_server_database_host", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sentry Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "SENTRY_service_env_safety_valve", "value": null }, { "desc": "Name of the ZooKeeper service that this Sentry service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Connecting Users parameter.", "display_name": "Suppress Parameter Validation: Allowed Connecting Users", "name": "service_config_suppression_sentry_service_allow_connect", "value": "false" }, { "desc": "Host name of Sentry Server database.", "display_name": "Sentry Server Database Host", "name": "sentry_server_database_host", "value": "localhost" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "sentry_server_config_safety_valve", "value": null }, { "desc": "Name of Sentry Server database.", "display_name": "Sentry Server Database Name", "name": "sentry_server_database_name", "value": "sentry" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "sentry" }, { "desc": "Port number to use for the Web UI.", "display_name": "Web UI Port", "name": "sentry_service_web_port", "value": "29000" }, { "desc": "List of users allowed to connect to the Sentry Server. These are usually service users such as hive and impala, and the list does not usually need to include end users.", "display_name": "Allowed Connecting Users", "name": "sentry_service_allow_connect", "value": "hive,impala,hue,hdfs,solr,kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Name parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Name", "name": "service_config_suppression_sentry_server_database_name", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "sentry" }, { "desc": "The health test thresholds of the overall Sentry Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Sentry Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Sentry Servers falls below the critical threshold.", "display_name": "Healthy Sentry Server Monitoring Thresholds", "name": "sentry_sentry_server_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_sentry_server_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Enables the Sentry Web UI. This must be enabled when using Sentry in high availability configuration.", "display_name": "Enable Web UI", "name": "sentry_service_web_enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Port number of Sentry Server database.", "display_name": "Sentry Server Database Port", "name": "sentry_server_database_port", "value": "3306" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin Groups parameter.", "display_name": "Suppress Parameter Validation: Admin Groups", "name": "service_config_suppression_sentry_service_admin_group", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "If an end user is in one of these admin groups, that user has administrative privileges on the Sentry Server.", "display_name": "Admin Groups", "name": "sentry_service_admin_group", "value": "hive,impala,hue,solr,kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Name of the HDFS service that this Sentry service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_sentry_service_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database User parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database User", "name": "service_config_suppression_sentry_server_database_user", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/impala-catalogserver.json0000666000175100017510000010075713245514472026564 0ustar zuulzuul00000000000000[ { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Catalog Server in Bytes", "name": "catalogd_embedded_jvm_heapsize", "value": "4294967296" }, { "desc": "Directory where the Catalog Server core dump is placed.", "display_name": "Catalog Server Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/catalogd" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Catalog Server Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Catalog Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Catalog Server", "name": "role_config_suppression_catalogd_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The password for the private key in the Catalog Server Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Catalog Server Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "Number of retry attempts the Catalog Server will make when connecting to the Hive Metastore Server. By default, the Catalog Server waits one second between retries.", "display_name": "Catalog Server Hive Metastore Connection Retries", "name": "hive_metastore_connect_retries", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "Port where Catalog Server debug web server runs.", "display_name": "Catalog Server HTTP Server Port", "name": "catalogserver_webserver_port", "value": "25020" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "catalogserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Port where Catalog Server is exported.", "display_name": "Catalog Server Service Port", "name": "catalog_service_port", "value": "26000" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_catalogserver_audit_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server HBase Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server HBase Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hbase_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Hive Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Hive Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hive_conf_safety_valve", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Impala Catalog Server logs. Typically used by log4j or logback.", "display_name": "Impala Catalog Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Username for Catalog Server web server authentication.", "display_name": "Catalog Server Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Password for Catalog Server web server authentication.", "display_name": "Catalog Server Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Directory where Catalog Server will place its log files.", "display_name": "Catalog Server Log Directory", "name": "log_dir", "value": "/var/log/catalogd" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml for this role only.", "display_name": "Catalog Server HDFS Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hdfs_site_conf_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_catalogserver_unexpected_exits", "value": "false" }, { "desc": "Enable/Disable Catalog Server web server. This web server contains useful information about Catalog Server daemon.", "display_name": "Enable Catalog Server Web Server", "name": "catalogd_enable_webserver", "value": "true" }, { "desc": "When computing the overall Impala Catalog Server health, consider the host's health.", "display_name": "Impala Catalog Server Host Health Test", "name": "catalogserver_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Directory for storing Catalog Server Breakpad dumps.", "display_name": "Catalog Server Breakpad Dump Directory", "name": "minidump_path", "value": "/var/log/impala-minidumps" }, { "desc": "The minimum log level for Impala Catalog Server logs", "display_name": "Impala Catalog Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_catalogserver_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_catalogserver_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_catalogserver_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_catalogserver_host_health", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "catalogserver_web_metric_collection_enabled", "value": "true" }, { "desc": "The amount of time to wait for the Catalog Server to fully start up and connect to the StateStore before enforcing the connectivity check.", "display_name": "Catalog Server Connectivity Tolerance at Startup", "name": "catalogserver_connectivity_tolerance", "value": "180" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "Enables the health test that the Impala Catalog Server's process state is consistent with the role configuration", "display_name": "Impala Catalog Server Process Health Test", "name": "catalogserver_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_cmd_args_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_catalogserver_web_metric_collection", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_catalogserver_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enables the health test that verifies the Catalog Server is connected to the StateStore", "display_name": "Catalog Server Connectivity Health Test", "name": "catalogserver_connectivity_health_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_catalogserver_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server HDFS Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server HDFS Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hdfs_site_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the StateStore Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: StateStore Connectivity", "name": "role_health_suppression_catalogserver_connectivity", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Catalog Server Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The amount of time allowed after this role is started that failures of health tests that rely on communication with this role will be tolerated.", "display_name": "Health Test Startup Tolerance", "name": "catalogserver_startup_tolerance", "value": "5" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "catalogserver_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Catalog Server", "name": "catalogd_java_opts", "value": "" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "Catalog Server Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "If true, loads catalog metadata in the background. If false, metadata is loaded lazily (on access). Only effective in CDH 5 and Impala 1.2.4 and higher.", "display_name": "Load Catalog in Background", "name": "load_catalog_in_background", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml for this role only.", "display_name": "Catalog Server HBase Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hbase_conf_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Breakpad Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Breakpad Dump Directory", "name": "role_config_suppression_minidump_path", "value": "false" }, { "desc": "Timeout for requests to the Hive Metastore Server from Catalog Server. Consider increasing this if you have tables with a lot of metadata and see timeout errors.", "display_name": "Catalog Server Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "3600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogserver_role_env_safety_valve", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Catalog Server command line flags. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_cmd_args_safety_valve", "value": null }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Maximum number of Breakpad dump files stored by Catalog Server Role.", "display_name": "Catalog Server Max Breakpad Dump Files", "name": "max_minidumps", "value": "9" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "CATALOGSERVER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_catalogserver_memory_rss_health", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml for this role only.", "display_name": "Catalog Server Hive Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hive_conf_safety_valve", "value": null }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above.", "display_name": "Catalog Server Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "Catalog Server Maximum Log Files", "name": "max_log_files", "value": "10" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-nfsgateway.json0000666000175100017510000007064013245514472025551 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_nfsgateway_role_env_safety_valve", "value": "false" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_nfsgateway_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_nfsgateway_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_nfsgateway_scm_health", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Log Directory parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Log Directory", "name": "role_config_suppression_nfsgateway_log_dir", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "nfsgateway_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "NFS clients often reorder writes. As a result, sequential writes can arrive at the NFS Gateway in random order. This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. Please make sure this directory has enough space. For example, if the application uploads 10 files with each having 100MB, it is recommended that this directory have roughly 1GB of space in case write reorder happens (in the worst case) to every file.", "display_name": "Temporary Dump Directory", "name": "dfs_nfs3_dump_dir", "value": "/tmp/.hdfs-nfs" }, { "desc": "Enables the health test that the NFS Gateway's process state is consistent with the role configuration", "display_name": "NFS Gateway Process Health Test", "name": "nfsgateway_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NFS Gateway in Bytes", "name": "nfsgateway_java_heapsize", "value": "268435456" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "nfsgateway_config_safety_valve", "value": null }, { "desc": "The port number of the system portmap or rpcbind service. This configuration is used by Cloudera Manager to verify if the system portmap or rpcbind service is running before starting NFS Gateway role. Cloudera Manager does not manage the system portmap or rpcbind service.", "display_name": "Portmap (or Rpcbind) Port", "name": "nfs3_portmap_port", "value": "111" }, { "desc": "The NFS Gateway server port.", "display_name": "NFS Gateway Server Port", "name": "nfs3_server_port", "value": "2049" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NFS Gateway", "name": "nfsgateway_java_opts", "value": "" }, { "desc": "The maximum number of rolled log files to keep for NFS Gateway logs. Typically used by log4j or logback.", "display_name": "NFS Gateway Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When computing the overall NFS Gateway health, consider the host's health.", "display_name": "NFS Gateway Host Health Test", "name": "nfsgateway_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Temporary Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Temporary Dump Directory Free Space Monitoring Percentage Thresholds", "name": "nfsgateway_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_nfsgateway_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "The base port where the secure NFS Gateway server web UI listens. Combined with the NFS Gateway server's hostname to build its secure web UI address.", "display_name": "Secure NFS Gateway Web UI Port (TLS/SSL)", "name": "nfs3_https_port", "value": "50579" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Temporary Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Temporary Dump Directory", "name": "role_config_suppression_dfs_nfs3_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "By default, NFS Gateway exported directories can be mounted by any client. For better access control, update this property with a list of host names and access privileges separated by whitespace characters. Host name format can be a single host, a Java regular expression, or an IPv4 address. The access privilege uses rw to specify readwrite and ro to specify readonly access. If the access privilege is not provided, the default is read-only. Examples of host name format and access privilege: \"192.168.0.0/22 rw\", \"host.*.example.com\", \"host1.test.org ro\".", "display_name": "Allowed Hosts and Privileges", "name": "dfs_nfs_exports_allowed_hosts", "value": "* rw" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_nfsgateway_audit_health", "value": "false" }, { "desc": "The port number of the mount daemon implemented inside the NFS Gateway server role.", "display_name": "NFS Gateway MountD Port", "name": "nfs3_mountd_port", "value": "4242" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NFS Gateway Environment Advanced Configuration Snippet (Safety Valve)", "name": "NFSGATEWAY_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NFS Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_nfsgateway_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for NFS Gateway logs. Typically used by log4j or logback.", "display_name": "NFS Gateway Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Hosts and Privileges parameter.", "display_name": "Suppress Parameter Validation: Allowed Hosts and Privileges", "name": "role_config_suppression_dfs_nfs_exports_allowed_hosts", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_nfsgateway_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Temporary Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Temporary Dump Directory Free Space", "name": "role_health_suppression_nfsgateway_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory.", "display_name": "Temporary Dump Directory Free Space Monitoring Absolute Thresholds", "name": "nfsgateway_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NFS Gateway parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NFS Gateway", "name": "role_config_suppression_nfsgateway_java_opts", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_nfsgateway_unexpected_exits", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for NFS Gateway logs", "display_name": "NFS Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_nfsgateway_config_safety_valve", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Directory where NFS Gateway will place its log files.", "display_name": "NFS Gateway Log Directory", "name": "nfsgateway_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "The base port where the NFS Gateway server web UI listens. Combined with the NFS Gateway server hostname to build its HTTP address.", "display_name": "NFS Gateway Web UI Port", "name": "nfs3_http_port", "value": "50079" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hive-server2-sentry-safety.xml0000666000175100017510000000065513245514472027441 0ustar zuulzuul00000000000000 hive.security.authorization.task.factory org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl hive.server2.session.hook org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook hive.sentry.conf.url file:///{{CMF_CONF_DIR}}/sentry-site.xml sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hive-site.xml0000666000175100017510000000303213245514472024172 0ustar zuulzuul00000000000000 hive.metastore.local false hive.metastore.uris thrift://test-master-001.novalocal:9083 hive.metastore.client.socket.timeout 300 hive.metastore.warehouse.dir /user/hive/warehouse hive.warehouse.subdir.inherit.perms true mapred.reduce.tasks -1 hive.exec.reducers.bytes.per.reducer 1073741824 hive.exec.copyfile.maxsize 33554432 hive.exec.reducers.max 999 hive.metastore.execute.setugi true hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.MemoryTokenStore hive.server2.enable.doAs true fs.hdfs.impl.disable.cache true hive.server2.use.SSL false sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hive-gateway.json0000666000175100017510000001222213245514472025041 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only, a string to be inserted into the client configuration for hive-site.xml.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "These are Java command-line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hive_client_java_opts", "value": "-XX:MaxPermSize=512M -Djava.net.preferIPv4Stack=true" }, { "desc": "Timeout for requests to the Hive Metastore Server. Consider increasing this if you have tables with a lot of metadata and see timeout errors. Used by most Hive Metastore clients such as Hive CLI and HiveServer2, but not by Impala. Impala has a separately configured timeout.", "display_name": "Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "300" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hive" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hive-env.sh", "display_name": "Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh", "name": "hive_client_env_safety_valve", "value": null }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hive_client_java_heapsize", "value": "2147483648" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh parameter.", "display_name": "Suppress Parameter Validation: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh", "name": "role_config_suppression_hive_client_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hive_client_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/ks_indexer-service.json0000666000175100017510000004661613245514472026256 0ustar zuulzuul00000000000000[ { "desc": "Name of the HBase service that this Key-Value Store Indexer service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Whether to suppress the results of the Lily HBase Indexer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Lily HBase Indexer Health", "name": "service_health_suppression_ks_indexer_hbase_indexers_healthy", "value": "false" }, { "desc": "The password for the HBase Indexer TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HBase Indexer TLS/SSL Certificate Trust Store Password", "name": "keystore_indexer_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_keystore_indexer_sentry_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_ks_indexer_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Lily HBase Indexer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Lily HBase Indexer Count Validator", "name": "service_config_suppression_hbase_indexer_count_validator", "value": "false" }, { "desc": "Name of the Solr service that this Key-Value Store Indexer service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_keystore_indexer_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Morphlines File parameter.", "display_name": "Suppress Parameter Validation: Morphlines File", "name": "service_config_suppression_morphlines_conf_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Grok Dictionary File parameter.", "display_name": "Suppress Parameter Validation: Grok Dictionary File", "name": "service_config_suppression_grok_dictionary_conf_file", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. Sentry is supported only on CDH 5.4 or later deployments and requires authentication to be turned on for HBase Indexer.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "keystore_indexer_sentry_enabled", "value": "false" }, { "desc": "The health test thresholds of the overall Lily HBase Indexer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Lily HBase Indexers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Lily HBase Indexers falls below the critical threshold.", "display_name": "Healthy Lily HBase Indexer Monitoring Thresholds", "name": "ks_indexer_indexers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Sentry Validator", "name": "service_config_suppression_keystore_indexer_sentry_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Custom Mime-types File parameter.", "display_name": "Suppress Parameter Validation: Custom Mime-types File", "name": "service_config_suppression_custom_mimetypes_file", "value": "false" }, { "desc": "Text that goes into morphlines.conf file used by HBase Indexers. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service.", "display_name": "Morphlines File", "name": "morphlines_conf_file", "value": "SOLR_LOCATOR : {\n # Name of solr collection\n collection : collection\n \n # ZooKeeper ensemble\n zkHost : \"$ZK_HOST\" \n}\n\n\nmorphlines : [\n{\nid : morphline\nimportCommands : [\"org.kitesdk.**\", \"com.ngdata.**\"]\n\ncommands : [ \n {\n extractHBaseCells {\n mappings : [\n {\n inputColumn : \"data:*\"\n outputField : \"data\" \n type : string \n source : value\n }\n ]\n }\n }\n\n\n { logDebug { format : \"output record: {}\", args : [\"@{}\"] } }\n]\n}\n]\n" }, { "desc": "Authentication mechanism used by HBase Indexer.", "display_name": "HBase Indexer Secure Authentication", "name": "hbase_indexer_security_authentication", "value": "simple" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "keystore_indexer_sentry_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Text that goes verbatim into grok-dictionary.conf file used by HBase Indexers.", "display_name": "Grok Dictionary File", "name": "grok_dictionary_conf_file", "value": "USERNAME [a-zA-Z0-9._-]+\nUSER %{USERNAME}\nINT (?:[+-]?(?:[0-9]+))\nBASE10NUM (?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))\nNUMBER (?:%{BASE10NUM})\nBASE16NUM (?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))\nUUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}\n\n# Networking\nMAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})\nCISCOMAC (?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})\nWINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})\nCOMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})\nIP (?/(?>[\\w_%!$@:.,-]+|\\\\.)*)+\n#UNIXPATH (?/dev/pts/%{NONNEGINT})\nBSDTTY (?>/dev/tty[pq][a-z0-9])\nTTY (?:%{BSDTTY}|%{LINUXTTY})\nWINPATH (?>[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+\nURIPROTO [A-Za-z]+(\\+[A-Za-z+]+)?\nURIHOST %{IPORHOST}(?::%{POSINT:port})?\n# uripath comes loosely from RFC1738, but mostly from what Firefox\n# doesn't turn into %XX\nURIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=#%_\\-]*)+\n#URIPARAM \\?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?\nURIPARAM \\?[A-Za-z0-9$.+!*'|(){},~#%&/=:;_?\\-\\[\\]]*\nURIPATHPARAM %{URIPATH}(?:%{URIPARAM})?\nURI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?\n\n# Months: January, Feb, 3, 03, 12, December\nMONTH \\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b\nMONTHNUM (?:0?[1-9]|1[0-2])\nMONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\n\n# Days: Monday, Tue, Thu, etc...\nDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)\n\n# Years?\nYEAR (?>\\d\\d){1,2}\n# Time: HH:MM:SS\n#TIME \\d{2}:\\d{2}(?::\\d{2}(?:\\.\\d+)?)?\n# I'm still on the fence about using grok to perform the time match,\n# since it's probably slower.\n# TIME %{POSINT<24}:%{POSINT<60}(?::%{POSINT<60}(?:\\.%{POSINT})?)?\nHOUR (?:2[0123]|[01]?[0-9])\nMINUTE (?:[0-5][0-9])\n# '60' is a leap second in most time standards and thus is valid.\nSECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\nTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])\n# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)\nDATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}\nDATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}\nISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))\nISO8601_SECOND (?:%{SECOND}|60)\nTIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?\nDATE %{DATE_US}|%{DATE_EU}\nDATESTAMP %{DATE}[- ]%{TIME}\nTZ (?:[PMCE][SD]T)\nDATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}\nDATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}\n\n# Syslog Dates: Month Day HH:MM:SS\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}\nPROG (?:[\\w._/%-]+)\nSYSLOGPROG %{PROG:program}(?:\\[%{POSINT:pid}\\])?\nSYSLOGHOST %{IPORHOST}\nSYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>\nHTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}\n\n# Shortcuts\nQS %{QUOTEDSTRING}\n\n# Log formats\nSYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:\nCOMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}\n\n# Log Levels\nLOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_keystore_indexer_provider_resource", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "ks_indexer_env_safety_valve", "value": null }, { "desc": "Text that goes verbatim into custom-mimetypes.xml file used by HBase Indexers.", "display_name": "Custom Mime-types File", "name": "custom_mimetypes_file", "value": "\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access.", "display_name": "Sentry User to Group Mapping Class", "name": "keystore_indexer_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_keystore_indexer_truststore_file", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HBase Indexer might connect to. This is used when HBase Indexer is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HBase Indexer TLS/SSL Certificate Trust Store File", "name": "keystore_indexer_truststore_file", "value": null }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "keystore_indexer_provider_resource", "value": "/user/hbaseindexer/sentry/sentry-provider.ini" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/zookeeper-service.json0000666000175100017510000003232513245514472026116 0ustar zuulzuul00000000000000[ { "desc": "Configures the path of the root znode under which all canary updates are performed", "display_name": "ZooKeeper Canary Root Znode Path", "name": "zookeeper_canary_root_path", "value": "/cloudera_manager_zookeeper_canary" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "zookeeper" }, { "desc": "Enables SASL authentication between ZooKeeper servers. Only applies when ZooKeeper Kerberos Authentication is enabled.", "display_name": "Enable Server to Server SASL Authentication", "name": "quorum_auth_enable_sasl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Server Count Validator", "name": "service_config_suppression_server_count_validator", "value": "false" }, { "desc": "Size of the thread pool quorum connection manager uses to manage connections between quorum servers. Only applies when ZooKeeper Server to Server SASL Authentication is enabled.", "display_name": "Quorum Connection Manager Thread Pool Size", "name": "quorum_cnxn_threads_size", "value": "20" }, { "desc": "When enabled, ZooKeeper auto purge feature retains this many most recent snapshots and the corresponding transaction logs in the dataDir and dataLogDir respectively and deletes the rest. Defaults to 5. Minimum value is 3.", "display_name": "Auto Purge Snapshots Retain Count", "name": "autopurgeSnapRetainCount", "value": "5" }, { "desc": "The time interval in hours for which the purge task has to be triggered. Set to a positive integer (1 and above) to enable the auto purging. Defaults to 24.", "display_name": "Auto Purge Time Interval", "name": "autopurgeInterval", "value": "24" }, { "desc": "Enable Kerberos authentication for ZooKeeper.", "display_name": "Enable Kerberos Authentication", "name": "enableSecurity", "value": "false" }, { "desc": "Whether to suppress the results of the ZooKeeper Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ZooKeeper Canary", "name": "service_health_suppression_zookeeper_canary_health", "value": "false" }, { "desc": "The number of snapshot files and corresponding transaction logs to keep when running the Cleanup command.", "display_name": "Cleanup Retain Count", "name": "cleanupRetainCount", "value": "5" }, { "desc": "Enables the health check that a client can connect to ZooKeeper and perform basic operations", "display_name": "ZooKeeper Canary Health Check", "name": "zookeeper_canary_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "zookeeper_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Canary Root Znode Path parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Canary Root Znode Path", "name": "service_config_suppression_zookeeper_canary_root_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_zookeeper_env_safety_valve", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "zookeeper" }, { "desc": "Configures the timeout used by the canary sessions with ZooKeeper servers", "display_name": "ZooKeeper Canary Session Timeout", "name": "zookeeper_canary_session_timeout", "value": "30000" }, { "desc": "Amount of time, in ticks, to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they are dropped.", "display_name": "Synchronization Limit", "name": "syncLimit", "value": "5" }, { "desc": "The maximum size of the data that can be stored in a znode in bytes.", "display_name": "Jute Max Buffer", "name": "jute_maxbuffer", "value": "4194304" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "zookeeper" }, { "desc": "Whether to suppress the results of the Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Server Health", "name": "service_health_suppression_zookeeper_servers_healthy", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Amount of time, in ticks, to allow followers to connect and sync to a leader. Increase this value as needed, if the amount of data managed by ZooKeeper is large.", "display_name": "Initialization Limit", "name": "initLimit", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Configures the timeout used by the canary for connection establishment with ZooKeeper servers", "display_name": "ZooKeeper Canary Connection Timeout", "name": "zookeeper_canary_connection_timeout", "value": "10000" }, { "desc": "Configures the timeout used by the canary for ZooKeeper operations", "display_name": "ZooKeeper Canary Operation Timeout", "name": "zookeeper_canary_operation_timeout", "value": "30000" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The length of time, in milliseconds, of a single tick, which is the basic time unit used by ZooKeeper. A tick is used to regulate heartbeats and timeouts.", "display_name": "Tick Time", "name": "tickTime", "value": "2000" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "The health test thresholds of the overall Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Servers falls below the critical threshold.", "display_name": "Healthy Server Monitoring Thresholds", "name": "zookeeper_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the ZooKeeper Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: ZooKeeper Server Count Validator", "name": "service_config_suppression_zookeeper_odd_number_of_servers_validator", "value": "false" }, { "desc": "Automatically create data directories at startup, if they do not exist. Enabling this configuration should be used with care as it will suppress any errors in setup of data directories.", "display_name": "Enable auto-creation of data directories", "name": "zookeeper_datadir_autocreate", "value": "false" }, { "desc": "Whether the leader accepts client connections.", "display_name": "Leader Serves", "name": "leaderServes", "value": "yes" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/priority-one-confs.json0000666000175100017510000000320513245514472026216 0ustar zuulzuul00000000000000[ "dfs_block_size", "dfs_umaskmode", "dfs_webhdfs_enabled", "dfs_permissions", "dfs_replication", "io_compression_codecs", "io_sort_mb", "dfs_datanode_du_reserved", "dfs_datanode_failed_volumes_tolerated", "dfs_name_dir_restore", "fs_trash_interval", "dfs_safemode_min_datanodes", "dfs_safemode_extension", "dfs_access_time_precision", "yarn_acl_enable", "yarn_admin_acl", "yarn_log_aggregation_enable", "yarn_log_aggregation_retain_seconds", "mapreduce_jobhistory_max_age_ms", "mapreduce_jobhistory_cleaner_interval", "mapreduce_reduce_memory_mb", "mapreduce_reduce_java_opts", "mapreduce_map_memory_mb", "mapreduce_map_java_opts", "yarn_nodemanager_container_manager_thread_count", "yarn_nodemanager_delete_thread_count", "yarn_nodemanager_heartbeat_interval_ms", "yarn_nodemanager_localizer_cache_cleanup_interval_ms", "yarn_nodemanager_localizer_client_thread_count", "yarn_nodemanager_localizer_cache_target_size_mb", "yarn_nodemanager_localizer_fetch_thread_count", "yarn_nodemanager_log_retain_seconds", "yarn_nodemanager_resource_memory_mb", "yarn_resourcemanager_client_thread_count", "yarn_resourcemanager_scheduler_client_thread_count", "yarn_resourcemanager_admin_client_thread_count", "yarn_resourcemanager_amliveliness_monitor_interval_ms", "yarn_am_liveness_monitor_expiry_interval_ms", "yarn_resourcemanager_am_max_retries", "yarn_scheduler_minimum_allocation_mb", "yarn_scheduler_maximum_allocation_mb", "yarn_app_mapreduce_am_command_opts", "yarn_app_mapreduce_am_resource_mb" ] sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hive-service.json0000666000175100017510000012404213245514472025044 0ustar zuulzuul00000000000000[ { "desc": "The number of partitions per RPC retrieved during Hive Metastore Sentry HDFS Sync Cache Initialization.", "display_name": "Number of Partitions per RPC on Initialization", "name": "sentry_hdfs_sync_metastore_cache_partitions_per_rpc", "value": "100" }, { "desc": "List of users that are allowed to bypass Sentry Authorization in the Hive metastore. These are usually service users that already ensure that all activity has been authorized, such as hive and impala. Only applies when Hive is using Sentry Service.", "display_name": "Bypass Sentry Authorization Users", "name": "sentry_metastore_service_users", "value": "hive,impala,hue,hdfs" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Sentry Validator", "name": "service_config_suppression_hive_sentry_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HiveServer2 Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HiveServer2 Count Validator", "name": "service_config_suppression_hiveserver2_count_validator", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hive" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HiveServer2 might connect to. This is used when HiveServer2 is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HiveServer2 TLS/SSL Certificate Trust Store File", "name": "hiveserver2_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Name parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Name", "name": "service_config_suppression_hive_metastore_database_name", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_service_config_safety_valve", "value": null }, { "desc": "Port number of Hive Metastore database", "display_name": "Hive Metastore Database Port", "name": "hive_metastore_database_port", "value": "3306" }, { "desc": "Smaller than this size, Hive uses a single-threaded copy; larger than this size, Hive uses DistCp.", "display_name": "Hive Copy Large File Size", "name": "hive_exec_copyfile_maxsize", "value": "33554432" }, { "desc": "Name of Hive Metastore database", "display_name": "Hive Metastore Database Name", "name": "hive_metastore_database_name", "value": "metastore" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop user to group mapping or local groups defined in the policy file. Hadoop user to group mapping may be configured in the Cloudera Manager HDFS service configuration page under the Security section.", "display_name": "Sentry User to Group Mapping Class", "name": "hive_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using the Sentry service instead, add the Sentry service as a dependency to the Hive service. The Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry is supported only on CDH 4.4 or later deployments. Before enabling Sentry, read the requirements and configuration steps in Setting Up Hive Authorization with Sentry.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Proxy Groups Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Proxy Groups Validator", "name": "service_config_suppression_hive_proxy_groups_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the Spark on YARN service that this Hive service instance depends on. If selected, Hive jobs can use the Spark execution engine instead of MapReduce2. Requires that Hive depends on YARN. See Configuring Hive on Spark for more information about Hive on Spark. In CDH releases lower than 5.7, Hive on Spark also requires setting Enable Hive on Spark to true.", "display_name": "Spark On YARN Service", "name": "spark_on_yarn_service", "value": null }, { "desc": "The password for the HiveServer2 JKS keystore file.", "display_name": "HiveServer2 TLS/SSL Server JKS Keystore File Password", "name": "hiveserver2_keystore_password", "value": null }, { "desc": "Instead of talking to Hive Metastore Server for Metastore information, Hive clients will talk directly to the Metastore database.", "display_name": "Bypass Hive Metastore Server", "name": "hive_bypass_metastore_server", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Host parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Host", "name": "service_config_suppression_hive_metastore_database_host", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The maximum number of retries allowed during Hive Metastore Sentry HDFS Sync cache initialization.", "display_name": "Max Number of Retries on Initialization", "name": "sentry_hdfs_sync_metastore_cache_retry_max_num", "value": "1" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_service_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "hive_server2_sentry_safety_valve", "value": null }, { "desc": "Size per reducer. If the input size is 10GiB and this is set to 1GiB, Hive will use 10 reducers.", "display_name": "Hive Bytes Per Reducer", "name": "hive_bytes_per_reducer", "value": "67108864" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_hiveserver2_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_hive_server2_sentry_safety_valve", "value": "false" }, { "desc": "Password for Hive Metastore database", "display_name": "Hive Metastore Database Password", "name": "hive_metastore_database_password", "value": "" }, { "desc": "The password for the HiveServer2 TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HiveServer2 TLS/SSL Certificate Trust Store Password", "name": "hiveserver2_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the WebHCat Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: WebHCat Server Count Validator", "name": "service_config_suppression_webhcat_count_validator", "value": "false" }, { "desc": "Directory name where Hive Metastore's database is stored (only for Derby)", "display_name": "Hive Metastore Derby Path", "name": "hive_metastore_derby_path", "value": "/var/lib/hive/cloudera_manager/derby/metastore_db" }, { "desc": "Perform DataNucleus validation of metadata during startup. Note: when enabled, Hive will log DataNucleus warnings even though Hive will function normally.", "display_name": "Hive Metastore Database DataNucleus Metadata Validation", "name": "hive_metastore_database_datanucleus_metadata_validation", "value": "false" }, { "desc": "Name of the Sentry service that this Hive service instance depends on. If selected, Hive uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "The number of threads the metastore uses when bulk adding partitions to the metastore. Each thread performs some metadata operations for each partition added, such as collecting statistics for the partition or checking if the partition directory exists. This config is also used to control the size of the threadpool used when scanning the filesystem to look for directories that could correspond to partitions, each thread performs a list status on each possible partition directory.", "display_name": "Metastore Bulk Partitions Thread Count", "name": "hive_metastore_fshandler_threads", "value": "15" }, { "desc": "Max number of reducers to use. If the configuration parameter Hive Reduce Tasks is negative, Hive will limit the number of reducers to the value of this parameter.", "display_name": "Hive Max Reducers", "name": "hive_max_reducers", "value": "1099" }, { "desc": "Type of Hive Metastore database. Note that Derby is not recommended and Cloudera Impala does not support Derby.", "display_name": "Hive Metastore Database Type", "name": "hive_metastore_database_type", "value": "mysql" }, { "desc": "Whether to suppress the results of the HiveServer2 Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HiveServer2 Health", "name": "service_health_suppression_hive_hiveserver2s_healthy", "value": "false" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "hiveserver2_ldap_uri", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hive_service_env_safety_valve", "value": "false" }, { "desc": "Host name of Hive Metastore database", "display_name": "Hive Metastore Database Host", "name": "hive_metastore_database_host", "value": "localhost" }, { "desc": "Prevent Metastore operations in the event of schema version incompatibility. Consider setting this to true to reduce probability of schema corruption during Metastore operations. Note that setting this property to true will also set datanucleus.autoCreateSchema property to false and datanucleus.fixedDatastore property to true. Any values set in Cloudera Manager for these properties will be overridden.", "display_name": "Strict Hive Metastore Schema Validation", "name": "hive_metastore_schema_verification", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hive on Spark Dependency Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive on Spark Dependency Validator", "name": "service_config_suppression_hive_on_spark_missing_dependency", "value": "false" }, { "desc": "This configuration overrides the value set for Hive Proxy User Groups configuration in HDFS service for use by Hive Metastore Server. Specify a comma-delimited list of groups that you want to allow access to Hive Metastore metadata and allow the Hive user to impersonate. A value of '*' allows all groups. The default value of empty inherits the value set for Hive Proxy User Groups configuration in the HDFS service.", "display_name": "Hive Metastore Access Control and Proxy User Groups Override", "name": "hive_proxy_user_groups_list", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_hive_sentry_provider_resource", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Warehouse Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Warehouse Directory", "name": "service_config_suppression_hive_warehouse_directory", "value": "false" }, { "desc": "Wait duration in milliseconds for each retry during Hive Metastore Sentry HDFS Sync Cache Initialization.", "display_name": "Retry Wait Time on Initialization", "name": "sentry_hdfs_sync_metastore_cache_retry_wait_duration_millis", "value": "1000" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "hive_core_site_safety_valve", "value": null }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "hive.metastore.client.socket.timeout60" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Bypass Sentry Authorization Users parameter.", "display_name": "Suppress Parameter Validation: Bypass Sentry Authorization Users", "name": "service_config_suppression_sentry_metastore_service_users", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_hiveserver2_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Replication Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hive_replication_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Auxiliary JARs Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Auxiliary JARs Directory", "name": "service_config_suppression_hive_aux_jars_path_dir", "value": "false" }, { "desc": "The health test thresholds of the overall WebHCat Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" WebHCat Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" WebHCat Servers falls below the critical threshold.", "display_name": "Healthy WebHCat Server Monitoring Thresholds", "name": "hive_webhcats_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml. Applies to all Hive Replication jobs.", "display_name": "Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_service_replication_config_safety_valve", "value": null }, { "desc": "This parameter is useful when authenticating against a non-Active Directory server, such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,<this parameter>. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "hiveserver2_ldap_basedn", "value": null }, { "desc": "If set to true, the Hive metastore will treat a problem with cache initialization as a fatal error.", "display_name": "Abort on Initialization Failure", "name": "sentry_hdfs_sync_metastore_cache_fail_on_partial_update", "value": "true" }, { "desc": "Whether Hive Metastore should try to use direct SQL queries instead of DataNucleus for certain read paths. This can improve metastore performance by orders of magnitude when fetching many partitions. In case of failure, execution will fall back to DataNucleus. This configuration is not supported and is disabled when Hive service is configured with Postgres.", "display_name": "Enable Direct SQL", "name": "hive_metastore_database_datanucleus_try_direct_sql", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Bypass Metastore Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Bypass Metastore Validator", "name": "service_config_suppression_hive_bypass_metastore_validator", "value": "false" }, { "desc": "Hive warehouse directory is the location in HDFS where Hive's tables are stored. Note that Hive's default value for its warehouse directory is '/user/hive/warehouse'.", "display_name": "Hive Warehouse Directory", "name": "hive_warehouse_directory", "value": "/user/hive/warehouse" }, { "desc": "User for Hive Metastore database", "display_name": "Hive Metastore Database User", "name": "hive_metastore_database_user", "value": "hive" }, { "desc": "Whether to suppress the results of the WebHCat Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: WebHCat Server Health", "name": "service_health_suppression_hive_webhcats_healthy", "value": "false" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "hiveserver2_ldap_domain", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Hive Metastore Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Metastore Server Count Validator", "name": "service_config_suppression_hivemetastore_count_validator", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "The number of tables per RPC retrieved during Hive Metastore Sentry HDFS Sync Cache Initialization.", "display_name": "Number of Tables per RPC on Initialization", "name": "sentry_hdfs_sync_metastore_cache_tables_per_rpc", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_hiveserver2_ldap_domain", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Derby Path parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Derby Path", "name": "service_config_suppression_hive_metastore_derby_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_hiveserver2_keystore_password", "value": "false" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "hiveserver2_enable_ldap_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_hive_core_site_safety_valve", "value": "false" }, { "desc": "Default number of reduce tasks per job. Usually set to a prime number close to the number of available hosts. Ignored when mapred.job.tracker is \"local\". Hadoop sets this to 1 by default, while Hive uses -1 as the default. When set to -1, Hive will automatically determine an appropriate number of reducers for each job.", "display_name": "Hive Reduce Tasks", "name": "hive_reduce_tasks", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Name for Sentry Authorization parameter.", "display_name": "Suppress Parameter Validation: Server Name for Sentry Authorization", "name": "service_config_suppression_hive_sentry_server", "value": "false" }, { "desc": "The health test thresholds of the overall Hive Metastore Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Hive Metastore Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Hive Metastore Servers falls below the critical threshold.", "display_name": "Healthy Hive Metastore Server Monitoring Thresholds", "name": "hive_hivemetastores_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The health test thresholds of the overall HiveServer2 health. The check returns \"Concerning\" health if the percentage of \"Healthy\" HiveServer2s falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" HiveServer2s falls below the critical threshold.", "display_name": "Healthy HiveServer2 Monitoring Thresholds", "name": "hive_hiveserver2s_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Automatically create or upgrade tables in the Hive Metastore database when needed. Consider setting this to false and managing the schema manually.", "display_name": "Auto Create and Upgrade Hive Metastore Database Schema", "name": "hive_metastore_database_auto_create_schema", "value": "false" }, { "desc": "Encrypt communication between clients and HiveServer2 using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HiveServer2", "name": "hiveserver2_enable_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_hiveserver2_ldap_uri", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of Hive replication jobs.", "display_name": "Hive Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_replication_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "service_config_suppression_hive_service_replication_config_safety_valve", "value": "false" }, { "desc": "Name of the HBase service that this Hive service instance depends on.", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Directory containing auxiliary JARs used by Hive. This should be a directory location and not a classpath containing one or more JARs. This directory must be created and managed manually on hosts that run the Hive Metastore Server, HiveServer2, or the Hive CLI. The directory location is set in the environment as HIVE_AUX_JARS_PATH and will generally override the hive.aux.jars.path property set in XML files, even if hive.aux.jars.path is set in an advanced configuration snippet.", "display_name": "Hive Auxiliary JARs Directory", "name": "hive_aux_jars_path_dir", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database User parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database User", "name": "service_config_suppression_hive_metastore_database_user", "value": "false" }, { "desc": "The server name used when defining privilege rules in Sentry authorization. Sentry uses this name as an alias for the Hive service. It does not correspond to any physical server name.", "display_name": "Server Name for Sentry Authorization", "name": "hive_sentry_server", "value": "server1" }, { "desc": "Whether to suppress the results of the Hive Metastore Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Metastore Server Health", "name": "service_health_suppression_hive_hivemetastores_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_hiveserver2_keystore_path", "value": "false" }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "hive_sentry_provider_resource", "value": "/user/hive/sentry/sentry-provider.ini" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_hiveserver2_ldap_basedn", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Hive service instance depends on.", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "The number of threads used during Hive Metastore Sentry HDFS Sync Cache Initialization.", "display_name": "Number of Threads on Initialization", "name": "sentry_hdfs_sync_metastore_cache_init_threads", "value": "10" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HiveServer2 is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HiveServer2 TLS/SSL Server JKS Keystore File Location", "name": "hiveserver2_keystore_path", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Hive Derby Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Derby Validator", "name": "service_config_suppression_hive_derby_validator", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hive" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hive" }, { "desc": "Let the table directories inherit the permission of the Warehouse or Database directory instead of being created with the permissions derived from dfs umask. This allows Impala to insert into tables created via Hive.", "display_name": "Hive Warehouse Subdirectories Inherit Permissions", "name": "hive_warehouse_subdir_inherit_perms", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Client TLS/SSL In Use With LDAP Authentication Validator configuration validator.", "display_name": "Suppress Configuration Validator: Client TLS/SSL In Use With LDAP Authentication Validator", "name": "service_config_suppression_hive_client_ssl_recommended_with_ldap_auth_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Password parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Password", "name": "service_config_suppression_hive_metastore_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Access Control and Proxy User Groups Override parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Access Control and Proxy User Groups Override", "name": "service_config_suppression_hive_proxy_user_groups_list", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Concurrency Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Concurrency Configuration Validator", "name": "service_config_suppression_hive_concurrency_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Allows URIs when defining privileges in per-database policy files. Warning: Typically, this configuration should be disabled. Enabling it would allow database policy file owner (which is generally not Hive admin user) to grant load privileges to any directory with read access to Hive admin user, including databases controlled by other database policy files.", "display_name": "Allow URIs in Database Policy File", "name": "sentry_allow_uri_db_policyfile", "value": "false" }, { "desc": "In unsecure mode, setting this property to true will cause the Metastore Server to execute DFS operations using the client's reported user and group permissions. Cloudera Manager will set this for all clients and servers.", "display_name": "Set User and Group Information", "name": "hive_set_ugi", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "service_config_suppression_hive_service_config_safety_valve", "value": "false" }, { "desc": "MapReduce jobs are run against this service.", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hbase-master.json0000666000175100017510000010555713245514472025040 0ustar zuulzuul00000000000000[ { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Directory where Master will place its log files.", "display_name": "Master Log Directory", "name": "hbase_master_log_dir", "value": "/var/log/hbase" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "master_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_master_log_directory_free_space", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase Master in Bytes", "name": "hbase_master_java_heapsize", "value": "1073741824" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Master Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When true, HBase Master will bind to 0.0.0.0. Only available with CDH 4.3 and later.", "display_name": "HBase Master Bind to Wildcard Address", "name": "hbase_master_bind_to_wildcard_address", "value": "true" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master Web UI Address parameter.", "display_name": "Suppress Parameter Validation: HBase Master Web UI Address", "name": "role_config_suppression_hbase_master_info_bindaddress", "value": "false" }, { "desc": "The port for the HBase Master web UI. Set to -1 to disable the HBase Master web UI.", "display_name": "HBase Master Web UI Port", "name": "hbase_master_info_port", "value": "60010" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Maximum time an HLog remains in the .oldlogdir directory until an HBase Master thread deletes it.", "display_name": "Maximum Time to Keep HLogs", "name": "hbase_master_logcleaner_ttl", "value": "60000" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the HBase Master Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HBase Master Canary", "name": "role_health_suppression_master_canary_health", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Master Environment Advanced Configuration Snippet (Safety Valve)", "name": "MASTER_role_env_safety_valve", "value": null }, { "desc": "The address for the HBase Master web UI", "display_name": "HBase Master Web UI Address", "name": "hbase_master_info_bindAddress", "value": null }, { "desc": "A comma-separated list of LogCleanerDelegate(s) that are used in LogsCleaner. WAL/HLog cleaner(s) are called in order, so put the log cleaner that prunes the most log files in the front. To implement your own LogCleanerDelegate, add it to HBase's classpath and add the fully-qualified class name here. You should always add the above default log cleaners in the list, unless you have a special reason not to.", "display_name": "HBase Master Log Cleaner Plugins", "name": "hbase_master_logcleaner_plugins", "value": null }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_master_web_metric_collection", "value": "false" }, { "desc": "The host name or IP address of the DNS name server which an HBase Master should use to determine the host name used for communication and display purposes.", "display_name": "HBase Master DNS Name Server", "name": "hbase_master_dns_nameserver", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Master Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_master_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"IPC Server handler.*ClosedChannelException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"IPC Server Responder, call.*output error\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Daughter regiondir does not exist: .*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"File.*might still be open.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"File.*might still be open.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Moving table .+ state to enabled but was already enabled\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Received OPENED for region.*but region was in the state.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Log Directory parameter.", "display_name": "Suppress Parameter Validation: Master Log Directory", "name": "role_config_suppression_hbase_master_log_dir", "value": "false" }, { "desc": "The amount of time allowed after this role is started that failures of health tests that rely on communication with this role will be tolerated.", "display_name": "Health Test Startup Tolerance", "name": "master_startup_tolerance", "value": "5" }, { "desc": "Number of pooled threads to handle region closing in the master.", "display_name": "Region Closing Threads", "name": "hbase_master_executor_closeregion_threads", "value": "5" }, { "desc": "When computing the overall Master health, consider the host's health.", "display_name": "Master Host Health Test", "name": "master_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "master_web_metric_collection_enabled", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Master logs. Typically used by log4j or logback.", "display_name": "Master Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "master_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_master_scm_health", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_master_config_safety_valve", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase Master", "name": "hbase_master_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase Master DNS Network Interface", "name": "role_config_suppression_hbase_master_dns_interface", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The minimum log level for Master logs", "display_name": "Master Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_master_audit_health", "value": "false" }, { "desc": "Number of pooled threads to handle the recovery of the RegionServers in the master.", "display_name": "RegionServer Recovery Threads", "name": "hbase_master_executor_serverops_threads", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Master Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_master_gc_duration", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "master_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Number of RPC Server instances spun up on HBase Master.", "display_name": "HBase Master Handler Count", "name": "hbase_master_handler_count", "value": "25" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_master_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase Master DNS Name Server", "name": "role_config_suppression_hbase_master_dns_nameserver", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Number of pooled threads to handle region opening in the master.", "display_name": "Region Opening Threads", "name": "hbase_master_executor_openregion_threads", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The name of the DNS network interface from which an HBase Master should report its IP address.", "display_name": "HBase Master DNS Network Interface", "name": "hbase_master_dns_interface", "value": null }, { "desc": "Enables the health test that the Master's process state is consistent with the role configuration", "display_name": "Master Process Health Test", "name": "master_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Coprocessor Master Classes parameter.", "display_name": "Suppress Parameter Validation: HBase Coprocessor Master Classes", "name": "role_config_suppression_hbase_coprocessor_master_classes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_master_config_safety_valve", "value": "false" }, { "desc": "Time period in seconds to reset long-running metrics (e.g. compactions). This is an HBase specific configuration.", "display_name": "Extended Period", "name": "hbase_metrics_extended_period", "value": "3600" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "master_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The port that the HBase Master binds to.", "display_name": "HBase Master Port", "name": "hbase_master_port", "value": "60000" }, { "desc": "The maximum number of rolled log files to keep for Master logs. Typically used by log4j or logback.", "display_name": "Master Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "List of org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are loaded by default on the active HMaster process. For any implemented coprocessor methods, the listed classes will be called in order. After implementing your own MasterObserver, just put it in HBase's classpath and add the fully qualified class name here.", "display_name": "HBase Coprocessor Master Classes", "name": "hbase_coprocessor_master_classes", "value": "" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_master_file_descriptor", "value": "false" }, { "desc": "Enables the health test that a client can connect to the HBase Master", "display_name": "HBase Master Canary Health Test", "name": "master_canary_health_enabled", "value": "true" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_master_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_master_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master Log Cleaner Plugins parameter.", "display_name": "Suppress Parameter Validation: HBase Master Log Cleaner Plugins", "name": "role_config_suppression_hbase_master_logcleaner_plugins", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase Master parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase Master", "name": "role_config_suppression_hbase_master_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_master_host_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/kms-service.json0000666000175100017510000001361213245514472024703 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "kms" }, { "desc": "Authentication type for the KMS. Can either be \"simple\" or \"kerberos\".", "display_name": "Authentication Type", "name": "hadoop_kms_authentication_type", "value": "simple" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_kms_service_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "kms" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Key Management Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Key Management Server Count Validator", "name": "service_config_suppression_kms_count_validator", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "KMS_service_env_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/ks_indexer-hbase_indexer.json0000666000175100017510000005562613245514472027417 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Kerberos principal used by the Lily HBase Indexer roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "hbase" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_hbase_indexer_process_groupname", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "ZooKeeper session timeout. Controls the amount of time the HBase Indexer will attempt to connect to ZooKeeper before timing out.", "display_name": "HBase Indexer ZooKeeper Session Timeout", "name": "hbase_indexer_zk_session_timeout", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Lily HBase Indexer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Lily HBase Indexer", "name": "role_config_suppression_hbase_indexer_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer Log Directory", "name": "role_config_suppression_hbase_indexer_log_dir", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_indexer_swap_memory_usage", "value": "false" }, { "desc": "HTTP port used by HBase Indexer.", "display_name": "HBase Indexer HTTP Port", "name": "hbase_indexer_http_port", "value": "11060" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbase_indexer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_indexer_host_health", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Lily HBase Indexer logs. Typically used by log4j or logback.", "display_name": "Lily HBase Indexer Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The user that the HBase Indexer process should run as.", "display_name": "System User", "name": "hbase_indexer_process_username", "value": "hbase" }, { "desc": "The group that the HBase Indexer process should run as.", "display_name": "System Group", "name": "hbase_indexer_process_groupname", "value": "hbase" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Lily HBase Indexer", "name": "hbase_indexer_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_indexer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_indexer_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_indexer_unexpected_exits", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only. A string to be inserted into hbase-indexer-site.xml for this role only.", "display_name": "Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml", "name": "hbase_indexer_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_hbase_indexer_audit_health", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASE_INDEXER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_hbase_indexer_process_username", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Lily HBase Indexer logs. Typically used by log4j or logback.", "display_name": "Lily HBase Indexer Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where HBase Indexer will place its log files.", "display_name": "HBase Indexer Log Directory", "name": "hbase_indexer_log_dir", "value": "/var/log/hbase-solr" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_indexer_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml", "name": "role_config_suppression_hbase_indexer_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When computing the overall Lily HBase Indexer health, consider the host's health.", "display_name": "Lily HBase Indexer Host Health Test", "name": "hbase_indexer_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbase_indexer_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_indexer_log_directory_free_space", "value": "false" }, { "desc": "Enables the health test that the Lily HBase Indexer's process state is consistent with the role configuration", "display_name": "Lily HBase Indexer Process Health Test", "name": "hbase_indexer_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for Lily HBase Indexer logs", "display_name": "Lily HBase Indexer Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Lily HBase Indexer in Bytes", "name": "hbase_indexer_java_heapsize", "value": "1073741824" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hbase-hbaserestserver.json0000666000175100017510000007355613245514472026757 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase REST Server", "name": "hbase_restserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_hbase_restserver_keystore_file", "value": "false" }, { "desc": "Maximum size of the HBase REST Server thread pool. The server can process this number of concurrent requests. Setting this too high can lead to out of memory errors.", "display_name": "HBase REST Server Maximum Threads", "name": "hbase_restserver_threads_max", "value": "100" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbaserestserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Log Directory", "name": "role_config_suppression_hbase_restserver_log_dir", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HBase REST Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASERESTSERVER_role_env_safety_valve", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_restserver_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_hbase_restserver_keystore_password", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_hbase_rest_server_audit_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbaserestserver_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_restserver_config_safety_valve", "value": null }, { "desc": "The password for the HBase REST Server JKS keystore file.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore File Password", "name": "hbase_restserver_keystore_password", "value": null }, { "desc": "The maximum number of rolled log files to keep for HBase REST Server logs. Typically used by log4j or logback.", "display_name": "HBase REST Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_rest_server_scm_health", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase REST Server in Bytes", "name": "hbase_restserver_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_rest_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Encrypt communication between clients and HBase REST Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HBase REST Server", "name": "hbase_restserver_ssl_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Enables the health test that the HBase REST Server's process state is consistent with the role configuration", "display_name": "HBase REST Server Process Health Test", "name": "hbaserestserver_scm_health_enabled", "value": "true" }, { "desc": "The port that HBase REST Server binds to.", "display_name": "HBase REST Server Port", "name": "hbase_restserver_port", "value": "20550" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server DNS Name Server", "name": "role_config_suppression_hbase_restserver_dns_nameserver", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_rest_server_file_descriptor", "value": "false" }, { "desc": "If true, HBase REST Server Web UI will bind to a wildcard address (0.0.0.0). Otherwise it will bind to a host name. Only available in CDH 4.3 and later.", "display_name": "HBase REST Server Web UI Bind to Wildcard Address", "name": "hbase_restserver_info_bind_to_wildcard", "value": "true" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HBase REST Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase REST Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore File Location", "name": "hbase_restserver_keystore_file", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_rest_server_swap_memory_usage", "value": "false" }, { "desc": "When false, all HTTP methods are permitted (GET/PUT/POST/DELETE). When true, only GET is permitted.", "display_name": "Enable HBase REST Server Read Only Mode", "name": "hbase_restserver_readonly", "value": "false" }, { "desc": "The port that HBase REST Server Web UI binds to.", "display_name": "HBase REST Server Web UI Port", "name": "hbase_restserver_info_port", "value": "8085" }, { "desc": "Directory where HBase REST Server will place its log files.", "display_name": "HBase REST Server Log Directory", "name": "hbase_restserver_log_dir", "value": "/var/log/hbase" }, { "desc": "When computing the overall HBase REST Server health, consider the host's health.", "display_name": "HBase REST Server Host Health Test", "name": "hbaserestserver_host_health_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase REST Server is acting as a TLS/SSL server.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore Key Password", "name": "hbase_restserver_keystore_keypassword", "value": null }, { "desc": "The maximum size, in megabytes, per log file for HBase REST Server logs. Typically used by log4j or logback.", "display_name": "HBase REST Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The host name or IP address of the DNS name server which an HBase REST Server should use to determine the host name used for communication and display purposes.", "display_name": "HBase REST Server DNS Name Server", "name": "hbase_restserver_dns_nameserver", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase REST Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase REST Server", "name": "role_config_suppression_hbase_restserver_java_opts", "value": "false" }, { "desc": "HBase REST Server will bind to this address.", "display_name": "HBase REST Server Host Address", "name": "hbase_restserver_host", "value": "0.0.0.0" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Host Address parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Host Address", "name": "role_config_suppression_hbase_restserver_host", "value": "false" }, { "desc": "The name of the DNS network interface from which an HBase REST Server should report its IP address.", "display_name": "HBase REST Server DNS Network Interface", "name": "hbase_restserver_dns_interface", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server DNS Network Interface", "name": "role_config_suppression_hbase_restserver_dns_interface", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_rest_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_hbase_restserver_keystore_keypassword", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_rest_server_host_health", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Minimum size of the HBase REST Server thread pool. The server will maintain at least this number of threads in the pool at all times. The thread pool can grow up to the maximum size set by hbase.rest.threads.max.", "display_name": "HBase REST Server Minimum Threads", "name": "hbase_restserver_threads_min", "value": "2" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_rest_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for HBase REST Server logs", "display_name": "HBase REST Server Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/yarn-resourcemanager.json0000666000175100017510000013115613245514472026610 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "resourcemanager_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For applications that request containers on particular racks, the minimum time in milliseconds the Fair Scheduler waits before accepting a placement on another rack. Requires Fair Scheduler continuous scheduling to be enabled. If continuous scheduling is disabled, yarn.scheduler.fair.locality.threshold.rack should be used instead.", "display_name": "Fair Scheduler Rack Locality Delay", "name": "yarn_scheduler_fair_locality_delay_rack_ms", "value": "4000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_resourcemanager_config_safety_valve", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The expiry interval to wait until a NodeManager is considered dead.", "display_name": "NodeManager Monitor Expiry", "name": "yarn_nm_liveness_monitor_expiry_interval_ms", "value": "600000" }, { "desc": "The HTTP port of the ResourceManager web application.", "display_name": "ResourceManager Web Application HTTP Port", "name": "resourcemanager_webserver_port", "value": "8088" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The address of the applications manager interface in the ResourceManager.", "display_name": "ResourceManager Address", "name": "yarn_resourcemanager_address", "value": "8032" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "resourcemanager_web_metric_collection_enabled", "value": "true" }, { "desc": "The periodic interval that the ResourceManager will check whether ApplicationMasters is still alive.", "display_name": "ApplicationMaster Monitor Interval", "name": "yarn_resourcemanager_amliveliness_monitor_interval_ms", "value": "1000" }, { "desc": "Number of threads used to handle the ResourceManager admin interface.", "display_name": "Admin Client Thread Count", "name": "yarn_resourcemanager_admin_client_thread_count", "value": "1" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "If enabled, the ResourceManager binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind ResourceManager to Wildcard Address", "name": "yarn_rm_bind_wildcard", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_fair_scheduler_configuration", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_resource_manager_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The address of the admin interface in the ResourceManager.", "display_name": "Administration Address", "name": "yarn_resourcemanager_admin_address", "value": "8033" }, { "desc": "The periodic interval that the ResourceManager will check whether NodeManagers are still alive.", "display_name": "NodeManager Monitor Interval", "name": "yarn_resourcemanager_nm_liveness_monitor_interval_ms", "value": "1000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_resource_manager_swap_memory_usage", "value": "false" }, { "desc": "The maximum number of application attempts. This is a global setting for all ApplicationMasters. Each ApplicationMaster can specify its individual maximum through the API, but if the individual maximum is more than the global maximum, the ResourceManager overrides it.", "display_name": "ApplicationMaster Maximum Attempts", "name": "yarn_resourcemanager_am_max_retries", "value": "2" }, { "desc": "For advanced use only. A string to be inserted into nodes_exclude.txt for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt", "name": "rm_hosts_exclude_safety_valve", "value": null }, { "desc": "Enable continuous scheduling in the Fair Scheduler. When enabled, scheduling decisions are decoupled from NodeManager heartbeats, leading to faster resource allocations.", "display_name": "Enable Fair Scheduler Continuous Scheduling", "name": "yarn_scheduler_fair_continuous_scheduling_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_resource_manager_web_metric_collection", "value": "false" }, { "desc": "The periodic interval that the ResourceManager will check whether containers are still alive.", "display_name": "Container Monitor Interval", "name": "yarn_resourcemanager_container_liveness_monitor_interval_ms", "value": "600000" }, { "desc": "The maximum size, in megabytes, per log file for ResourceManager logs. Typically used by log4j or logback.", "display_name": "ResourceManager Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "resourcemanager_mapred_safety_valve", "value": null }, { "desc": "For applications that request containers on particular racks, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another rack. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. If not set, this means don't pass up any scheduling opportunities. Requires Fair Scheduler continuous scheduling to be disabled. If continuous scheduling is enabled, yarn.scheduler.fair.locality-delay-rack-ms should be used instead.", "display_name": "Fair Scheduler Rack Locality Threshold", "name": "resourcemanager_fair_scheduler_locality_threshold_rack", "value": null }, { "desc": "The class to use as the resource scheduler. FairScheduler is only supported in CDH 4.2.1 and later.", "display_name": "Scheduler Class", "name": "yarn_resourcemanager_scheduler_class", "value": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When enabled, any applications that were running on the cluster when the ResourceManager died will be recovered when the ResourceManager next starts. Note: If RM-HA is enabled, then this configuration is always enabled.", "display_name": "Enable ResourceManager Recovery", "name": "yarn_resourcemanager_recovery_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Maximum Attempts parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Maximum Attempts", "name": "role_config_suppression_yarn_resourcemanager_am_max_retries", "value": "false" }, { "desc": "The expiry interval to wait until an ApplicationMaster is considered dead.", "display_name": "ApplicationMaster Monitor Expiry", "name": "yarn_am_liveness_monitor_expiry_interval_ms", "value": "600000" }, { "desc": "Enter an XML string that represents the Capacity Scheduler configuration.", "display_name": "Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)", "name": "resourcemanager_capacity_scheduler_configuration", "value": "\n\n \n yarn.scheduler.capacity.root.queues\n default\n \n \n yarn.scheduler.capacity.root.capacity\n 100\n \n \n yarn.scheduler.capacity.root.default.capacity\n 100\n \n\n" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_resource_manager_audit_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The largest amount of physical memory, in MiB, that can be requested for a container.", "display_name": "Container Memory Maximum", "name": "yarn_scheduler_maximum_allocation_mb", "value": "65536" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt", "name": "role_config_suppression_rm_hosts_exclude_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When enabled, if a pool's minimum share is not met for some period of time, the Fair Scheduler preempts applications in other pools. Preemption guarantees that production applications are not starved while also allowing the cluster to be used for experimental and research applications. To minimize wasted computation, the Fair Scheduler preempts the most recently launched applications.", "display_name": "Enable Fair Scheduler Preemption", "name": "resourcemanager_fair_scheduler_preemption", "value": "false" }, { "desc": "For applications that request containers on particular nodes, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another node. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. If not set, this means don't pass up any scheduling opportunities. Requires Fair Scheduler continuous scheduling to be disabled. If continuous scheduling is enabled, yarn.scheduler.fair.locality-delay-node-ms should be used instead.", "display_name": "Fair Scheduler Node Locality Threshold", "name": "resourcemanager_fair_scheduler_locality_threshold_node", "value": null }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "For applications that request containers on particular nodes, the minimum time in milliseconds the Fair Scheduler waits before accepting a placement on another node. Requires Fair Scheduler continuous scheduling to be enabled. If continuous scheduling is disabled, yarn.scheduler.fair.locality.threshold.node should be used instead.", "display_name": "Fair Scheduler Node Locality Delay", "name": "yarn_scheduler_fair_locality_delay_node_ms", "value": "2000" }, { "desc": "The minimum log level for ResourceManager logs", "display_name": "ResourceManager Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The amount of time allowed after this role is started that failures of health tests that rely on communication with this role will be tolerated.", "display_name": "Health Test Startup Tolerance", "name": "resourcemanager_startup_tolerance_minutes", "value": "5" }, { "desc": "When enabled, ResourceManager has proxy user privileges.", "display_name": "Enable ResourceManager Proxy User Privileges", "name": "yarn_resourcemanager_proxy_user_privileges_enabled", "value": "true" }, { "desc": "An XML string that will be inserted verbatim into the Fair Scheduler allocations file. For CDH5, overrides the configuration set using the Pools configuration UI. For CDH4, this is the only way to configure the Fair Scheduler for YARN.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "resourcemanager_fair_scheduler_configuration", "value": null }, { "desc": "For advanced use only. A string to be inserted into nodes_allow.txt for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt", "name": "rm_hosts_allow_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_resource_manager_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_resource_manager_log_directory_free_space", "value": "false" }, { "desc": "The maximum number of completed applications that the ResourceManager keeps.", "display_name": "Max Completed Applications", "name": "yarn_resourcemanager_max_completed_applications", "value": "10000" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "resourcemanager_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_capacity_scheduler_configuration", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt", "name": "role_config_suppression_rm_hosts_allow_safety_valve", "value": "false" }, { "desc": "The address of the scheduler interface in the ResourceManager.", "display_name": "Scheduler Address", "name": "yarn_resourcemanager_scheduler_address", "value": "8030" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "resourcemanager_gc_duration_window", "value": "5" }, { "desc": "The timeout for the ResourceManager session with ZooKeeper. The session expires if the ZooKeeper ensemble does not hear from the ResourceManager within the specified timeout period (no heartbeat). Session expiration is managed by the ZooKeeper ensemble, not by the ResourceManager.", "display_name": "ZooKeeper Session Timeout", "name": "yarn_resourcemanager_zk_timeout_ms", "value": "60000" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_resource_manager_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "resourcemanager_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_resource_manager_gc_duration", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The smallest number of virtual CPU cores that can be requested for a container. If using the Capacity or FIFO scheduler (or any scheduler, prior to CDH 5), virtual core requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores Minimum", "name": "yarn_scheduler_minimum_allocation_vcores", "value": "1" }, { "desc": "If using the Fair Scheduler, memory requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 5.", "display_name": "Container Memory Increment", "name": "yarn_scheduler_increment_allocation_mb", "value": "512" }, { "desc": "When computing the overall ResourceManager health, consider the host's health.", "display_name": "ResourceManager Host Health Test", "name": "resourcemanager_host_health_enabled", "value": "true" }, { "desc": "When enabled, the Fair Scheduler will assign shares to individual apps based on their size, rather than providing an equal share to all apps regardless of size.", "display_name": "Fair Scheduler Size-Based Weight", "name": "resourcemanager_fair_scheduler_size_based_weight", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The address of the resource tracker interface in the ResourceManager.", "display_name": "Resource Tracker Address", "name": "yarn_resourcemanager_resource_tracker_address", "value": "8031" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_resource_manager_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Log Directory parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Log Directory", "name": "role_config_suppression_resource_manager_log_dir", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for ResourceManager", "name": "resource_manager_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dlibrary.leveldbjni.path={{CONF_DIR}}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "ResourceManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Enables multiple Fair Scheduler container assignments in one heartbeat, which improves cluster throughput when there are many small tasks to run.", "display_name": "Fair Scheduler Assign Multiple Tasks", "name": "resourcemanager_fair_scheduler_assign_multiple", "value": "true" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "resourcemanager_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_resourcemanager_mapred_safety_valve", "value": "false" }, { "desc": "Directory where ResourceManager will place its log files.", "display_name": "ResourceManager Log Directory", "name": "resource_manager_log_dir", "value": "/var/log/hadoop-yarn" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Number of threads to handle resource tracker calls.", "display_name": "Resource Tracker Thread Count", "name": "yarn_resourcemanager_resource_tracker_client_thread_count", "value": "50" }, { "desc": "If using the Fair Scheduler, virtual core requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 5.", "display_name": "Container Virtual CPU Cores Increment", "name": "yarn_scheduler_increment_allocation_vcores", "value": "1" }, { "desc": "The HTTPS port of the ResourceManager web application.", "display_name": "ResourceManager Web Application HTTPS Port (TLS/SSL)", "name": "resourcemanager_webserver_https_port", "value": "8090" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of ResourceManager in Bytes", "name": "resource_manager_java_heapsize", "value": "1073741824" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "The number of threads used to handle applications manager requests.", "display_name": "Client Thread Count", "name": "yarn_resourcemanager_client_thread_count", "value": "50" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_resource_manager_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for ResourceManager logs. Typically used by log4j or logback.", "display_name": "ResourceManager Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "ResourceManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "RESOURCEMANAGER_role_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The utilization threshold after which preemption kicks in. The utilization is computed as the maximum ratio of usage to capacity among all resources.", "display_name": "Fair Scheduler Preemption Utilization Threshold", "name": "yarn_scheduler_fair_preemption_cluster_utilization_threshold", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for ResourceManager parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for ResourceManager", "name": "role_config_suppression_resource_manager_java_opts", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The number of threads used to handle requests through the scheduler interface.", "display_name": "Scheduler Thread Count", "name": "yarn_resourcemanager_scheduler_client_thread_count", "value": "50" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "When set to true, the Fair Scheduler uses the username as the default pool name, in the event that a pool name is not specified. When set to false, all applications are run in a shared pool, called default.", "display_name": "Fair Scheduler User As Default Queue", "name": "resourcemanager_fair_scheduler_user_as_default_queue", "value": "true" }, { "desc": "Enables the health test that the ResourceManager's process state is consistent with the role configuration", "display_name": "ResourceManager Process Health Test", "name": "resourcemanager_scm_health_enabled", "value": "true" }, { "desc": "The largest number of virtual CPU cores that can be requested for a container. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores Maximum", "name": "yarn_scheduler_maximum_allocation_vcores", "value": "32" }, { "desc": "The smallest amount of physical memory, in MiB, that can be requested for a container. If using the Capacity or FIFO scheduler (or any scheduler, prior to CDH 5), memory requests will be rounded up to the nearest multiple of this number.", "display_name": "Container Memory Minimum", "name": "yarn_scheduler_minimum_allocation_mb", "value": "1024" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hue-hue_load_balancer.json0000666000175100017510000004154413245514472026646 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hue_load_balancer_scm_health", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Hue Load Balancer is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue Load Balancer TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_certificate_key", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_ssl_certificate_key", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hue_load_balancer_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hue_load_balancer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf parameter.", "display_name": "Suppress Parameter Validation: Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf", "name": "role_config_suppression_hue_load_balancer_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_ssl_certificate", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Load Balancer Environment Advanced Configuration Snippet (Safety Valve)", "name": "HUE_LOAD_BALANCER_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into httpd.conf for this role only. This can only add options to the configuration, and cannot override previously defined options.", "display_name": "Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf", "name": "hue_load_balancer_safety_valve", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hue_load_balancer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Port to use to connect to the Hue through the Load Balancer.", "display_name": "Hue Load Balancer Port", "name": "listen", "value": "8889" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hue_load_balancer_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hue_load_balancer_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Load Balancer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Load Balancer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hue_load_balancer_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Directory where Hue Load Balancer will place its log files.", "display_name": "Hue Load Balancer Log Directory", "name": "hue_load_balancer_log_dir", "value": "/var/log/hue-httpd" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When computing the overall Load Balancer health, consider the host's health.", "display_name": "Load Balancer Host Health Test", "name": "hue_load_balancer_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer Log Directory", "name": "role_config_suppression_hue_load_balancer_log_dir", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hue_load_balancer_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Enables the health test that the Load Balancer's process state is consistent with the role configuration", "display_name": "Load Balancer Process Health Test", "name": "hue_load_balancer_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hue_load_balancer_swap_memory_usage", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Hue Load Balancer is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue Load Balancer TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_certificate", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_hue_load_balancer_audit_health", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-datanode.json0000666000175100017510000013131713245514472025157 0ustar zuulzuul00000000000000[ { "desc": "Specifies the maximum number of threads to use for transferring data in and out of the DataNode.", "display_name": "Maximum Number of Transfer Threads", "name": "dfs_datanode_max_xcievers", "value": "4096" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Transceiver Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Transceiver Usage", "name": "role_health_suppression_data_node_transceivers_usage", "value": "false" }, { "desc": "In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This behavior will always be disabled for workloads that read only short sections of a block (e.g HBase random-IO workloads). This property is supported in CDH3u3 or later deployments.", "display_name": "Enable purging cache after reads", "name": "dfs_datanode_drop_cache_behind_reads", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_data_node_audit_health", "value": "false" }, { "desc": "Whether DataNodes should use DataNode hostnames when connecting to DataNodes for data transfer. This property is supported in CDH3u4 or later deployments.", "display_name": "Use DataNode Hostname", "name": "dfs_datanode_use_datanode_hostname", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for DataNode", "name": "datanode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Enables the health test that the DataNode's process state is consistent with the role configuration", "display_name": "DataNode Process Health Test", "name": "datanode_scm_health_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "DataNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of DataNode in Bytes", "name": "datanode_java_heapsize", "value": "1073741824" }, { "desc": "Directory where DataNode will place its log files.", "display_name": "DataNode Log Directory", "name": "datanode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_data_node_web_metric_collection", "value": "false" }, { "desc": "While reading block files, the DataNode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader's position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the DataNode will attempt to read ahead. A value of 0 disables this feature. This property is supported in CDH3u3 or later deployments.", "display_name": "Number of read ahead bytes", "name": "dfs_datanode_readahead_bytes", "value": "4194304" }, { "desc": "Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls how much DataNode volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis.", "display_name": "Available Space Policy Balanced Threshold", "name": "dfs_datanode_available_space_balanced_threshold", "value": "10737418240" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "datanode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a DataNode Data Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "DataNode Data Directory Free Space Monitoring Percentage Thresholds", "name": "datanode_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Port for the various DataNode Protocols. Combined with the DataNode's hostname to build its IPC port address.", "display_name": "DataNode Protocol Port", "name": "dfs_datanode_ipc_port", "value": "50020" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "datanode_pause_duration_window", "value": "5" }, { "desc": "In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This property is supported in CDH3u3 or later deployments.", "display_name": "Enable purging cache after writes", "name": "dfs_datanode_drop_cache_behind_writes", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory.", "display_name": "DataNode Data Directory Free Space Monitoring Absolute Thresholds", "name": "datanode_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Permissions for the directories on the local file system where the DataNode stores its blocks. The permissions must be octal. 755 and 700 are typical values.", "display_name": "DataNode Data Directory Permissions", "name": "dfs_datanode_data_dir_perm", "value": "700" }, { "desc": "Maximum amount of bandwidth that each DataNode can use for balancing. Specified in bytes per second.", "display_name": "DataNode Balancing Bandwidth", "name": "dfs_balance_bandwidthPerSec", "value": "10485760" }, { "desc": "The health test thresholds of transceivers usage in a DataNode. Specified as a percentage of the total configured number of transceivers.", "display_name": "DataNode Transceivers Usage Thresholds", "name": "datanode_transceivers_usage_thresholds", "value": "{\"critical\":\"95.0\",\"warning\":\"75.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory parameter.", "display_name": "Suppress Parameter Validation: DataNode Data Directory", "name": "role_config_suppression_dfs_data_dir_list", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_data_node_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 5, \"content\":\"Datanode registration failed\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Got a command from standby NN - ignoring command:.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Failed Volumes Tolerated Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Failed Volumes Tolerated Validator", "name": "role_config_suppression_datanode_failed_volumes_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Data Directory Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Data Directory Status", "name": "role_health_suppression_data_node_volume_failures", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the DataNode Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: DataNode Data Directory Free Space", "name": "role_health_suppression_datanode_data_directories_free_space", "value": "false" }, { "desc": "When computing the overall DataNode health, consider the host's health.", "display_name": "DataNode Host Health Test", "name": "datanode_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for DataNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for DataNode", "name": "role_config_suppression_datanode_java_opts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for DataNode logs. Typically used by log4j or logback.", "display_name": "DataNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "datanode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DateNode Plugins parameter.", "display_name": "Suppress Parameter Validation: DateNode Plugins", "name": "role_config_suppression_dfs_datanode_plugins_list", "value": "false" }, { "desc": "Whether to suppress the results of the NameNode Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Connectivity", "name": "role_health_suppression_data_node_ha_connectivity", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Port for the DataNode HTTP web UI. Combined with the DataNode's hostname to build its HTTP address.", "display_name": "DataNode HTTP Web UI Port", "name": "dfs_datanode_http_port", "value": "50075" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of free space in a DataNode. Specified as a percentage of the capacity on the DataNode.", "display_name": "DataNode Free Space Monitoring Thresholds", "name": "datanode_free_space_thresholds", "value": "{\"critical\":\"10.0\",\"warning\":\"20.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Port for DataNode's XCeiver Protocol. Combined with the DataNode's hostname to build its address.", "display_name": "DataNode Transceiver Port", "name": "dfs_datanode_port", "value": "50010" }, { "desc": "The health test thresholds of the number of blocks on a DataNode", "display_name": "DataNode Block Count Thresholds", "name": "datanode_block_count_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"500000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds of failed volumes in a DataNode.", "display_name": "DataNode Volume Failures Thresholds", "name": "datanode_volume_failures_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_data_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "The minimum log level for DataNode logs", "display_name": "DataNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_data_node_scm_health", "value": "false" }, { "desc": "DataNode Policy for picking which volume should get a new block. The Available Space policy is only available starting with CDH 4.3.", "display_name": "DataNode Volume Choosing Policy", "name": "dfs_datanode_volume_choosing_policy", "value": "org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory Permissions parameter.", "display_name": "Suppress Parameter Validation: DataNode Data Directory Permissions", "name": "role_config_suppression_dfs_datanode_data_dir_perm", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "datanode_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: DataNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: DataNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_datanode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_data_node_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Reserved Space Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Reserved Space Validator", "name": "role_config_suppression_datanode_reserved_space_validator", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "datanode_config_safety_valve", "value": null }, { "desc": "The maximum amount of memory a DataNode may use to cache data blocks in memory. Setting it to zero will disable caching.", "display_name": "Maximum Memory Used for Caching", "name": "dfs_datanode_max_locked_memory", "value": "4294967296" }, { "desc": "Enables the health test that verifies the DataNode is connected to the NameNode", "display_name": "DataNode Connectivity Health Test", "name": "datanode_connectivity_health_enabled", "value": "true" }, { "desc": "Minimum number of running threads for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Min Threadcount", "name": "dfs_thrift_threads_min", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_datanode_config_safety_valve", "value": "false" }, { "desc": "The base port where the secure DataNode web UI listens. Combined with the DataNode's hostname to build its secure web UI address.", "display_name": "Secure DataNode Web UI Port (TLS/SSL)", "name": "dfs_datanode_https_port", "value": "50475" }, { "desc": "The amount of time to wait for the DataNode to fully start up and connect to the NameNode before enforcing the connectivity check.", "display_name": "DataNode Connectivity Tolerance at Startup", "name": "datanode_connectivity_tolerance", "value": "180" }, { "desc": "Maximum number of running threads for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Max Threadcount", "name": "dfs_thrift_threads_max", "value": "20" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Free Space", "name": "role_health_suppression_data_node_free_space_remaining", "value": "false" }, { "desc": "If this configuration is enabled, the DataNode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. This property is supported in CDH3u3 or later deployments.", "display_name": "Enable immediate enqueuing of data to disk after writes", "name": "dfs_datanode_sync_behind_writes", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_data_node_swap_memory_usage", "value": "false" }, { "desc": "Comma-separated list of DataNode plug-ins to be activated. If one plug-in cannot be loaded, all the plug-ins are ignored.", "display_name": "DateNode Plugins", "name": "dfs_datanode_plugins_list", "value": "" }, { "desc": "Comma-delimited list of directories on the local file system where the DataNode stores HDFS block data. Typical values are /data/N/dfs/dn for N = 1, 2, 3.... In CDH 5.7 and higher, these directories can be optionally tagged with their storage types, for example, [SSD]/data/1/dns/dn. HDFS supports the following storage types: [DISK], [SSD], [ARCHIVE], [RAM_DISK]. The default storage type of a directory will be [DISK] if it does not have a storage type tagged explicitly. These directories should be mounted using the noatime option, and the disks should be configured using JBOD. RAID is not recommended. Warning: Be very careful when modifying this property. Removing or changing entries can result in data loss. To hot swap drives in CDH 5.4 and higher, override the value of this property for the specific DataNode role instance that has the drive to be hot-swapped; do not modify the property value in the role group. See Configuring Hot Swap for DataNodes for more information.", "display_name": "DataNode Data Directory", "name": "dfs_data_dir_list", "value": null }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_data_node_pause_duration", "value": "false" }, { "desc": "The number of volumes that are allowed to fail before a DataNode stops offering service. By default, any volume failure will cause a DataNode to shutdown.", "display_name": "DataNode Failed Volumes Tolerated", "name": "dfs_datanode_failed_volumes_tolerated", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "If enabled, the DataNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind DataNode to Wildcard Address", "name": "dfs_datanode_bind_wildcard", "value": "false" }, { "desc": "The number of server threads for the DataNode.", "display_name": "Handler Count", "name": "dfs_datanode_handler_count", "value": "3" }, { "desc": "The maximum number of rolled log files to keep for DataNode logs. Typically used by log4j or logback.", "display_name": "DataNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "DataNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "DATANODE_role_env_safety_valve", "value": null }, { "desc": "Timeout in seconds for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Timeout", "name": "dfs_thrift_timeout", "value": "60" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_data_node_log_directory_free_space", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Block Count heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Block Count", "name": "role_health_suppression_data_node_block_count", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: DataNode Log Directory", "name": "role_config_suppression_datanode_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of DataNode in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of DataNode in Bytes", "name": "role_config_suppression_datanode_java_heapsize", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Reserved space in bytes per volume for non Distributed File System (DFS) use.", "display_name": "Reserved Space for Non DFS Use", "name": "dfs_datanode_du_reserved", "value": "10737418240" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations.", "display_name": "Available Space Policy Balanced Preference", "name": "dfs_datanode_available_space_balanced_preference", "value": "0.75" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_data_node_unexpected_exits", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "datanode_web_metric_collection_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/impala-service.json0000666000175100017510000011332513245514472025356 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Impala Llama ApplicationMaster Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Llama ApplicationMaster Health", "name": "service_health_suppression_impala_llamas_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Bypass Hive Metastore Validator configuration validator.", "display_name": "Suppress Configuration Validator: Bypass Hive Metastore Validator", "name": "service_config_suppression_impala_bypass_hms_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "The user that this Impala's processes should run as (except Llama, which has its own user).", "display_name": "Impala System User (except Llama)", "name": "process_username", "value": "impala" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. Typically edited using the Rules configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Configuration Rules (Staged)", "name": "impala_schedule_rules_draft", "value": null }, { "desc": "Name of the HDFS service that this Impala service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Maximum amount of time (in milliseconds) that a request waits to be admitted before timing out. Must be a positive integer.", "display_name": "Admission Control Queue Timeout", "name": "admission_control_queue_timeout", "value": "60000" }, { "desc": "Use Dynamic Resource Pools to configure Impala admission control or RM for this Impala service. These features are only supported in Impala 1.3 or higher deployments.", "display_name": "Enable Dynamic Resource Pools", "name": "admission_control_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Local Path Access User Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Local Path Access User Validator", "name": "service_config_suppression_local_path_access_user_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Short-Circuit Read Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Short-Circuit Read Validator", "name": "service_config_suppression_impala_dfs_client_read_shortcircuit_validator", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using Sentry service instead, add Sentry service as a dependency to Impala service. Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry is supported only on Impala 1.1 or later deployments.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_impalad_sentry_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The time period over which to compute the assignment locality ratio. Specified in minutes.", "display_name": "Assignment Locality Monitoring Period", "name": "impala_assignment_locality_window", "value": "15" }, { "desc": "Timeout in milliseconds for all HBase RPCs made by Impala. Overrides configuration in HBase service.", "display_name": "HBase RPC Timeout", "name": "hbase_rpc_timeout", "value": "3000" }, { "desc": "Use debug build of Impala binaries when starting roles. Useful when performing diagnostic activities to get more information in the stacktrace or core dump.", "display_name": "Use Debug Build", "name": "use_debug_build", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Pattern parameter.", "display_name": "Suppress Parameter Validation: LDAP Pattern", "name": "service_config_suppression_ldap_bind_pattern", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Impala is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_private_key", "value": null }, { "desc": "The minimum number of assignments that must occur during the test time period before the threshold values will be checked. Until this number of assignments have been observed in the test time period the health test will be disabled.", "display_name": "Assignment Locality Minimum Assignments", "name": "impala_assignment_locality_minimum", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP Configuration Validator", "name": "service_config_suppression_impala_ldap_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala System Group (except Llama) parameter.", "display_name": "Suppress Parameter Validation: Impala System Group (except Llama)", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress the results of the Assignment Locality heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Assignment Locality", "name": "service_health_suppression_impala_assignment_locality", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Daemon Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Daemon Count Validator", "name": "service_config_suppression_impalad_count_validator", "value": "false" }, { "desc": "Specifies the set of authorized proxy users (users who can impersonate other users during authorization) and whom they are allowed to impersonate. Input is a semicolon-separated list of key=value pairs of authorized proxy users to the user(s) they can impersonate. These users are specified as a comma separated list of short usernames, or '*' to indicate all users. For example: joe=alice,bob;hue=*;admin=*. Only valid when Sentry is enabled.", "display_name": "Proxy User Configuration", "name": "impala_authorized_proxy_user_config", "value": "hue=*" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Catalog Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Catalog Server Count Validator", "name": "service_config_suppression_catalogserver_count_validator", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Configures the maximum number of concurrently running queries for admission control when using a single pool. -1 indicates no limit and 0 indicates all incoming requests will be rejected. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Max Running Queries", "name": "admission_control_single_pool_max_requests", "value": "200" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "ldap_domain", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_impala_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Llama Supported Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Llama Supported Validator", "name": "service_config_suppression_impala_llama_supported_validator", "value": "false" }, { "desc": "Name of YARN service to use for resource management integration between Impala and YARN. This service dependency and the existence of a Llama role is required for using said integration.", "display_name": "YARN Service for Resource Management", "name": "yarn_service", "value": null }, { "desc": "Name of the Sentry service that this Impala service instance depends on. If selected, Impala uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Time in seconds before Impala Daemon or Catalog Server times out with the StateStore.", "display_name": "StateStoreSubscriber Timeout", "name": "statestore_subscriber_timeout", "value": "30" }, { "desc": "Configures the max memory of running queries for admission control when using a single pool. -1 or 0 indicates no limit. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Mem Limit", "name": "admission_control_single_pool_mem_limit", "value": "-1" }, { "desc": "Maximum number of HBase client retries for Impala. Used as a maximum for all operations such as fetching of the root region from the root RegionServer, getting a cell's value, and starting a row update. Overrides configuration in the HBase service.", "display_name": "Maximum HBase Client Retries", "name": "hbase_client_retries_number", "value": "3" }, { "desc": "The health test thresholds of the overall Impala Llama ApplicationMaster health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Impala Llama ApplicationMasters falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Impala Llama ApplicationMasters falls below the critical threshold.", "display_name": "Healthy Impala Llama ApplicationMaster Monitoring Thresholds", "name": "impala_llamas_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala System User (except Llama) parameter.", "display_name": "Suppress Parameter Validation: Impala System User (except Llama)", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Proxy User Configuration parameter.", "display_name": "Suppress Parameter Validation: Proxy User Configuration", "name": "service_config_suppression_impala_authorized_proxy_user_config", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Daemon Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Health", "name": "service_health_suppression_impala_impalads_healthy", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Impala is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_server_certificate", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL CA Certificate parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL CA Certificate", "name": "service_config_suppression_ssl_client_ca_certificate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Permissions Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Permissions Validator", "name": "service_config_suppression_short_circuit_reads_data_directory_permissions_validator", "value": "false" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. The value specified here is deployed to all hosts. This configuration has effect only on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Configuration Rules (Deployed)", "name": "impala_schedule_rules", "value": "[]" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "enable_ldap_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala StateStore Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala StateStore Count Validator", "name": "service_config_suppression_statestore_count_validator", "value": "false" }, { "desc": "Use Impala Admission Control to throttle Impala requests. Unless 'Enable Dynamic Resource Pools' is enabled, Impala uses a single, default pool that is configured using the Single Pool configurations below. These features are only supported in Impala 1.3 or higher deployments.", "display_name": "Enable Impala Admission Control", "name": "all_admission_control_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_ldap_domain", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_ldap_basedn", "value": "false" }, { "desc": "Name of the Kudu service that this Impala service instance depends on", "display_name": "Kudu Service", "name": "kudu_service", "value": null }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take across all schedules. Typically edited using the Pools configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Allocations (Staged)", "name": "impala_scheduled_allocations_draft", "value": null }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "impalad_sentry_safety_valve", "value": null }, { "desc": "Configures the maximum number of queued queries for admission control when using a single pool. -1 or 0 disables queuing, i.e. incoming requests are rejected if they can not be executed immediately. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Max Queued Queries", "name": "admission_control_single_pool_max_queued", "value": "200" }, { "desc": "The health test thresholds of the overall Impala Daemon health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Impala Daemons falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Impala Daemons falls below the critical threshold.", "display_name": "Healthy Impala Daemon Monitoring Thresholds", "name": "impala_impalads_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Enable HDFS Block Metadata API Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Enable HDFS Block Metadata API Configuration Validator", "name": "service_config_suppression_impala_hdfs_dfs_datanode_hdfs_blocks_metadata_enabled_set_validator", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Impala Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "impala_service_env_safety_valve", "value": null }, { "desc": "If checked and Impala is using Kerberos, Impala will use the hadoop.security.auth_to_local rules from HDFS configuration. The relevant HDFS configuration is derived from Additional Rules to Map Kerberos Principals to Short Names.", "display_name": "Use HDFS Rules to Map Kerberos Principals to Short Names", "name": "load_hdfs_auth_to_local_rules", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_impala_cmd_args_safety_valve", "value": "false" }, { "desc": "Number of minutes between reestablishing our ticket with the Kerberos server.", "display_name": "Kerberos Re-init Interval", "name": "kerberos_reinit_interval", "value": "60" }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "enable_ldap_tls", "value": "false" }, { "desc": "When set, this parameter allows arbitrary mapping from usernames into a Distinguished Name (DN). The string specified must have a placeholder named \"#UID\" inside it, and that #UID is replaced with the username. For example, you could mimic the behavior of LDAP BaseDN by specifying \"uid=#UID,ou=People,dc=cloudera,dc=com\". When the username of \"mike\" comes in, it replaces the #UID and the result is \"uid=mike,ou=People,dc=cloudera,dc=com\". This option should be used when more control over the DN is needed. This parameter is mutually exclusive with LDAP Domain and LDAP BaseDN.", "display_name": "LDAP Pattern", "name": "ldap_bind_pattern", "value": null }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "impala_ldap_uri", "value": null }, { "desc": "Controls which queries a non-admin user can see in the queries list view", "display_name": "Non-Admin Users Query List Visibility Settings", "name": "user_query_list_settings", "value": "ALL" }, { "desc": "The location on disk of the certificate, in PEM format, used to confirm the authenticity of SSL/TLS servers that the Impala daemons might connect to. Because the Impala daemons connect to each other, this should also include the CA certificate used to sign all the SSL/TLS Certificates. Without this parameter, SSL/TLS between Impala daemons will not be enabled.", "display_name": "Impala TLS/SSL CA Certificate", "name": "ssl_client_ca_certificate", "value": null }, { "desc": "Used to generate a core dump to get more information about an Impala crash. Unless otherwise configured systemwide using /proc/sys/kernel/core_pattern, the dump is generated in the 'current directory' of the Impala process (usually a subdirectory of the /var/run/cloudera-scm-agent/process directory). The core file can be very large.", "display_name": "Enable Core Dump", "name": "enable_core_dump", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "true" }, { "desc": "The password for the private key in the Impala TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Impala TLS/SSL Private Key Password", "name": "ssl_private_key_password", "value": null }, { "desc": "Whether to suppress the results of the Impala Catalog Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Catalog Server Health", "name": "service_health_suppression_impala_catalogserver_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala DSSD Short-Circuit Read with Kerberos Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala DSSD Short-Circuit Read with Kerberos Validator", "name": "service_config_suppression_impala_dssd_kerberos_scr_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Impala StateStore Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala StateStore Health", "name": "service_health_suppression_impala_statestore_health", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take across all schedules. This configuration has effect only on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Allocations (Deployed)", "name": "impala_scheduled_allocations", "value": "{\"defaultFairSharePreemptionThreshold\":null,\"defaultFairSharePreemptionTimeout\":null,\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":null,\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":null,\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null,\"type\":null}],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxChildResources\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null,\"type\":null}],\"userMaxAppsDefault\":null,\"users\":[]}" }, { "desc": "When computing the overall IMPALA health, consider Impala Catalog Server's health", "display_name": "Impala Catalog Server Role Health Test", "name": "impala_catalogserver_health_enabled", "value": "true" }, { "desc": "Name of the HBase service that this Impala service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Server Private Key File (PEM Format)", "name": "service_config_suppression_ssl_private_key", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CA Certificate File Validator configuration validator.", "display_name": "Suppress Configuration Validator: CA Certificate File Validator", "name": "service_config_suppression_pem_ca_cert_recommended_for_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Llama ApplicationMaster Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Llama ApplicationMaster Count Validator", "name": "service_config_suppression_llama_count_validator", "value": "false" }, { "desc": "This parameter is useful when authenticating against a non-Active Directory server, such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,<this parameter>. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "ldap_baseDN", "value": null }, { "desc": "Name of the ZooKeeper service to use for leader election and fencing when Llama is configured for high availability. This service dependency is required when more than one Llama role is present.", "display_name": "ZooKeeper Service for Llama HA", "name": "zookeeper_service", "value": null }, { "desc": "The health test thresholds for the assignment locality health test. Specified as a percentage of total assignments.", "display_name": "Assignment Locality Ratio Thresholds", "name": "impala_assignment_locality_thresholds", "value": "{\"critical\":\"5.0\",\"warning\":\"80.0\"}" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "impala" }, { "desc": "The group that this Impala's processes should run as (except Llama, which has its own group).", "display_name": "Impala System Group (except Llama)", "name": "process_groupname", "value": "impala" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Server Certificate File (PEM Format)", "name": "service_config_suppression_ssl_server_certificate", "value": "false" }, { "desc": "Controls which queries admin users can see in the queries list view", "display_name": "Admin Users Query List Visibility Settings", "name": "admin_query_list_settings", "value": "ALL" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Short-Circuit Read Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Short-Circuit Read Validator", "name": "service_config_suppression_impala_dssd_shortcircuit_read_enable_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Encrypt communication between clients and Impala using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Impala", "name": "client_services_ssl_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Private Key Password", "name": "service_config_suppression_ssl_private_key_password", "value": "false" }, { "desc": "When computing the overall IMPALA health, consider Impala StateStore's health", "display_name": "Impala StateStore Role Health Test", "name": "impala_statestore_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Impala Daemon command-line flags. Applies to all roles in this service. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "impala_cmd_args_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_impala_ldap_uri", "value": "false" }, { "desc": "Name of the Hive service that this Impala service instance depends on", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Sentry Validator", "name": "service_config_suppression_impala_sentry_validator", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hive-hivemetastore.json0000666000175100017510000006512013245514472026264 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Hive Metastore Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Hive Metastore Server", "name": "role_config_suppression_hive_metastore_java_opts", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "hivemetastore_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_hivemetastore_audit_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Metastore Max Message Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Metastore Max Message Size Validator", "name": "role_config_suppression_hive_metastore_max_message_size_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hivemetastore_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hivemetastore_host_health", "value": "false" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "hivemetastore_pause_duration_window", "value": "5" }, { "desc": "Maximum number of worker threads in the Hive Metastore Server's thread pool", "display_name": "Max Hive Metastore Server Threads", "name": "hive_metastore_max_threads", "value": "100000" }, { "desc": "Enables the health test that checks that basic Hive Metastore operations succeed", "display_name": "Hive Metastore Canary Health Test", "name": "metastore_canary_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hivemetastore_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Hive Metastore Server", "name": "hive_metastore_java_opts", "value": "-XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Time-to-live in seconds for notifications present in NOTIFICATION_LOG. Only used when Enable Stored Notifications in Database is enabled.", "display_name": "Time-to-live for Database Notifications", "name": "hive_metastore_event_db_listener_timetolive", "value": "172800" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_metastore_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hivemetastore_swap_memory_usage", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Hive Metastore Server logs. Typically used by log4j or logback.", "display_name": "Hive Metastore Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hivemetastore_scm_health", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enable stored notifications of metadata changes. When enabled, each metadata change will be stored in NOTIFICATION_LOG.", "display_name": "Enable Stored Notifications in Database", "name": "hive_enable_db_notification", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "When computing the overall Hive Metastore Server health, consider the host's health.", "display_name": "Hive Metastore Server Host Health Test", "name": "hivemetastore_host_health_enabled", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Controls whether the Hive metrics subsystem is enabled for the role.", "display_name": "Enable Metrics Subsystem", "name": "hive_metrics_enabled", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Hive Metastore Server in Bytes", "name": "hive_metastore_java_heapsize", "value": "8589934592" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hivemetastore_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hivemetastore_heap_dump_directory_free_space", "value": "false" }, { "desc": "Port on which Hive Metastore Server will listen for connections.", "display_name": "Hive Metastore Server Port", "name": "hive_metastore_port", "value": "9083" }, { "desc": "Whether to suppress configuration warnings produced by the Hive MetaStore Metadata Notification Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive MetaStore Metadata Notification Validator", "name": "role_config_suppression_hive_metastore_db_notification_validator", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_metastore_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Hive Metastore Server logs. Typically used by log4j or logback.", "display_name": "Hive Metastore Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_hivemetastore_pause_duration", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Hive Metastore Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Metastore Canary", "name": "role_health_suppression_hivemetastore_canary_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hive_metrics_sample_file_location", "value": "false" }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hive/metrics-hivemetastore/metrics.log. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Location", "name": "hive_metrics_sample_file_location", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hivemetastore_unexpected_exits", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Logging Frequency", "name": "hive_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_metastore_config_safety_valve", "value": "false" }, { "desc": "Maximum message size Hive MetaStore accepts.", "display_name": "Max Message Size for Hive MetaStore", "name": "hive_metastore_server_max_message_size", "value": "104857600" }, { "desc": "Enables the health test that the Hive Metastore Server's process state is consistent with the role configuration", "display_name": "Hive Metastore Server Process Health Test", "name": "hivemetastore_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Log Directory", "name": "role_config_suppression_hive_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Minimum number of worker threads in the Hive Metastore Server's thread pool", "display_name": "Min Hive Metastore Server Threads", "name": "hive_metastore_min_threads", "value": "200" }, { "desc": "Directory where Hive Metastore Server will place its log files.", "display_name": "Hive Metastore Server Log Directory", "name": "hive_log_dir", "value": "/var/log/hive" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The delegation token store implementation class. Use DBTokenStore for Highly Available Metastore Configuration.", "display_name": "Hive Metastore Delegation Token Store", "name": "hive_metastore_delegation_token_store", "value": "org.apache.hadoop.hive.thrift.MemoryTokenStore" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The minimum log level for Hive Metastore Server logs", "display_name": "Hive Metastore Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_metastore_env_safety_valve", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hbase-service.json0000666000175100017510000011455113245514472025177 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hbase" }, { "desc": "Enables the canary that checks HBase region availability by scanning a row from every region.", "display_name": "HBase Region Health Canary", "name": "hbase_region_health_canary_enabled", "value": "true" }, { "desc": "Name of the HDFS service that this HBase service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Period of time, in milliseconds, to pause between connection retries to ZooKeeper. Used together with ${zookeeper.retries} in an exponential backoff fashion when making queries to ZooKeeper.", "display_name": "ZooKeeper Connection Retry Pause Duration", "name": "zookeeper_pause", "value": null }, { "desc": "Allow HBase tables to be replicated.", "display_name": "Enable Replication", "name": "hbase_enable_replication", "value": "false" }, { "desc": "When computing the overall HBase cluster health, consider the active HBase Master's health.", "display_name": "Active Master Health Test", "name": "hbase_master_health_enabled", "value": "true" }, { "desc": "Size of the threadpool used for hedged reads in hdfs clients. If a read from a block is slow, a parallel 'hedged' read will be started against a different block replica. The first one to return with a result is used while the other one is cancelled. This 'hedged' read feature helps rein in the outliers. A value of zero disables the feature.", "display_name": "HDFS Hedged Read Threadpool Size", "name": "hbase_server_dfs_client_hedged_read_threadpool_size", "value": "0" }, { "desc": "Use this to enable Http server usage on thrift, which is mainly needed for \"doAs\" functionality.", "display_name": "Enable HBase Thrift Http Server", "name": "hbase_thriftserver_http", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase User to Impersonate parameter.", "display_name": "Suppress Parameter Validation: HBase User to Impersonate", "name": "service_config_suppression_hbase_user_to_impersonate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the AWS S3 Secret Access Key for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: AWS S3 Secret Access Key for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_secret_access_key", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Timeout for all HBase RPCs in milliseconds.", "display_name": "RPC Timeout", "name": "hbase_rpc_timeout", "value": "60000" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "hbase_ssl_server_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hbase_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "service_config_suppression_hbase_service_config_safety_valve", "value": "false" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "The number of times to retry connections to ZooKeeper. Used for reading and writing root region location. Used together with ${zookeeper.pause} in an exponential backoff fashion when making queries to ZooKeeper.", "display_name": "ZooKeeper Connection Retries", "name": "zookeeper_retries", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Timeout for graceful shutdown of this HBase service. Once this timeout is reached, any remaining running roles are abruptly shutdown. A value of 0 means no timeout.", "display_name": "Graceful Shutdown Timeout", "name": "hbase_graceful_stop_timeout", "value": "180" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_hbase_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The health test thresholds of the overall RegionServer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" RegionServers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" RegionServers falls below the critical threshold.", "display_name": "Healthy RegionServer Monitoring Thresholds", "name": "hbase_regionservers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress the results of the RegionServer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: RegionServer Health", "name": "service_health_suppression_hbase_region_servers_healthy", "value": "false" }, { "desc": "An alert is published if the HBase region health canary detects at least this many unhealthy regions. This setting takes precedence over the hbase_canary_alert_unhealthy_region_percent_threshold config.", "display_name": "HBase Canary Unhealthy Region Count Alert Threshold", "name": "hbase_canary_alert_unhealthy_region_count_threshold", "value": null }, { "desc": "Set to true to cause the hosting server (Master or RegionServer) to abort if a coprocessor throws a Throwable object that is not IOException or a subclass of IOException. Setting it to true might be useful in development environments where one wants to terminate the server as soon as possible to simplify coprocessor failure analysis.", "display_name": "HBase Coprocessor Abort on Error", "name": "hbase_coprocessor_abort_on_error", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_hbase_core_site_safety_valve", "value": "false" }, { "desc": "The user the management services impersonate when connecting to HBase. If no value is specified, the HBase superuser is used.", "display_name": "HBase User to Impersonate", "name": "hbase_user_to_impersonate", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Amazon S3 Path for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: Amazon S3 Path for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_path", "value": "false" }, { "desc": "Timeout (in ms) for the distributed log splitting manager to receive response from a worker.", "display_name": "SplitLog Manager Timeout", "name": "hbase_service_splitlog_manager_timeout", "value": "120000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Scheduler Pool for Remote Snapshots in AWS S3 parameter.", "display_name": "Suppress Parameter Validation: Scheduler Pool for Remote Snapshots in AWS S3", "name": "service_config_suppression_hbase_snapshot_s3_scheduler_pool", "value": "false" }, { "desc": "Allow indexing of tables in HBase by Lily HBase Indexer. Note: Replication must be enabled for indexing to work.", "display_name": "Enable Indexing", "name": "hbase_enable_indexing", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Superusers parameter.", "display_name": "Suppress Parameter Validation: HBase Superusers", "name": "service_config_suppression_hbase_superuser", "value": "false" }, { "desc": "The password for the HBase JKS keystore file.", "display_name": "HBase TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Ratio of Lily HBase Indexers used by each HBase RegionServer while doing replication.", "display_name": "Replication Source Ratio", "name": "hbase_replication_source_ratio", "value": "1.0" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "hbase_core_site_safety_valve", "value": null }, { "desc": "The timeout before injecting a snapshot timeout error when waiting for a snapshot completion.", "display_name": "HBase Master Snapshot Waiting Timeout", "name": "hbase_snapshot_master_timeoutMillis", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Authentication And Authorization Validation configuration validator.", "display_name": "Suppress Configuration Validator: HBase Authentication And Authorization Validation", "name": "service_config_suppression_hbase_authentication_and_authorization_validator", "value": "false" }, { "desc": "The maximum amount of time the Hbase RegionServer waits for a snapshot to complete.", "display_name": "HBase RegionServer Snapshot Timeout", "name": "hbase_snapshot_region_timeout", "value": "60000" }, { "desc": "If this is set to \"kerberos\", HBase REST Server will authenticate its clients. HBase Proxy User Hosts and Groups should be configured to allow specific users to access HBase through REST Server.", "display_name": "HBase REST Authentication", "name": "hbase_restserver_security_authentication", "value": "simple" }, { "desc": "Specifies the combined maximum allowed size of a KeyValue instance. This option configures an upper boundary for a single entry saved in a storage file. This option prevents a region from splitting if the data is too large. Set this option to a fraction of the maximum region size. To disable this check, use a value of zero or less.", "display_name": "Maximum Size of HBase Client KeyValue", "name": "hbase_client_keyvalue_maxsize", "value": "10485760" }, { "desc": "Enable snapshots. Disabling snapshots requires deletion of all snapshots before restarting the HBase master; the HBase master will not start if snapshots are disabled and snapshots exist.", "display_name": "Enable Snapshots", "name": "hbase_snapshot_enabled", "value": "true" }, { "desc": "Maximum number of client retries. Used as a maximum for all operations such as fetching of the root region from the root RegionServer, getting a cell's value, and starting a row update.", "display_name": "Maximum HBase Client Retries", "name": "hbase_client_retries_number", "value": "35" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "zookeeper.recovery.retry0zookeeper.recovery.retry.intervalmill3000hbase.zookeeper.recoverable.waittime1000zookeeper.session.timeout30000hbase.rpc.timeout10000hbase.client.retries.number1hbase.client.rpc.maxattempts1hbase.client.operation.timeout10000" }, { "desc": "Access key ID required to access Amazon S3 to store remote snapshots.", "display_name": "Amazon S3 Access Key ID for Remote Snapshots", "name": "hbase_snapshot_s3_access_key_id", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Amazon S3 Access Key ID for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: Amazon S3 Access Key ID for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_access_key_id", "value": "false" }, { "desc": "Whether to suppress the results of the HBase Master Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HBase Master Health", "name": "service_health_suppression_hbase_master_health", "value": "false" }, { "desc": "Enable HBase authorization.", "display_name": "HBase Secure Authorization", "name": "hbase_security_authorization", "value": "false" }, { "desc": "Period of time, in milliseconds, to pause between searches for work. Used as a sleep interval by service threads such as a META scanner and log roller.", "display_name": "HBase Server Thread Wake Frequency", "name": "hbase_server_thread_wakefrequency", "value": "10000" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Thrift Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Thrift Server Count Validator", "name": "service_config_suppression_hbasethriftserver_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Root Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Root Directory", "name": "service_config_suppression_hdfs_rootdir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the RegionServer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: RegionServer Count Validator", "name": "service_config_suppression_regionserver_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the ZooKeeper Max Session Timeout Validator configuration validator.", "display_name": "Suppress Configuration Validator: ZooKeeper Max Session Timeout Validator", "name": "service_config_suppression_zookeeper_max_session_timeout_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode Rootserver parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode Rootserver", "name": "service_config_suppression_zookeeper_znode_rootserver", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode Parent parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode Parent", "name": "service_config_suppression_zookeeper_znode_parent", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Region Health Canary Exclude Tables parameter.", "display_name": "Suppress Parameter Validation: HBase Region Health Canary Exclude Tables", "name": "service_config_suppression_hbase_region_health_canary_exclude_tables", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HBase Proxy User Groups", "name": "service_config_suppression_hbase_proxy_user_groups_list", "value": "false" }, { "desc": "The root znode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file paths are configured with a relative path, so they will all go under this directory unless changed.", "display_name": "ZooKeeper Znode Parent", "name": "zookeeper_znode_parent", "value": "/hbase" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "AWS secret access key required to access S3 to store remote snapshots.", "display_name": "AWS S3 Secret Access Key for Remote Snapshots", "name": "hbase_snapshot_s3_secret_access_key", "value": null }, { "desc": "Number of rows to fetch when calling next on a scanner if it is not served from memory. Higher caching values enable faster scanners but require more memory and some calls of next may take longer when the cache is empty.", "display_name": "HBase Client Scanner Caching", "name": "hbase_client_scanner_caching", "value": "100" }, { "desc": "Tables to exclude in the HBase Region Health Canary which will scan a row from every region.", "display_name": "HBase Region Health Canary Exclude Tables", "name": "hbase_region_health_canary_exclude_tables", "value": "" }, { "desc": "The maximum amount of time the HBase master waits for a snapshot to complete.", "display_name": "HBase Master Snapshot Timeout", "name": "hbase_snapshot_master_timeout_millis", "value": "60000" }, { "desc": "Amazon S3 path where remote snapshots should be stored.", "display_name": "Amazon S3 Path for Remote Snapshots", "name": "hbase_snapshot_s3_path", "value": null }, { "desc": "Start a process to periodically check that RegionServer is alive when RegionServer is started. Note: This canary is different from the Cloudera Service Monitoring canary and is provided by the HBase service itself.", "display_name": "Enable HBase Canary", "name": "hbase_regionserver_enable_canary", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "true" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase is acting as a TLS/SSL server.", "display_name": "HBase TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Path to ZooKeeper Node holding root region location. This is written by the HBase Master and read by clients and RegionServers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, the root location is stored at /hbase/root-region-server.", "display_name": "ZooKeeper Znode Rootserver", "name": "zookeeper_znode_rootserver", "value": "root-region-server" }, { "desc": "When computing the overall HBase cluster health, consider the health of the backup HBase Masters.", "display_name": "Backup Masters Health Test", "name": "hbase_backup_masters_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "An alert is published if the HBase region health canary runs slowly.", "display_name": "HBase Region Health Canary Slow Run Alert Enabled", "name": "hbase_region_health_canary_slow_run_alert_enabled", "value": "true" }, { "desc": "Name of the scheduler pool to use for MR jobs created during export/import of remote snapshots in AWS S3.", "display_name": "Scheduler Pool for Remote Snapshots in AWS S3", "name": "hbase_snapshot_s3_scheduler_pool", "value": null }, { "desc": "Configure the type of encrypted communication to be used with RPC.", "display_name": "HBase Transport Security", "name": "hbase_rpc_protection", "value": "authentication" }, { "desc": "An alert is published if the HBase region health canary detects at least this percentage of total regions are unhealthy. This threshold is used if the explicit count is not set via the hbase_canary_alert_unhealthy_region_count_threshold config.", "display_name": "HBase Canary Unhealthy Region Percentage Alert Threshold", "name": "hbase_canary_alert_unhealthy_region_percent_threshold", "value": "0.1" }, { "desc": "Duration to wait before starting up a 'hedged' read.", "display_name": "HDFS Hedged Read Delay Threshold", "name": "hbase_server_dfs_client_hedged_read_threshold_millis", "value": "500" }, { "desc": "Enable TLS/SSL encryption for HBase web UIs.", "display_name": "Web UI TLS/SSL Encryption Enabled", "name": "hbase_hadoop_ssl_enabled", "value": "false" }, { "desc": "List of users or groups, who are allowed full privileges, regardless of stored ACLs, across the cluster. Only used when HBase security is enabled.", "display_name": "HBase Superusers", "name": "hbase_superuser", "value": "" }, { "desc": "Use this to allow proxy users on thrift gateway, which is mainly needed for \"doAs\" functionality.", "display_name": "Enable HBase Thrift Proxy Users", "name": "hbase_thriftserver_support_proxyuser", "value": "false" }, { "desc": "Maximum number of hlog entries to replicate in one go. If this is large, and a consumer takes a while to process the events, the HBase RPC call will time out.", "display_name": "Replication Batch Size", "name": "hbase_replication_source_nb_capacity", "value": "1000" }, { "desc": "Whether to suppress configuration warnings produced by the Master Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Master Count Validator", "name": "service_config_suppression_master_count_validator", "value": "false" }, { "desc": "ZooKeeper session timeout in milliseconds. HBase passes this to the ZooKeeper quorum as the suggested maximum time for a session. See http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions The client sends a requested timeout, the server responds with the timeout that it can give the client.", "display_name": "ZooKeeper Session Timeout", "name": "zookeeper_session_timeout", "value": "60000" }, { "desc": "Enable HBase row-level authorization.", "display_name": "HBase Row-Level Authorization", "name": "hbase_row_level_authorization", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HBase user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'. Note: This property is used only if HBase REST/Thrift Server Authentication is enabled.", "display_name": "HBase Proxy User Hosts", "name": "hbase_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HBase Proxy User Hosts", "name": "service_config_suppression_hbase_proxy_user_hosts_list", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the HBase user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'. Note: This property is used only if HBase REST/Thrift Server Authentication is enabled.", "display_name": "HBase Proxy User Groups", "name": "hbase_proxy_user_groups_list", "value": "*" }, { "desc": "The tolerance window that will be used in HBase service tests that depend on detection of the active HBase Master.", "display_name": "HBase Active Master Detection Window", "name": "hbase_active_master_detecton_window", "value": "3" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_service_config_safety_valve", "value": null }, { "desc": "Choose the authentication mechanism used by HBase.", "display_name": "HBase Secure Authentication", "name": "hbase_security_authentication", "value": "simple" }, { "desc": "A general client pause time value. Used mostly as a time period to wait before retrying operations such as a failed get or region lookup.", "display_name": "HBase Client Pause", "name": "hbase_client_pause", "value": "100" }, { "desc": "Name of the ZooKeeper service that this HBase service instance depends on.", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Write buffer size in bytes. A larger buffer requires more memory on both the client and the server because the server instantiates the passed write buffer to process it but reduces the number of remote procedure calls (RPC). To estimate the amount of server memory used, multiply the value of 'hbase.client.write.buffer' by the value of 'hbase.regionserver.handler.count'.", "display_name": "HBase Client Write Buffer", "name": "hbase_client_write_buffer", "value": "2097152" }, { "desc": "If this is set, HBase Thrift Server authenticates its clients. HBase Proxy User Hosts and Groups should be configured to allow specific users to access HBase through Thrift Server.", "display_name": "HBase Thrift Authentication", "name": "hbase_thriftserver_security_authentication", "value": "none" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hbase" }, { "desc": "Whether to suppress configuration warnings produced by the HBase REST Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase REST Server Count Validator", "name": "service_config_suppression_hbaserestserver_count_validator", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hbase" }, { "desc": "The HDFS directory shared by HBase RegionServers.", "display_name": "HDFS Root Directory", "name": "hdfs_rootdir", "value": "/hbase" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hbase_service_env_safety_valve", "value": null }, { "desc": "Whether the step to reload regions back onto the original RegionServers should be skipped during rolling restart. This can be used to increase the speed of rolling restart or upgrade operations, but can result in regions being moved multiple times, decreasing performance for clients during rolling restart.", "display_name": "Skip Region Reload During Rolling Restart", "name": "hbase_skip_reload_during_rr", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HBase DSSD Short-Circuit Read with Kerberos Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase DSSD Short-Circuit Read with Kerberos Validator", "name": "service_config_suppression_hbase_dssd_kerberos_scr_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-gateway.json0000666000175100017510000001403613245514472025037 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_hdfs_client_config_safety_valve", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hadoop" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Trash Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Trash Enabled Validator", "name": "role_config_suppression_hdfs_trash_disabled_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether HDFS clients will use the legacy block reader.", "display_name": "Use Legacy Blockreader", "name": "dfs_client_use_legacy_blockreader", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh", "display_name": "HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "hdfs_client_env_safety_valve", "value": null }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for hdfs-site.xml.", "display_name": "HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "hdfs_client_config_safety_valve", "value": null }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "false" }, { "desc": "These are Java command-line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hbase_client_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Move deleted files to the trash so that they can be recovered if necessary. This client side configuration takes effect only if the HDFS service-wide trash is disabled (NameNode Filesystem Trash Interval set to 0) and is ignored otherwise. The trash is not automatically emptied when enabled with this configuration.", "display_name": "Use Trash", "name": "dfs_client_use_trash", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hdfs_client_java_heapsize", "value": "268435456" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "role_config_suppression_hdfs_client_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hbase_client_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/cdh_config.sh0000777000175100017510000000021413245514472024174 0ustar zuulzuul00000000000000#!/bin/bash # prereqs - virtualenv virtualenv venv . venv/bin/activate pip install cm-api python cdh_config.py deactivate rm -rf venv sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/zookeeper-server.json0000666000175100017510000011761213245514472025767 0ustar zuulzuul00000000000000[ { "desc": "The port to monitor for inter-server communication", "display_name": "Quorum Port", "name": "quorumPort", "value": "3181" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Log Directory parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Log Directory", "name": "role_config_suppression_zk_server_log_dir", "value": "false" }, { "desc": "Specifies the name of the user that has read-only privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Name of User with Read-Only access to the JMX Agent", "name": "jmx_passwd_file_readonly_user", "value": "monitorRole" }, { "desc": "Enables the health test that the Server's process state is consistent with the role configuration", "display_name": "Server Process Health Test", "name": "zookeeper_server_scm_health_enabled", "value": "true" }, { "desc": "The address (IPv4, IPv6, or hostname) to monitor for client connections. This is the address that clients attempt to connect to. This setting is optional, because by default, ZooKeeper binds in such a way that any connection to the client port for any address/interface/NIC on the server will be accepted.", "display_name": "Client Port Address", "name": "clientPortAddress", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Specifies the password of the user that has read-only privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Password of User with Read-Only Access to the JMX agent", "name": "jmx_passwd_file_readonly_user_password", "value": "MONITOR" }, { "desc": "Whether to suppress the results of the Quorum Membership heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Quorum Membership", "name": "role_health_suppression_zookeeper_server_quorum_membership", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Name of User with Read-Write Access to the JMX Agent parameter.", "display_name": "Suppress Parameter Validation: Name of User with Read-Write Access to the JMX Agent", "name": "role_config_suppression_jmx_passwd_file_readwrite_user", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_zookeeper_server_audit_health", "value": "false" }, { "desc": "The health check thresholds of the weighted average size of the ZooKeeper Server connection count over a recent period. See ZooKeeper Server Connection Count Monitoring Period.", "display_name": "ZooKeeper Server Connection Count Thresholds", "name": "zookeeper_server_connection_count_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into zoo.cfg for this role only.", "display_name": "Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg", "name": "zookeeper_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Transaction Log Directory.", "display_name": "Transaction Log Directory Free Space Monitoring Absolute Thresholds", "name": "zookeeper_server_data_log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The port to monitor for client connections. This is the port that clients attempt to connect to.", "display_name": "Client Port", "name": "clientPort", "value": "2181" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg parameter.", "display_name": "Suppress Parameter Validation: Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg", "name": "role_config_suppression_zookeeper_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Password of User with Read-Only Access to the JMX agent parameter.", "display_name": "Suppress Parameter Validation: Password of User with Read-Only Access to the JMX agent", "name": "role_config_suppression_jmx_passwd_file_readonly_user_password", "value": "false" }, { "desc": "The port used by the ZooKeeper Server's RMI registry. This is required to enable JMX access through RMI which is required for Cloudera Manager ZooKeeper monitoring. This is added as \"-Dcom.sun.management.jmxremote.port\" to the ZooKeeper Server's JVM command line.", "display_name": "JMX Remote Port", "name": "server_jmx_agent_port", "value": "9010" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_zookeeper_server_gc_duration", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_zookeeper_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Unique identifier for each ZooKeeper server, typically starts at 1", "display_name": "ZooKeeper Server ID", "name": "serverId", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "zookeeper_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_zookeeper_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_zookeeper_server_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The disk location that ZooKeeper will use to store its transaction logs.", "display_name": "Transaction Log Directory", "name": "dataLogDir", "value": "/var/lib/zookeeper" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Data Directory.", "display_name": "Data Directory Free Space Monitoring Absolute Thresholds", "name": "zookeeper_server_data_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The maximum size, in megabytes, per log file for Server logs. Typically used by log4j or logback.", "display_name": "Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Password of user with read-write access to the JMX agent parameter.", "display_name": "Suppress Parameter Validation: Password of user with read-write access to the JMX agent", "name": "role_config_suppression_jmx_passwd_file_readwrite_user_password", "value": "false" }, { "desc": "The minimum session timeout, in milliseconds, that the ZooKeeper Server will allow the client to negotiate", "display_name": "Minimum Session Timeout", "name": "minSessionTimeout", "value": "4000" }, { "desc": "The percentage thresholds of the ratio of the maximum request latency to the maximum client-negotiable session timeout since the server was started.", "display_name": "Maximum Latency Monitoring Thresholds", "name": "zookeeper_server_max_latency_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"75.0\"}" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Enables authentication when interacting with the JMX agent on the ZooKeeper server.", "display_name": "Enable Authenticated Communication with the JMX Agent", "name": "enable_jmx_authentication", "value": "false" }, { "desc": "Enables the quorum membership check for this ZooKeeper Server.", "display_name": "Enable the Quorum Membership Check", "name": "zookeeper_server_quorum_membership_enabled", "value": "true" }, { "desc": "The port used by the ZooKeeper Server's RMI server to handle JMX RMI requests. This is added as \"-Dcom.sun.management.jmxremote.rmi.port=<port>\" to the ZooKeeper Server's JVM command line. This has an effect only in Oracle JDK 7u4 and higher. If the setting is left blank, the JMX Remote Port value is used. If set to 0 or -1, this setting is ignored. When this setting is not in effect, the JVM uses a random port for the RMI server.", "display_name": "JMX RMI Server Port", "name": "server_jmx_rmi_port", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_zookeeper_server_host_health", "value": "false" }, { "desc": "Directory where ZooKeeper will place its log files.", "display_name": "ZooKeeper Log Directory", "name": "zk_server_log_dir", "value": "/var/log/zookeeper" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Transaction Log Directory parameter.", "display_name": "Suppress Parameter Validation: Transaction Log Directory", "name": "role_config_suppression_datalogdir", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The minimum log level for Server logs", "display_name": "Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SERVER_role_env_safety_valve", "value": null }, { "desc": "The period to review when computing the moving average of the outstanding requests queue size. Specified in minutes.", "display_name": "ZooKeeper Server Outstanding Requests Monitoring Period", "name": "zookeeper_server_outstanding_requests_window", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Port Address parameter.", "display_name": "Suppress Parameter Validation: Client Port Address", "name": "role_config_suppression_clientportaddress", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Zookeeper Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Zookeeper Server", "name": "role_config_suppression_zk_server_java_opts", "value": "false" }, { "desc": "The disk location that ZooKeeper will use to store its database snapshots.", "display_name": "Data Directory", "name": "dataDir", "value": "/var/lib/zookeeper" }, { "desc": "The tolerance window that will be used in the detection of a ZooKeeper server's membership in a quorum. Specified in minutes.", "display_name": "Quorum Membership Detection Window", "name": "zookeeper_server_quorum_membership_detection_window", "value": "3" }, { "desc": "The health check thresholds of the weighted average size of the ZooKeeper Server outstanding requests queue over a recent period. See ZooKeeper Server Outstanding Requests Monitoring Period.", "display_name": "ZooKeeper Server Outstanding Requests Thresholds", "name": "zookeeper_server_outstanding_requests_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_server_role_env_safety_valve", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Specifies the name of the user that has read-write privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Name of User with Read-Write Access to the JMX Agent", "name": "jmx_passwd_file_readwrite_user", "value": "controlRole" }, { "desc": "The port to monitor for leadership election", "display_name": "Election Port", "name": "electionPort", "value": "4181" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The maximum number of concurrent connections (at the socket level) that a single client, identified by the IP address, may make to a single member of the ZooKeeper ensemble. This setting is used to prevent certain classes of DoS attacks, including file descriptor exhaustion. To remove the limit on concurrent connections, set this value to 0.", "display_name": "Maximum Client Connections", "name": "maxClientCnxns", "value": "60" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Zookeeper Server", "name": "zk_server_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Maximum Request Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Maximum Request Latency", "name": "role_health_suppression_zookeeper_server_max_latency", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_zookeeper_server_unexpected_exits", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "zookeeper_server_gc_duration_window", "value": "5" }, { "desc": "Specifies the password of the user that has read-write privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Password of user with read-write access to the JMX agent", "name": "jmx_passwd_file_readwrite_user_password", "value": "CONTROL" }, { "desc": "Enables the JMX agent on the ZooKeeper server. Turning this off on any of the ZooKeeper servers that are part of a service will prevent Cloudera Manager from being able to monitor that server and may affect the monitoring provided on the entire service.", "display_name": "Enable JMX Agent", "name": "enable_jmx_agent", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_zookeeper_server_scm_health", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Transaction Log Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Transaction Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Transaction Log Directory Free Space Monitoring Percentage Thresholds", "name": "zookeeper_server_data_log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Data Directory parameter.", "display_name": "Suppress Parameter Validation: Data Directory", "name": "role_config_suppression_datadir", "value": "false" }, { "desc": "Whether to suppress the results of the Outstanding Requests heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Outstanding Requests", "name": "role_health_suppression_zookeeper_server_outstanding_requests", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Name of User with Read-Only access to the JMX Agent parameter.", "display_name": "Suppress Parameter Validation: Name of User with Read-Only access to the JMX Agent", "name": "role_config_suppression_jmx_passwd_file_readonly_user", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "zookeeper_server_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of ZooKeeper Server in Bytes", "name": "zookeeper_server_java_heapsize", "value": "1073741824" }, { "desc": "The maximum number of rolled log files to keep for Server logs. Typically used by log4j or logback.", "display_name": "Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_zookeeper_server_log_directory_free_space", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Data Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Data Directory Free Space Monitoring Percentage Thresholds", "name": "zookeeper_server_data_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "The period to review when computing the moving average of the connection count. Specified in minutes.", "display_name": "ZooKeeper Server Connection Count Monitoring Period", "name": "zookeeper_server_connection_count_window", "value": "3" }, { "desc": "When computing the overall Server health, consider the host's health.", "display_name": "Server Host Health Test", "name": "zookeeper_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The maximum session timeout, in milliseconds, that the ZooKeeper Server will allow the client to negotiate", "display_name": "Maximum Session Timeout", "name": "maxSessionTimeout", "value": "40000" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Transaction Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Transaction Log Directory Free Space", "name": "role_health_suppression_zookeeper_server_data_log_directory_free_space", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Data Directory Free Space", "name": "role_health_suppression_zookeeper_server_data_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Connection Count heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Connection Count", "name": "role_health_suppression_zookeeper_server_connection_count", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/spark-gateway.json0000666000175100017510000003063713245514472025240 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Data Serializer parameter.", "display_name": "Suppress Parameter Validation: Spark Data Serializer", "name": "role_config_suppression_spark_data_serializer", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Extra Listeners parameter.", "display_name": "Suppress Parameter Validation: Spark Extra Listeners", "name": "role_config_suppression_spark_extralisteners", "value": "false" }, { "desc": "When dynamic allocation is enabled, time after which idle executors will be stopped.", "display_name": "Executor Idle Timeout", "name": "spark_dynamic_allocation_idle_timeout", "value": "60" }, { "desc": "Whether to encrypt temporary shuffle and cache files stored by Spark on the local disks.", "display_name": "Enable I/O Encryption", "name": "spark_io_encryption_enabled", "value": "false" }, { "desc": "Enables the external shuffle service. The external shuffle service preserves shuffle files written by executors so that the executors can be deallocated without losing work. Must be enabled if Enable Dynamic Allocation is enabled. Recommended and enabled by default for CDH 5.5 and higher.", "display_name": "Enable Shuffle Service", "name": "spark_shuffle_service_enabled", "value": "true" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties parameter.", "display_name": "Suppress Parameter Validation: Gateway Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "role_config_suppression_navigator_lineage_client_config_safety_valve", "value": "false" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "51" }, { "desc": "The minimum log level for the Spark shell.", "display_name": "Shell Logging Threshold", "name": "spark_gateway_shell_logging_threshold", "value": "WARN" }, { "desc": "Whether to enable the Spark Web UI on individual applications. It's recommended that the UI be disabled in secure clusters.", "display_name": "Enable Spark Web UI", "name": "spark_ui_enabled", "value": "true" }, { "desc": "Enable dynamic allocation of executors in Spark applications.", "display_name": "Enable Dynamic Allocation", "name": "spark_dynamic_allocation_enabled", "value": "true" }, { "desc": "Which deploy mode to use by default. Can be overridden by users when launching applications.", "display_name": "Default Application Deploy Mode", "name": "spark_deploy_mode", "value": "client" }, { "desc": "Spark SQL QueryExecutionListener that will listen to query executions and write out the lineage info to the lineage directory if lineage is enabled.", "display_name": "Spark SQL Query Execution Listeners", "name": "spark_sql_queryexecutionlisteners", "value": "com.cloudera.spark.lineage.ClouderaNavigatorListener" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-env.sh.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "When dynamic allocation is enabled, maximum number of executors to allocate. By default, Spark relies on YARN to control the maximum number of executors for the application.", "display_name": "Maximum Executor Count", "name": "spark_dynamic_allocation_max_executors", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "When dynamic allocation is enabled, timeout before requesting new executors when there are backlogged tasks.", "display_name": "Scheduler Backlog Timeout", "name": "spark_dynamic_allocation_scheduler_backlog_timeout", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Extra Python Path parameter.", "display_name": "Suppress Parameter Validation: Extra Python Path", "name": "role_config_suppression_spark_python_path", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-defaults.conf.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", "name": "spark-conf/spark-defaults.conf_client_config_safety_valve", "value": null }, { "desc": "Whether to encrypt communication between Spark processes belonging to the same application. Requires authentication (spark.authenticate) to be enabled.", "display_name": "Enable Network Encryption", "name": "spark_network_encryption_enabled", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When dynamic allocation is enabled, timeout before requesting new executors after the initial backlog timeout has already expired. By default this is the same value as the initial backlog timeout.", "display_name": "Sustained Scheduler Backlog Timeout", "name": "spark_dynamic_allocation_sustained_scheduler_backlog_timeout", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the GATEWAY Lineage Log Directory parameter.", "display_name": "Suppress Parameter Validation: GATEWAY Lineage Log Directory", "name": "role_config_suppression_lineage_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark SQL Query Execution Listeners parameter.", "display_name": "Suppress Parameter Validation: Spark SQL Query Execution Listeners", "name": "role_config_suppression_spark_sql_queryexecutionlisteners", "value": "false" }, { "desc": "When dynamic allocation is enabled, time after which idle executors with cached RDDs blocks will be stopped. By default, they're never stopped. This configuration is only available starting in CDH 5.5.", "display_name": "Caching Executor Idle Timeout", "name": "spark_dynamic_allocation_cached_idle_timeout", "value": null }, { "desc": "Write Spark application history logs to HDFS.", "display_name": "Enable History", "name": "spark_history_enabled", "value": "true" }, { "desc": "When dynamic allocation is enabled, minimum number of executors to keep alive while the application is running.", "display_name": "Minimum Executor Count", "name": "spark_dynamic_allocation_min_executors", "value": "0" }, { "desc": "Name of class implementing org.apache.spark.serializer.Serializer to use in Spark applications.", "display_name": "Spark Data Serializer", "name": "spark_data_serializer", "value": "org.apache.spark.serializer.KryoSerializer" }, { "desc": "Python library paths to add to PySpark applications.", "display_name": "Extra Python Path", "name": "spark_python_path", "value": "" }, { "desc": "Whether to allow users to kill running stages from the Spark Web UI.", "display_name": "Enable Kill From UI", "name": "spark_gateway_ui_kill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", "name": "role_config_suppression_spark-conf/spark-defaults.conf_client_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Spark listener that will write out the end of application marker when the application ends.", "display_name": "Spark Extra Listeners", "name": "spark_extralisteners", "value": "com.cloudera.spark.lineage.ClouderaNavigatorListener" }, { "desc": "When dynamic allocation is enabled, number of executors to allocate when the application starts. By default, this is the same value as the minimum number of executors.", "display_name": "Initial Executor Count", "name": "spark_dynamic_allocation_initial_executors", "value": null }, { "desc": "The directory in which GATEWAY lineage log files are written. If changed from the default, Cloudera Manager will not be able to provide lineage information without restarting the Cloudera Manager Agent(s).", "display_name": "GATEWAY Lineage Log Directory", "name": "lineage_event_log_dir", "value": "/var/log/spark/lineage" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only. A string to be inserted into navigator.lineage.client.properties for this role only.", "display_name": "Gateway Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "navigator_lineage_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "role_config_suppression_spark-conf/spark-env.sh_client_config_safety_valve", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-service.json0000666000175100017510000024250713245514472025044 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_core_site_safety_valve", "value": "false" }, { "desc": "Timeout in milliseconds for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations(). This value is only emitted for Impala.", "display_name": "HDFS File Block Storage Location Timeout", "name": "dfs_client_file_block_storage_locations_timeout", "value": "10000" }, { "desc": "Select an Object Store service to enable cloud storage support. Once enabled, the cloud storage can be used in Impala and Hue services, via fully-qualified URIs.", "display_name": "Object Store Service", "name": "object_store_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_shell_cmd_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the FailoverProxyProvider Class parameter.", "display_name": "Suppress Parameter Validation: FailoverProxyProvider Class", "name": "service_config_suppression_dfs_ha_proxy_provider", "value": "false" }, { "desc": "The domain to use for the HTTP cookie that stores the authentication token. In order for authentiation to work correctly across all Hadoop nodes' web-consoles the domain must be correctly set. Important: when using IP addresses, browsers ignore cookies with domain settings. For this setting to work properly all nodes in the cluster must be configured to generate URLs with hostname.domain names on it.", "display_name": "Hadoop HTTP Authentication Cookie Domain", "name": "hadoop_http_auth_cookie_domain", "value": "" }, { "desc": "The user that this service's processes should run as (except the HttpFS server, which has its own user)", "display_name": "System User", "name": "process_username", "value": "hdfs" }, { "desc": "Comma-delimited list of groups that you want to allow the Hue user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hue Proxy User Groups", "name": "hue_proxy_user_groups_list", "value": "*" }, { "desc": "The password for the TLS/SSL truststore.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Truststore Password", "name": "hadoop_group_mapping_ldap_keystore_passwd", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Groups", "name": "service_config_suppression_flume_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Failover Controller Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Failover Controller Count Validator", "name": "service_config_suppression_failovercontroller_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hue Proxy User Hosts", "name": "hue_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Groups", "name": "service_config_suppression_oozie_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Admin Users parameter.", "display_name": "Suppress Parameter Validation: Authorized Admin Users", "name": "service_config_suppression_hadoop_authorized_admin_users", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_replication_env_safety_valve", "value": "false" }, { "desc": "The service monitor will use these permissions to create the directory and files to test if the hdfs service is healthy. Permissions are specified using the 10-character unix-symbolic format e.g. '-rwxr-xr-x'.", "display_name": "HDFS Health Canary Directory Permissions", "name": "firehose_hdfs_canary_directory_permissions", "value": "-rwxrwxrwx" }, { "desc": "The health test thresholds of the overall DataNode health. The check returns \"Concerning\" health if the percentage of \"Healthy\" DataNodes falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" DataNodes falls below the critical threshold.", "display_name": "Healthy DataNode Monitoring Thresholds", "name": "hdfs_datanodes_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Class for user to group mapping (get groups for a given user).", "display_name": "Hadoop User Group Mapping Implementation", "name": "hadoop_security_group_mapping", "value": "org.apache.hadoop.security.ShellBasedUnixGroupsMapping" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop HTTP Authentication Cookie Domain parameter.", "display_name": "Suppress Parameter Validation: Hadoop HTTP Authentication Cookie Domain", "name": "service_config_suppression_hadoop_http_auth_cookie_domain", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the YARN user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "YARN Proxy User Groups", "name": "yarn_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Count Validator", "name": "service_config_suppression_namenode_count_validator", "value": "false" }, { "desc": "Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Oozie Proxy User Groups", "name": "oozie_proxy_user_groups_list", "value": "*" }, { "desc": "Comma-separated list of compression codecs that can be used in job or map compression.", "display_name": "Compression Codecs", "name": "io_compression_codecs", "value": "org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec" }, { "desc": "Comma-separated list of users authorized to used Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Users", "name": "hadoop_authorized_users", "value": "*" }, { "desc": "Quality of protection for secured RPC connections between NameNode and HDFS clients. For effective RPC protection, enable Kerberos authentication.", "display_name": "Hadoop RPC Protection", "name": "hadoop_rpc_protection", "value": "authentication" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_datanode_read_shortcircuit", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Checkpoint Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Checkpoint Configuration Validator", "name": "service_config_suppression_nameservice_checkpoint_configuration_validator", "value": "false" }, { "desc": "Distinguished name of the user to bind as. This is used to connect to LDAP/AD for searching user and group information. This may be left blank if the LDAP server supports anonymous binds.", "display_name": "Hadoop User Group Mapping LDAP Bind User Distinguished Name", "name": "hadoop_group_mapping_ldap_bind_user", "value": "" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Balancer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Balancer Count Validator", "name": "service_config_suppression_balancer_count_validator", "value": "false" }, { "desc": "Password for the TLS/SSL client truststore. Defines a cluster-wide default that can be overridden by individual services.", "display_name": "Cluster-Wide Default TLS/SSL Client Truststore Password", "name": "ssl_client_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Bind User Distinguished Name parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Bind User Distinguished Name", "name": "service_config_suppression_hadoop_group_mapping_ldap_bind_user", "value": "false" }, { "desc": "The password of the bind user.", "display_name": "Hadoop User Group Mapping LDAP Bind User Password", "name": "hadoop_group_mapping_ldap_bind_passwd", "value": "" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping Search Base parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping Search Base", "name": "service_config_suppression_hadoop_group_mapping_ldap_base", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Groups", "name": "service_config_suppression_yarn_proxy_user_groups_list", "value": "false" }, { "desc": "Enable/Disable the Log and Query Redaction Policy for this cluster.", "display_name": "Enable Log and Query Redaction", "name": "redaction_policy_enabled", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to all roles and client configurations in this HDFS service as well as all its dependent services. Any configs added here will be overridden by their default values in HDFS (which can be found in hdfs-default.xml).", "display_name": "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "core_site_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Authentication And Authorization Validation configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Authentication And Authorization Validation", "name": "service_config_suppression_hdfs_authentication_and_authorization_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Proxy User Groups", "name": "service_config_suppression_smon_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Local Path Access Users parameter.", "display_name": "Suppress Parameter Validation: DataNode Local Path Access Users", "name": "service_config_suppression_dfs_block_local_path_access_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue's Kerberos Principal Short Name parameter.", "display_name": "Suppress Parameter Validation: Hue's Kerberos Principal Short Name", "name": "service_config_suppression_hue_kerberos_principal_shortname", "value": "false" }, { "desc": "The default block size in bytes for new HDFS files. Note that this value is also used as the HBase Region Server HLog block size.", "display_name": "HDFS Block Size", "name": "dfs_block_size", "value": "134217728" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Private Keys for SSH Fencing Strategy parameter.", "display_name": "Suppress Parameter Validation: Private Keys for SSH Fencing Strategy", "name": "service_config_suppression_dfs_ha_fencing_ssh_private_key_files", "value": "false" }, { "desc": "Enable WebHDFS interface", "display_name": "Enable WebHDFS", "name": "dfs_webhdfs_enabled", "value": "true" }, { "desc": "The short name of the Hue Kerberos principal. Normally, you do not need to specify this configuration. Cloudera Manager auto-configures this property so that Hue and Cloudera Manamgent Service work properly.", "display_name": "Hue's Kerberos Principal Short Name", "name": "hue_kerberos_principal_shortname", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Search Filter parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Search Filter", "name": "service_config_suppression_hadoop_group_mapping_ldap_group_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Groups", "name": "service_config_suppression_hue_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the JournalNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: JournalNode Count Validator", "name": "service_config_suppression_journalnode_count_validator", "value": "false" }, { "desc": "Typically, HDFS clients and servers communicate by opening sockets via an IP address. In certain networking configurations, it is preferable to open sockets after doing a DNS lookup on the hostname. Enable this property to open sockets after doing a DNS lookup on the hostname. This property is supported in CDH3u4 or later deployments.", "display_name": "Use DataNode Hostname", "name": "dfs_client_use_datanode_hostname", "value": "false" }, { "desc": "Enter a FailoverProxyProvider implementation to configure two URIs to connect to during fail-over. The first configured address is tried first, and on a fail-over event the other address is tried.", "display_name": "FailoverProxyProvider Class", "name": "dfs_ha_proxy_provider", "value": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "The search base for the LDAP connection. This is a distinguished name, and will typically be the root of the LDAP directory.", "display_name": "Hadoop User Group Mapping Search Base", "name": "hadoop_group_mapping_ldap_base", "value": "" }, { "desc": "If false, permission checking is turned off for files in HDFS.", "display_name": "Check HDFS Permissions", "name": "dfs_permissions", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the NFS Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NFS Gateway Count Validator", "name": "service_config_suppression_nfsgateway_count_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Corrupt Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Corrupt Blocks", "name": "service_health_suppression_hdfs_blocks_with_corrupt_replicas", "value": "false" }, { "desc": "Comma-separated list of groups authorized to used Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Groups", "name": "hadoop_authorized_groups", "value": "" }, { "desc": "If AES/CTR/NoPadding is chosen for the Data Transfer Encryption Algorithm, this specifies the length (in bits) of the AES key. When this parameter is changed, a full, non-rolling restart of the cluster must be performed.", "display_name": "Data Transfer Cipher Suite Key Strength", "name": "dfs_encrypt_data_transfer_cipher_keybits", "value": "256" }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted shuffle and encrypted web UIs. Applies to all configurations of daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Oozie Proxy User Hosts", "name": "oozie_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Authorization Provider Group parameter.", "display_name": "Suppress Parameter Validation: Sentry Authorization Provider Group", "name": "service_config_suppression_sentry_authorization_provider_hdfs_group", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Cloudera Service Monitor user to impersonate other users. The default '*' allows all hosts. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'.", "display_name": "Service Monitor Proxy User Hosts", "name": "smon_proxy_user_hosts_list", "value": "*" }, { "desc": "Comma-delimited list of groups that you want to allow the mapred user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Mapred Proxy User Groups", "name": "mapred_proxy_user_groups_list", "value": "*" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_service_env_safety_valve", "value": null }, { "desc": "For paths where authorization is enforced by Sentry Synchronization, file permissions will use this parameter as the group. This group should normally include the hive and impala users.", "display_name": "Sentry Authorization Provider Group", "name": "sentry_authorization_provider_hdfs_group", "value": "hive" }, { "desc": "Additional mapping rules that will be inserted before rules generated from the list of trusted realms and before the default rule. After changing this value and restarting the service, any services depending on this one must be restarted as well. The hadoop.security.auth_to_local property is configured using this information.", "display_name": "Additional Rules to Map Kerberos Principals to Short Names", "name": "extra_auth_to_local_rules", "value": null }, { "desc": "Password for the server keystore file used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "service_config_suppression_hadoop_policy_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the SecondaryNameNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: SecondaryNameNode Count Validator", "name": "service_config_suppression_secondarynamenode_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "service_config_suppression_hdfs_service_config_safety_valve", "value": "false" }, { "desc": "Enables Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Note: This is effective only if Kerberos is enabled for the HDFS service.", "display_name": "Enable Kerberos Authentication for HTTP Web-Consoles", "name": "hadoop_secure_web_ui", "value": "false" }, { "desc": "For advanced use only. Key-value pairs (one on each line) to be inserted into the HDFS replication configuration for hadoop-env.sh.", "display_name": "HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "hdfs_replication_haoop_env_sh_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Secure Web UI Validator configuration validator.", "display_name": "Suppress Configuration Validator: Secure Web UI Validator", "name": "service_config_suppression_hadoop_secure_web_ui", "value": "false" }, { "desc": "Comma-delimited list of groups to allow the HDFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "HDFS Proxy User Groups", "name": "hdfs_proxy_user_groups_list", "value": "*" }, { "desc": "The service monitor will use this directory to create files to test if the hdfs service is healthy. The directory and files are created with permissions specified by 'HDFS Health Canary Directory Permissions'", "display_name": "HDFS Health Canary Directory", "name": "firehose_hdfs_canary_directory", "value": "/tmp/.cloudera_health_monitoring_canary_files" }, { "desc": "Comma-delimited list of groups that you want to allow the Hive user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hive Proxy User Groups", "name": "hive_proxy_user_groups_list", "value": "*" }, { "desc": "Comma-separated list of users authorized to perform admin operations on Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Admin Users", "name": "hadoop_authorized_admin_users", "value": "*" }, { "desc": "The health check thresholds of the number of missing blocks. Specified as a percentage of the total number of blocks.", "display_name": "Missing Block Monitoring Thresholds", "name": "hdfs_missing_blocks_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The amount of time after NameNode(s) start that the lack of an active NameNode will be tolerated. This is intended to allow either the auto-failover daemon to make a NameNode active, or a specifically issued failover command to take effect. This is an advanced option that does not often need to be changed.", "display_name": "NameNode Activation Startup Tolerance", "name": "hdfs_namenode_activation_startup_tolerance", "value": "180" }, { "desc": "

Note: Do not edit this property in the classic layout. Switch to the new layout to use preconfigured redaction rules and test your rules inline.

Use this property to define a list of rules to be followed for redacting sensitive information from log files and query strings. Click + to add a new redaction rule. You can choose one of the preconfigured rules or add a custom rule. When specifying a custom rule, the Search field should contain a regular expression that will be matched against the data. If a match is found, it is replaced by the contents of the Replace field.

Trigger is an optional field. It can be used to specify a simple string to be searched in the data. If the string is found, the redactor attempts to find a match for the Search regex. If no trigger is specified, redaction occurs by matching the Search regular expression. Use the Trigger field to enhance performance: simple string matching is faster than regular expression matching.

Test your rules by entering sample text into the Test Redaction Rules text box and clicking Test Redaction. If no rules match, the text you entered is returned unchanged.

", "display_name": "Log and Query Redaction Policy", "name": "redaction_policy", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Hosts", "name": "service_config_suppression_hue_proxy_user_hosts_list", "value": "false" }, { "desc": "Comma-delimited list of groups to allow the HttpFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "HttpFS Proxy User Groups", "name": "httpfs_proxy_user_groups_list", "value": "*" }, { "desc": "Allows the flume user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Flume Proxy User Groups", "name": "flume_proxy_user_groups_list", "value": "*" }, { "desc": "Comma-delimited list of hosts where you want to allow the mapred user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Mapred Proxy User Hosts", "name": "mapred_proxy_user_hosts_list", "value": "*" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "dfs.socket.timeout3000dfs.datanode.socket.write.timeout3000ipc.client.connect.max.retries1fs.permissions.umask-mode000" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Count Validator", "name": "service_config_suppression_datanode_count_validator", "value": "false" }, { "desc": "Path to the TLS/SSL client truststore file. Defines a cluster-wide default that can be overridden by individual services. This truststore must be in JKS format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. The contents of the truststore can be modified without restarting any roles. By default, changes to its contents are picked up within ten seconds. If not set, the default Java truststore is used to verify certificates.", "display_name": "Cluster-Wide Default TLS/SSL Client Truststore Location", "name": "ssl_client_truststore_location", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Enable TLS/SSL encryption for HDFS, MapReduce, and YARN web UIs, as well as encrypted shuffle for MapReduce and YARN.", "display_name": "Hadoop TLS/SSL Enabled", "name": "hdfs_hadoop_ssl_enabled", "value": "false" }, { "desc": "List of Kerberos realms that Hadoop services should trust. If empty, defaults to the default_realm property configured in the krb5.conf file. After changing this value and restarting the service, all services depending on this service must also be restarted. Adds mapping rules for each domain to the hadoop.security.auth_to_local property in core-site.xml.", "display_name": "Trusted Kerberos Realms", "name": "trusted_realms", "value": "" }, { "desc": "Whether to suppress the results of the Failover Controllers Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Failover Controllers Health", "name": "service_health_suppression_hdfs_failover_controllers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Proxy User Hosts", "name": "service_config_suppression_smon_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress the results of the DataNode Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: DataNode Health", "name": "service_health_suppression_hdfs_data_nodes_healthy", "value": "false" }, { "desc": "Enables the health check that verifies that the failover controllers associated with this service are healthy and running.", "display_name": "Failover Controllers Healthy", "name": "failover_controllers_healthy_enabled", "value": "true" }, { "desc": "The attribute of the group object that identifies the users that are members of the group. The default will usually be appropriate for any LDAP installation.", "display_name": "Hadoop User Group Mapping LDAP Group Membership Attribute", "name": "hadoop_group_mapping_ldap_member_attr", "value": "member" }, { "desc": "Comma separated list of users allowed to do short circuit read. A short circuit read allows a client co-located with the data to read HDFS file blocks directly from HDFS. If empty, will default to the DataNode process' user.", "display_name": "DataNode Local Path Access Users", "name": "dfs_block_local_path_access_user", "value": null }, { "desc": "The timeout, in milliseconds, to use with the Cloudera Manager agent-based fencer.", "display_name": "Timeout for Cloudera Manager Fencing Strategy", "name": "dfs_ha_fencing_cloudera_manager_timeout_millis", "value": "10000" }, { "desc": "Maximum bandwidth used for image transfer in bytes per second. This can help keep normal NameNode operations responsive during checkpointing. A default value of 0 indicates that throttling is disabled.", "display_name": "FsImage Transfer Bandwidth", "name": "dfs_image_transfer_bandwidthPerSec", "value": "0" }, { "desc": "The health check thresholds of the number of blocks that have at least one corrupt replica. Specified as a percentage of the total number of blocks.", "display_name": "Blocks With Corrupt Replicas Monitoring Thresholds", "name": "hdfs_blocks_with_corrupt_replicas_thresholds", "value": "{\"critical\":\"1.0\",\"warning\":\"0.5\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Groups", "name": "service_config_suppression_mapred_proxy_user_groups_list", "value": "false" }, { "desc": "The user the management services impersonates when connecting to HDFS. If no value is specified, the HDFS superuser is used.", "display_name": "HDFS User to Impersonate", "name": "hdfs_user_to_impersonate", "value": null }, { "desc": "File path to a jks-format truststore containing the TLS/SSL certificate used sign the LDAP server's certificate. Note that in previous releases this was erroneously referred to as a \"keystore\".", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Truststore", "name": "hadoop_group_mapping_ldap_keystore", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Synchronization Path Prefixes parameter.", "display_name": "Suppress Parameter Validation: Sentry Synchronization Path Prefixes", "name": "service_config_suppression_hdfs_sentry_sync_path_prefixes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-Wide Default TLS/SSL Client Truststore Password parameter.", "display_name": "Suppress Parameter Validation: Cluster-Wide Default TLS/SSL Client Truststore Password", "name": "service_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "The name of the group of superusers.", "display_name": "Superuser Group", "name": "dfs_permissions_supergroup", "value": "supergroup" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP URL parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP URL", "name": "service_config_suppression_hadoop_group_mapping_ldap_url", "value": "false" }, { "desc": "Allows the Cloudera Service Monitor user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "Service Monitor Proxy User Groups", "name": "smon_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Groups parameter.", "display_name": "Suppress Parameter Validation: Authorized Groups", "name": "service_config_suppression_hadoop_authorized_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Groups", "name": "service_config_suppression_httpfs_proxy_user_groups_list", "value": "false" }, { "desc": "The minimal block replication.", "display_name": "Minimal Block Replication", "name": "dfs_replication_min", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the UNIX Domain Socket path parameter.", "display_name": "Suppress Parameter Validation: UNIX Domain Socket path", "name": "service_config_suppression_dfs_domain_socket_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Hadoop Group Name parameter.", "display_name": "Suppress Parameter Validation: Shared Hadoop Group Name", "name": "service_config_suppression_hdfs_hadoop_group_name", "value": "false" }, { "desc": "The maximal block replication.", "display_name": "Maximal Block Replication", "name": "dfs_replication_max", "value": "512" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "Enable authorization", "display_name": "Hadoop Secure Authorization", "name": "hadoop_security_authorization", "value": "false" }, { "desc": "The Key Management Server used by HDFS. This must be set to use encryption for data at rest.", "display_name": "KMS Service", "name": "kms_service", "value": null }, { "desc": "The attribute of the group object that identifies the group name. The default will usually be appropriate for all LDAP systems.", "display_name": "Hadoop User Group Mapping LDAP Group Name Attribute", "name": "hadoop_group_mapping_ldap_group_name_attr", "value": "cn" }, { "desc": "Enables DataNode support for the experimental DistributedFileSystem.getFileVBlockStorageLocations API. Applicable to CDH 4.1 and onwards.", "display_name": "Enable HDFS Block Metadata API", "name": "dfs_datanode_hdfs_blocks_metadata_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Groups", "name": "service_config_suppression_hdfs_proxy_user_groups_list", "value": "false" }, { "desc": "The tolerance window that will be used in HDFS service tests that depend on detection of the active NameNode.", "display_name": "Active NameNode Detection Window", "name": "hdfs_active_namenode_detecton_window", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Hosts", "name": "service_config_suppression_flume_proxy_user_hosts_list", "value": "false" }, { "desc": "Default block replication. The number of replications to make when the file is created. The default value is used if a replication number is not specified.", "display_name": "Replication Factor", "name": "dfs_replication", "value": "3" }, { "desc": "Comma-delimited list of groups that you want to allow the HTTP user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. This is used by WebHCat.", "display_name": "HTTP Proxy User Groups", "name": "HTTP_proxy_user_groups_list", "value": "*" }, { "desc": "The name of the system group shared by all the core Hadoop services.", "display_name": "Shared Hadoop Group Name", "name": "hdfs_hadoop_group_name", "value": "hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP User Search Filter parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP User Search Filter", "name": "service_config_suppression_hadoop_group_mapping_ldap_user_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS High Availability Fencing Methods parameter.", "display_name": "Suppress Parameter Validation: HDFS High Availability Fencing Methods", "name": "service_config_suppression_dfs_ha_fencing_methods", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Hosts", "name": "service_config_suppression_hive_proxy_user_hosts_list", "value": "false" }, { "desc": "The amount of time to wait for HDFS filesystem image transfer from NameNode to complete.", "display_name": "FsImage Transfer Timeout", "name": "dfs_image_transfer_timeout", "value": "60000" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS snapshot shell command.", "display_name": "HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_shell_cmd_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codecs parameter.", "display_name": "Suppress Parameter Validation: Compression Codecs", "name": "service_config_suppression_io_compression_codecs", "value": "false" }, { "desc": "An additional filter to use when searching for LDAP users. The default will usually be appropriate for Active Directory installations. If connecting to a generic LDAP server, ''sAMAccountName'' will likely be replaced with ''uid''. {0} is a special string used to denote where the username fits into the filter.", "display_name": "Hadoop User Group Mapping LDAP User Search Filter", "name": "hadoop_group_mapping_ldap_user_filter", "value": "(&(objectClass=user)(sAMAccountName={0}))" }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "List of fencing methods to use for service fencing. Setting this to shell(true) enables the built-in HDFS fencing mechanism, which causes the NameNode to exit if it attempts a write operation when it is not active. In almost all cases, this is the best choice. The sshfence method uses SSH. If using custom fencers (that may communicate with shared store, power units, or network switches), use the shell to invoke them.", "display_name": "HDFS High Availability Fencing Methods", "name": "dfs_ha_fencing_methods", "value": "shell(true)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Hosts", "name": "service_config_suppression_hdfs_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Health Canary Directory Permissions parameter.", "display_name": "Suppress Parameter Validation: HDFS Health Canary Directory Permissions", "name": "service_config_suppression_firehose_hdfs_canary_directory_permissions", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Health Canary Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Health Canary Directory", "name": "service_config_suppression_firehose_hdfs_canary_directory", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS replication jobs.", "display_name": "HDFS Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_replication_env_safety_valve", "value": null }, { "desc": "Enables the health check that a client can create, read, write, and delete files", "display_name": "HDFS Canary Health Check", "name": "hdfs_canary_health_enabled", "value": "true" }, { "desc": "Whether to serve logs over HTTP from HDFS web servers. This includes listing the logs directory at the /logs endpoint, which may be a security concern.", "display_name": "Serve logs over HTTP", "name": "http_logs_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP TLS/SSL Truststore Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP TLS/SSL Truststore Password", "name": "service_config_suppression_hadoop_group_mapping_ldap_keystore_passwd", "value": "false" }, { "desc": "Path on the DataNode's local file system to a UNIX domain socket used for communication between the DataNode and local HDFS clients. This socket is used for Short Circuit Reads. Only the HDFS System User and \"root\" should have write access to the parent directory and all of its ancestors. This property is supported in CDH 4.2 or later deployments.", "display_name": "UNIX Domain Socket path", "name": "dfs_domain_socket_path", "value": "/var/run/hdfs-sockets/dn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User's Home Directory parameter.", "display_name": "Suppress Parameter Validation: System User's Home Directory", "name": "service_config_suppression_hdfs_user_home_dir", "value": "false" }, { "desc": "Algorithm to encrypt data transfer between DataNodes and clients, and among DataNodes. If 3des or rc4 are chosen, the entire communication is encrypted with that algorithm. In CDH 5.4 and higher, if AES/CTR/NoPadding is chosen, 3des is used for the initial key exchange, and then AES/CTR/NoPadding is used for the transfer of data. This is the most secure option, and is recommended for clusters running CDH 5.4 or higher. It also requires that the \"openssl-devel\" package be installed on all machines in the cluster. When this parameter is changed, a full, nonrolling restart of the cluster must be performed.", "display_name": "Data Transfer Encryption Algorithm", "name": "dfs_encrypt_data_transfer_algorithm", "value": "rc4" }, { "desc": "The health check thresholds of the number of under-replicated blocks. Specified as a percentage of the total number of blocks.", "display_name": "Under-replicated Block Monitoring Thresholds", "name": "hdfs_under_replicated_blocks_thresholds", "value": "{\"critical\":\"40.0\",\"warning\":\"10.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "hdfs_service_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Hosts", "name": "service_config_suppression_yarn_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress the results of the HDFS Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HDFS Canary", "name": "service_health_suppression_hdfs_canary_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "service_config_suppression_hdfs_ssl_client_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HttpFS Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HttpFS Count Validator", "name": "service_config_suppression_httpfs_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. This is used by WebHCat.", "display_name": "HTTP Proxy User Hosts", "name": "HTTP_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress the results of the Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Free Space", "name": "service_health_suppression_hdfs_free_space_remaining", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Groups", "name": "service_config_suppression_http_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Membership Attribute parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Membership Attribute", "name": "service_config_suppression_hadoop_group_mapping_ldap_member_attr", "value": "false" }, { "desc": "Whether to suppress the results of the Under-Replicated Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Under-Replicated Blocks", "name": "service_health_suppression_hdfs_under_replicated_blocks", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Heap Size Validator", "name": "service_config_suppression_nameservice_namenodes_heap_size_validator", "value": "false" }, { "desc": "Choose the authentication mechanism used by Hadoop", "display_name": "Hadoop Secure Authentication", "name": "hadoop_security_authentication", "value": "simple" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Replication Factor", "name": "service_config_suppression_dfs_replication", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "hadoop_policy_config_safety_valve", "value": null }, { "desc": "Enable encryption of data transfer between DataNodes and clients, and among DataNodes. For effective data transfer protection, enable Kerberos authentication and choose Privacy Quality of RPC Protection.", "display_name": "Enable Data Transfer Encryption", "name": "dfs_encrypt_data_transfer", "value": "false" }, { "desc": "Comma-delimited list of hosts that you want to allow the YARN user to impersonate. The default '*' allows all hosts. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'.", "display_name": "YARN Proxy User Hosts", "name": "yarn_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Bind User Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Bind User Password", "name": "service_config_suppression_hadoop_group_mapping_ldap_bind_passwd", "value": "false" }, { "desc": "When computing the overall HDFS cluster health, consider the active NameNode's health", "display_name": "Active NameNode Role Health Check", "name": "hdfs_namenode_health_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "hdfs_ssl_server_safety_valve", "value": null }, { "desc": "The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory.", "display_name": "System User's Home Directory", "name": "hdfs_user_home_dir", "value": "/var/lib/hadoop-hdfs" }, { "desc": "Whether to suppress the results of the Missing Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Missing Blocks", "name": "service_health_suppression_hdfs_missing_blocks", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "SASL protection mode for secured connections to the DataNodes when reading or writing data.", "display_name": "DataNode Data Transfer Protection", "name": "dfs_data_transfer_protection", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Users parameter.", "display_name": "Suppress Parameter Validation: Authorized Users", "name": "service_config_suppression_hadoop_authorized_users", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the flume user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Flume Proxy User Hosts", "name": "flume_proxy_user_hosts_list", "value": "*" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "Hadoop User Group Mapping LDAP URL", "name": "hadoop_group_mapping_ldap_url", "value": null }, { "desc": "SSH connection timeout, in milliseconds, to use with the built-in sshfence fencer.", "display_name": "Timeout for SSH Fencing Strategy", "name": "dfs_ha_fencing_ssh_connect_timeout", "value": "30000" }, { "desc": "Enable automatic synchronization of HDFS ACLs with Sentry privileges. HDFS Access Control Lists and Check HDFS Permissions must be enabled when this feature is enabled. Use Sentry Synchronization Path Prefixes to define the HDFS regions where authorization is enforced using Sentry information. For more information, see Synchronizing HDFS ACLs and Sentry Authorization.", "display_name": "Enable Sentry Synchronization", "name": "hdfs_sentry_sync_enable", "value": "false" }, { "desc": "Comma-delimited list of hosts where you allow the HttpFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HttpFS Proxy User Hosts", "name": "httpfs_proxy_user_hosts_list", "value": "*" }, { "desc": "Name of the ZooKeeper service that this HDFS service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Mountpoints Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Mountpoints Validator", "name": "service_config_suppression_nameservice_mountpoints_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Admin Groups parameter.", "display_name": "Suppress Parameter Validation: Authorized Admin Groups", "name": "service_config_suppression_hadoop_authorized_admin_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Hosts", "name": "service_config_suppression_mapred_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Superuser Group parameter.", "display_name": "Suppress Parameter Validation: Superuser Group", "name": "service_config_suppression_dfs_permissions_supergroup", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Auto Failover Validator configuration validator.", "display_name": "Suppress Configuration Validator: Auto Failover Validator", "name": "service_config_suppression_auto_failover_validator", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hdfs" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Encryption Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Encryption Validator", "name": "service_config_suppression_hdfs_encryption_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Hosts", "name": "service_config_suppression_oozie_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Hosts", "name": "service_config_suppression_http_proxy_user_hosts_list", "value": "false" }, { "desc": "The group that this service's processes should run as (except the HttpFS server, which has its own group)", "display_name": "System Group", "name": "process_groupname", "value": "hdfs" }, { "desc": "ACLs (Access Control Lists) enhance the existing HDFS permission model to support controlling file access for arbitrary combinations of users and groups instead of a single owner, single group, and all other users. When ACLs are disabled, the NameNode rejects all attempts to set an ACL.", "display_name": "Enable Access Control Lists", "name": "dfs_namenode_acls_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Trusted Kerberos Realms parameter.", "display_name": "Suppress Parameter Validation: Trusted Kerberos Realms", "name": "service_config_suppression_trusted_realms", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the NFS High Availability Validator configuration validator.", "display_name": "Suppress Configuration Validator: NFS High Availability Validator", "name": "service_config_suppression_nfs_ha_validator", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Whether to suppress the results of the NameNode Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Health", "name": "service_health_suppression_hdfs_ha_namenode_health", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hive Proxy User Hosts", "name": "hive_proxy_user_hosts_list", "value": "*" }, { "desc": "Default umask for file and directory creation, specified in an octal value (with a leading 0)", "display_name": "Default Umask", "name": "dfs_umaskmode", "value": "022" }, { "desc": "The health check thresholds of free space in HDFS. Specified as a percentage of total HDFS capacity.", "display_name": "HDFS Free Space Monitoring Thresholds", "name": "hdfs_free_space_thresholds", "value": "{\"critical\":\"10.0\",\"warning\":\"20.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS User to Impersonate parameter.", "display_name": "Suppress Parameter Validation: HDFS User to Impersonate", "name": "service_config_suppression_hdfs_user_to_impersonate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Groups", "name": "service_config_suppression_hive_proxy_user_groups_list", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Comma-separated list of groups authorized to perform admin operations on Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Admin Groups", "name": "hadoop_authorized_admin_groups", "value": "" }, { "desc": "An additional filter to use when searching for groups.", "display_name": "Hadoop User Group Mapping LDAP Group Search Filter", "name": "hadoop_group_mapping_ldap_group_filter", "value": "(objectClass=group)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Rules to Map Kerberos Principals to Short Names parameter.", "display_name": "Suppress Parameter Validation: Additional Rules to Map Kerberos Principals to Short Names", "name": "service_config_suppression_extra_auth_to_local_rules", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-Wide Default TLS/SSL Client Truststore Location parameter.", "display_name": "Suppress Parameter Validation: Cluster-Wide Default TLS/SSL Client Truststore Location", "name": "service_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Hosts", "name": "service_config_suppression_httpfs_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_hdfs_ssl_server_safety_valve", "value": "false" }, { "desc": "A list of path prefixes that define the HDFS regions where authorization is enforced using Sentry information. Only relevant when Sentry Synchronization is enabled.", "display_name": "Sentry Synchronization Path Prefixes", "name": "hdfs_sentry_sync_path_prefixes", "value": "/user/hive/warehouse" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Log and Query Redaction Policy parameter.", "display_name": "Suppress Parameter Validation: Log and Query Redaction Policy", "name": "service_config_suppression_redaction_policy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Name Attribute parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Name Attribute", "name": "service_config_suppression_hadoop_group_mapping_ldap_group_name_attr", "value": "false" }, { "desc": "When computing the overall HDFS cluster health, consider the health of the standby NameNode.", "display_name": "Standby NameNode Health Check", "name": "hdfs_standby_namenodes_health_enabled", "value": "true" }, { "desc": "The SSH private key files to use with the built-in sshfence fencer. These are to be accessible to the hdfs user on the machines running the NameNodes.", "display_name": "Private Keys for SSH Fencing Strategy", "name": "dfs_ha_fencing_ssh_private_key_files", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "service_config_suppression_hdfs_replication_haoop_env_sh_safety_valve", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HDFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HDFS Proxy User Hosts", "name": "hdfs_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether or not to use TLS/SSL when connecting to the LDAP server.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Enabled", "name": "hadoop_group_mapping_ldap_use_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Redaction Policy Validator configuration validator.", "display_name": "Suppress Configuration Validator: Redaction Policy Validator", "name": "service_config_suppression_redaction_policy_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into ssl-client.xml. Applies cluster-wide, but can be overridden by individual services.", "display_name": "HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "hdfs_ssl_client_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP TLS/SSL Truststore parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP TLS/SSL Truststore", "name": "service_config_suppression_hadoop_group_mapping_ldap_keystore", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hbase-gateway.json0000666000175100017510000001636613245514472025205 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Scanner Timeout, in milliseconds, for HBase Clients. Scanner related RPCs will apply this timeout against the RegionServers they talk to.", "display_name": "HBase Client Scanner Timeout", "name": "hbase_client_scanner_timeout_period", "value": "60000" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_client_config_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for hbase-site.xml.", "display_name": "HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_client_config_safety_valve", "value": null }, { "desc": "If a multiget operation is performed with 'Consistency.TIMELINE', the read RPC is sent to the primary RegionServer first. After this timeout, a parallel RPC for secondary region replicas is also sent if the primary does not respond. After this, the result is returned from whichever RPC is finished first. If the response returns from the primary region replica, that the data is the most recent. Result.isStale() API has been added to inspect the staleness. If the result is from a secondary region, Result.isStale() is set to true.", "display_name": "HBase Client Multiget Timeout For Secondary Region Replicas", "name": "hbase_client_primaryCallTimeout_multiget", "value": "10" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hbase" }, { "desc": "Whether to enable interruption of RPC threads at the client. The default value of true enables primary RegionServers to access data from other regions' secondary replicas.", "display_name": "Enable Client RPC Threads Interruption", "name": "hbase_ipc_client_allowsInterrupt", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hbase-env.sh", "display_name": "HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh", "name": "hbase_client_env_safety_valve", "value": null }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh parameter.", "display_name": "Suppress Parameter Validation: HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh", "name": "role_config_suppression_hbase_client_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Client Scanner Timeout exceeds Lease Period Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Client Scanner Timeout exceeds Lease Period Validator", "name": "role_config_suppression_client_server_scanner_rpc_timeout_validator", "value": "false" }, { "desc": "If a get operation is performed with 'Consistency.TIMELINE', the read RPC is sent to the primary RegionServer first. After this timeout, parallel RPC for secondary region replicas is also sent if the primary does not respond. After this, the result is returned from whichever RPC is finished first. If the response returns from the primary region replica, that data is the most recent. Result.isStale() API has been added to inspect the staleness. If the result is from a secondary region, Result.isStale() is set to true.", "display_name": "HBase Client Get Timeout For Secondary Region Replicas", "name": "hbase_client_primaryCallTimeout_get", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "These are Java command-line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hbase_client_java_opts", "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hbase_client_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hbase_client_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/sqoop-sqoop_server.json0000666000175100017510000006114313245514472026343 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_sqoop_server_scm_health", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Password for Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Password", "name": "sqoop_repository_database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_sqoop_server_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties", "name": "role_config_suppression_sqoop_config_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Directory where Sqoop 2 Server will place its log files.", "display_name": "Sqoop 2 Server Log Directory", "name": "sqoop_log_dir", "value": "/var/log/sqoop2" }, { "desc": "Admin port of Sqoop 2 Server. (Note: This configuration only applies from CDH 4.3 onwards.)", "display_name": "Sqoop 2 Admin Port", "name": "sqoop_admin_port", "value": "8005" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum number of clients allowed to connect to the Sqoop 2 Server.", "display_name": "Maximum Client Connections", "name": "max_client_connections", "value": "10" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only. A string to be inserted into sqoop.properties for this role only.", "display_name": "Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties", "name": "sqoop_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Log Directory", "name": "role_config_suppression_sqoop_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_sqoop_server_unexpected_exits", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Sqoop 2 Server logs. Typically used by log4j or logback.", "display_name": "Sqoop 2 Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_sqoop_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_sqoop_server_file_descriptor", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SQOOP_SERVER_role_env_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "sqoop_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Password parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Password", "name": "role_config_suppression_sqoop_repository_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database User parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database User", "name": "role_config_suppression_sqoop_repository_database_user", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Sqoop 2 Server in Bytes", "name": "sqoop_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Directory where the Sqoop 2 Server places its metastore data. This is used only when Sqoop Repository Database Type is Derby.", "display_name": "Sqoop 2 Server Metastore Directory", "name": "sqoop_metastore_data_dir", "value": "/var/lib/sqoop2" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Sqoop 2 Server", "name": "sqoop_java_opts", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Type of Sqoop repository database.", "display_name": "Sqoop Repository Database Type", "name": "sqoop_repository_database_type", "value": "derby" }, { "desc": "User for Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database User", "name": "sqoop_repository_database_user", "value": "sa" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_sqoop_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Name of Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Name", "name": "sqoop_repository_database_name", "value": "sqoop" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Sqoop 2 Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Sqoop 2 Server", "name": "role_config_suppression_sqoop_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Metastore Directory parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Metastore Directory", "name": "role_config_suppression_sqoop_metastore_data_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Sqoop 2 Server logs. Typically used by log4j or logback.", "display_name": "Sqoop 2 Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Host parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Host", "name": "role_config_suppression_sqoop_repository_database_host", "value": "false" }, { "desc": "When computing the overall Sqoop 2 Server health, consider the host's health.", "display_name": "Sqoop 2 Server Host Health Test", "name": "sqoop_server_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Enables the health test that the Sqoop 2 Server's process state is consistent with the role configuration", "display_name": "Sqoop 2 Server Process Health Test", "name": "sqoop_server_scm_health_enabled", "value": "true" }, { "desc": "HTTP port of Sqoop 2 Server. (Note: This configuration only applies from CDH 4.3 onwards.)", "display_name": "Sqoop 2 HTTP Port", "name": "sqoop_http_port", "value": "12000" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_sqoop_server_audit_health", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_sqoop_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Host name of Sqoop repository database. If the port is non-default for your database type, use host:port notation. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Host", "name": "sqoop_repository_database_host", "value": "localhost" }, { "desc": "The minimum log level for Sqoop 2 Server logs", "display_name": "Sqoop 2 Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Name parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Name", "name": "role_config_suppression_sqoop_repository_database_name", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_sqoop_server_host_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/0000775000175100017510000000000013245515026023014 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/common/0000775000175100017510000000000013245515026024304 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/common/install_extjs0000666000175100017510000000115213245514476027123 0ustar zuulzuul00000000000000#!/bin/bash EXTJS_DESTINATION_DIR="/var/lib/oozie" EXTJS_DOWNLOAD_URL="http://tarballs.openstack.org/sahara/dist/common-artifacts/ext-2.2.zip" extjs_archive=/tmp/$(basename $EXTJS_DOWNLOAD_URL) if [ ! -n "$EXTJS_DESTINATION_DIR" ]; then if [ $test_only -eq 0 ]; then wget -O $extjs_archive $EXTJS_DOWNLOAD_URL mkdir -p $EXTJS_DESTINATION_DIR if [ -z "${EXTJS_NO_UNPACK:-}" ]; then unzip -d "$EXTJS_DESTINATION_DIR" $extjs_archive rm -f $extjs_archive else mv $extjs_archive $EXTJS_DESTINATION_DIR fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/common/add_jar0000666000175100017510000000132113245514476025622 0ustar zuulzuul00000000000000#!/usr/bin/env bash hadoop="2.6.0" HADOOP_SWIFT_JAR_NAME="hadoop-openstack-$hadoop-cdh$plugin_version.jar" HADOOP_SWIFT_BUILD_LOCATION="http://tarballs.openstack.org/sahara/dist/hadoop-openstack/master" SWIFT_LIB_URI="$HADOOP_SWIFT_BUILD_LOCATION/hadoop-openstack-${hadoop}.jar" HADOOP_SWIFT_JAR_NAME="hadoop-openstack.jar" if [ ! -f $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME ]; then if [ $test_only -eq 0 ]; then wget -O $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME $SWIFT_LIB_URI if [ $? -ne 0 ]; then echo -e "Could not download Swift Hadoop FS implementation.\nAborting" exit 1 fi chmod 0644 $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/common/install_cloudera0000666000175100017510000000130113245514472027554 0ustar zuulzuul00000000000000#!/bin/bash HADOOP_OPENSTACK_5_9_x_URL="https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-openstack/2.6.0-cdh$plugin_version/hadoop-openstack-2.6.0-cdh$plugin_version.jar" dest=/usr/lib/hadoop-mapreduce/hadoop-openstack.jar if [ ! -L "/usr/lib/oozie/oozie-sharelib-yarn.tar.gz" ]; then if [ $test_only -eq 0 ]; then wget -O $dest $HADOOP_OPENSTACK_5_9_x_URL ln -s /usr/lib/oozie/oozie-sharelib-yarn /usr/lib/oozie/oozie-sharelib-yarn.tar.gz ln -s /usr/lib/oozie/oozie-sharelib-mr1 /usr/lib/oozie/oozie-sharelib-mr1.tar.gz ln -s /usr/lib/oozie/oozie-sharelib-yarn.tar.gz /usr/lib/oozie/oozie-sharelib.tar.gz else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/common/unlimited_security_artifacts0000666000175100017510000000055513245514472032223 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -n /tmp/UnlimitedPolicy/ ]; then if [ $test_only -eq 0 ]; then mkdir /tmp/UnlimitedPolicy/ wget $unlimited_security_location/local_policy.jar -O /tmp/UnlimitedPolicy/local_policy.jar wget $unlimited_security_location/US_export_policy.jar -O /tmp/UnlimitedPolicy/US_export_policy.jar else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/image.yaml0000666000175100017510000000622213245514472024772 0ustar zuulzuul00000000000000arguments: java_distro: default: cloudera-jdk description: The distribution of Java to install. Defaults to openjdk. choices: - openjdk - oracle-java - cloudera-jdk plugin_version: default: 5.11.0 description: The distribution of CDH to install. Defaults to 5.11.0. hidden: True required: False choices: - 5.11.0 - 5.11.1 - 5.11.2 hdfs_lib_dir: default: /usr/lib/hadoop-mapreduce description: The path to HDFS_LIB_DIR. Default to /usr/lib/hadoop-mapreduce required: False unlimited_security_location: default: http://sahara-files.mirantis.com/kerberos-artifacts/ description: Place where UnlimitedSecurity polices are located required: False validators: - script: common/install_extjs - os_case: - centos: - package: wget - script: centos/wget_repo: env_vars: [plugin_version] - ubuntu: - script: ubuntu/wget_repo: env_vars: [plugin_version] - argument_case: argument_name: java_distro cases: openjdk: - any: - all: - package: java-1.8.0-openjdk-devel - all: - package: java-1.7.0-openjdk-devel cloudera-jdk: - all: - package: oracle-j2sdk1.7 - package: ntp - package: - cloudera-manager-agent - cloudera-manager-daemons - cloudera-manager-server - cloudera-manager-server-db-2 - package: - hadoop-hdfs-namenode - hadoop-hdfs-datanode - hadoop-hdfs-secondarynamenode - hadoop-mapreduce - hadoop-mapreduce-historyserver - hadoop-yarn-nodemanager - hadoop-yarn-resourcemanager - package: - hbase - hbase-solr - package: - hive-hcatalog - hive-metastore - hive-server2 - hive-webhcat-server - hue - package: - oozie - spark-core - os_case: - centos: - package: spark-history-server - package: zookeeper - package: unzip - package: flume-ng - package: hadoop-kms - package: - impala - impala-server - impala-state-store - impala-catalog - impala-shell - package: keytrustee-keyprovider - package: - sentry - solr-server - solr-doc - search - sqoop2 - package: - kafka - kafka-server - script: common/install_cloudera: env_vars: [plugin_version] - os_case: - centos: - script: centos/turn_off_services - ubuntu: - script: ubuntu/turn_off_services - script: common/add_jar: env_vars: [plugin_version, hdfs_lib_dir] - script: common/unlimited_security_artifacts: env_vars: [unlimited_security_location] - os_case: - centos: - package: - krb5-server - krb5-libs - krb5-workstation - rng-tools - package: iptables-services - script: centos/selinux_permissive - script: centos/disable_firewall - package: nmap-ncat - ubuntu: - package: - krb5-admin-server - libpam-krb5 - ldap-utils - krb5-user - rng-tools sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/centos/0000775000175100017510000000000013245515026024307 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/centos/turn_off_services0000666000175100017510000000147013245514472027767 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files oozie.service | grep 'enabled' | wc -l) if [ "$check" != "on" ]; then if [ $test_only -eq 0 ]; then for i in cloudera-scm-agent \ cloudera-scm-server \ cloudera-scm-server-db \ hadoop-hdfs-datanode \ hadoop-hdfs-namenode \ hadoop-hdfs-secondarynamenode \ hadoop-mapreduce-historyserver \ hadoop-yarn-nodemanager \ hadoop-yarn-resourcemanager \ hive-metastore \ hive-server2 \ hive-webhcat-server \ hue \ oozie \ postgresql \ impala-catalog \ impala-server \ impala-state-store \ solr-server \ spark-history-server do chkconfig $i off done else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/centos/selinux_permissive0000666000175100017510000000046113245514472030176 0ustar zuulzuul00000000000000#!/bin/bash check=$(cat /etc/selinux/config | grep "SELINUX=permissive" | wc -l) echo $check if [ $check -eq 0 ]; then if [ $test_only -eq 0 ]; then echo "SELINUX=permissive" > /etc/selinux/config echo "SELINUXTYPE=targeted" >> /etc/selinux/config else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/centos/wget_repo0000666000175100017510000000426613245514472026243 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -f /etc/yum.repos.d/cloudera-cdh5.repo ]; then if [ $test_only -eq 0 ]; then echo '[cloudera-cdh5]' > /etc/yum.repos.d/cloudera-cdh5.repo echo "name=Cloudera's Distribution for Hadoop, Version 5" >> /etc/yum.repos.d/cloudera-cdh5.repo echo "baseurl=http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/$plugin_version/" >> /etc/yum.repos.d/cloudera-cdh5.repo echo "gpgkey = http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-cdh5.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/cloudera-cdh5.repo echo '[cloudera-manager]' > /etc/yum.repos.d/cloudera-manager.repo echo 'name=Cloudera Manager' >> /etc/yum.repos.d/cloudera-manager.repo echo "baseurl=http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/$plugin_version/" >> /etc/yum.repos.d/cloudera-manager.repo echo "gpgkey = http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-manager.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/cloudera-manager.repo echo '[navigator-keytrustee]' > /etc/yum.repos.d/kms.repo echo "name=Cloudera's Distribution for navigator-Keytrustee, Version 5" >> /etc/yum.repos.d/kms.repo echo "baseurl=http://archive.cloudera.com/navigator-keytrustee5/redhat/7/x86_64/navigator-keytrustee/$plugin_version/" >> /etc/yum.repos.d/kms.repo echo "gpgkey = http://archive.cloudera.com/navigator-keytrustee5/redhat/7/x86_64/navigator-keytrustee/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/kms.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/kms.repo echo "[cloudera-kafka]" > /etc/yum.repos.d/cloudera-kafka.repo echo "name=Cloudera's Distribution for kafka, Version 2.0.2" >> /etc/yum.repos.d/cloudera-kafka.repo echo "baseurl=http://archive.cloudera.com/kafka/redhat/7/x86_64/kafka/2.0.2/" >> /etc/yum.repos.d/cloudera-kafka.repo echo "gpgkey = http://archive.cloudera.com/kafka/redhat/7/x86_64/kafka/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-kafka.repo echo "gpgcheck = 1" >> /etc/yum.repos.d/cloudera-kafka.repo yum clean all else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/centos/disable_firewall0000666000175100017510000000103513245514472027527 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files iptables.service | grep 'enabled' | wc -l) if [ $check -eq 1 ]; then if [ $test_only -eq 0 ]; then if type -p systemctl && [[ "$(systemctl --no-pager list-unit-files firewalld)" =~ 'enabled' ]]; then systemctl disable firewalld fi if type -p service; then service ip6tables save service iptables save chkconfig ip6tables off chkconfig iptables off fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/ubuntu/0000775000175100017510000000000013245515026024336 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/ubuntu/turn_off_services0000666000175100017510000000150013245514472030010 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files oozie.service | grep 'enabled' | wc -l) if [ "$check" != "on" ]; then if [ $test_only -eq 0 ]; then for i in cloudera-scm-agent \ cloudera-scm-server \ cloudera-scm-server-db \ hadoop-hdfs-datanode \ hadoop-hdfs-namenode \ hadoop-hdfs-secondarynamenode \ hadoop-mapreduce-historyserver \ hadoop-yarn-nodemanager \ hadoop-yarn-resourcemanager \ hive-metastore \ hive-server2 \ hive-webhcat-server \ hue \ oozie \ postgresql \ impala-catalog \ impala-server \ impala-state-store \ solr-server \ spark-history-server do update-rc.d -f $i remove done else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/images/ubuntu/wget_repo0000666000175100017510000000374613245514472026274 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -f /etc/apt/sources.list.d/cdh5.list ]; then if [ $test_only -eq 0 ]; then # Add repository with postgresql package (it's dependency of cloudera packages) # Base image doesn't contain this repo echo -e 'deb http://nova.clouds.archive.ubuntu.com/ubuntu/ xenial universe multiverse main' >> /etc/apt/sources.list # Cloudera repositories echo "deb [arch=amd64] http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh$plugin_version contrib" > /etc/apt/sources.list.d/cdh5.list echo "deb-src http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh$plugin_version contrib" >> /etc/apt/sources.list.d/cdh5.list wget -qO - http://archive-primary.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/archive.key | apt-key add - echo "deb [arch=amd64] http://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm xenial-cm$plugin_version contrib" > /etc/apt/sources.list.d/cm5.list echo "deb-src http://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm xenial-cm$plugin_version contrib" >> /etc/apt/sources.list.d/cm5.list wget -qO - http://archive-primary.cloudera.com/cm5/ubuntu/xenial/amd64/cm/archive.key | apt-key add - wget -O /etc/apt/sources.list.d/kms.list http://archive.cloudera.com/navigator-keytrustee5/ubuntu/xenial/amd64/navigator-keytrustee/cloudera.list wget -qO - http://archive.cloudera.com/navigator-keytrustee5/ubuntu/xenial/amd64/navigator-keytrustee/archive.key | apt-key add - # add Kafka repository echo -e 'deb http://archive.cloudera.com/kafka/ubuntu/xenial/amd64/kafka/ xenial-kafka2.2.0 contrib' >> /etc/apt/sources.list wget -qO - https://archive.cloudera.com/kafka/ubuntu/xenial/amd64/kafka/archive.key | apt-key add - #change repository priority echo -e 'Package: zookeeper\nPin: origin "archive.cloudera.com"\nPin-Priority: 1001' > /etc/apt/preferences.d/cloudera-pin apt-get update else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/kms-kms.json0000666000175100017510000011343513245514472024041 0ustar zuulzuul00000000000000[ { "desc": "Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Oozie Proxy User Groups", "name": "hadoop_kms_proxyuser_oozie_groups", "value": "*" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_mapred_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml", "name": "role_config_suppression_kms-site.xml_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_oozie_groups", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kms_kms_file_descriptor", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_kms_kms_audit_health", "value": "false" }, { "desc": "Enables the health test that the Key Management Server's process state is consistent with the role configuration", "display_name": "Key Management Server Process Health Test", "name": "kms_scm_health_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into kms-acls.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml", "name": "kms-acls.xml_role_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kms_kms_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kms_role_env_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Key Management Server TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_flume_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Allows the yarn superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "YARN Proxy User Groups", "name": "hadoop_kms_proxyuser_yarn_groups", "value": "*" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kms_kms_unexpected_exits", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_mapred_hosts", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Key Management Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Key Management Server TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "Directory of the keystore file kms.keystore used by JavaKeyStoreProvider that backs the KMS.", "display_name": "JavaKeyStoreProvider Directory", "name": "hadoop_security_key_provider_dir", "value": "/var/lib/kms" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Maximum heap size of the KMS.", "display_name": "KMS Heap Size", "name": "kms_heap_size", "value": "1073741824" }, { "desc": "Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Oozie Proxy User Hosts", "name": "hadoop_kms_proxyuser_oozie_hosts", "value": "*" }, { "desc": "

The number of threads to be used to accept connections. Increase this value on a multi CPU machine, although you would never really need more than 2. Also, with a lot of non keep alive connections, you might want to increase this value as well.

This setting only works with Tomcat's NIO handler protocol.

This configuration is only supported in CDH 5.11 and up.

", "display_name": "KMS Acceptor Thread Count", "name": "kms_acceptor_thread_count", "value": "1" }, { "desc": "The maximum size, in megabytes, per log file for Key Management Server logs. Typically used by log4j or logback.", "display_name": "Key Management Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Additional Java Configuration Options for KMS", "name": "kms_java_opts", "value": "" }, { "desc": "Allows the hdfs superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HDFS Proxy User Groups", "name": "hadoop_kms_proxyuser_hdfs_groups", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_yarn_hosts", "value": "false" }, { "desc": "The log directory for log files of the role Key Management Server.", "display_name": "Key Management Server Log Directory", "name": "log_dir", "value": "/var/log/hadoop-kms" }, { "desc": "Comma-delimited list of hosts where you want to allow the hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hue Proxy User Hosts", "name": "hadoop_kms_proxyuser_hue_hosts", "value": "*" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Comma-delimited list of hosts where you want to allow the hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hive Proxy User Hosts", "name": "hadoop_kms_proxyuser_hive_hosts", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_oozie_hosts", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the flume user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Flume Proxy User Hosts", "name": "hadoop_kms_proxyuser_flume_hosts", "value": "*" }, { "desc": "Port used to access the KMS' embedded Tomcat admin console.", "display_name": "KMS Admin Port", "name": "kms_admin_port", "value": "16001" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_flume_groups", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into core-site.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "core-site.xml_role_safety_valve", "value": null }, { "desc": "Allows the flume superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Flume Proxy User Groups", "name": "hadoop_kms_proxyuser_flume_groups", "value": "*" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Key Management Server TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_yarn_groups", "value": "false" }, { "desc": "The minimum log level for Key Management Server logs", "display_name": "Key Management Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kms_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Allows the mapred superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Mapred Proxy User Groups", "name": "hadoop_kms_proxyuser_mapred_groups", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_http_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hdfs_groups", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the yarn user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "YARN Proxy User Hosts", "name": "hadoop_kms_proxyuser_yarn_hosts", "value": "*" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hue_hosts", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HTTP Proxy User Hosts", "name": "hadoop_kms_proxyuser_HTTP_hosts", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "

The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused.

This configuration is only supported in CDH 5.11 and up.

", "display_name": "KMS Accept Count", "name": "kms_accept_count", "value": "500" }, { "desc": "Encrypt communication between clients and Key Management Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Key Management Server", "name": "ssl_enabled", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Key Management Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "KMS_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_http_groups", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml", "name": "role_config_suppression_kms-acls.xml_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the KMS Staging Directory parameter.", "display_name": "Suppress Parameter Validation: KMS Staging Directory", "name": "role_config_suppression_kms_staging_dir", "value": "false" }, { "desc": "When computing the overall Key Management Server health, consider the host's health.", "display_name": "Key Management Server Host Health Test", "name": "kms_host_health_enabled", "value": "true" }, { "desc": "A comma-separated list of users (no spaces) for whom to disallow access to key material. These users can still fetch key metadata and create encrypted encryption keys, but are unable to do any other KMS operations. Typically, HDFS superusers will be specified here.", "display_name": "KMS Blacklist Users", "name": "kms_blacklist_users", "value": "" }, { "desc": "Allows the httpfs superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HttpFS Proxy User Groups", "name": "hadoop_kms_proxyuser_httpfs_groups", "value": "*" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Allows the HTTP superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HTTP Proxy User Groups", "name": "hadoop_kms_proxyuser_HTTP_groups", "value": "*" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hive_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hdfs_hosts", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the mapred user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Mapred Proxy User Hosts", "name": "hadoop_kms_proxyuser_mapred_hosts", "value": "*" }, { "desc": "Comma-delimited list of hosts where you want to allow the httpfs user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HttpFS Proxy User Hosts", "name": "hadoop_kms_proxyuser_httpfs_hosts", "value": "*" }, { "desc": "Maximum number of threads used to handle KMS requests.", "display_name": "KMS Max Threads", "name": "kms_max_threads", "value": "250" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_httpfs_groups", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Key Management Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "

The Tomcat handler protocol. The default HTTP/1.1 handler is a thread-per-request model.

This configuration is only supported in CDH 5.11 and up.

", "display_name": "KMS Handler Protocol", "name": "kms_protocol", "value": "HTTP/1.1" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kms_kms_host_health", "value": "false" }, { "desc": "Port used by clients to interact with the KMS.", "display_name": "KMS HTTP Port", "name": "kms_http_port", "value": "16000" }, { "desc": "The maximum number of rolled log files to keep for Key Management Server logs. Typically used by log4j or logback.", "display_name": "Key Management Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Allows the hive superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hive Proxy User Groups", "name": "hadoop_kms_proxyuser_hive_groups", "value": "*" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Comma-delimited list of hosts where you want to allow the hdfs user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HDFS Proxy User Hosts", "name": "hadoop_kms_proxyuser_hdfs_hosts", "value": "*" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_httpfs_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Allows the hue superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hue Proxy User Groups", "name": "hadoop_kms_proxyuser_hue_groups", "value": "*" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kms_kms_scm_health", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into kms-site.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml", "name": "kms-site.xml_role_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hive_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "role_config_suppression_core-site.xml_role_safety_valve", "value": "false" }, { "desc": "The password for the Key Management Server JKS keystore file.", "display_name": "Key Management Server TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the KMS Blacklist Users parameter.", "display_name": "Suppress Parameter Validation: KMS Blacklist Users", "name": "role_config_suppression_kms_blacklist_users", "value": "false" }, { "desc": "Directory where configuration and binaries are staged before starting KMS. Does not normally need to be modified.", "display_name": "KMS Staging Directory", "name": "kms_staging_dir", "value": "/var/lib/hadoop-kms" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hue_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JavaKeyStoreProvider Directory parameter.", "display_name": "Suppress Parameter Validation: JavaKeyStoreProvider Directory", "name": "role_config_suppression_hadoop_security_key_provider_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the KMS Acceptor Thread Count parameter.", "display_name": "Suppress Parameter Validation: KMS Acceptor Thread Count", "name": "role_config_suppression_kms_acceptor_thread_count", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Java Configuration Options for KMS parameter.", "display_name": "Suppress Parameter Validation: Additional Java Configuration Options for KMS", "name": "role_config_suppression_kms_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hbase-regionserver.json0000666000175100017510000017346713245514472026264 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_regionserver_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the HBase RegionServer Multiple HDFS WAL with Replication Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase RegionServer Multiple HDFS WAL with Replication Validator", "name": "role_config_suppression_hbase_regionserver_multiwal_replication_validator", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Maximum number of HStoreFiles to compact per minor compaction.", "display_name": "Maximum Number of HStoreFiles Compaction", "name": "hbase_hstore_compaction_max", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: RegionServer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_regionserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase HRegion Major Compaction parameter.", "display_name": "Suppress Parameter Validation: HBase HRegion Major Compaction", "name": "role_config_suppression_hbase_hregion_majorcompaction", "value": "false" }, { "desc": "The address for the HBase RegionServer web UI", "display_name": "HBase RegionServer Web UI Address", "name": "hbase_regionserver_info_bindAddress", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Number of threads to use while loading and unloading regions to or from a RegionServer. Can be used to increase the speed of decommissioning or rolling restart operations.", "display_name": "Region Mover Threads", "name": "hbase_regionserver_regionmover_thread_count", "value": "1" }, { "desc": "The health test thresholds of the average size of the HBase RegionServer flush queue over a recent period. See HBase RegionServer Flush Queue Monitoring Period.", "display_name": "HBase RegionServer Flush Queue Monitoring Thresholds", "name": "regionserver_flush_queue_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "The amount of time to wait for the HBase Region Server to fully start up and connect to the HBase Master before enforcing the connectivity check.", "display_name": "HBase Region Server Connectivity Tolerance at Startup", "name": "regionserver_connectivity_tolerance", "value": "180" }, { "desc": "Where to store the contents of the BucketCache. Either \"offheap\" or file:/path/to/file -- this should be a file in the local file system (not HDFS), and is generally a file on ramdisk or SSD (not spinning disk). If this is set to \"offheap\" then Java's -XX:MaxDirectMemorySize is set to the value of hbase.bucketcache.size plus 1GB for short-circuit reads.", "display_name": "BucketCache IOEngine", "name": "hbase_bucketcache_ioengine", "value": null }, { "desc": "Whether to suppress the results of the Store File Index Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Store File Index Size", "name": "role_health_suppression_region_server_store_file_idx_size", "value": "false" }, { "desc": "The hashing algorithm for use in HashFunction. Two values are supported: 'murmur' (for MurmurHash) and 'jenkins' (for JenkinsHash).", "display_name": "HBase Hash Type", "name": "hbase_hash_type", "value": "murmur" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: RegionServer DNS Network Interface", "name": "role_config_suppression_hbase_regionserver_dns_interface", "value": "false" }, { "desc": "The port for the HBase RegionServer web UI. Set to -1 to disable RegionServer web UI.", "display_name": "HBase RegionServer Web UI port", "name": "hbase_regionserver_info_port", "value": "60030" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "regionserver_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The period over which to compute the moving average of the compaction queue size.", "display_name": "HBase RegionServer Compaction Queue Monitoring Period", "name": "regionserver_compaction_queue_window", "value": "5" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase RegionServer", "name": "hbase_regionserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "RegionServer Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The health test thresholds of the latency that the RegionServer sees for HDFS read operations", "display_name": "HBase RegionServer HDFS Read Latency Thresholds", "name": "regionserver_read_latency_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"50.0\"}" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds of the size used by the HStoreFile index. Specified as a percentage of the total heap size.", "display_name": "Percentage of Heap Used by HStoreFile Index", "name": "regionserver_store_file_idx_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "Maximum number of Write-Ahead Log (WAL) files. This value multiplied by HDFS Block Size (dfs.blocksize) is the size of the WAL that will need to be replayed when a server crashes. This value is inversely proportional to the frequency of flushes to disk.", "display_name": "Maximum number of Write-Ahead Log (WAL) files", "name": "hbase_regionserver_maxlogs", "value": "32" }, { "desc": "List of coprocessors that are loaded by default on all tables. For any override coprocessor method, these classes will be called in order. After implementing your own coprocessor, just put it in HBase's classpath and add the fully qualified class name here. A coprocessor can also be loaded on demand by setting HTableDescriptor.", "display_name": "HBase Coprocessor Region Classes", "name": "hbase_coprocessor_region_classes", "value": "" }, { "desc": "When BucketCache is enabled, use it as an L2 cache for LruBlockCache. If set to true, indexes and Bloom filters are kept in the LruBlockCache and the data blocks are kept in the BucketCache.", "display_name": "Enable Combined BucketCache", "name": "hbase_bucketcache_combinedcache_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_region_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_region_server_scm_health", "value": "false" }, { "desc": "The Hierarchical Storage Management policy that should be used by the RegionServer for the write-ahead-log. Using an SSD policy will have no effect unless HDFS HSM is configured to know which drives are SSDs. See Enabling HSM with HBase.", "display_name": "WAL HSM Storage Policy", "name": "hbase_regionserver_wal_storage_policy", "value": "NONE" }, { "desc": "Blocks writes if the size of the memstore increases to the value of 'hbase.hregion.block.memstore' multiplied by the value of 'hbase.hregion.flush.size' bytes. This setting is useful for preventing runaway memstore during spikes in update traffic. Without an upper-bound, memstore fills such that when it flushes, the resultant process of flushing files take a long time to compact or split, or worse, an \"out of memory\" error occurs.", "display_name": "HBase Memstore Block Multiplier", "name": "hbase_hregion_memstore_block_multiplier", "value": "2" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Sync the HLog to HDFS after this interval, in milliseconds, if it has not accumulated the number of HLog Entries specified to trigger a sync.", "display_name": "Synch Interval of HLog Entries", "name": "hbase_regionserver_optionallogflushinterval", "value": "1000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \"IPC Server handler.*took.*appending an edit to hlog.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"ABORTING region server serverName.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"DFSOutputStream ResponseProcessor exception.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Error Recovery for block blk.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Failed init\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Problem renewing lease for DFSClient.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"remote error telling master we are up\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Session.*for server.*closing socket connection and attempting reconnect.\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting .*compaction on region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_REGION_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^completed compaction on region (.+) after (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_REGION_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting compaction on (.+) in region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"COLUMN_FAMILY\", \"group1\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^completed compaction: regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Completed compaction: Request = regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^aborted compaction: regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_ABORTED\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Finished memstore flush of .+ for region (.+) in (.+), sequenceid=(.+), compaction requested=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DURATION\", \"group2\": \"SEQUENCE_ID\", \"group3\": \"COMPACTION_REQUESTED\" },\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Flush of region (.+) due to global heap pressure$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_DUE_TO_HEAP_PRESSURE\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"WARN\", \"content\": \"^Region (.+) has too many store files; delaying flush up to 90000ms$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_DELAYED_TOO_MANY_STORE_FILES\", \"attribute:SEVERITY\": \"CRITICAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting split of region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Running rollback/cleanup of failed split of (.+);.+$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_ABORTED\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Region split, hbase:meta updated, and report to master. Parent=(.+), new regions: (.+,.*,.+), (.+,.*,.+). Split took (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DAUGHTER_REGIONS\", \"group2\": \"DAUGHTER_REGIONS\", \"group3\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Region split, META updated, and report to master. Parent=(.+), new regions: (.+,.*,.+), (.+,.*,.+). Split took (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DAUGHTER_REGIONS\", \"group2\": \"DAUGHTER_REGIONS\", \"group3\": \"DURATION\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "The total size of the BucketCache, in megabytes. The size to configure depends on the amount of memory available to HBase, or the size of a local SSD. If hbase.bucketcache.ioengine is set to \"offheap\", then the bucketcache consumes the configured amount of memory from Java's Direct Memory.", "display_name": "BucketCache Size", "name": "hbase_bucketcache_size", "value": "1024" }, { "desc": "Whether to suppress the results of the Cluster Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Cluster Connectivity", "name": "role_health_suppression_region_server_master_connectivity", "value": "false" }, { "desc": "The host name or IP address of the DNS name server which a RegionServer should use to determine the host name used by the HBase Master for communication and display purposes.", "display_name": "RegionServer DNS Nameserver", "name": "hbase_regionserver_dns_nameserver", "value": null }, { "desc": "Number of threads for completing small compactions.", "display_name": "RegionServer Small Compactions Thread Count", "name": "hbase_regionserver_thread_compaction_small", "value": "1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds of the total size of RegionServer's memstores. Specified as a percentage of the configured upper limit. See Maximum Size of All Memstores in RegionServer.", "display_name": "HBase RegionServer Memstore Size Thresholds", "name": "regionserver_memstore_size_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Secure Bulk Load Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Secure Bulk Load Directory", "name": "role_config_suppression_hbase_bulkload_staging_dir", "value": "false" }, { "desc": "Percentage of maximum heap (-Xmx setting) to allocate to block cache used by HFile/StoreFile. To disable, set this value to 0 .", "display_name": "HFile Block Cache Size", "name": "hfile_block_cache_size", "value": "0.4" }, { "desc": "The maximum size, in megabytes, per log file for RegionServer logs. Typically used by log4j or logback.", "display_name": "RegionServer Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Memstore will be flushed to disk if size of the memstore exceeds this value in number of bytes. This value is checked by a thread that runs the frequency specified by hbase.server.thread.wakefrequency.", "display_name": "HBase Memstore Flush Size", "name": "hbase_hregion_memstore_flush_size", "value": "134217728" }, { "desc": "The HLog file reader implementation.", "display_name": "HLog Reader Implementation", "name": "hbase_regionserver_hlog_reader_impl", "value": null }, { "desc": "If this number of HStoreFiles in any one HStore is exceeded, then a compaction is run to rewrite all HStoreFiles files as one HStoreFile. (One HStoreFile is written per flush of memstore.) You can delay compaction by specifying a larger number, but the compaction will take longer when it does run. During a compaction, updates cannot be flushed to disk. Long compactions require memory sufficient to carry the logging of all updates across the duration of the compaction. If too large, clients timeout during compaction.", "display_name": "HStore Compaction Threshold", "name": "hbase_hstore_compactionThreshold", "value": "3" }, { "desc": "The period over which to compute the moving average of the HDFS sync latency of the HBase RegionServer.", "display_name": "HBase RegionServer HDFS Sync Latency Monitoring Period", "name": "regionserver_sync_latency_window", "value": "5" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_region_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Codecs parameter.", "display_name": "Suppress Parameter Validation: RegionServer Codecs", "name": "role_config_suppression_hbase_regionserver_codecs", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The maximum size byte array that should come from the MSLAB, in bytes.", "display_name": "Maximum Byte Array from MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_max_allocation", "value": "262144" }, { "desc": "The health test thresholds for the latency of HDFS write operations that the RegionServer detects", "display_name": "HBase RegionServer HDFS Sync Latency Thresholds", "name": "regionserver_sync_latency_thresholds", "value": "{\"critical\":\"5000.0\",\"warning\":\"500.0\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_region_server_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where RegionServer will place its log files.", "display_name": "RegionServer Log Directory", "name": "hbase_regionserver_log_dir", "value": "/var/log/hbase" }, { "desc": "The period of time that an HRegion will block updates after reaching the HStoreFile limit that is specified by 'hbase.hstore.blockingStoreFiles'. After this time has elapsed, the HRegion will stop blocking updates even if a compaction has not been completed.", "display_name": "HStore Blocking Wait Time", "name": "hbase_hstore_blockingWaitTime", "value": "90000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Flush Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Flush Queue Size", "name": "role_health_suppression_region_server_flush_queue", "value": "false" }, { "desc": "Timeout for Canary to perform its checks.", "display_name": "Canary Timeout", "name": "hbase_regionserver_canary_timeout", "value": "15" }, { "desc": "Duration between consecutive checks done by the Canary.", "display_name": "Canary Interval", "name": "hbase_regionserver_canary_interval", "value": "6" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Coprocessor Region Classes parameter.", "display_name": "Suppress Parameter Validation: HBase Coprocessor Region Classes", "name": "role_config_suppression_hbase_coprocessor_region_classes", "value": "false" }, { "desc": "The lease period, in milliseconds, for the HBase RegionServer. Clients must report in within this period or else they are considered dead.", "display_name": "HBase RegionServer Lease Period", "name": "hbase_regionserver_lease_period", "value": "60000" }, { "desc": "Period, in milliseconds, at which to roll the commit log.", "display_name": "HBase RegionServer Log Roll Period", "name": "hbase_regionserver_logroll_period", "value": "3600000" }, { "desc": "The minimum log level for RegionServer logs", "display_name": "RegionServer Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "regionserver_web_metric_collection_enabled", "value": "true" }, { "desc": "The name of the DNS Network Interface from which a RegionServer should report its IP address.", "display_name": "RegionServer DNS Network Interface", "name": "hbase_regionserver_dns_interface", "value": null }, { "desc": "The number of reservation blocks that are used to prevent unstable RegionServers caused by an OOME.", "display_name": "RegionServer Reservation Blocks", "name": "hbase_regionserver_nbreservationblocks", "value": "4" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HLog Reader Implementation parameter.", "display_name": "Suppress Parameter Validation: HLog Reader Implementation", "name": "role_config_suppression_hbase_regionserver_hlog_reader_impl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase RegionServer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase RegionServer", "name": "role_config_suppression_hbase_regionserver_java_opts", "value": "false" }, { "desc": "The implementation that should be used by the RegionServer for the write-ahead-log.", "display_name": "WAL Provider", "name": "hbase_regionserver_wal_provider", "value": "multiwal" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "regionserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The size of the chunks allocated by MSLAB, in bytes.", "display_name": "Chunk Size Allocated by MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_chunksize", "value": "2097152" }, { "desc": "Whether to suppress the results of the Memstore Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Memstore Size", "name": "role_health_suppression_region_server_memstore_size", "value": "false" }, { "desc": "If the memstores in a region are this size or larger when closing, run a pre-flush process to clear out memstores before putting up the region closed flag and taking the region offline. On close, a flush process is run under the close flag up to empty memory. During this time, the region is offline and no writes are taken. If the memstore content is large, the flush process could take a long time to complete. The pre-flush process cleans out the bulk of the memstore before putting up the close flag and taking the region offline, so that the flush process that runs under the close flag has little to do.", "display_name": "HBase Memstore Pre-close Flush Size", "name": "hbase_hregion_preclose_flush_size", "value": "5242880" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_regionserver_config_safety_valve", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: RegionServer Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period over which to compute the moving average of the HDFS read latency of the HBase RegionServer.", "display_name": "HBase RegionServer HDFS Read Latency Monitoring Period", "name": "regionserver_read_latency_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of HBase RegionServer in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of HBase RegionServer in Bytes", "name": "role_config_suppression_hbase_regionserver_java_heapsize", "value": "false" }, { "desc": "The directory for HBase secure bulk Load.", "display_name": "HBase Secure Bulk Load Directory", "name": "hbase_bulkload_staging_dir", "value": "/tmp/hbase-staging" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "An interface that is assignable to HRegionInterface. Used in HBase Client for opening a proxy to a remote HBase RegionServer.", "display_name": "HBase RegionServer Interface Class", "name": "hbase_regionserver_class", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase RegionServer Web UI Address parameter.", "display_name": "Suppress Parameter Validation: HBase RegionServer Web UI Address", "name": "role_config_suppression_hbase_regionserver_info_bindaddress", "value": "false" }, { "desc": "When true, HBase RegionServer will bind to 0.0.0.0. Only available in CDH 4.3 and later.", "display_name": "HBase RegionServer Bind to Wildcard Address", "name": "hbase_regionserver_bind_to_wildcard_address", "value": "true" }, { "desc": "The time, in milliseconds, between 'major' compactions of all HStoreFiles in a region. To disable automated major compactions, set this value to 0.", "display_name": "HBase HRegion Major Compaction", "name": "hbase_hregion_majorcompaction", "value": "604800000" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enable MemStore-Local Allocation Buffer (MSLAB) Allocation Scheme. Note: This feature is experimental in CDH3.", "display_name": "Enable MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_enabled", "value": "true" }, { "desc": "Enables the health test that the RegionServer is connected to the Master", "display_name": "HBase RegionServer to Master Connectivity Test", "name": "regionserver_master_connectivity_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": "32768" }, { "desc": "Limit for the number of regions after which no more region splitting should take place. This is not a hard limit for the number of regions but acts as a guideline for the RegionServer to stop splitting after a certain limit.", "display_name": "HBase Region Split Limit", "name": "hbase_regionserver_regionSplitLimit", "value": "2147483647" }, { "desc": "When using Multiple HDFS WAL as the WALProvider, sets how many write-ahead-logs each RegionServer should run. Will result in this number of HDFS pipelines. Writes for a given Region only go to a single pipeline, spreading total RegionServer load.", "display_name": "Per-RegionServer Number of WAL Pipelines", "name": "hbase_regionserver_wal_pipelines", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HLog Writer Implementation parameter.", "display_name": "Suppress Parameter Validation: HLog Writer Implementation", "name": "role_config_suppression_hbase_regionserver_hlog_writer_impl", "value": "false" }, { "desc": "Enables the health test that the RegionServer's process state is consistent with the role configuration", "display_name": "RegionServer Process Health Test", "name": "regionserver_scm_health_enabled", "value": "true" }, { "desc": "Interval, in milliseconds, between messages from the RegionServer to the HBase Master. Use a high value such as 3000 for clusters that have more than 10 hosts.", "display_name": "HBase RegionServer Message Interval", "name": "hbase_regionserver_msginterval", "value": "3000" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "When memstores are being forced to flush to make room in memory, keep flushing until this amount is reached. If this amount is equal to 'hbase.regionserver.global.memstore.upperLimit', then minimum possible flushing will occur when updates are blocked due to memstore limiting.", "display_name": "Low Watermark for Memstore Flush", "name": "hbase_regionserver_global_memstore_lowerLimit", "value": "0.38" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Log Directory parameter.", "display_name": "Suppress Parameter Validation: RegionServer Log Directory", "name": "role_config_suppression_hbase_regionserver_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the RegionServer File Descriptor Limit Validator configuration validator.", "display_name": "Suppress Configuration Validator: RegionServer File Descriptor Limit Validator", "name": "role_config_suppression_regionserver_fd_limit_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_region_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the BucketCache IOEngine parameter.", "display_name": "Suppress Parameter Validation: BucketCache IOEngine", "name": "role_config_suppression_hbase_bucketcache_ioengine", "value": "false" }, { "desc": "Whether asynchronous WAL replication to the secondary region replicas is enabled.\u2028 If enabled, a replication peer named 'region_replica_replication' is created\u2028 that tails the logs and replicates the mutations to region replicas for tables that\u2028 have region replication > 1. Disabling this replication also\u2028 requires disabling the replication peer using shell or the ReplicationAdmin Java class.\u2028 Replication to secondary region replicas works over standard intercluster replication. If disabled explicitly, enable replication by setting 'hbase.replication' to true for this feature to work.\u2028", "display_name": "Enable Replication To Secondary Region Replicas", "name": "hbase_region_replica_replication_enabled", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "regionserver_gc_duration_window", "value": "5" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "RegionServer Environment Advanced Configuration Snippet (Safety Valve)", "name": "REGIONSERVER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_region_server_host_health", "value": "false" }, { "desc": "Time period in seconds to reset long-running metrics (e.g. compactions). This is an HBase specific configuration.", "display_name": "Extended Period", "name": "hbase_metrics_extended_period", "value": "3600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Comma-separated list of codecs that the RegionServer requires to start. Use this setting to make sure that all RegionServers joining a cluster are installed with a particular set of codecs.", "display_name": "RegionServer Codecs", "name": "hbase_regionserver_codecs", "value": "" }, { "desc": "Maximum size of all memstores in a RegionServer before new updates are blocked and flushes are forced.", "display_name": "Maximum Size of All Memstores in RegionServer", "name": "hbase_regionserver_global_memstore_upperLimit", "value": "0.4" }, { "desc": "Number of handlers for processing priority requests in a RegionServer.", "display_name": "HBase RegionServer Meta-Handler Count", "name": "hbase_regionserver_metahandler_count", "value": "10" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_region_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_region_server_audit_health", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Compaction Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Compaction Queue Size", "name": "role_health_suppression_region_server_compaction_queue", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_region_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Jitter outer bound for major compactions.\u2028 On each RegionServer, the hbase.region.majorcompaction\u2028 interval is multiplied by a random fraction that is inside the bounds of this\u2028 maximum. This + or - product is added to when the next\u2028 major compaction is to run. Major compaction\u2028 should not occur on every RegionServer at the same time. The\u2028 smaller this number, the closer together the compactions.", "display_name": "HBase HRegion Major Compaction Jitter", "name": "hbase_hregion_majorcompaction_jitter", "value": "0.5" }, { "desc": "The period over which to compute the moving average of the flush queue size.", "display_name": "HBase RegionServer Flush Queue Monitoring Period", "name": "regionserver_flush_queue_window", "value": "5" }, { "desc": "Maximum HStoreFile size. If any one of a column families' HStoreFiles has grown to exceed this value, the hosting HRegion is split in two.", "display_name": "HBase Maximum File Size", "name": "hbase_hregion_max_filesize", "value": "10737418240" }, { "desc": "The health test thresholds of the weighted average size of the HBase RegionServer compaction queue over a recent period. See HBase RegionServer Compaction Queue Monitoring Period.", "display_name": "HBase RegionServer Compaction Queue Monitoring Thresholds", "name": "regionserver_compaction_queue_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_region_server_gc_duration", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Read threadpool size used by the RegionServer HBase IPC Server.", "display_name": "RegionServer IPC Read Threadpool Size", "name": "hbase_ipc_server_read_threadpool_size", "value": "10" }, { "desc": "If there are more than this number of HStoreFiles in any one HStore, then updates are blocked for this HRegion until a compaction is completed, or until the value specified for 'hbase.hstore.blockingWaitTime' has been exceeded.", "display_name": "HStore Blocking Store Files", "name": "hbase_hstore_blockingStoreFiles", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "regionserver_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The maximum number of rolled log files to keep for RegionServer logs. Typically used by log4j or logback.", "display_name": "RegionServer Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The HLog file writer implementation.", "display_name": "HLog Writer Implementation", "name": "hbase_regionserver_hlog_writer_impl", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer DNS Nameserver parameter.", "display_name": "Suppress Parameter Validation: RegionServer DNS Nameserver", "name": "role_config_suppression_hbase_regionserver_dns_nameserver", "value": "false" }, { "desc": "The port that an HBase RegionServer binds to.", "display_name": "HBase RegionServer Port", "name": "hbase_regionserver_port", "value": "60020" }, { "desc": "Number of RPC Server instances spun up on RegionServers.", "display_name": "HBase RegionServer Handler Count", "name": "hbase_regionserver_handler_count", "value": "30" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase RegionServer in Bytes", "name": "hbase_regionserver_java_heapsize", "value": "4294967296" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase RegionServer Interface Class parameter.", "display_name": "Suppress Parameter Validation: HBase RegionServer Interface Class", "name": "role_config_suppression_hbase_regionserver_class", "value": "false" }, { "desc": "When computing the overall RegionServer health, consider the host's health.", "display_name": "RegionServer Host Health Test", "name": "regionserver_host_health_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/impala-llama.json0000666000175100017510000007466313245514472025017 0ustar zuulzuul00000000000000[ { "desc": "Kerberos principal used by the Impala Llama ApplicationMaster roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "llama" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "ACL for Impala ApplicationMaster clients. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. \"alice,bob users,wheel\". A special value of \"*\" means all users are allowed. These take effect only if security is enabled.", "display_name": "Client ACLs", "name": "llama_am_server_thrift_client_acl", "value": "*" }, { "desc": "When computing the overall Impala Llama ApplicationMaster health, consider the host's health.", "display_name": "Impala Llama ApplicationMaster Host Health Test", "name": "llama_host_health_enabled", "value": "true" }, { "desc": "Port on which the Llama ApplicationMaster listens to HTTP requests.", "display_name": "Llama HTTP Port", "name": "llama_http_port", "value": "15001" }, { "desc": "Maximum number of retries for a client notification. After the maximum number of client notification retries has been reached without success the client is considered lost and all its reservations are released. A successful client notification resets the retries count.", "display_name": "Maximum Client Notification Retries", "name": "llama_am_server_thrift_client_notifier_max_retries", "value": "5" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Client notification retry interval, in milliseconds.", "display_name": "Client Notification Retry Interval", "name": "llama_am_server_thrift_client_notifier_retry_interval_ms", "value": "5000" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Maximum number of threads used by the Llama ApplicationMaster auxiliary service uses for serving client requests.", "display_name": "Thrift Server Maximum Threads", "name": "llama_am_server_thrift_server_max_threads", "value": "50" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve)", "name": "LLAMA_role_env_safety_valve", "value": null }, { "desc": "Interval of time without any new allocation that will trigger the Impala ApplicationMaster anti-deadlock logic.", "display_name": "Anti-Deadlock No Allocation Limit Interval", "name": "llama_am_gang_anti_deadlock_no_allocation_limit_ms", "value": "30000" }, { "desc": "Time in milliseconds after which Llama will discard its AM for a queue that has been empty of reservations. Does not apply to queues specified with the Core Queues property.", "display_name": "Queue Expiration Age", "name": "llama_am_queue_expire_ms", "value": "300000" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_llama_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "llama_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_llama_unexpected_exits", "value": "false" }, { "desc": "Enables the health test that the Impala Llama ApplicationMaster's process state is consistent with the role configuration", "display_name": "Impala Llama ApplicationMaster Process Health Test", "name": "llama_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where Llama will place its log files.", "display_name": "Llama Log Directory", "name": "llama_log_dir", "value": "/var/log/impala-llama" }, { "desc": "Whether Llama should cache allocated resources on release.", "display_name": "Enable Resource Caching", "name": "llama_am_cache_enabled", "value": "true" }, { "desc": "The user that the Llama process should run as.", "display_name": "Llama System User", "name": "process_username", "value": "llama" }, { "desc": "Queues Llama ApplicationMaster should connect to at start up.", "display_name": "Core Queues", "name": "llama_am_core_queues", "value": "" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama System Group parameter.", "display_name": "Suppress Parameter Validation: Llama System Group", "name": "role_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml", "name": "role_config_suppression_llama_config_valve", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Impala Llama ApplicationMaster logs. Typically used by log4j or logback.", "display_name": "Impala Llama ApplicationMaster Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Impala Llama ApplicationMaster in Bytes", "name": "llama_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Administrative Interface ACLs parameter.", "display_name": "Suppress Parameter Validation: Administrative Interface ACLs", "name": "role_config_suppression_llama_am_server_thrift_admin_acl", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Minimum number of threads used by the Llama ApplicationMaster auxiliary service uses for serving client requests.", "display_name": "Thrift Server Minimum Threads", "name": "llama_am_server_thrift_server_min_threads", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Core Queues parameter.", "display_name": "Suppress Parameter Validation: Core Queues", "name": "role_config_suppression_llama_am_core_queues", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Port on which the Llama ApplicationMaster serves its Thrift interface.", "display_name": "Llama Thrift Port", "name": "llama_port", "value": "15000" }, { "desc": "For advanced use only. A string to be inserted into llama-site.xml for this role only.", "display_name": "Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml", "name": "llama_config_valve", "value": null }, { "desc": "Socket time, in milliseconds, used Llama ApplicationMaster auxiliary service for all its server and client Thrift connections.", "display_name": "Thrift Transport Timeout", "name": "llama_am_server_thrift_transport_timeout_ms", "value": "60000" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The minimum log level for Impala Llama ApplicationMaster logs", "display_name": "Impala Llama ApplicationMaster Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_llama_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Log Directory parameter.", "display_name": "Suppress Parameter Validation: Llama Log Directory", "name": "role_config_suppression_llama_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_llama_file_descriptor", "value": "false" }, { "desc": "Enter an XML string that will be inserted verbatim into the Fair Scheduler allocations file. Overrides the configuration set using the Pools configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "llama_fair_scheduler_safety_valve", "value": null }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Minimum amount of time the backed off reservations will be in 'backed off' state. The actual amount time is a random value between the minimum and the maximum.", "display_name": "Anti-Deadlock Minimum Delay", "name": "llama_am_gang_anti_deadlock_min_delay_ms", "value": "10000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client ACLs parameter.", "display_name": "Suppress Parameter Validation: Client ACLs", "name": "role_config_suppression_llama_am_server_thrift_client_acl", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If enabled, the Impala Llama ApplicationMaster binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind Impala Llama ApplicationMaster to Wildcard Address", "name": "llama_bind_wildcard", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Llama Server", "name": "llama_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama System User parameter.", "display_name": "Suppress Parameter Validation: Llama System User", "name": "role_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_llama_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to break resource requests into smaller requests of standard size before the resource cache. The sizes are taken from Yarn settings Container Memory Increment and Container Virtual CPU Cores Increment.", "display_name": "Enable Resource Cache Normalization", "name": "llama_am_resource_normalizing_enabled", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "ACL for Impala ApplicationMaster admins. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. \"alice,bob users,wheel\". A special value of \"*\" means all users are allowed. These take effect only if security is enabled.", "display_name": "Administrative Interface ACLs", "name": "llama_am_server_thrift_admin_acl", "value": "*" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_llama_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Maximum amount of time the backed off reservations will be in 'backed off' state. The actual amount time is a random value between the minimum and the maximum.", "display_name": "Anti-Deadlock Maximum Delay", "name": "llama_am_gang_anti_deadlock_max_delay_ms", "value": "30000" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Port on which the Llama ApplicationMaster listens to administrative requests on its administrative Thrift interface.", "display_name": "Llama Thrift Admin Port", "name": "llama_am_server_thrift_admin_address", "value": "15002" }, { "desc": "The maximum number of rolled log files to keep for Impala Llama ApplicationMaster logs. Typically used by log4j or logback.", "display_name": "Impala Llama ApplicationMaster Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Timeout policy for resources being cached.", "display_name": "Resource Caching Idle Timeout", "name": "llama_am_cache_eviction_timeout_policy_idle_timeout_ms", "value": "30000" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The group that the Llama processes should run as.", "display_name": "Llama System Group", "name": "process_groupname", "value": "llama" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_llama_role_env_safety_valve", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Llama ApplicationMaster heartbeat interval, in milliseconds. On each heartbeat the ApplicationMaster submits new reservations to YARN ResourceManager and gets updates from it.", "display_name": "AM Heartbeat Interval", "name": "llama_am_server_thrift_client_notifier_heartbeat_ms", "value": "5000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_llama_fair_scheduler_safety_valve", "value": "false" }, { "desc": "Percentage of resources that will be backed off by the Impala ApplicationMaster anti-deadlock logic. Random reservations will be backed off until the percentage of backed off resources reaches this percentage.", "display_name": "Anti-Deadlock Backoff Percentage", "name": "llama_am_gang_anti_deadlock_backoff_percent", "value": "30" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_llama_audit_health", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_llama_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Llama Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Llama Server", "name": "role_config_suppression_llama_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/yarn-gateway.json0000666000175100017510000007114413245514472025067 0ustar zuulzuul00000000000000[ { "desc": "For MapReduce job outputs that are compressed, specify the compression codec to use. Will be part of generated client configuration.", "display_name": "Compression Codec of MapReduce Job Output", "name": "mapred_output_compression_codec", "value": "org.apache.hadoop.io.compress.DefaultCodec" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for mapred-site.xml.", "display_name": "MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "mapreduce_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Reduce Task Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: Reduce Task Java Opts Base", "name": "role_config_suppression_mapreduce_reduce_java_opts", "value": "false" }, { "desc": "Whether map tasks should attempt to use the optimized native implementation of the map-side output collector. This can improve performance of many jobs that are shuffle-intensive. Experimental in CDH 5.2.", "display_name": "Enable Optimized Map-side Output Collector", "name": "mapreduce_enable_native_map_output_collector", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Map Task Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Map Task Maximum Heap Size Validator", "name": "role_config_suppression_mapreduce_map_java_opts_max_heap_mapreduce_map_memory_mb_validator", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the ApplicationMaster Java Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: ApplicationMaster Java Maximum Heap Size Validator", "name": "role_config_suppression_yarn_app_mapreduce_am_max_heap_yarn_app_mapreduce_am_resource_mb_validator", "value": "false" }, { "desc": "The number of streams to merge at the same time while sorting files. That is, the number of sort heads to use during the merge sort on the reducer side. This determines the number of open file handles. Merging more files in parallel reduces merge sort iterations and improves run time by eliminating disk I/O. Note that merging more files in parallel uses more memory. If 'io.sort.factor' is set too high or the maximum JVM heap is set too low, excessive garbage collection will occur. The Hadoop default is 10, but Cloudera recommends a higher value. Will be part of generated client configuration.", "display_name": "I/O Sort Factor", "name": "io_sort_factor", "value": "64" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "92" }, { "desc": "Location to store the job history files of running jobs. This is a path on the host where the JobTracker is running.", "display_name": "Running Job History Location", "name": "hadoop_job_history_dir", "value": "/var/log/hadoop-mapreduce/history" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh", "display_name": "Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "mapreduce_client_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codec of MapReduce Job Output parameter.", "display_name": "Suppress Parameter Validation: Compression Codec of MapReduce Job Output", "name": "role_config_suppression_mapred_output_compression_codec", "value": "false" }, { "desc": "If enabled, multiple instances of some reduce tasks may be executed in parallel.", "display_name": "Reduce Tasks Speculative Execution", "name": "mapred_reduce_tasks_speculative_execution", "value": "false" }, { "desc": "The application framework to run jobs with. If not set, jobs will be run with the local job runner.", "display_name": "Application Framework", "name": "mapreduce_framework_name", "value": "yarn" }, { "desc": "Whether to suppress configuration warnings produced by the Job Submit Replication Validator configuration validator.", "display_name": "Suppress Configuration Validator: Job Submit Replication Validator", "name": "role_config_suppression_mapreduce_replication_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Redacted MapReduce Job Properties parameter.", "display_name": "Suppress Parameter Validation: Redacted MapReduce Job Properties", "name": "role_config_suppression_mapreduce_job_redacted_properties", "value": "false" }, { "desc": "Base sleep time between failover attempts. Used only if RM HA is enabled.", "display_name": "Client Failover Sleep Base Time", "name": "client_failover_sleep_base", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Running Job History Location parameter.", "display_name": "Suppress Parameter Validation: Running Job History Location", "name": "role_config_suppression_hadoop_job_history_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for yarn-site.xml.", "display_name": "YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_mapreduce_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Compression level for the codec used to compress MapReduce outputs. Default compression is a balance between speed and compression ratio.", "display_name": "Compression Level of Codecs", "name": "zlib_compress_level", "value": "DEFAULT_COMPRESSION" }, { "desc": "The maximum Java heap size, in bytes, of the map processes. This number will be formatted and concatenated with 'Map Task Java Opts Base' to pass to Hadoop.", "display_name": "Map Task Maximum Heap Size", "name": "mapreduce_map_java_opts_max_heap", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the Reduce Task Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Reduce Task Maximum Heap Size Validator", "name": "role_config_suppression_mapreduce_reduce_java_opts_max_heap_mapreduce_reduce_memory_mb_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Maximum Number of Attempts for MapReduce Jobs parameter.", "display_name": "Suppress Parameter Validation: Maximum Number of Attempts for MapReduce Jobs", "name": "role_config_suppression_mapreduce_am_max_attempts", "value": "false" }, { "desc": "The default number of parallel transfers run by reduce during the copy (shuffle) phase. This number is calculated by the following formula: min(number_of_nodes, n * min(number_of_cores_per_node, number_of_spindles_per_node)) where the n represents how many streams you want to run per core/spindle. A value of 10 for n is appropriate in most cases. Will be part of generated client configuration.", "display_name": "Default Number of Parallel Transfers During Shuffle", "name": "mapred_reduce_parallel_copies", "value": "10" }, { "desc": "The maximum heap size, in bytes, of the Java MapReduce ApplicationMaster. This number will be formatted and concatenated with 'ApplicationMaster Java Opts Base' to pass to Hadoop.", "display_name": "ApplicationMaster Java Maximum Heap Size", "name": "yarn_app_mapreduce_am_max_heap", "value": "825955249" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codec of MapReduce Map Output parameter.", "display_name": "Suppress Parameter Validation: Compression Codec of MapReduce Map Output", "name": "role_config_suppression_mapred_map_output_compression_codec", "value": "false" }, { "desc": "The replication level for submitted job files.", "display_name": "Mapreduce Submit Replication", "name": "mapred_submit_replication", "value": "10" }, { "desc": "The total amount of memory buffer, in megabytes, to use while sorting files. Note that this memory comes out of the user JVM heap size (meaning total user JVM heap - this amount of memory = total user usable heap space. Note that Cloudera's default differs from Hadoop's default; Cloudera uses a bigger buffer by default because modern machines often have more RAM. The smallest value across all TaskTrackers will be part of generated client configuration.", "display_name": "I/O Sort Memory Buffer (MiB)", "name": "io_sort_mb", "value": "256" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_yarn_client_config_safety_valve", "value": "false" }, { "desc": "Whether to enable ubertask optimization, which runs \"sufficiently small\" jobs sequentially within a single JVM. \"Small\" is defined by the mapreduce.job.ubertask.maxmaps, mapreduce.job.ubertask.maxreduces, and mapreduce.job.ubertask.maxbytes settings.", "display_name": "Enable Ubertask Optimization", "name": "mapreduce_job_ubertask_enabled", "value": "false" }, { "desc": "The ratio of heap size to container size for both map and reduce tasks. The heap should be smaller than the container size to allow for some overhead of the JVM.", "display_name": "Heap to Container Size Ratio", "name": "mapreduce_job_heap_memory_mb_ratio", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_mapreduce_client_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The amount of physical memory, in MiB, allocated for each reduce task of a job. For versions before CDH 5.5, if not specified, by default it is set to 1024. For CDH 5.5 and higher, a value less than 128 is not supported but if it is specified as 0, the amount of physical memory to request is inferred from Reduce Task Maximum Heap Size and Heap to Container Size Ratio. If Reduce Task Maximum Heap Size is not specified, by default the amount of physical memory to request is set to 1024. This parameter has no effect prior to CDH 4.4.", "display_name": "Reduce Task Memory", "name": "mapreduce_reduce_memory_mb", "value": "0" }, { "desc": "The maximum permissible size of the split metainfo file. The JobTracker won't attempt to read split metainfo files bigger than the configured value. No limits if set to -1.", "display_name": "JobTracker MetaInfo Maxsize", "name": "mapreduce_jobtracker_split_metainfo_maxsize", "value": "10000000" }, { "desc": "Fraction of the number of map tasks in the job which should be completed before reduce tasks are scheduled for the job.", "display_name": "Number of Map Tasks to Complete Before Reduce Tasks", "name": "mapred_reduce_slowstart_completed_maps", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Classpath parameter.", "display_name": "Suppress Parameter Validation: MR Application Classpath", "name": "role_config_suppression_mapreduce_application_classpath", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Environment parameter.", "display_name": "Suppress Parameter Validation: MR Application Environment", "name": "role_config_suppression_mapreduce_admin_user_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Environment parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Environment", "name": "role_config_suppression_yarn_app_mapreduce_am_admin_user_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Framework Path parameter.", "display_name": "Suppress Parameter Validation: MR Application Framework Path", "name": "role_config_suppression_mapreduce_application_framework_path", "value": "false" }, { "desc": "These are Java command-line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "mapreduce_client_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "The maximum Java heap size, in bytes, of the reduce processes. This number will be formatted and concatenated with 'Reduce Task Java Opts Base' to pass to Hadoop.", "display_name": "Reduce Task Maximum Heap Size", "name": "mapreduce_reduce_java_opts_max_heap", "value": "0" }, { "desc": "The physical memory requirement, in MiB, for the ApplicationMaster.", "display_name": "ApplicationMaster Memory", "name": "yarn_app_mapreduce_am_resource_mb", "value": "1024" }, { "desc": "A comma-separated list of job properties to redact in MapReduce.", "display_name": "Redacted MapReduce Job Properties", "name": "mapreduce_job_redacted_properties", "value": "fs.s3a.access.key,fs.s3a.secret.key" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "mapreduce_client_java_heapsize", "value": "825955249" }, { "desc": "The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string.", "display_name": "Mapreduce Task Timeout", "name": "mapred_task_timeout", "value": "600000" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Environment variables for the MapReduce ApplicationMaster. These settings can be overridden in the ApplicationMaster User Environment (yarn.app.mapreduce.am.env).", "display_name": "ApplicationMaster Environment", "name": "yarn_app_mapreduce_am_admin_user_env", "value": "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH" }, { "desc": "Limit on the number of counter groups allowed per job.", "display_name": "Job Counter Groups Limit", "name": "mapreduce_job_counter_groups_limit", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Map Task Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: Map Task Java Opts Base", "name": "role_config_suppression_mapreduce_map_java_opts", "value": "false" }, { "desc": "The amount of physical memory, in MiB, allocated for each map task of a job. For versions before CDH 5.5, if not specified, by default it is set to 1024. For CDH 5.5 and higher, a value less than 128 is not supported but if it is specified as 0, the amount of physical memory to request is inferred from Map Task Maximum Heap Size and Heap to Container Size Ratio. If Map Task Maximum Heap Size is not specified, by default the amount of physical memory to request is set to 1024.", "display_name": "Map Task Memory", "name": "mapreduce_map_memory_mb", "value": "0" }, { "desc": "Threshold for number of reduces, beyond which a job is considered too big for ubertask optimization. Note: As of CDH 5, MR2 does not support more than one reduce in an ubertask. (Zero is valid.)", "display_name": "Ubertask Maximum Reduces", "name": "mapreduce_job_ubertask_maxreduces", "value": "1" }, { "desc": "The number of virtual CPU cores for each reduce task of a job.", "display_name": "Reduce Task CPU Virtual Cores", "name": "mapreduce_reduce_cpu_vcores", "value": "1" }, { "desc": "For MapReduce map outputs that are compressed, specify the compression codec to use. Will be part of generated client configuration.", "display_name": "Compression Codec of MapReduce Map Output", "name": "mapred_map_output_compression_codec", "value": "org.apache.hadoop.io.compress.SnappyCodec" }, { "desc": "Classpaths to include for MapReduce applications. During evaluation, the string '{version}' in the value of this parameter will be replaced by the actual MapReduce version.", "display_name": "MR Application Classpath", "name": "mapreduce_application_classpath", "value": "$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH" }, { "desc": "Java command line arguments passed to the MapReduce ApplicationMaster.", "display_name": "ApplicationMaster Java Opts Base", "name": "yarn_app_mapreduce_am_command_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Maximum sleep time between failover attempts. Used only if RM HA is enabled.", "display_name": "Client Failover Sleep Max Time", "name": "client_failover_sleep_max", "value": "2000" }, { "desc": "Path for MapReduce application framework. During evaluation, the string '{version}' in the value of this parameter will be replaced by the actual MapReduce version.", "display_name": "MR Application Framework Path", "name": "mapreduce_application_framework_path", "value": "/user/yarn/mapreduce/mr-framework/{version}-mr-framework.tar.gz#mr-framework" }, { "desc": "Compress the output of MapReduce jobs. Will be part of generated client configuration.", "display_name": "Compress MapReduce Job Output", "name": "mapred_output_compress", "value": "false" }, { "desc": "For MapReduce job outputs that are compressed as SequenceFiles, you can select one of these compression type options: NONE, RECORD or BLOCK. Cloudera recommends BLOCK. Will be part of generated client configuration.", "display_name": "Compression Type of MapReduce Job Output", "name": "mapred_output_compression_type", "value": "BLOCK" }, { "desc": "Java opts for the map processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp pass a value of: \"-verbose:gc -Xloggc:/tmp/@taskid@.gc\". The configuration variable 'Map Task Memory' can be used to control the maximum memory of the map processes.", "display_name": "Map Task Java Opts Base", "name": "mapreduce_map_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Java Opts Base", "name": "role_config_suppression_yarn_app_mapreduce_am_command_opts", "value": "false" }, { "desc": "Additional execution environment entries for map and reduce task processes.", "display_name": "MR Application Environment", "name": "mapreduce_admin_user_env", "value": "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH" }, { "desc": "The soft limit in either the buffer or record collection buffers. When this limit is reached, a thread will begin to spill the contents to disk in the background. Note that this does not imply any chunking of data to the spill. A value less than 0.5 is not recommended. The syntax is in decimal units; the default is 80% and is formatted 0.8. Will be part of generated client configuration.", "display_name": "I/O Sort Spill Percent", "name": "io_sort_spill_percent", "value": "0.8" }, { "desc": "The default number of reduce tasks per job. Will be part of generated client configuration.", "display_name": "Default Number of Reduce Tasks per Job", "name": "mapred_reduce_tasks", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Limit on the number of counters allowed per job.", "display_name": "Job Counters Limit", "name": "mapreduce_job_counters_limit", "value": "120" }, { "desc": "The virtual CPU cores requirement, for the ApplicationMaster. This parameter has no effect prior to CDH 4.4.", "display_name": "ApplicationMaster Virtual CPU Cores", "name": "yarn_app_mapreduce_am_resource_cpu_vcores", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the I/O Sort Factor parameter.", "display_name": "Suppress Parameter Validation: I/O Sort Factor", "name": "role_config_suppression_io_sort_factor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "role_config_suppression_mapreduce_client_env_safety_valve", "value": "false" }, { "desc": "Size of buffer for read and write operations of SequenceFiles.", "display_name": "SequenceFile I/O Buffer Size", "name": "io_file_buffer_size", "value": "65536" }, { "desc": "If enabled, multiple instances of some map tasks may be executed in parallel.", "display_name": "Map Tasks Speculative Execution", "name": "mapred_map_tasks_speculative_execution", "value": "false" }, { "desc": "The number of virtual CPU cores allocated for each map task of a job. This parameter has no effect prior to CDH 4.4.", "display_name": "Map Task CPU Virtual Cores", "name": "mapreduce_map_cpu_vcores", "value": "1" }, { "desc": "Java opts for the reduce processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp pass a value of: \"-verbose:gc -Xloggc:/tmp/@taskid@.gc\". The configuration variable 'Reduce Task Memory' can be used to control the maximum memory of the reduce processes.", "display_name": "Reduce Task Java Opts Base", "name": "mapreduce_reduce_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "If enabled, uses compression on the map outputs before they are sent across the network. Will be part of generated client configuration.", "display_name": "Use Compression on Map Outputs", "name": "mapred_compress_map_output", "value": "true" }, { "desc": "The maximum number of application attempts for MapReduce jobs. The value of this parameter overrides ApplicationMaster Maximum Attempts for MapReduce jobs.", "display_name": "Maximum Number of Attempts for MapReduce Jobs", "name": "mapreduce_am_max_attempts", "value": "2" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Threshold for number of maps, beyond which a job is considered too big for ubertask optimization.", "display_name": "Ubertask Maximum Maps", "name": "mapreduce_job_ubertask_maxmaps", "value": "9" }, { "desc": "File format the AM will use when generating the .jhist file. Valid values are \"json\" for text output and \"binary\" for faster parsing.", "display_name": "JHist File Format", "name": "mapred_jobhistory_jhist_format", "value": "binary" }, { "desc": "Threshold for number of input bytes, beyond which a job is considered too big for ubertask optimization. If no value is specified, dfs.block.size is used as a default.", "display_name": "Ubertask Maximum Job Size", "name": "mapreduce_job_ubertask_maxbytes", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hue-hue_server.json0000666000175100017510000006054213245514472025405 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Top Banner Custom HTML parameter.", "display_name": "Suppress Parameter Validation: Top Banner Custom HTML", "name": "role_config_suppression_banner_html", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hue TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hue TLS/SSL Validator", "name": "role_config_suppression_hue_ssl_validator", "value": "false" }, { "desc": "Encrypt communication between clients and Hue using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Hue", "name": "ssl_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_ssl_private_key", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hue_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "Hue Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hue_server_hive_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Private Key Password", "name": "role_config_suppression_ssl_private_key_password", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hue_server_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Directory where Hue Server will place its log files.", "display_name": "Hue Server Log Directory", "name": "hue_server_log_dir", "value": "/var/log/hue" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_hue_server_audit_health", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hue_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hue Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hue_server_role_env_safety_valve", "value": "false" }, { "desc": "Timeout in seconds for Thrift calls to HiveServer2 and Impala.", "display_name": "HiveServer2 and Impala Thrift Connection Timeout", "name": "hs2_conn_timeout", "value": "120" }, { "desc": "An optional, custom one-line HTML code to display as a banner on top of all Hue Server web pages. Useful in displaying cluster identity of the Hue Server.", "display_name": "Top Banner Custom HTML", "name": "banner_html", "value": null }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_hue_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Server Log Directory", "name": "role_config_suppression_hue_server_log_dir", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "hue_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hue_server_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hue_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hue_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Location on HDFS where the jobsub examples and templates are stored.", "display_name": "Jobsub Examples and Templates Directory", "name": "hue_server_remote_data_dir", "value": "/user/hue/jobsub" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The path to the TLS/SSL file containing the certificate of the certificate authority (CA) and any intermediate certificates used to sign the server certificate. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format, and is usually created by concatenating all of the appropriate root and intermediate certificates.", "display_name": "Hue TLS/SSL Server CA Certificate (PEM Format)", "name": "ssl_cacerts", "value": null }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_ssl_certificate", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hue_server_host_health", "value": "false" }, { "desc": "When computing the overall Hue Server health, consider the host's health.", "display_name": "Hue Server Host Health Test", "name": "hue_server_host_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to impalad_flags for this role only. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Hue Server Advanced Configuration Snippet (Safety Valve) for impalad_flags", "name": "hue_impalad_flags_safety_valve", "value": null }, { "desc": "Port to use to connect to the Hue server.", "display_name": "Hue HTTP Port", "name": "hue_http_port", "value": "8888" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Jobsub Examples and Templates Directory parameter.", "display_name": "Suppress Parameter Validation: Jobsub Examples and Templates Directory", "name": "role_config_suppression_hue_server_remote_data_dir", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file.", "display_name": "Metrics Sample File Logging Frequency", "name": "hue_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Secret Key parameter.", "display_name": "Suppress Parameter Validation: Secret Key", "name": "role_config_suppression_secret_key", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Hue Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HUE_SERVER_role_env_safety_valve", "value": null }, { "desc": "For advanced use only. A string to be inserted into hue_safety_valve_server.ini for this role only.", "display_name": "Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini", "name": "hue_server_hue_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hue_metrics_sample_file_location", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_private_key", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Advanced Configuration Snippet (Safety Valve) for impalad_flags parameter.", "display_name": "Suppress Parameter Validation: Hue Server Advanced Configuration Snippet (Safety Valve) for impalad_flags", "name": "role_config_suppression_hue_impalad_flags_safety_valve", "value": "false" }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hue/metrics-hue_server/metrics.log.", "display_name": "Metrics Sample File Location", "name": "hue_metrics_sample_file_location", "value": null }, { "desc": "The password for the private key in the Hue TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Hue TLS/SSL Private Key Password", "name": "ssl_private_key_password", "value": null }, { "desc": "If enabled, the Hue server binds to the wildcard address (\"0.0.0.0\") for its ports.", "display_name": "Bind Hue Server to Wildcard Address", "name": "hue_server_bind_wildcard", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini parameter.", "display_name": "Suppress Parameter Validation: Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini", "name": "role_config_suppression_hue_server_hue_safety_valve", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "hue_server_web_metric_collection_enabled", "value": "true" }, { "desc": "Enables the health test that the Hue Server's process state is consistent with the role configuration", "display_name": "Hue Server Process Health Test", "name": "hue_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server CA Certificate (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server CA Certificate (PEM Format)", "name": "role_config_suppression_ssl_cacerts", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_certificate", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hue_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hue Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hue_server_hive_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Random string used for secure hashing in the session store.", "display_name": "Secret Key", "name": "secret_key", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hive-metastore-sentry-safety.xml0000666000175100017510000000135213245514472030047 0ustar zuulzuul00000000000000 hive.metastore.client.impl org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient Sets custom Hive metastore client which Sentry uses to filter out metadata. hive.metastore.pre.event.listeners org.apache.sentry.binding.metastore.MetastoreAuthzBinding list of comma separated listeners for metastore events. hive.metastore.event.listeners org.apache.sentry.binding.metastore.SentryMetastorePostEventListener list of comma separated listeners for metastore, post events. sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hive-webhcat.json0000666000175100017510000005314113245514472025022 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_webhcat_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_webhcat_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_webhcat_log_directory_free_space", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "webhcat_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Port on which WebHCat Server will listen for connections.", "display_name": "WebHCat Server Port", "name": "hive_webhcat_address_port", "value": "50111" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_webhcat_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The maximum number of rolled log files to keep for WebHCat Server logs. Typically used by log4j or logback.", "display_name": "WebHCat Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Directory where WebHCat Server will place its log files.", "display_name": "WebHCat Server Log Directory", "name": "hcatalog_log_dir", "value": "/var/log/hcatalog" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of WebHCat Server in Bytes", "name": "hive_webhcat_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Log Directory", "name": "role_config_suppression_hcatalog_log_dir", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_webhcat_hive_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_webhcat_scm_health", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "WebHCat Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "WebHCat Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_webhcat_env_safety_valve", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for WebHCat Server", "name": "hive_webhcat_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_webhcat_audit_health", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Enables the health test that the WebHCat Server's process state is consistent with the role configuration", "display_name": "WebHCat Server Process Health Test", "name": "webhcat_scm_health_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_webhcat_swap_memory_usage", "value": "false" }, { "desc": "When computing the overall WebHCat Server health, consider the host's health.", "display_name": "WebHCat Server Host Health Test", "name": "webhcat_host_health_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into webhcat-site.xml for this role only.", "display_name": "WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml", "name": "hive_webhcat_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_webhcat_hive_config_safety_valve", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for WebHCat Server logs. Typically used by log4j or logback.", "display_name": "WebHCat Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_webhcat_host_health", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_webhcat_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for WebHCat Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for WebHCat Server", "name": "role_config_suppression_hive_webhcat_java_opts", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml", "name": "role_config_suppression_hive_webhcat_config_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for WebHCat Server logs", "display_name": "WebHCat Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/mapred-site.xml0000666000175100017510000001050113245514472024506 0ustar zuulzuul00000000000000 mapreduce.job.split.metainfo.maxsize 10000000 mapreduce.job.counters.max 120 mapreduce.output.fileoutputformat.compress false mapreduce.output.fileoutputformat.compress.type BLOCK mapreduce.output.fileoutputformat.compress.codec org.apache.hadoop.io.compress.DefaultCodec mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.compress true zlib.compress.level DEFAULT_COMPRESSION mapreduce.task.io.sort.factor 64 mapreduce.map.sort.spill.percent 0.8 mapreduce.reduce.shuffle.parallelcopies 10 mapreduce.task.timeout 600000 mapreduce.client.submit.file.replication 10 mapreduce.job.reduces 1 mapreduce.task.io.sort.mb 256 mapreduce.map.speculative false mapreduce.reduce.speculative false mapreduce.job.reduce.slowstart.completedmaps 0.8 mapreduce.jobhistory.address test-master-001.novalocal:10020 mapreduce.jobhistory.webapp.address test-master-001.novalocal:19888 mapreduce.jobhistory.webapp.https.address test-master-001.novalocal:19890 mapreduce.jobhistory.admin.address test-master-001.novalocal:10033 mapreduce.framework.name yarn yarn.app.mapreduce.am.staging-dir /user yarn.app.mapreduce.am.resource.mb 1024 yarn.app.mapreduce.am.resource.cpu-vcores 1 mapreduce.job.ubertask.enable false yarn.app.mapreduce.am.command-opts -Djava.net.preferIPv4Stack=true -Xmx825955249 mapreduce.map.java.opts -Djava.net.preferIPv4Stack=true -Xmx825955249 mapreduce.reduce.java.opts -Djava.net.preferIPv4Stack=true -Xmx825955249 yarn.app.mapreduce.am.admin.user.env LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH mapreduce.map.memory.mb 1024 mapreduce.map.cpu.vcores 1 mapreduce.reduce.memory.mb 1024 mapreduce.reduce.cpu.vcores 1 mapreduce.application.classpath $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH mapreduce.admin.user.env LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH mapreduce.shuffle.max.connections 80 sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/sqoop-service.json0000666000175100017510000001600113245514472025245 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_sqoop_env_safety_valve", "value": "false" }, { "desc": "When computing the overall SQOOP health, consider Sqoop 2 Server's health", "display_name": "Sqoop 2 Server Role Health Test", "name": "sqoop_sqoop_server_health_enabled", "value": "true" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "sqoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Sqoop 2 Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Sqoop 2 Server Count Validator", "name": "service_config_suppression_sqoop_server_count_validator", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "sqoop_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "sqoop2" }, { "desc": "Whether to suppress the results of the Sqoop 2 Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sqoop 2 Server Health", "name": "service_health_suppression_sqoop_sqoop_server_health", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "sqoop2" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "MapReduce jobs are run against this service.", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/solr-gateway.json0000666000175100017510000000437313245514472025075 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/solr" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/impala-impalad.json0000666000175100017510000012770713245514472025336 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Impala Daemon Scratch Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Scratch Directories Free Space", "name": "role_health_suppression_impalad_scratch_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Lineage Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Lineage Enabled Validator", "name": "role_config_suppression_impala_lineage_enabled_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemons Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Impala Daemons Load Balancer", "name": "role_config_suppression_impalad_load_balancer", "value": "false" }, { "desc": "The health test thresholds for failures encountered when monitoring queries within a recent period specified by the impala_query_monitoring_failure_window configuration for the role.", "display_name": "Query Monitoring Failures Thresholds", "name": "impala_query_monitoring_failure_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_hdfs_site_conf_safety_valve", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Impala Daemon command-line flags. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "impalad_cmd_args_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Impala Audit Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Audit Enabled Validator", "name": "role_config_suppression_impala_audit_enabled_validator", "value": "false" }, { "desc": "Directory where an Impala Daemon core dump is placed.", "display_name": "Impala Daemon Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/impalad" }, { "desc": "Port on which HiveServer2 client requests are served by Impala Daemons.", "display_name": "Impala Daemon HiveServer2 Port", "name": "hs2_port", "value": "21050" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_fair_scheduler_safety_valve", "value": "false" }, { "desc": "Maximum number of seconds that Impala attempts to register or re-register with Llama. If registration is unsuccessful, Impala cancels the action with an error, which could result in an impalad startup failure or a cancelled query. A setting of -1 seconds means try indefinitely.", "display_name": "Llama Registration Timeout Seconds", "name": "llama_registration_timeout_secs", "value": "30" }, { "desc": "When computing the overall Impala Daemon health, consider the host's health.", "display_name": "Impala Daemon Host Health Test", "name": "impalad_host_health_enabled", "value": "true" }, { "desc": "The directory in which Impala daemon audit event log files are written. If \"Impala Audit Event Generation\" property is enabled, Impala will generate its audit logs in this directory.", "display_name": "Impala Daemon Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/impalad/audit" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "A list of key-value pairs of additional query options to pass to the Impala Daemon command line, separated by ','.", "display_name": "Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve)", "name": "default_query_options", "value": "" }, { "desc": "Port on which Beeswax client requests are served by Impala Daemons.", "display_name": "Impala Daemon Beeswax Port", "name": "beeswax_port", "value": "21000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Enable or disable the Impala Daemon web server. This web server contains useful information about Impala Daemon.", "display_name": "Enable Impala Daemon Web Server", "name": "impalad_enable_webserver", "value": "true" }, { "desc": "An XML snippet to append to llama-site.xml for Impala Daemons. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve)", "name": "impala_llama_site_conf_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "impalad_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)", "name": "IMPALAD_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "Enables the health test that verifies the Impala Daemon is connected to the StateStore.", "display_name": "Impala Daemon Connectivity Health Test", "name": "impalad_connectivity_health_enabled", "value": "true" }, { "desc": "Enables the health check that determines if query collection for the Impala Daemon is successful.", "display_name": "Impala Daemon Query Collection Status Health Check", "name": "impala_query_monitoring_status_check_enabled", "value": "true" }, { "desc": "The amount of time at Impala Daemon startup allowed for the Impala Daemon to start accepting new queries for processing.", "display_name": "Impala Daemon Ready Status Startup Tolerance", "name": "impalad_ready_status_check_startup_tolerance", "value": "180" }, { "desc": "The maximum size (in queries) of the Impala Daemon audit event log file before a new one is created.", "display_name": "Impala Daemon Maximum Audit Log File Size", "name": "max_audit_event_log_file_size", "value": "5000" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_impalad_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Local UDF Library Dir parameter.", "display_name": "Suppress Parameter Validation: Local UDF Library Dir", "name": "role_config_suppression_local_library_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Server CA Certificate parameter.", "display_name": "Suppress Parameter Validation: LDAP Server CA Certificate", "name": "role_config_suppression_impalad_ldap_ca_certificate", "value": "false" }, { "desc": "The directory in which Impala daemon lineage log files are written. If \"Impala Lineage Generation\" property is enabled, Impala generates its lineage logs in this directory.", "display_name": "Impala Daemon Lineage Log Directory", "name": "lineage_event_log_dir", "value": "/var/log/impalad/lineage" }, { "desc": "The password for the private key in the Impala Daemon Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Impala Daemon Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_llama_site_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Scratch Directories parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Scratch Directories", "name": "role_config_suppression_scratch_dirs", "value": "false" }, { "desc": "Number of retry attempts the Impala Daemon will make when connecting to the Hive Metastore Server. By default, the Impala Daemon waits one second between retries.", "display_name": "Impala Daemon Hive Metastore Connection Retries", "name": "hive_metastore_connect_retries", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Breakpad Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Breakpad Dump Directory", "name": "role_config_suppression_minidump_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "User-defined function (UDF) libraries are copied from HDFS into this local directory.", "display_name": "Local UDF Library Dir", "name": "local_library_dir", "value": "/var/lib/impala/udfs" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Audit Log Directory", "name": "role_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "impalad_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_impalad_unexpected_exits", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Impala Daemon logs. Typically used by log4j or logback.", "display_name": "Impala Daemon Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Username for Impala Daemon webserver authentication.", "display_name": "Impala Daemon Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Password for Impala Daemon webserver authentication.", "display_name": "Impala Daemon Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Directory where Impala Daemon will place its log files.", "display_name": "Impala Daemon Log Directory", "name": "log_dir", "value": "/var/log/impalad" }, { "desc": "An XML string to use verbatim for the contents of fair-scheduler.xml for Impala Daemons. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve)", "name": "impalad_fair_scheduler_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_cmd_args_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_default_query_options", "value": "false" }, { "desc": "Port on which ImpalaBackendService is exported.", "display_name": "Impala Daemon Backend Port", "name": "be_port", "value": "22000" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml for this role only.", "display_name": "Impala Daemon HBase Advanced Configuration Snippet (Safety Valve)", "name": "impalad_hbase_conf_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml for this role only.", "display_name": "Impala Daemon Hive Advanced Configuration Snippet (Safety Valve)", "name": "impala_hive_conf_safety_valve", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_impalad_audit_health", "value": "false" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon HBase Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon HBase Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_hbase_conf_safety_valve", "value": "false" }, { "desc": "Directory for storing Impala Daemon Breakpad dumps.", "display_name": "Impala Daemon Breakpad Dump Directory", "name": "minidump_path", "value": "/var/log/impala-minidumps" }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above. ", "display_name": "Impala Daemon Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_impalad_memory_rss_health", "value": "false" }, { "desc": "Encrypt and verify the integrity of all data spilled to disk as part of a query. This feature is only supported for Impala 2.0 and higher and CDH 5.2 and higher (which includes Impala 2.0).", "display_name": "Disk Spill Encryption", "name": "disk_spill_encryption", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_impalad_swap_memory_usage", "value": "false" }, { "desc": "The timeout used by the Cloudera Manager Agent's query monitor when communicating with the Impala Daemon web server, specified in seconds.", "display_name": "Query Monitoring Timeout", "name": "executing_queries_timeout_seconds", "value": "5.0" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_impalad_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Hive Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Hive Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_hive_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "Enables audit event generation by Impala daemons. The audit log file will be placed in the directory specified by 'Impala Daemon Audit Log Directory' parameter.", "display_name": "Enable Impala Audit Event Generation", "name": "enable_audit_event_log", "value": "false" }, { "desc": "Number of seconds to wait between attempts during Llama registration.", "display_name": "Llama Registration Wait Seconds", "name": "llama_registration_wait_secs", "value": "3" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_impalad_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_impalad_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Daemon Ready Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Ready Check", "name": "role_health_suppression_impalad_ready_status", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"any\"}" }, { "desc": "Enables the health check that determines if the Impala daemon is ready to process queries.", "display_name": "Impala Daemon Ready Status Health Check", "name": "impalad_ready_status_check_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "The polling period of the Impala query monitor in the Cloudera Manager Agent, specified in seconds. If set to zero, query monitoring is disabled.", "display_name": "Query Monitoring Period", "name": "query_monitoring_period_seconds", "value": "1.0" }, { "desc": "Whether to suppress the results of the Query Monitoring Status Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Query Monitoring Status Check", "name": "role_health_suppression_impalad_query_monitoring_status", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_impalad_log_directory_free_space", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Impala Daemon Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Impala Daemon Scratch Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Impala Daemon Scratch Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Impala Daemon Scratch Directories Free Space Monitoring Percentage Thresholds", "name": "impalad_scratch_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Maximum number of query results a client may request to be cached on a per-query basis to support restarting fetches. This option guards against unreasonably large result caches requested by clients. Requests exceeding this maximum will be rejected.", "display_name": "Result Cache Maximum Size", "name": "impalad_result_cache_max_size", "value": "100000" }, { "desc": "Whether to suppress the results of the StateStore Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: StateStore Connectivity", "name": "role_health_suppression_impalad_connectivity", "value": "false" }, { "desc": "Abort Impala startup if there are improper configs or running on unsupported hardware.", "display_name": "Abort on Config Error", "name": "abort_on_config_error", "value": "true" }, { "desc": "Directories where Impala Daemon will write data such as spilling information to disk to free up memory. This can potentially be large amounts of data.", "display_name": "Impala Daemon Scratch Directories", "name": "scratch_dirs", "value": null }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "Impala Daemon Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "Enables lineage generation by Impala daemons. The lineage log file is placed in the directory specified by the 'Impala Daemon Lineage Log Directory' parameter.", "display_name": "Enable Impala Lineage Generation", "name": "enable_lineage_log", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_impalad_host_health", "value": "false" }, { "desc": "The maximum size (in entries) of the Impala daemon lineage log file before a new one is created.", "display_name": "Impala Daemon Maximum Lineage Log File Size", "name": "max_lineage_log_file_size", "value": "5000" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Impala Daemon Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "The location on disk of the certificate, in .pem format, used to confirm the authenticity of the LDAP server certificate. This is the Certificate Authority (CA) certificate, and it was used to sign the LDAP server certificate. If not set, all certificates are trusted, which means that an attacker could potentially intercept otherwise encrypted usernames and passwords.", "display_name": "LDAP Server CA Certificate", "name": "impalad_ldap_ca_certificate", "value": null }, { "desc": "Timeout for requests to the Hive Metastore Server from Impala. Consider increasing this if you have tables with a lot of metadata and see timeout errors.", "display_name": "Impala Daemon Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "3600" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The period to review when computing query monitoring failures.", "display_name": "Monitoring Period For Query Monitoring Failures", "name": "impala_query_monitoring_failure_window", "value": "5" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Maximum number of Breakpad dump files stored by Impala Daemon Role.", "display_name": "Impala Daemon Max Breakpad Dump Files", "name": "max_minidumps", "value": "9" }, { "desc": "Port where StateStoreSubscriberService is running.", "display_name": "StateStoreSubscriber Service Port", "name": "state_store_subscriber_port", "value": "23000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Lineage Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Lineage Log Directory", "name": "role_config_suppression_lineage_event_log_dir", "value": "false" }, { "desc": "Address of the load balancer used for Impala daemons. Should be specified in host:port format. If this is specified and Kerberos is enabled, Cloudera Manager adds a principal for 'impala/<load_balancer_host>@<realm>' to the keytab for all Impala daemons.", "display_name": "Impala Daemons Load Balancer", "name": "impalad_load_balancer", "value": null }, { "desc": "Maximum number of times a request to reserve, expand, or release resources is attempted until the request is cancelled.", "display_name": "Llama Maximum Request Attempts", "name": "llama_max_request_attempts", "value": "5" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The minimum log level for Impala Daemon logs", "display_name": "Impala Daemon Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Port where Impala debug web server runs.", "display_name": "Impala Daemon HTTP Server Port", "name": "impalad_webserver_port", "value": "25000" }, { "desc": "Enables the health test that the Impala Daemon's process state is consistent with the role configuration", "display_name": "Impala Daemon Process Health Test", "name": "impalad_scm_health_enabled", "value": "true" }, { "desc": "Memory limit in bytes for Impala Daemon, enforced by the daemon itself. If reached, queries running on the Impala Daemon may be killed. Leave it blank to let Impala pick its own limit. Use a value of -1 B to specify no limit.", "display_name": "Impala Daemon Memory Limit", "name": "impalad_memory_limit", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of time to wait for the Impala Daemon to fully start up and connect to the StateStore before enforcing the connectivity check.", "display_name": "Impala Daemon Connectivity Tolerance at Startup", "name": "impalad_connectivity_tolerance", "value": "180" }, { "desc": "Port where Llama notification callback should be started", "display_name": "Llama Callback Port", "name": "llama_callback_port", "value": "28000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Impala Daemon Scratch Directories.", "display_name": "Impala Daemon Scratch Directories Free Space Monitoring Absolute Thresholds", "name": "impalad_scratch_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml for this role only.", "display_name": "Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve)", "name": "impala_hdfs_site_conf_safety_valve", "value": null }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "impalad_web_metric_collection_enabled", "value": "true" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "Impala Maximum Log Files", "name": "max_log_files", "value": "10" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-namenode.json0000666000175100017510000016057613245514472025177 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Checkpoint Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Checkpoint Status", "name": "role_health_suppression_name_node_ha_checkpoint_age", "value": "false" }, { "desc": "Name of the journal located on each JournalNode filesystem.", "display_name": "Quorum-based Storage Journal name", "name": "dfs_namenode_quorum_journal_name", "value": null }, { "desc": "Timeout when starting a new edit segment with JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Start Segment Timeout", "name": "dfs_qjournal_start_segment_timeout_ms", "value": "20000" }, { "desc": "Indicate whether or not to avoid reading from stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Stale DataNodes are moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.", "display_name": "Avoid Reading Stale DataNode", "name": "dfs_namenode_avoid_read_stale_datanode", "value": "false" }, { "desc": "The base port where the DFS NameNode web UI listens. If the port number is 0, then the server starts on a free port. Combined with the NameNode's hostname to build its HTTP address.", "display_name": "NameNode Web UI Port", "name": "dfs_http_port", "value": "50070" }, { "desc": "Enables the health test that the NameNode is not in safemode", "display_name": "NameNode Safemode Health Test", "name": "namenode_safe_mode_enabled", "value": "true" }, { "desc": "Number of minutes between trash checkpoints. After a .Trash directory checkpoint is created, the Filesystem Trash Interval will define the time until permanent deletion. If set to 0, the value will be considered equal to the Filesytem Trash Interval value, which can cause the permanent deletion of entries in Trash to take over twice as long. The value for this must not exceed the Filesystem Trash Interval value.", "display_name": "Filesystem Trash Checkpoint Interval", "name": "fs_trash_checkpoint_interval", "value": "60" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Timeout when accepting recovery of an edit segment from JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Accept Recovery Timeout", "name": "dfs_qjournal_accept_recovery_timeout_ms", "value": "120000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_name_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "namenode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health check thresholds of the NameNode's RPC latency.", "display_name": "NameNode RPC Latency Thresholds", "name": "namenode_rpc_latency_thresholds", "value": "{\"critical\":\"5000.0\",\"warning\":\"1000.0\"}" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "namenode_pause_duration_window", "value": "5" }, { "desc": "Timeout when writing edits to a JournalNode. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Write Transactions Timeout", "name": "dfs_qjournal_write_txns_timeout_ms", "value": "20000" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the RPC Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: RPC Latency", "name": "role_health_suppression_name_node_rpc_latency", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_name_node_file_descriptor", "value": "false" }, { "desc": "The maximum number of outgoing replication threads a node can have at one time. This limit is waived for the highest priority replications. Configure dfs.namenode.replication.max-streams-hard-limit to set the absolute limit, including the highest-priority replications.", "display_name": "Maximum Number of Replication Threads on a DataNode", "name": "dfs_namenode_replication_max_streams", "value": "20" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When the ratio of number stale DataNodes to total DataNodes marked is greater than this ratio, permit writing to stale nodes to prevent causing hotspots.", "display_name": "Write Stale DataNode Ratio", "name": "dfs_namenode_write_stale_datanode_ratio", "value": "0.5" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "namenode_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_name_node_swap_memory_usage", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the NameNode Data Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Data Directories Free Space", "name": "role_health_suppression_name_node_data_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Filesystem Trash Interval On Validator configuration validator.", "display_name": "Suppress Configuration Validator: Filesystem Trash Interval On Validator", "name": "role_config_suppression_fs_trash_interval_minimum_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_namenode_config_safety_valve", "value": "false" }, { "desc": "Enables the health test that the NameNode's process state is consistent with the role configuration", "display_name": "NameNode Process Health Test", "name": "namenode_scm_health_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt", "name": "role_config_suppression_namenode_hosts_allow_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: NameNode Log Directory", "name": "role_config_suppression_namenode_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Plugins parameter.", "display_name": "Suppress Parameter Validation: NameNode Plugins", "name": "role_config_suppression_dfs_namenode_plugins_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Nameservice parameter.", "display_name": "Suppress Parameter Validation: NameNode Nameservice", "name": "role_config_suppression_dfs_federation_namenode_nameservice", "value": "false" }, { "desc": "Enables the health test of the rolling metadata upgrade status of the NameNode. This covers rolling metadata upgrades. Nonrolling metadata upgrades are covered in a separate health test.", "display_name": "HDFS Rolling Metadata Upgrade Status Health Test", "name": "namenode_rolling_upgrade_status_enabled", "value": "true" }, { "desc": "Mount points that are mapped to this NameNode's nameservice.", "display_name": "Mount Points", "name": "nameservice_mountpoints", "value": "/" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "namenode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"},\n {\"alert\": false, \"rate\": 1, \"threshold\":\"INFO\", \"content\":\"Triggering checkpoint.*\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_name_node_audit_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NameNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NameNode", "name": "role_config_suppression_namenode_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Edits Directories parameter.", "display_name": "Suppress Parameter Validation: NameNode Edits Directories", "name": "role_config_suppression_dfs_namenode_edits_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt", "name": "role_config_suppression_namenode_hosts_exclude_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Service Handler Count Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Service Handler Count Minimum Validator", "name": "role_config_suppression_dfs_namenode_service_handler_count_minimum_validator", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Timeout when preparing recovery of an edit segment with JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Prepare Recovery Timeout", "name": "dfs_qjournal_prepare_recovery_timeout_ms", "value": "120000" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NameNode in Bytes", "name": "namenode_java_heapsize", "value": "4294967296" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the JournalNode Sync Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JournalNode Sync Status", "name": "role_health_suppression_name_node_journal_node_sync_status", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for NameNode logs. Typically used by log4j or logback.", "display_name": "NameNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Determines where on the local file system the NameNode should store the name table (fsimage). For redundancy, enter a comma-delimited list of directories to replicate the name table in all of the directories. Typical values are /data/N/dfs/nn where N=1..3.", "display_name": "NameNode Data Directories", "name": "dfs_name_dir_list", "value": null }, { "desc": "Whether to suppress the results of the Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Upgrade Status", "name": "role_health_suppression_name_node_upgrade_status", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Edits Directory parameter.", "display_name": "Suppress Parameter Validation: Shared Edits Directory", "name": "role_config_suppression_dfs_namenode_shared_edits_dir", "value": "false" }, { "desc": "Default time interval for marking a DataNode as \"stale\". If the NameNode has not received heartbeat messages from a DataNode for more than this time interval, the DataNode is marked and treated as \"stale\" by default.", "display_name": "Stale DataNode Time Interval", "name": "dfs_namenode_stale_datanode_interval", "value": "30000" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_name_node_pause_duration", "value": "false" }, { "desc": "The port where the NameNode runs the HDFS protocol. Combined with the NameNode's hostname to build its address.", "display_name": "NameNode Port", "name": "namenode_port", "value": "8020" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The period to review when computing the moving average of the NameNode's RPC latency.", "display_name": "NameNode RPC Latency Monitoring Window", "name": "namenode_rpc_latency_window", "value": "5" }, { "desc": "Optional port for the service-rpc address which can be used by HDFS daemons instead of sharing the RPC address used by the clients.", "display_name": "NameNode Service RPC Port", "name": "dfs_namenode_servicerpc_address", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Quorum-based Storage Journal name parameter.", "display_name": "Suppress Parameter Validation: Quorum-based Storage Journal name", "name": "role_config_suppression_dfs_namenode_quorum_journal_name", "value": "false" }, { "desc": "The health check thresholds for the number of out-of-sync JournalNodes for this NameNode.", "display_name": "NameNode Out-Of-Sync JournalNodes Thresholds", "name": "namenode_out_of_sync_journal_nodes_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories.", "display_name": "NameNode Data Directories Free Space Monitoring Absolute Thresholds", "name": "namenode_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The absolute maximum number of outgoing replication threads a given node can have at one time. The regular limit (dfs.namenode.replication.max-streams) is waived for highest-priority block replications. Highest replication priority is for blocks that are at a very high risk of loss if the disk or server on which they remain fails. These are usually blocks with only one copy, or blocks with zero live copies but a copy in a node being decommissioned. dfs.namenode.replication.max-streams-hard-limit provides a limit on the total number of outgoing replication threads, including threads of all priorities.", "display_name": "Hard Limit on the Number of Replication Threads on a Datanode", "name": "dfs_namenode_replication_max_streams_hard_limit", "value": "40" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "NAMENODE_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where NameNode will place its log files.", "display_name": "NameNode Log Directory", "name": "namenode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Enables the health test of the metadata upgrade status of the NameNode. This covers nonrolling metadata upgrades. Rolling metadata upgrades are covered in a separate health test.", "display_name": "HDFS Metadata Upgrade Status Health Test", "name": "namenode_upgrade_status_enabled", "value": "true" }, { "desc": "Nameservice of this NameNode. The Nameservice represents the interface to this NameNode and its High Availability partner. The Nameservice also represents the namespace associated with a federated NameNode.", "display_name": "NameNode Nameservice", "name": "dfs_federation_namenode_nameservice", "value": null }, { "desc": "The minimum log level for NameNode logs", "display_name": "NameNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.replication.min. Enter a value less than or equal to 0 to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safemode permanent.", "display_name": "Safemode Threshold Percentage", "name": "dfs_safemode_threshold_pct", "value": "0.999" }, { "desc": "Directories on the local file system to store the NameNode edits. If not set, the edits are stored in the NameNode's Data Directories. The value of this configuration is automatically generated to be the Quorum-based Storage URI if there are JournalNodes and this NameNode is not Highly Available.", "display_name": "NameNode Edits Directories", "name": "dfs_namenode_edits_dir", "value": null }, { "desc": "If set to false and if one of the replicas of the NameNode storage fails, such as temporarily failure of NFS, this directory is not used until the NameNode restarts. If enabled, failed storage is re-checked on every checkpoint and, if it becomes valid, the NameNode will try to restore the edits and fsimage.", "display_name": "Restore NameNode Directories at Checkpoint Time", "name": "dfs_name_dir_restore", "value": "false" }, { "desc": "Timeout when creating new epoch number with JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode New Epoch Timeout", "name": "dfs_qjournal_new_epoch_timeout_ms", "value": "120000" }, { "desc": "The amount of time allowed after this role is started that failures of health tests that rely on communication with this role will be tolerated.", "display_name": "Health Test Startup Tolerance", "name": "namenode_startup_tolerance", "value": "5" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NameNode", "name": "namenode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the Java Heap Size of NameNode in Bytes Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: Java Heap Size of NameNode in Bytes Minimum Validator", "name": "role_config_suppression_namenode_java_heapsize_minimum_validator", "value": "false" }, { "desc": "Enable Automatic Failover to maintain High Availability. Requires a ZooKeeper service and a High Availability NameNode partner.", "display_name": "Enable Automatic Failover", "name": "autofailover_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NameNode Data Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NameNode Data Directories Free Space Monitoring Percentage Thresholds", "name": "namenode_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Timeout when finalizing current edit segment with JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Finalize Segment Timeout", "name": "dfs_qjournal_finalize_segment_timeout_ms", "value": "120000" }, { "desc": "For advanced use only. A string to be inserted into dfs_hosts_exclude.txt for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt", "name": "namenode_hosts_exclude_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Safe Mode Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Safe Mode Status", "name": "role_health_suppression_name_node_safe_mode", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_namenode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Rolling Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Rolling Upgrade Status", "name": "role_health_suppression_name_node_rolling_upgrade_status", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "This determines the percentage amount of block invalidations (deletes) to do over a single DataNode heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DataNode.", "display_name": "Invalidate Work Percentage Per Iteration", "name": "dfs_namenode_invalidate_work_pct_per_iteration", "value": "0.32" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "namenode_web_metric_collection_enabled", "value": "true" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_name_node_log_directory_free_space", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "namenode_config_safety_valve", "value": null }, { "desc": "The time between two periodic file system checkpoints.", "display_name": "Filesystem Checkpoint Period", "name": "fs_checkpoint_period", "value": "3600" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When computing the overall NameNode health, consider the host's health.", "display_name": "NameNode Host Health Test", "name": "namenode_host_health_enabled", "value": "true" }, { "desc": "The access time for HDFS file is precise upto this value. Setting the value of 0 disables access times for HDFS. When using the NFS Gateway role, make sure this property is enabled.", "display_name": "Access Time Precision", "name": "dfs_access_time_precision", "value": "3600000" }, { "desc": "Minimum number of running threads for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Min Threadcount", "name": "dfs_thrift_threads_min", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topology Script File Name parameter.", "display_name": "Suppress Parameter Validation: Topology Script File Name", "name": "role_config_suppression_topology_script_file_name", "value": "false" }, { "desc": "The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired.", "display_name": "Filesystem Checkpoint Transaction Threshold", "name": "fs_checkpoint_txns", "value": "1000000" }, { "desc": "Maximum number of running threads for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Max Threadcount", "name": "dfs_thrift_threads_max", "value": "20" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Handler Count Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Handler Count Minimum Validator", "name": "role_config_suppression_dfs_namenode_handler_count_minimum_validator", "value": "false" }, { "desc": "Full path to a custom topology script on the host file system. The topology script is used to determine the rack location of nodes. If left blank, a topology script will be provided that uses your hosts' rack information, visible in the \"Hosts\" page.", "display_name": "Topology Script File Name", "name": "topology_script_file_name", "value": null }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The minimum log level for NameNode block state change log messages. Setting this to WARN or higher greatly reduces the amount of log output related to block state changes.", "display_name": "NameNode Block State Change Logging Threshold", "name": "namenode_blockstatechange_log_threshold", "value": "INFO" }, { "desc": "Indicate whether or not to avoid writing to stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Writes avoid using stale DataNodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of DataNodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads.", "display_name": "Avoid Writing Stale DataNode", "name": "dfs_namenode_avoid_write_stale_datanode", "value": "false" }, { "desc": "The health test thresholds of failed status directories in a NameNode.", "display_name": "NameNode Directory Failures Thresholds", "name": "namenode_directory_failures_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_name_node_unexpected_exits", "value": "false" }, { "desc": "Controls the number of minutes after which a trash checkpoint directory is deleted permanently. To disable the trash feature, enter 0. The checkpointing frequency of .Trash directory contents is separately controlled by Filesystem Trash Checkpoint Interval.", "display_name": "Filesystem Trash Interval", "name": "fs_trash_interval", "value": "1440" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "This determines the total amount of block transfers to begin in parallel at a DataNode for replication, when such a command list is being sent over a DataNode heartbeat by the NameNode. The actual number is obtained by multiplying this value by the total number of live nodes in the cluster. The result number is the number of blocks to transfer immediately, per DataNode heartbeat.", "display_name": "Replication Work Multiplier Per Iteration", "name": "dfs_namenode_replication_work_multiplier_per_iteration", "value": "10" }, { "desc": "The maximum number of rolled log files to keep for NameNode logs. Typically used by log4j or logback.", "display_name": "NameNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Determines extension of safemode in milliseconds after the threshold level is reached.", "display_name": "Safemode Extension", "name": "dfs_safemode_extension", "value": "30000" }, { "desc": "Timeout in seconds for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Timeout", "name": "dfs_thrift_timeout", "value": "60" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The number of server threads for the NameNode used for service calls. Only used when NameNode Service RPC Port is configured.", "display_name": "NameNode Service Handler Count", "name": "dfs_namenode_service_handler_count", "value": "30" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Data Directories parameter.", "display_name": "Suppress Parameter Validation: NameNode Data Directories", "name": "role_config_suppression_dfs_name_dir_list", "value": "false" }, { "desc": "The health test thresholds of the number of transactions since the last HDFS namespace checkpoint. Specified as a percentage of the configured checkpointing transaction limit.", "display_name": "Filesystem Checkpoint Transactions Monitoring Thresholds", "name": "namenode_checkpoint_transactions_thresholds", "value": "{\"critical\":\"400.0\",\"warning\":\"200.0\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_name_node_web_metric_collection", "value": "false" }, { "desc": "The health test thresholds of the age of the HDFS namespace checkpoint. Specified as a percentage of the configured checkpoint interval.", "display_name": "Filesystem Checkpoint Age Monitoring Thresholds", "name": "namenode_checkpoint_age_thresholds", "value": "{\"critical\":\"400.0\",\"warning\":\"200.0\"}" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_name_node_scm_health", "value": "false" }, { "desc": "The base port where the secure NameNode web UI listens.", "display_name": "Secure NameNode Web UI Port (TLS/SSL)", "name": "dfs_https_port", "value": "50470" }, { "desc": "For advanced use only. A string to be inserted into dfs_hosts_allow.txt for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt", "name": "namenode_hosts_allow_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mount Points parameter.", "display_name": "Suppress Parameter Validation: Mount Points", "name": "role_config_suppression_nameservice_mountpoints", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "If enabled, the NameNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind NameNode to Wildcard Address", "name": "namenode_bind_wildcard", "value": "false" }, { "desc": "Comma-separated list of NameNode plug-ins to be activated. If one plug-in cannot be loaded, all the plug-ins are ignored.", "display_name": "NameNode Plugins", "name": "dfs_namenode_plugins_list", "value": "" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_name_node_host_health", "value": "false" }, { "desc": "Directory on a shared storage device, such as a Quorum-based Storage URI or a local directory that is an NFS mount from a NAS, to store the NameNode edits. The value of this configuration is automatically generated to be the Quourm Journal URI if there are JournalNodes and this NameNode is Highly Available.", "display_name": "Shared Edits Directory", "name": "dfs_namenode_shared_edits_dir", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The number of server threads for the NameNode.", "display_name": "NameNode Handler Count", "name": "dfs_namenode_handler_count", "value": "30" }, { "desc": "Timeout when selecting input streams on JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Select Input Streams Timeout", "name": "dfs_qjournal_select_input_streams_timeout_ms", "value": "20000" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Timeout when getting current states from JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Get State Timeout", "name": "dfs_qjournal_get_journal_state_timeout_ms", "value": "120000" }, { "desc": "Whether to suppress the results of the Name Directory Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Name Directory Status", "name": "role_health_suppression_name_node_directory_failures", "value": "false" }, { "desc": "Specifies the number of DataNodes that must be live before the name node exits safemode. Enter a value less than or equal to 0 to take the number of live DataNodes into account when deciding whether to remain in safemode during startup. Values greater than the number of DataNodes in the cluster will make safemode permanent.", "display_name": "Safemode Minimum DataNodes", "name": "dfs_safemode_min_datanodes", "value": "1" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/spark-service.json0000666000175100017510000002317513245514472025236 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Spark Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_spark_on_yarn_service_env_safety_valve", "value": "false" }, { "desc": "Name of the HBase service that this Spark service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Enable whether the Spark communication protocols do authentication using a shared secret. If using Spark2, ensure that value of this property is the same in both services.", "display_name": "Spark Authentication", "name": "spark_authenticate", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark History Location (HDFS) parameter.", "display_name": "Suppress Parameter Validation: Spark History Location (HDFS)", "name": "service_config_suppression_spark_history_log_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into spark-conf/spark-env.sh. Applies to configurations of all roles in this service except client configuration.", "display_name": "Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_service_safety_valve", "value": null }, { "desc": "The location of Spark application history logs in HDFS. Changing this value will not move existing logs to the new location.", "display_name": "Spark History Location (HDFS)", "name": "spark_history_log_dir", "value": "/user/spark/applicationHistory" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark JAR Location (HDFS) parameter.", "display_name": "Suppress Parameter Validation: Spark JAR Location (HDFS)", "name": "service_config_suppression_spark_jar_hdfs_path", "value": "false" }, { "desc": "The location of the Spark JAR in HDFS. If left blank, Cloudera Manager will use the Spark JAR installed on the cluster nodes.", "display_name": "Spark JAR Location (HDFS)", "name": "spark_jar_hdfs_path", "value": "" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "spark" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Spark Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "SPARK_ON_YARN_service_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Name of the YARN (MR2 Included) service that this Spark service instance depends on", "display_name": "YARN (MR2 Included) Service", "name": "yarn_service", "value": null }, { "desc": "The port the Spark Shuffle Service listens for fetch requests. If using Spark2, ensure that value of this property is the same in both services.", "display_name": "Spark Shuffle Service Port", "name": "spark_shuffle_service_port", "value": "7337" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "service_config_suppression_spark-conf/spark-env.sh_service_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the History Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: History Server Count Validator", "name": "service_config_suppression_spark_yarn_history_server_count_validator", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-secondarynamenode.json0000666000175100017510000010015413245514472027071 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"},\n {\"alert\": false, \"rate\": 1, \"threshold\":\"INFO\", \"content\":\"Triggering checkpoint.*\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_secondary_name_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Secondary NameNode", "name": "secondarynamenode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "secondarynamenode_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories.", "display_name": "HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds", "name": "secondarynamenode_checkpoint_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Secondary NameNode in Bytes", "name": "secondary_namenode_java_heapsize", "value": "4294967296" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Nameservice parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Nameservice", "name": "role_config_suppression_dfs_secondarynamenode_nameservice", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_secondary_name_node_scm_health", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "secondarynamenode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_secondary_name_node_web_metric_collection", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Enables the health test that the SecondaryNameNode's process state is consistent with the role configuration", "display_name": "SecondaryNameNode Process Health Test", "name": "secondarynamenode_scm_health_enabled", "value": "true" }, { "desc": "The maximum number of rolled log files to keep for SecondaryNameNode logs. Typically used by log4j or logback.", "display_name": "SecondaryNameNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The time between two periodic file system checkpoints.", "display_name": "Filesystem Checkpoint Period", "name": "fs_checkpoint_period", "value": "3600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Checkpoint Directories parameter.", "display_name": "Suppress Parameter Validation: HDFS Checkpoint Directories", "name": "role_config_suppression_fs_checkpoint_dir_list", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "secondarynamenode_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_secondary_name_node_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "secondarynamenode_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Secondary NameNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Secondary NameNode", "name": "role_config_suppression_secondarynamenode_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_secondary_name_node_gc_duration", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "SECONDARYNAMENODE_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_secondary_name_node_unexpected_exits", "value": "false" }, { "desc": "If enabled, the SecondaryNameNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind SecondaryNameNode to Wildcard Address", "name": "secondary_namenode_bind_wildcard", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired.", "display_name": "Filesystem Checkpoint Transaction Threshold", "name": "fs_checkpoint_txns", "value": "1000000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_secondarynamenode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the HDFS Checkpoint Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HDFS Checkpoint Directories Free Space", "name": "role_health_suppression_secondary_name_node_checkpoint_directories_free_space", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_secondary_name_node_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Log Directory", "name": "role_config_suppression_secondarynamenode_log_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "HDFS Checkpoint Directories Free Space Monitoring Percentage Thresholds", "name": "secondarynamenode_checkpoint_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When computing the overall SecondaryNameNode health, consider the host's health.", "display_name": "SecondaryNameNode Host Health Test", "name": "secondarynamenode_host_health_enabled", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for SecondaryNameNode logs. Typically used by log4j or logback.", "display_name": "SecondaryNameNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_secondarynamenode_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "secondarynamenode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The base port where the secure SecondaryNameNode web UI listens.", "display_name": "Secure SecondaryNameNode Web UI Port (TLS/SSL)", "name": "dfs_secondary_https_port", "value": "50495" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_secondary_name_node_audit_health", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_secondary_name_node_log_directory_free_space", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "secondarynamenode_gc_duration_window", "value": "5" }, { "desc": "Determines where on the local file system the HDFS SecondaryNameNode should store the temporary images to merge. For redundancy, enter a comma-delimited list of directories to replicate the image in all of the directories. Typical values are /data/N/dfs/snn for N = 1, 2, 3...", "display_name": "HDFS Checkpoint Directories", "name": "fs_checkpoint_dir_list", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_secondary_name_node_file_descriptor", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Nameservice of this SecondaryNameNode", "display_name": "SecondaryNameNode Nameservice", "name": "dfs_secondarynamenode_nameservice", "value": null }, { "desc": "The minimum log level for SecondaryNameNode logs", "display_name": "SecondaryNameNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Directory where SecondaryNameNode will place its log files.", "display_name": "SecondaryNameNode Log Directory", "name": "secondarynamenode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "The SecondaryNameNode HTTP port. If the port is 0, then the server starts on a free port. Combined with the SecondaryNameNode's hostname to build its HTTP address.", "display_name": "SecondaryNameNode Web UI Port", "name": "dfs_secondary_http_port", "value": "50090" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/yarn-jobhistory.json0000666000175100017510000010027113245514472025614 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Kerberos principal used by the JobHistory Server roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "mapred" }, { "desc": "

This file contains the rules that govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule is used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that can be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages for which contents match this regular expression.
  • exceptiontype - match only those messages that are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule sends events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, because the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for JobHistory Server", "name": "mr2_jobhistory_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dlibrary.leveldbjni.path={{CONF_DIR}}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_jobhistory_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Enables the health test that the JobHistory Server's process state is consistent with the role configuration", "display_name": "JobHistory Server Process Health Test", "name": "jobhistory_scm_health_enabled", "value": "true" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_jobhistory_gc_duration", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "JobHistory Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of JobHistory Server in Bytes", "name": "mr2_jobhistory_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_jobhistory_scm_health", "value": "false" }, { "desc": "When computing the overall JobHistory Server health, consider the host's health.", "display_name": "JobHistory Server Host Health Test", "name": "jobhistory_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_history_process_groupname", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for JobHistory Server logs. Typically used by log4j or logback.", "display_name": "JobHistory Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_jobhistory_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_jobhistory_host_health", "value": "false" }, { "desc": "Directory where JobHistory Server will place its log files.", "display_name": "JobHistory Server Log Directory", "name": "mr2_jobhistory_log_dir", "value": "/var/log/hadoop-mapreduce" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_jobhistory_swap_memory_usage", "value": "false" }, { "desc": "Time interval for history cleaner to check for files to delete. Files are only deleted if they are older than mapreduce.jobhistory.max-age-ms.", "display_name": "Job History Files Cleaner Interval", "name": "mapreduce_jobhistory_cleaner_interval", "value": "86400000" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "jobhistory_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "jobhistory_gc_duration_window", "value": "5" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_jobhistory_config_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MapReduce ApplicationMaster Staging Root Directory parameter.", "display_name": "Suppress Parameter Validation: MapReduce ApplicationMaster Staging Root Directory", "name": "role_config_suppression_yarn_app_mapreduce_am_staging_dir", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "jobhistory_config_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "JobHistory Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "JOBHISTORY_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Log Directory", "name": "role_config_suppression_mr2_jobhistory_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_jobhistory_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_jobhistory_file_descriptor", "value": "false" }, { "desc": "The port of the MapReduce JobHistory Server. Together with the hostname of the JobHistory role, forms the address.", "display_name": "MapReduce JobHistory Server Port", "name": "mapreduce_jobhistory_address", "value": "10020" }, { "desc": "The group that the JobHistory Server process should run as.", "display_name": "System Group", "name": "history_process_groupname", "value": "hadoop" }, { "desc": "Job history files older than this time duration will deleted when the history cleaner runs.", "display_name": "Job History Files Maximum Age", "name": "mapreduce_jobhistory_max_age_ms", "value": "604800000" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for JobHistory Server logs. Typically used by log4j or logback.", "display_name": "JobHistory Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_history_process_username", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "jobhistory_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The HTTP port of the MapReduce JobHistory Server web application. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Web Application HTTP Port", "name": "mapreduce_jobhistory_webapp_address", "value": "19888" }, { "desc": "The root HDFS directory of the staging area for users' MR2 jobs; for example /user. The staging directories are always named after the user.", "display_name": "MapReduce ApplicationMaster Staging Root Directory", "name": "yarn_app_mapreduce_am_staging_dir", "value": "/user" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "jobhistory_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The minimum log level for JobHistory Server logs", "display_name": "JobHistory Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The port of the MapReduce JobHistory Server administrative interface. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Server Admin Interface Port", "name": "mapreduce_jobhistory_admin_address", "value": "10033" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "jobhistory_mapred_safety_valve", "value": null }, { "desc": "If enabled, the JobHistory Server binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind JobHistory Server to Wildcard Address", "name": "yarn_jobhistory_bind_wildcard", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum allowed connections for the shuffle. Set to 0 (zero) to indicate no limit on the number of connections.", "display_name": "Max Shuffle Connections", "name": "mapreduce_shuffle_max_connections", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JobHistory Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for JobHistory Server", "name": "role_config_suppression_mr2_jobhistory_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_jobhistory_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The user that the JobHistory Server process should run as.", "display_name": "System User", "name": "history_process_username", "value": "mapred" }, { "desc": "The HTTPS port of the MapReduce JobHistory Server web application. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Web Application HTTPS Port (TLS/SSL)", "name": "mapreduce_jobhistory_webapp_https_address", "value": "19890" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_jobhistory_mapred_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "jobhistory_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_jobhistory_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_jobhistory_audit_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/flume-service.json0000666000175100017510000002436513245514472025230 0ustar zuulzuul00000000000000[ { "desc": "Name of the HBase service that this Flume service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Flume might connect to. This is used when Flume is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Flume TLS/SSL Certificate Trust Store File", "name": "flume_truststore_file", "value": null }, { "desc": "Whether to suppress the results of the Agent Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Agent Health", "name": "service_health_suppression_flume_agents_healthy", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "flume" }, { "desc": "Name of the HDFS service that this Flume service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Agent Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Agent Count Validator", "name": "service_config_suppression_agent_count_validator", "value": "false" }, { "desc": "Name of the Solr service that this Flume service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Flume Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "flume_env_safety_valve", "value": null }, { "desc": "Sets the maximum number of Flume components that will be returned under Flume Metric Details. Increasing this value will negatively impact the interactive performance of the Flume Metrics Details page.", "display_name": "Maximum displayed Flume metrics components", "name": "flume_context_groups_request_limit", "value": "1000" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "flume" }, { "desc": "The health test thresholds of the overall Agent health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Agents falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Agents falls below the critical threshold.", "display_name": "Healthy Agent Monitoring Thresholds", "name": "flume_agents_healthy_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"95.0\"}" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "flume" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Flume TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_flume_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "The password for the Flume TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Flume TLS/SSL Certificate Trust Store Password", "name": "flume_truststore_password", "value": null }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Flume TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_flume_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Flume Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_flume_env_safety_valve", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/kafka-service.json0000666000175100017510000005544413245514472025177 0ustar zuulzuul00000000000000[ { "desc": "If automatic leader rebalancing is enabled, the controller tries to balance leadership for partitions among the brokers by periodically returning leadership for each partition to the preferred replica, if it is available.", "display_name": "Enable Automatic Leader Rebalancing", "name": "auto.leader.rebalance.enable", "value": "true" }, { "desc": "The frequency, in ms, with which the log flusher checks whether any log is eligible to be flushed to disk.", "display_name": "Log Flush Scheduler Interval", "name": "log.flush.scheduler.interval.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Minimum Number of Replicas in ISR parameter.", "display_name": "Suppress Parameter Validation: Minimum Number of Replicas in ISR", "name": "service_config_suppression_min.insync.replicas", "value": "false" }, { "desc": "The amount of time between updates of the Sentry privileges cache. Any change in role or privilege can take up to the specified time for changes to reflect in the cache.", "display_name": "Sentry Privileges Cache Refresh Interval", "name": "sentry.kafka.caching.ttl.ms", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Session Timeout parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Session Timeout", "name": "service_config_suppression_zookeeper.session.timeout.ms", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Leader Imbalance Allowed Per Broker parameter.", "display_name": "Suppress Parameter Validation: Leader Imbalance Allowed Per Broker", "name": "service_config_suppression_leader.imbalance.per.broker.percentage", "value": "false" }, { "desc": "The percentage of leader imbalance allowed per broker. The controller rebalances leadership if this ratio goes above the configured value per broker.", "display_name": "Leader Imbalance Allowed Per Broker", "name": "leader.imbalance.per.broker.percentage", "value": "10" }, { "desc": "If a follower has not sent any fetch requests, nor has it consumed up to the leader's log end offset during this time, the leader removes the follower from the ISR set.", "display_name": "Allowed Replica Time Lag", "name": "replica.lag.time.max.ms", "value": "10000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Root parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Root", "name": "service_config_suppression_zookeeper.chroot", "value": "false" }, { "desc": "Enables Sentry privileges caching. Only applicable if authorization via Sentry is enabled.", "display_name": "Enable Sentry Privileges Caching", "name": "sentry.kafka.caching.enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Maximum Message Size parameter.", "display_name": "Suppress Parameter Validation: Maximum Message Size", "name": "service_config_suppression_message.max.bytes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Controlled Shutdown Maximum Attempts parameter.", "display_name": "Suppress Parameter Validation: Controlled Shutdown Maximum Attempts", "name": "service_config_suppression_controlled.shutdown.max.retries", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Offset Commit Topic Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Offset Commit Topic Replication Factor", "name": "service_config_suppression_offsets.topic.replication.factor", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "kafka" }, { "desc": "Enables the log cleaner to compact topics with cleanup.policy=compact on this cluster.", "display_name": "Enable Log Compaction", "name": "log.cleaner.enable", "value": "true" }, { "desc": "The number of background threads to use for log cleaning.", "display_name": "Number of Log Cleaner Threads", "name": "log.cleaner.threads", "value": "1" }, { "desc": "The number of partitions for the offset commit topic. Since changing this after deployment is currently unsupported, Cloudera recommends using a higher setting for production (for example, 100-200).", "display_name": "Offset Commit Topic Number of Partitions", "name": "offsets.topic.num.partitions", "value": "50" }, { "desc": "The default replication factor for automatically created topics.", "display_name": "Default Replication Factor", "name": "default.replication.factor", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of Replica Fetchers parameter.", "display_name": "Suppress Parameter Validation: Number of Replica Fetchers", "name": "service_config_suppression_num.replica.fetchers", "value": "false" }, { "desc": "The amount of time to retain delete messages for log compacted topics. Once a consumer has seen an original message you need to ensure it also sees the delete message. If you removed the delete message too quickly, this might not happen. As a result there is a configurable delete retention time.", "display_name": "Log Compaction Delete Record Retention Time", "name": "log.cleaner.delete.retention.ms", "value": "604800000" }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "If a replica falls more than this number of messages behind the leader, the leader removes the follower from the ISR and treats it as dead. This property is deprecated in Kafka 1.4.0; higher versions use only replica.lag.time.max.ms.", "display_name": "Allowed Replica Message Lag", "name": "replica.lag.max.messages", "value": "4000" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "kafka" }, { "desc": "Name of the Sentry service that this Kafka service instance depends on", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Leader Imbalance Check Interval parameter.", "display_name": "Suppress Parameter Validation: Leader Imbalance Check Interval", "name": "service_config_suppression_leader.imbalance.check.interval.seconds", "value": "false" }, { "desc": "ZNode in ZooKeeper that should be used as a root for this Kafka cluster.", "display_name": "ZooKeeper Root", "name": "zookeeper.chroot", "value": "" }, { "desc": "The default number of partitions for automatically created topics.", "display_name": "Default Number of Partitions", "name": "num.partitions", "value": "1" }, { "desc": "Any producer distinguished by clientId will get throttled if it produces more bytes than this value per-second. Only respected by Kafka 2.0 or later.", "display_name": "Default Producer Quota", "name": "quota.producer.default", "value": null }, { "desc": "The number of messages written to a log partition before triggering an fsync on the log. Setting this lower syncs data to disk more often, but has a major impact on performance. Cloudera recommends use of replication for durability rather than depending on single-server fsync; however, this setting can be used to be extra certain. If used in conjunction with log.flush.interval.ms, the log is flushed when either criteria is met.", "display_name": "Log Flush Message Interval", "name": "log.flush.interval.messages", "value": null }, { "desc": "Enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so might result in data loss.", "display_name": "Enable Unclean Leader Election", "name": "unclean.leader.election.enable", "value": "false" }, { "desc": "If Sentry privileges cache fails to update for specified number of times, all authorization requests will be denied.", "display_name": "Sentry Privileges Cache Maximum Update Failures", "name": "sentry.kafka.caching.update.failures.count", "value": "3" }, { "desc": "Number of unsuccessful controlled shutdown attempts before executing an unclean shutdown. For example, the default value of 3 means that the system will attempt a controlled shutdown 3 times before executing an unclean shutdown.", "display_name": "Controlled Shutdown Maximum Attempts", "name": "controlled.shutdown.max.retries", "value": "3" }, { "desc": "The total memory used for log deduplication across all cleaner threads. This memory is statically allocated and will not cause GC problems.", "display_name": "Log Cleaner Deduplication Buffer Size", "name": "log.cleaner.dedupe.buffer.size", "value": "134217728" }, { "desc": "The minimum number of replicas in the in-sync replica needed to satisfy a produce request where required.acks=-1 (that is, all).", "display_name": "Minimum Number of Replicas in ISR", "name": "min.insync.replicas", "value": "1" }, { "desc": "Enables controlled shutdown of the broker. If enabled, the broker moves all leaders on it to other brokers before shutting itself down. This reduces the unavailability window during shutdown.", "display_name": "Enable Controlled Shutdown", "name": "controlled.shutdown.enable", "value": "true" }, { "desc": "Any consumer distinguished by clientId/consumer group will get throttled if it fetches more bytes than this value per-second. Only respected by Kafka 2.0 or later.", "display_name": "Default Consumer Quota", "name": "quota.consumer.default", "value": null }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Enables topic deletion using admin tools. When delete topic is disabled, deleting topics through the admin tools has no effect.", "display_name": "Enable Delete Topic", "name": "delete.topic.enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Number of Partitions parameter.", "display_name": "Suppress Parameter Validation: Default Number of Partitions", "name": "service_config_suppression_num.partitions", "value": "false" }, { "desc": "Number of threads used to replicate messages from leaders. Increasing this value increases the degree of I/O parallelism in the follower broker.", "display_name": "Number of Replica Fetchers", "name": "num.replica.fetchers", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Kafka service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "The maximum size of a message that the server can receive. It is important that this property be in sync with the maximum fetch size the consumers use, or else an unruly producer could publish messages too large for consumers to consume.", "display_name": "Maximum Message Size", "name": "message.max.bytes", "value": "1000000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Replica Maximum Fetch Size parameter.", "display_name": "Suppress Parameter Validation: Replica Maximum Fetch Size", "name": "service_config_suppression_replica.fetch.max.bytes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Default Replication Factor", "name": "service_config_suppression_default.replication.factor", "value": "false" }, { "desc": "The maximum number of bytes to fetch for each partition in fetch requests replicas send to the leader. This value should be larger than message.max.bytes.", "display_name": "Replica Maximum Fetch Size", "name": "replica.fetch.max.bytes", "value": "1048576" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Kafka MirrorMaker Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kafka MirrorMaker Count Validator", "name": "service_config_suppression_kafka_mirror_maker_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Kafka Broker Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kafka Broker Count Validator", "name": "service_config_suppression_kafka_broker_count_validator", "value": "false" }, { "desc": "Enable Kerberos authentication for this KAFKA service.", "display_name": "Enable Kerberos Authentication", "name": "kerberos.auth.enable", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the List of Metric Reporters parameter.", "display_name": "Suppress Parameter Validation: List of Metric Reporters", "name": "service_config_suppression_kafka.metrics.reporters", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "The replication factor for the offset commit topic. A higher setting is recommended in order to ensure higher availability (for example, 3 or 4) . If the offsets topic is created when there are fewer brokers than the replication factor, then the offsets topic is created with fewer replicas.", "display_name": "Offset Commit Topic Replication Factor", "name": "offsets.topic.replication.factor", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The maximum time between fsync calls on the log. If used in conjuction with log.flush.interval.messages, the log is flushed when either criteria is met.", "display_name": "Log Flush Time Interval", "name": "log.flush.interval.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Offset Commit Topic Number of Partitions parameter.", "display_name": "Suppress Parameter Validation: Offset Commit Topic Number of Partitions", "name": "service_config_suppression_offsets.topic.num.partitions", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_kafka_service_env_safety_valve", "value": "false" }, { "desc": "Enables Kafka monitoring.", "display_name": "Enable Kafka Monitoring (Note: Requires Kafka-1.3.0 parcel or higher)", "name": "monitoring.enabled", "value": "true" }, { "desc": "List of metric reporter class names. HTTP reporter is included by default.", "display_name": "List of Metric Reporters", "name": "kafka.metrics.reporters", "value": "nl.techop.kafka.KafkaHttpMetricsReporter" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Kafka Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_service_env_safety_valve", "value": null }, { "desc": "If the server fails to send a heartbeat to ZooKeeper within this period of time, it is considered dead. If set too low, ZooKeeper might falsely consider a server dead; if set too high, ZooKeeper might take too long to recognize a dead server.", "display_name": "ZooKeeper Session Timeout", "name": "zookeeper.session.timeout.ms", "value": "6000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Enables auto creation of topics on the server. If this is set to true, then attempts to produce, consume, or fetch metadata for a non-existent topic automatically create the topic with the default replication factor and number of partitions.", "display_name": "Topic Auto Creation", "name": "auto.create.topics.enable", "value": "true" }, { "desc": "The frequency with which to check for leader imbalance.", "display_name": "Leader Imbalance Check Interval", "name": "leader.imbalance.check.interval.seconds", "value": "300" }, { "desc": "Controls how frequently the log cleaner will attempt to clean the log. This ratio bounds the maximum space wasted in the log by duplicates. For example, at 0.5 at most 50% of the log could be duplicates. A higher ratio will mean fewer, more efficient cleanings but will mean more wasted space in the log.", "display_name": "Log Cleaner Clean Ratio", "name": "log.cleaner.min.cleanable.ratio", "value": "0.5" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/hdfs-httpfs.json0000666000175100017510000006560213245514472024713 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "httpfs_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_httpfs_scm_health", "value": "false" }, { "desc": "Kerberos principal used by the HttpFS roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "httpfs" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_httpfs_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_httpfs_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_httpfs_host_health", "value": "false" }, { "desc": "Address of the load balancer used for HttpFS roles. Should be specified in host:port format. Note: Changing this property will regenerate Kerberos keytabs for all HttpFS roles.", "display_name": "HttpFS Load Balancer", "name": "httpfs_load_balancer", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Log Directory parameter.", "display_name": "Suppress Parameter Validation: HttpFS Log Directory", "name": "role_config_suppression_httpfs_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_httpfs_unexpected_exits", "value": "false" }, { "desc": "The user that the HttpFS server process should run as.", "display_name": "System User", "name": "httpfs_process_username", "value": "httpfs" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The group that the HttpFS server process should run as.", "display_name": "System Group", "name": "httpfs_process_groupname", "value": "httpfs" }, { "desc": "When computing the overall HttpFS health, consider the host's health.", "display_name": "HttpFS Host Health Test", "name": "httpfs_host_health_enabled", "value": "true" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HttpFS logs. Typically used by log4j or logback.", "display_name": "HttpFS Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HttpFS Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Enables the health test that the HttpFS's process state is consistent with the role configuration", "display_name": "HttpFS Process Health Test", "name": "httpfs_scm_health_enabled", "value": "true" }, { "desc": "The password for the HttpFS JKS keystore file.", "display_name": "HttpFS TLS/SSL Server JKS Keystore File Password", "name": "httpfs_https_keystore_password", "value": null }, { "desc": "The password for the HttpFS TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HttpFS TLS/SSL Certificate Trust Store Password", "name": "httpfs_https_truststore_password", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_httpfs_https_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Signature Secret parameter.", "display_name": "Suppress Parameter Validation: Signature Secret", "name": "role_config_suppression_hdfs_httpfs_signature_secret", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml", "name": "role_config_suppression_httpfs_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_httpfs_https_keystore_password", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_httpfs_audit_health", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HttpFS parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HttpFS", "name": "role_config_suppression_httpfs_java_opts", "value": "false" }, { "desc": "Directory where HttpFS will place its log files.", "display_name": "HttpFS Log Directory", "name": "httpfs_log_dir", "value": "/var/log/hadoop-httpfs" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HttpFS Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_httpfs_process_groupname", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into httpfs-site.xml for this role only.", "display_name": "HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml", "name": "httpfs_config_safety_valve", "value": null }, { "desc": "The maximum size, in megabytes, per log file for HttpFS logs. Typically used by log4j or logback.", "display_name": "HttpFS Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HttpFS", "name": "httpfs_java_opts", "value": "" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_httpfs_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HttpFS Environment Advanced Configuration Snippet (Safety Valve)", "name": "HTTPFS_role_env_safety_valve", "value": null }, { "desc": "The secret to use for signing client authentication tokens.", "display_name": "Signature Secret", "name": "hdfs_httpfs_signature_secret", "value": "hadoop httpfs secret" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HttpFS Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_httpfs_role_env_safety_valve", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HttpFS is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HttpFS TLS/SSL Server JKS Keystore File Location", "name": "httpfs_https_keystore_file", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HttpFS in Bytes", "name": "httpfs_java_heapsize", "value": "268435456" }, { "desc": "The port for the administration interface.", "display_name": "Administration Port", "name": "hdfs_httpfs_admin_port", "value": "14001" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HttpFS might connect to. This is used when HttpFS is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HttpFS TLS/SSL Certificate Trust Store File", "name": "httpfs_https_truststore_file", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The port where the REST interface to HDFS is available. The REST interface is served over HTTPS if TLS/SSL is enabled for HttpFS, or over HTTP otherwise.", "display_name": "REST Port", "name": "hdfs_httpfs_http_port", "value": "14000" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_httpfs_https_keystore_file", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_httpfs_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_httpfs_https_truststore_password", "value": "false" }, { "desc": "The minimum log level for HttpFS logs", "display_name": "HttpFS Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Encrypt communication between clients and HttpFS using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HttpFS", "name": "httpfs_use_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Load Balancer parameter.", "display_name": "Suppress Parameter Validation: HttpFS Load Balancer", "name": "role_config_suppression_httpfs_load_balancer", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_httpfs_process_username", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/oozie-service.json0000666000175100017510000003405113245514472025236 0ustar zuulzuul00000000000000[ { "desc": "Namespace used by this Oozie service in ZooKeeper when High Availability is enabled.", "display_name": "ZooKeeper Namespace", "name": "oozie_zookeeper_namespace", "value": "oozie" }, { "desc": "Name of the ZooKeeper service that this Oozie service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "The number of threads that Oozie will use for the Upload ShareLib command.", "display_name": "Oozie Upload ShareLib Command Concurrency Level", "name": "oozie_upload_sharelib_concurrency", "value": "8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Database Dump File parameter.", "display_name": "Suppress Parameter Validation: Database Dump File", "name": "service_config_suppression_database_dump_file", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "oozie" }, { "desc": "The timeout in seconds used for the Oozie Upload ShareLib command. When the value is zero, there is no timeout for the command.", "display_name": "Oozie Upload ShareLib Command Timeout", "name": "oozie_upload_sharelib_cmd_timeout", "value": "270" }, { "desc": "URL of the JMS Broker used by the Oozie service in JMS integration is enabled.", "display_name": "JMS Broker", "name": "oozie_jms_broker", "value": "tcp://localhost:61616" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ShareLib Root Directory parameter.", "display_name": "Suppress Parameter Validation: ShareLib Root Directory", "name": "service_config_suppression_oozie_sharelib_rootdir", "value": "false" }, { "desc": "Encrypt communication between clients and Oozie using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Oozie", "name": "oozie_use_ssl", "value": "false" }, { "desc": "List of event listeners used by the Oozie service. Listeners needed for JMS or SLA integration are automatically emitted if they are enabled.", "display_name": "Oozie Event Listeners", "name": "oozie_event_listeners", "value": "" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Oozie Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Oozie Server Health", "name": "service_health_suppression_oozie_oozie_servers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_oozie_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JMS Broker parameter.", "display_name": "Suppress Parameter Validation: JMS Broker", "name": "service_config_suppression_oozie_jms_broker", "value": "false" }, { "desc": "Whether to configure Oozie properties needed for SLA integration", "display_name": "Enable SLA Integration", "name": "oozie_use_sla", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "oozie" }, { "desc": "File where the database gets dumped to or loaded from.", "display_name": "Database Dump File", "name": "database_dump_file", "value": "/tmp/oozie_database_dump.zip" }, { "desc": "Root of the directory in HDFS where the Oozie ShareLibs are stored. The libraries are stored in the share/lib subdirectory under the specified root directory.", "display_name": "ShareLib Root Directory", "name": "oozie_sharelib_rootdir", "value": "/user/oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Event Listeners parameter.", "display_name": "Suppress Parameter Validation: Oozie Event Listeners", "name": "service_config_suppression_oozie_event_listeners", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Oozie Load Balancer", "name": "service_config_suppression_oozie_load_balancer", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "oozie" }, { "desc": "Whether to configure Oozie properties needed for JMS integration", "display_name": "Enable JMS Integration", "name": "oozie_use_jms", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Oozie Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "oozie_env_safety_valve", "value": null }, { "desc": "A list of credential class mappings for CredentialsProvider.", "display_name": "Oozie Credential Classes", "name": "oozie_credential_classes", "value": "hcat=org.apache.oozie.action.hadoop.HCatCredentials,hbase=org.apache.oozie.action.hadoop.HbaseCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Address of the load balancer used if Oozie HA is enabled. Should be specified in host:port format.", "display_name": "Oozie Load Balancer", "name": "oozie_load_balancer", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the Spark on Yarn service that this Oozie service instance depends on", "display_name": "Spark on Yarn Service", "name": "spark_on_yarn_service", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The health test thresholds of the overall Oozie Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Oozie Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Oozie Servers falls below the critical threshold.", "display_name": "Healthy Oozie Server Monitoring Thresholds", "name": "oozie_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Credential Classes parameter.", "display_name": "Suppress Parameter Validation: Oozie Credential Classes", "name": "service_config_suppression_oozie_credential_classes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Namespace parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Namespace", "name": "service_config_suppression_oozie_zookeeper_namespace", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Default timeout for a Coordinator Action input check (in minutes) for a Coordinator Job.", "display_name": "Coordinator Action Input Check Default Timeout", "name": "oozie_service_coord_normal_default_timeout", "value": "120" }, { "desc": "Name of the Hive service that this Oozie service instance depends on. This is used to configure Oozie HCat integration.", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Oozie Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Oozie Server Count Validator", "name": "service_config_suppression_oozie_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Service to run MapReduce jobs against", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null }, { "desc": "Coordinator Job Lookup trigger command is scheduled at this interval (in seconds).", "display_name": "Coordinator Job Lookup Interval", "name": "oozie_service_coord_lookup_interval", "value": "300" }, { "desc": "Use ACLs on Znode while a secure ZooKeeper is used for Oozie High Availability. Note: This config is not emitted if ZooKeeper is not secure.", "display_name": "Use ACLs on Znode", "name": "oozie_zk_secure", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_11_0/resources/kafka-kafka_broker.json0000666000175100017510000010130513245514472026144 0ustar zuulzuul00000000000000[ { "desc": "The log for a topic partition is stored as a directory of segment files. This setting controls the size to which a segment file can grow before a new segment is rolled over in the log. This value should be larger than message.max.bytes.", "display_name": "Segment File Size", "name": "log.segment.bytes", "value": "1073741824" }, { "desc": "For advanced use only. A string to be inserted into kafka-monitoring.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties", "name": "kafka-monitoring.properties_role_safety_valve", "value": null }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The maximum time before a new log segment is rolled out. If both log.retention.ms and log.retention.bytes are set, a segment is deleted when either limit is exceeded. The special value of -1 is interpreted as unlimited. This property is used in Kafka 1.4.0 and later in place of log.retention.hours.", "display_name": "Data Retention Time", "name": "log.retention.ms", "value": null }, { "desc": "The amount of data to retain in the log for each topic-partition. This is the limit per partition: multiply by the number of partitions to get the total data retained for the topic. The special value of -1 is interpreted as unlimited. If both log.retention.ms and log.retention.bytes are set, a segment is deleted when either limit is exceeded.", "display_name": "Data Retention Size", "name": "log.retention.bytes", "value": "-1" }, { "desc": "Enables the health test that the Kafka Broker's process state is consistent with the role configuration", "display_name": "Kafka Broker Process Health Test", "name": "kafka_broker_scm_health_enabled", "value": "true" }, { "desc": "Kafka broker secure port.", "display_name": "TLS/SSL Port", "name": "ssl_port", "value": "9093" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kafka_broker_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The password for the Kafka Broker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Kafka Broker TLS/SSL Certificate Trust Store Password", "name": "ssl_client_truststore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka Broker is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "Host the HTTP metric reporter binds to.", "display_name": "HTTP Metric Report Host", "name": "kafka.http.metrics.host", "value": "0.0.0.0" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Advertised Host parameter.", "display_name": "Suppress Parameter Validation: Advertised Host", "name": "role_config_suppression_advertised.host.name", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out (in hours). Secondary to the log.retention.ms property. The special value of -1 is interpreted as unlimited. This property is deprecated in Kafka 1.4.0. Use log.retention.ms.", "display_name": "Data Retention Hours", "name": "log.retention.hours", "value": "168" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The password that protects the private key contained in the JKS keystore used when Kafka Broker is acting as a TLS/SSL server.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": "" }, { "desc": "The number of I/O threads that the server uses for executing requests. You should have at least as many threads as you have disks.", "display_name": "Number of I/O Threads", "name": "num.io.threads", "value": "8" }, { "desc": "Whether to suppress the results of the Audit Pipeline Test heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Audit Pipeline Test", "name": "role_health_suppression_kafka_kafka_broker_audit_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties", "name": "role_config_suppression_kafka.properties_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kafka_kafka_broker_host_health", "value": "false" }, { "desc": "Kafka broker port.", "display_name": "TCP Port", "name": "port", "value": "9092" }, { "desc": "The maximum size, in megabytes, per log file for Kafka Broker logs. Typically used by log4j or logback.", "display_name": "Kafka Broker Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The password for the Kafka Broker JKS keystore file.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "The log directory for log files of the role Kafka Broker.", "display_name": "Kafka Broker Log Directory", "name": "log_dir", "value": "/var/log/kafka" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kafka_kafka_broker_file_descriptor", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka Broker might connect to. This is used when Kafka Broker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Kafka Broker TLS/SSL Certificate Trust Store File", "name": "ssl_client_truststore_location", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties", "name": "role_config_suppression_kafka-monitoring.properties_role_safety_valve", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out (in hours). This property is deprecated in Cloudera Kafka 1.4.0; use log.roll.ms.", "display_name": "Data Log Roll Hours", "name": "log.roll.hours", "value": "168" }, { "desc": "Port for JMX.", "display_name": "JMX Port", "name": "jmx_port", "value": "9393" }, { "desc": "The frequency, in milliseconds, that the log cleaner checks whether any log segment is eligible for deletion, per retention policies.", "display_name": "Data Retention Check Interval", "name": "log.retention.check.interval.ms", "value": "300000" }, { "desc": "The timeout in milliseconds to wait for graceful shutdown to complete.", "display_name": "Graceful Shutdown Timeout", "name": "graceful_stop_timeout", "value": "30000" }, { "desc": "For advanced use only. A string to be inserted into ssl.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties", "name": "ssl.properties_role_safety_valve", "value": null }, { "desc": "Protocol to be used for inter-broker communication. INFERRED will use the same protocol configured for external clients.", "display_name": "Inter Broker Protocol", "name": "security.inter.broker.protocol", "value": "INFERRED" }, { "desc": "ID uniquely identifying each broker. Never set this property at the group level; it should always be overridden on instance level.", "display_name": "Broker ID", "name": "broker.id", "value": null }, { "desc": "The maximum time before a new log segment is rolled out. This property is used in Cloudera Kafka 1.4.0 and later in place of log.roll.hours.", "display_name": "Data Log Roll Time", "name": "log.roll.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of I/O Threads parameter.", "display_name": "Suppress Parameter Validation: Number of I/O Threads", "name": "role_config_suppression_num.io.threads", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "The minimum log level for Kafka Broker logs", "display_name": "Kafka Broker Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Data Directories parameter.", "display_name": "Suppress Parameter Validation: Data Directories", "name": "role_config_suppression_log.dirs", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kafka_kafka_broker_scm_health", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Port the HTTP metric reporter listens on.", "display_name": "HTTP Metric Report Port", "name": "kafka.http.metrics.port", "value": "24042" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Client authentication mode for SSL connections. Default is none, could be set to \"required\", i.e., client authentication is required or to \"requested\", i.e., client authentication is requested and client without certificates can still connect.", "display_name": "SSL Client Authentication", "name": "ssl.client.auth", "value": "none" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "Encrypt communication between clients and Kafka Broker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Kafka Broker", "name": "ssl_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "These arguments are passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags are passed here.", "display_name": "Additional Broker Java Options", "name": "broker_java_opts", "value": "-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kafka_broker_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_BROKER_role_env_safety_valve", "value": null }, { "desc": "When computing the overall Kafka Broker health, consider the host's health.", "display_name": "Kafka Broker Host Health Test", "name": "kafka_broker_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Broker Java Options parameter.", "display_name": "Suppress Parameter Validation: Additional Broker Java Options", "name": "role_config_suppression_broker_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Broker ID parameter.", "display_name": "Suppress Parameter Validation: Broker ID", "name": "role_config_suppression_broker.id", "value": "false" }, { "desc": "Maximum number of connections allowed from each IP address.", "display_name": "Maximum Connections per IP Address", "name": "max.connections.per.ip", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Segment File Size parameter.", "display_name": "Suppress Parameter Validation: Segment File Size", "name": "role_config_suppression_log.segment.bytes", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Kafka Broker Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Metric Report Host parameter.", "display_name": "Suppress Parameter Validation: HTTP Metric Report Host", "name": "role_config_suppression_kafka.http.metrics.host", "value": "false" }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in megabytes. Kafka does not generally require setting large heap sizes. It is better to let the file system cache utilize the available memory.", "display_name": "Java Heap Size of Broker", "name": "broker_max_heap_size", "value": "1024" }, { "desc": "For advanced use only. A string to be inserted into kafka.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties", "name": "kafka.properties_role_safety_valve", "value": null }, { "desc": "If set, this is the hostname given out to producers, consumers, and other brokers to use in establishing connections. Never set this property at the group level; it should always be overriden on instance level.", "display_name": "Advertised Host", "name": "advertised.host.name", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of Broker parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of Broker", "name": "role_config_suppression_broker_max_heap_size", "value": "false" }, { "desc": "Users who are allowed to perform any action on the Kafka cluster.", "display_name": "Super users", "name": "super.users", "value": "kafka" }, { "desc": "Cloudera Manager agent monitors each service and each of its role by publishing metrics to the Cloudera Manager Service Monitor. Setting it to false will stop Cloudera Manager agent from publishing any metric for corresponding service/roles. This is usually helpful for services that generate large amount of metrics which Service Monitor is not able to process.", "display_name": "Enable Metric Collection", "name": "process_should_monitor", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Super users parameter.", "display_name": "Suppress Parameter Validation: Super users", "name": "role_config_suppression_super.users", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Authenticate a SASL connection with zookeeper, if Kerberos authentication is enabled. It also allows a broker to set SASL ACL on zookeeper nodes which locks these nodes down so that only kafka broker can modify.", "display_name": "Authenticate Zookeeper Connection", "name": "authenticate.zookeeper.connection", "value": "true" }, { "desc": "The maximum number of rolled log files to keep for Kafka Broker logs. Typically used by log4j or logback.", "display_name": "Kafka Broker Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kafka_kafka_broker_unexpected_exits", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON-formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The port to give out to producers, consumers, and other brokers to use in establishing connections. This only needs to be set if this port is different from the port the server should bind to.", "display_name": "Advertised Port", "name": "advertised.port", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kafka_kafka_broker_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties", "name": "role_config_suppression_ssl.properties_role_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "A list of one or more directories in which Kafka data is stored. Each new partition created is placed in the directory that currently has the fewest partitions. Each directory should be on its own separate drive.", "display_name": "Data Directories", "name": "log.dirs", "value": "/var/local/kafka/data" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" } ]sahara-8.0.0/sahara/plugins/cdh/versionfactory.py0000666000175100017510000000360613245514472022125 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import re from sahara.utils import general class VersionFactory(object): versions = None modules = None initialized = False @staticmethod def get_instance(): if not VersionFactory.initialized: src_dir = os.path.join(os.path.dirname(__file__), '') versions = ( [name[1:].replace('_', '.') for name in os.listdir(src_dir) if (os.path.isdir(os.path.join(src_dir, name)) and re.match(r'^v\d+(_\d+)*$', name))]) versions.sort(key=general.natural_sort_key) VersionFactory.versions = versions VersionFactory.modules = {} for version in VersionFactory.versions: module_name = 'sahara.plugins.cdh.v%s.versionhandler' % ( version.replace('.', '_')) module_class = getattr( __import__(module_name, fromlist=['sahara']), 'VersionHandler') module = module_class() VersionFactory.modules[version] = module VersionFactory.initialized = True return VersionFactory() def get_versions(self): return VersionFactory.versions def get_version_handler(self, version): return VersionFactory.modules[version] sahara-8.0.0/sahara/plugins/cdh/plugin.py0000666000175100017510000001153313245514472020344 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara.i18n import _ from sahara.plugins.cdh import versionfactory as vhf from sahara.plugins import provisioning as p class CDHPluginProvider(p.ProvisioningPluginBase): def __init__(self): self.version_factory = vhf.VersionFactory.get_instance() def get_title(self): return "Cloudera Plugin" def get_description(self): return _('The Cloudera Sahara plugin provides the ability to ' 'launch the Cloudera distribution of Apache Hadoop ' '(CDH) with Cloudera Manager management console.') def get_labels(self): default = {'enabled': {'status': True}, 'stable': {'status': True}} result = {'plugin_labels': copy.deepcopy(default)} deprecated = {'enabled': {'status': True}, 'deprecated': {'status': True}} result['version_labels'] = { '5.11.0': copy.deepcopy(default), '5.9.0': copy.deepcopy(default), '5.7.0': copy.deepcopy(default), '5.5.0': copy.deepcopy(deprecated), } return result def _get_version_handler(self, hadoop_version): return self.version_factory.get_version_handler(hadoop_version) def get_versions(self): return self.version_factory.get_versions() def get_node_processes(self, hadoop_version): return self._get_version_handler(hadoop_version).get_node_processes() def get_configs(self, hadoop_version): return self._get_version_handler(hadoop_version).get_plugin_configs() def configure_cluster(self, cluster): return self._get_version_handler( cluster.hadoop_version).configure_cluster(cluster) def start_cluster(self, cluster): return self._get_version_handler( cluster.hadoop_version).start_cluster(cluster) def validate(self, cluster): return self._get_version_handler( cluster.hadoop_version).validate(cluster) def scale_cluster(self, cluster, instances): return self._get_version_handler( cluster.hadoop_version).scale_cluster(cluster, instances) def decommission_nodes(self, cluster, instances): return self._get_version_handler( cluster.hadoop_version).decommission_nodes(cluster, instances) def validate_scaling(self, cluster, existing, additional): return self._get_version_handler( cluster.hadoop_version).validate_scaling(cluster, existing, additional) def get_edp_engine(self, cluster, job_type): return self._get_version_handler( cluster.hadoop_version).get_edp_engine(cluster, job_type) def get_edp_job_types(self, versions=None): res = {} for vers in self.version_factory.get_versions(): if not versions or vers in versions: vh = self.version_factory.get_version_handler(vers) res[vers] = vh.get_edp_job_types() return res def get_edp_config_hints(self, job_type, version): version_handler = ( self.version_factory.get_version_handler(version)) return version_handler.get_edp_config_hints(job_type) def get_open_ports(self, node_group): return self._get_version_handler( node_group.cluster.hadoop_version).get_open_ports(node_group) def recommend_configs(self, cluster, scaling=False): return self._get_version_handler( cluster.hadoop_version).recommend_configs(cluster, scaling) def get_health_checks(self, cluster): return self._get_version_handler( cluster.hadoop_version).get_health_checks(cluster) def get_image_arguments(self, hadoop_version): return self._get_version_handler(hadoop_version).get_image_arguments() def pack_image(self, hadoop_version, remote, test_only=False, image_arguments=None): version = self._get_version_handler(hadoop_version) version.pack_image(hadoop_version, remote, test_only=test_only, image_arguments=image_arguments) def validate_images(self, cluster, test_only=False, image_arguments=None): self._get_version_handler(cluster.hadoop_version).validate_images( cluster, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/cdh/confighints_helper.py0000666000175100017510000000275013245514472022721 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp.oozie.workflow_creator import workflow_factory from sahara.utils import xmlutils def get_possible_hive_config_from(file_name): '''Return the possible configs, args, params for a Hive job.''' config = { 'configs': xmlutils.load_hadoop_xml_defaults(file_name), 'params': {} } return config def get_possible_mapreduce_config_from(file_name): '''Return the possible configs, args, params for a MapReduce job.''' config = { 'configs': get_possible_pig_config_from(file_name).get('configs') } config['configs'] += workflow_factory.get_possible_mapreduce_configs() return config def get_possible_pig_config_from(file_name): '''Return the possible configs, args, params for a Pig job.''' config = { 'configs': xmlutils.load_hadoop_xml_defaults(file_name), 'args': [], 'params': {} } return config sahara-8.0.0/sahara/plugins/cdh/config_helper.py0000666000175100017510000003131013245514476021651 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json from sahara.plugins import provisioning as p from sahara.utils import files as f class ConfigHelper(object): path_to_config = '' CDH5_REPO_URL = p.Config( 'CDH5 repo list URL', 'general', 'cluster', priority=1, default_value="") CDH5_REPO_KEY_URL = p.Config( 'CDH5 repo key URL (for debian-based only)', 'general', 'cluster', priority=1, default_value="") CM5_REPO_URL = p.Config( 'CM5 repo list URL', 'general', 'cluster', priority=1, default_value="") CM5_REPO_KEY_URL = p.Config( 'CM5 repo key URL (for debian-based only)', 'general', 'cluster', priority=1, default_value="") ENABLE_HBASE_COMMON_LIB = p.Config( 'Enable HBase Common Lib', 'general', 'cluster', config_type='bool', priority=1, default_value=True) ENABLE_SWIFT = p.Config( 'Enable Swift', 'general', 'cluster', config_type='bool', priority=1, default_value=True) DEFAULT_SWIFT_LIB_URL = ( 'https://repository.cloudera.com/artifactory/repo/org' '/apache/hadoop/hadoop-openstack/2.6.0-cdh5.5.0' '/hadoop-openstack-2.6.0-cdh5.5.0.jar') SWIFT_LIB_URL = p.Config( 'Hadoop OpenStack library URL', 'general', 'cluster', priority=1, default_value=DEFAULT_SWIFT_LIB_URL, description=("Library that adds Swift support to CDH. The file" " will be downloaded by VMs.")) DEFAULT_EXTJS_LIB_URL = ( 'http://tarballs.openstack.org/sahara/dist/common-artifacts/' 'ext-2.2.zip') EXTJS_LIB_URL = p.Config( "ExtJS library URL", 'general', 'cluster', priority=1, default_value=DEFAULT_EXTJS_LIB_URL, description=("Ext 2.2 library is required for Oozie Web Console. " "The file will be downloaded by VMs with oozie.")) _default_executor_classpath = ":".join( ['/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar', '/usr/lib/hadoop-mapreduce/hadoop-openstack.jar']) EXECUTOR_EXTRA_CLASSPATH = p.Config( 'Executor extra classpath', 'Spark', 'cluster', priority=2, default_value=_default_executor_classpath, description='Value for spark.executor.extraClassPath in ' 'spark-defaults.conf (default: %s)' % _default_executor_classpath) KMS_REPO_URL = p.Config( 'KMS repo list URL', 'general', 'cluster', priority=1, default_value="") KMS_REPO_KEY_URL = p.Config( 'KMS repo key URL (for debian-based only)', 'general', 'cluster', priority=1, default_value="") REQUIRE_ANTI_AFFINITY = p.Config( 'Require Anti Affinity', 'general', 'cluster', config_type='bool', priority=2, default_value=True) AWAIT_AGENTS_TIMEOUT = p.Config( 'Await Cloudera agents timeout', 'general', 'cluster', config_type='int', priority=1, default_value=300, is_optional=True, description="Timeout for Cloudera agents connecting to" " Cloudera Manager, in seconds") AWAIT_MANAGER_STARTING_TIMEOUT = p.Config( 'Timeout for Cloudera Manager starting', 'general', 'cluster', config_type='int', priority=1, default_value=300, is_optional=True, description='Timeout for Cloudera Manager starting, in seconds') def __new__(cls): # make it a singleton if not hasattr(cls, '_instance'): cls._instance = super(ConfigHelper, cls).__new__(cls) setattr(cls, '__init__', cls.decorate_init(cls.__init__)) return cls._instance @classmethod def decorate_init(cls, f): """decorate __init__ to prevent multiple calling.""" def wrap(*args, **kwargs): if not hasattr(cls, '_init'): f(*args, **kwargs) cls._init = True return wrap def __init__(self): self.ng_plugin_configs = [] self.priority_one_confs = {} def _load_json(self, path_to_file): data = f.get_file_text(path_to_file) return json.loads(data) def _init_ng_configs(self, confs, app_target, scope): prepare_value = lambda x: x.replace('\n', ' ') if x else "" cfgs = [] for cfg in confs: priority = 1 if cfg['name'] in self.priority_one_confs else 2 c = p.Config(cfg['name'], app_target, scope, priority=priority, default_value=prepare_value(cfg['value']), description=cfg['desc'], is_optional=True) cfgs.append(c) return cfgs def _init_all_ng_plugin_configs(self): self.hdfs_confs = self._load_and_init_configs( 'hdfs-service.json', 'HDFS', 'cluster') self.namenode_confs = self._load_and_init_configs( 'hdfs-namenode.json', 'NAMENODE', 'node') self.datanode_confs = self._load_and_init_configs( 'hdfs-datanode.json', 'DATANODE', 'node') self.secnamenode_confs = self._load_and_init_configs( 'hdfs-secondarynamenode.json', 'SECONDARYNAMENODE', 'node') self.hdfs_gateway_confs = self._load_and_init_configs( 'hdfs-gateway.json', 'HDFS_GATEWAY', 'node') self.journalnode_confs = self._load_and_init_configs( 'hdfs-journalnode.json', 'JOURNALNODE', 'node') self.yarn_confs = self._load_and_init_configs( 'yarn-service.json', 'YARN', 'cluster') self.resourcemanager_confs = self._load_and_init_configs( 'yarn-resourcemanager.json', 'RESOURCEMANAGER', 'node') self.nodemanager_confs = self._load_and_init_configs( 'yarn-nodemanager.json', 'NODEMANAGER', 'node') self.jobhistory_confs = self._load_and_init_configs( 'yarn-jobhistory.json', 'JOBHISTORY', 'node') self.yarn_gateway_conf = self._load_and_init_configs( 'yarn-gateway.json', 'YARN_GATEWAY', 'node') self.oozie_service_confs = self._load_and_init_configs( 'oozie-service.json', 'OOZIE', 'cluster') self.oozie_role_confs = self._load_and_init_configs( 'oozie-oozie_server.json', 'OOZIE', 'node') self.hive_service_confs = self._load_and_init_configs( 'hive-service.json', 'HIVE', 'cluster') self.hive_metastore_confs = self._load_and_init_configs( 'hive-hivemetastore.json', 'HIVEMETASTORE', 'node') self.hive_hiveserver_confs = self._load_and_init_configs( 'hive-hiveserver2.json', 'HIVESERVER', 'node') self.hive_webhcat_confs = self._load_and_init_configs( 'hive-webhcat.json', 'WEBHCAT', 'node') self.hue_service_confs = self._load_and_init_configs( 'hue-service.json', 'HUE', 'cluster') self.hue_role_confs = self._load_and_init_configs( 'hue-hue_server.json', 'HUE', 'node') self.spark_service_confs = self._load_and_init_configs( 'spark-service.json', 'SPARK_ON_YARN', 'cluster') self.spark_role_confs = self._load_and_init_configs( 'spark-spark_yarn_history_server.json', 'SPARK_ON_YARN', 'node') self.zookeeper_server_confs = self._load_and_init_configs( 'zookeeper-service.json', 'ZOOKEEPER', 'cluster') self.zookeeper_service_confs = self._load_and_init_configs( 'zookeeper-server.json', 'ZOOKEEPER', 'node') self.hbase_confs = self._load_and_init_configs( 'hbase-service.json', 'HBASE', 'cluster') self.master_confs = self._load_and_init_configs( 'hbase-master.json', 'MASTER', 'node') self.regionserver_confs = self._load_and_init_configs( 'hbase-regionserver.json', 'REGIONSERVER', 'node') self.flume_service_confs = self._load_and_init_configs( 'flume-service.json', 'FLUME', 'cluster') self.flume_agent_confs = self._load_and_init_configs( 'flume-agent.json', 'FLUME', 'node') self.sentry_service_confs = self._load_and_init_configs( 'sentry-service.json', 'SENTRY', 'cluster') self.sentry_server_confs = self._load_and_init_configs( 'sentry-sentry_server.json', 'SENTRY', 'node') self.solr_service_confs = self._load_and_init_configs( 'solr-service.json', 'SOLR', 'cluster') self.solr_server_confs = self._load_and_init_configs( 'solr-solr_server.json', 'SOLR', 'node') self.sqoop_service_confs = self._load_and_init_configs( 'sqoop-service.json', 'SQOOP', 'cluster') self.sqoop_server_confs = self._load_and_init_configs( 'sqoop-sqoop_server.json', 'SQOOP', 'node') self.ks_indexer_service_confs = self._load_and_init_configs( 'ks_indexer-service.json', 'KS_INDEXER', 'cluster') self.ks_indexer_role_confs = self._load_and_init_configs( 'ks_indexer-hbase_indexer.json', 'KS_INDEXER', 'node') self.impala_service_confs = self._load_and_init_configs( 'impala-service.json', 'IMPALA', 'cluster') self.impala_catalogserver_confs = self._load_and_init_configs( 'impala-catalogserver.json', 'CATALOGSERVER', 'node') self.impala_impalad_confs = self._load_and_init_configs( 'impala-impalad.json', 'IMPALAD', 'node') self.impala_statestore_confs = self._load_and_init_configs( 'impala-statestore.json', 'STATESTORE', 'node') self.kms_service_confs = self._load_and_init_configs( 'kms-service.json', 'KMS', 'cluster') self.kms_kms_confs = self._load_and_init_configs( 'kms-kms.json', 'KMS', 'node') self.kafka_service = self._load_and_init_configs( 'kafka-service.json', 'KAFKA', 'cluster') self.kafka_kafka_broker = self._load_and_init_configs( 'kafka-kafka_broker.json', 'KAFKA', 'node') self.kafka_kafka_mirror_maker = self._load_and_init_configs( 'kafka-kafka_mirror_maker.json', 'KAFKA', 'node') def _load_and_init_configs(self, filename, app_target, scope): confs = self._load_json(self.path_to_config + filename) cfgs = self._init_ng_configs(confs, app_target, scope) self.ng_plugin_configs += cfgs return cfgs def _get_ng_plugin_configs(self): return self.ng_plugin_configs def _get_cluster_plugin_configs(self): return [self.CDH5_REPO_URL, self.CDH5_REPO_KEY_URL, self.CM5_REPO_URL, self.CM5_REPO_KEY_URL, self.ENABLE_SWIFT, self.SWIFT_LIB_URL, self.ENABLE_HBASE_COMMON_LIB, self.EXTJS_LIB_URL, self.AWAIT_MANAGER_STARTING_TIMEOUT, self.AWAIT_AGENTS_TIMEOUT, self.EXECUTOR_EXTRA_CLASSPATH, self.KMS_REPO_URL, self.KMS_REPO_KEY_URL, self.REQUIRE_ANTI_AFFINITY] def get_plugin_configs(self): cluster_wide = self._get_cluster_plugin_configs() ng_wide = self._get_ng_plugin_configs() return cluster_wide + ng_wide def _get_config_value(self, cluster, key): return cluster.cluster_configs.get( 'general', {}).get(key.name, key.default_value) def get_cdh5_repo_url(self, cluster): return self._get_config_value(cluster, self.CDH5_REPO_URL) def get_cdh5_key_url(self, cluster): return self._get_config_value(cluster, self.CDH5_REPO_KEY_URL) def get_cm5_repo_url(self, cluster): return self._get_config_value(cluster, self.CM5_REPO_URL) def get_cm5_key_url(self, cluster): return self._get_config_value(cluster, self.CM5_REPO_KEY_URL) def is_swift_enabled(self, cluster): return self._get_config_value(cluster, self.ENABLE_SWIFT) def is_hbase_common_lib_enabled(self, cluster): return self._get_config_value(cluster, self.ENABLE_HBASE_COMMON_LIB) def get_swift_lib_url(self, cluster): return self._get_config_value(cluster, self.SWIFT_LIB_URL) def get_extjs_lib_url(self, cluster): return self._get_config_value(cluster, self.EXTJS_LIB_URL) def get_kms_key_url(self, cluster): return self._get_config_value(cluster, self.KMS_REPO_KEY_URL) def get_required_anti_affinity(self, cluster): return self._get_config_value(cluster, self.REQUIRE_ANTI_AFFINITY) sahara-8.0.0/sahara/plugins/cdh/v5_7_0/0000775000175100017510000000000013245515027017463 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_7_0/deploy.py0000666000175100017510000001260213245514472021337 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ from sahara.plugins.cdh import commands as cmd from sahara.plugins.cdh import deploy as common_deploy from sahara.plugins.cdh.v5_7_0 import cloudera_utils as cu from sahara.plugins import utils as gu from sahara.service.edp import hdfs_helper as h from sahara.utils import cluster_progress_ops as cpo CU = cu.ClouderaUtilsV570() PACKAGES = common_deploy.PACKAGES def configure_cluster(cluster): instances = gu.get_instances(cluster) if not cmd.is_pre_installed_cdh(CU.pu.get_manager(cluster).remote()): CU.pu.configure_os(instances) CU.pu.install_packages(instances, PACKAGES) CU.pu.start_cloudera_agents(instances) CU.pu.start_cloudera_manager(cluster) CU.update_cloudera_password(cluster) CU.configure_rack_awareness(cluster) CU.await_agents(cluster, instances) CU.create_mgmt_service(cluster) CU.create_services(cluster) CU.configure_services(cluster) CU.configure_instances(instances, cluster) CU.deploy_configs(cluster) @cpo.event_wrapper( True, step=_("Start roles: NODEMANAGER, DATANODE"), param=('cluster', 0)) def _start_roles(cluster, instances): for instance in instances: if 'HDFS_DATANODE' in instance.node_group.node_processes: hdfs = CU.get_service_by_role('DATANODE', instance=instance) CU.start_roles(hdfs, CU.pu.get_role_name(instance, 'DATANODE')) if 'YARN_NODEMANAGER' in instance.node_group.node_processes: yarn = CU.get_service_by_role('NODEMANAGER', instance=instance) CU.start_roles(yarn, CU.pu.get_role_name(instance, 'NODEMANAGER')) def scale_cluster(cluster, instances): if not instances: return if not cmd.is_pre_installed_cdh(instances[0].remote()): CU.pu.configure_os(instances) CU.pu.install_packages(instances, PACKAGES) CU.pu.start_cloudera_agents(instances) CU.await_agents(cluster, instances) CU.configure_rack_awareness(cluster) CU.configure_instances(instances, cluster) CU.update_configs(instances) common_deploy.prepare_scaling_kerberized_cluster( cluster, CU, instances) CU.pu.configure_swift(cluster, instances) _start_roles(cluster, instances) CU.refresh_datanodes(cluster) CU.refresh_yarn_nodes(cluster) CU.restart_stale_services(cluster) def decommission_cluster(cluster, instances): dns = [] dns_to_delete = [] nms = [] nms_to_delete = [] for i in instances: if 'HDFS_DATANODE' in i.node_group.node_processes: dns.append(CU.pu.get_role_name(i, 'DATANODE')) dns_to_delete.append( CU.pu.get_role_name(i, 'HDFS_GATEWAY')) if 'YARN_NODEMANAGER' in i.node_group.node_processes: nms.append(CU.pu.get_role_name(i, 'NODEMANAGER')) nms_to_delete.append( CU.pu.get_role_name(i, 'YARN_GATEWAY')) if dns: CU.decommission_nodes( cluster, 'DATANODE', dns, dns_to_delete) if nms: CU.decommission_nodes( cluster, 'NODEMANAGER', nms, nms_to_delete) CU.delete_instances(cluster, instances) CU.refresh_datanodes(cluster) CU.refresh_yarn_nodes(cluster) CU.restart_stale_services(cluster) @cpo.event_wrapper(True, step=_("Prepare cluster"), param=('cluster', 0)) def _prepare_cluster(cluster): if CU.pu.get_oozie(cluster): CU.pu.install_extjs(cluster) if CU.pu.get_hive_metastore(cluster): CU.pu.configure_hive(cluster) if CU.pu.get_sentry(cluster): CU.pu.configure_sentry(cluster) @cpo.event_wrapper( True, step=_("Finish cluster starting"), param=('cluster', 0)) def _finish_cluster_starting(cluster): if CU.pu.get_hive_metastore(cluster): CU.pu.put_hive_hdfs_xml(cluster) server = CU.pu.get_hbase_master(cluster) if CU.pu.c_helper.is_hbase_common_lib_enabled(cluster) and server: with server.remote() as r: h.create_hbase_common_lib(r) if CU.pu.get_flumes(cluster): flume = CU.get_service_by_role('AGENT', cluster) CU.start_service(flume) def start_cluster(cluster): _prepare_cluster(cluster) CU.first_run(cluster) CU.pu.configure_swift(cluster) if len(CU.pu.get_jns(cluster)) > 0: CU.enable_namenode_ha(cluster) # updating configs for NameNode role on needed nodes CU.update_role_config(CU.pu.get_secondarynamenode(cluster), 'HDFS_NAMENODE') if CU.pu.get_stdb_rm(cluster): CU.enable_resourcemanager_ha(cluster) # updating configs for ResourceManager on needed nodes CU.update_role_config(CU.pu.get_stdb_rm(cluster), 'YARN_STANDBYRM') _finish_cluster_starting(cluster) common_deploy.setup_kerberos_for_cluster(cluster, CU) def get_open_ports(node_group): ports = common_deploy.get_open_ports(node_group) return ports sahara-8.0.0/sahara/plugins/cdh/v5_7_0/validation.py0000666000175100017510000000136513245514472022201 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_7_0 import plugin_utils as pu from sahara.plugins.cdh import validation class ValidatorV570(validation.Validator): PU = pu.PluginUtilsV570() sahara-8.0.0/sahara/plugins/cdh/v5_7_0/versionhandler.py0000666000175100017510000000270213245514472023066 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import abstractversionhandler as avm from sahara.plugins.cdh.v5_7_0 import cloudera_utils from sahara.plugins.cdh.v5_7_0 import config_helper from sahara.plugins.cdh.v5_7_0 import deploy from sahara.plugins.cdh.v5_7_0 import edp_engine from sahara.plugins.cdh.v5_7_0 import images from sahara.plugins.cdh.v5_7_0 import plugin_utils from sahara.plugins.cdh.v5_7_0 import validation class VersionHandler(avm.BaseVersionHandler): def __init__(self): super(VersionHandler, self).__init__() self.config_helper = config_helper.ConfigHelperV570() self.cloudera_utils = cloudera_utils.ClouderaUtilsV570() self.plugin_utils = plugin_utils.PluginUtilsV570() self.deploy = deploy self.edp_engine = edp_engine self.images = images self.validation = validation.ValidatorV570() sahara-8.0.0/sahara/plugins/cdh/v5_7_0/config_helper.py0000666000175100017510000000737013245514472022655 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import config_helper as c_h from sahara.plugins import provisioning as p from sahara.utils import files as f class ConfigHelperV570(c_h.ConfigHelper): path_to_config = 'plugins/cdh/v5_7_0/resources/' CDH5_UBUNTU_REPO = ( 'deb [arch=amd64] http://archive.cloudera.com/cdh5' '/ubuntu/trusty/amd64/cdh trusty-cdh5.7.0 contrib' '\ndeb-src http://archive.cloudera.com/cdh5/ubuntu' '/trusty/amd64/cdh trusty-cdh5.7.0 contrib') DEFAULT_CDH5_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/cdh5/ubuntu' '/trusty/amd64/cdh/archive.key') CM5_UBUNTU_REPO = ( 'deb [arch=amd64] http://archive.cloudera.com/cm5' '/ubuntu/trusty/amd64/cm trusty-cm5.7.0 contrib' '\ndeb-src http://archive.cloudera.com/cm5/ubuntu' '/trusty/amd64/cm trusty-cm5.7.0 contrib') DEFAULT_CM5_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/cm5/ubuntu' '/trusty/amd64/cm/archive.key') CDH5_CENTOS_REPO = ( '[cloudera-cdh5]' '\nname=Cloudera\'s Distribution for Hadoop, Version 5' '\nbaseurl=http://archive.cloudera.com/cdh5/redhat/6' '/x86_64/cdh/5.7.0/' '\ngpgkey = http://archive.cloudera.com/cdh5/redhat/6' '/x86_64/cdh/RPM-GPG-KEY-cloudera' '\ngpgcheck = 1') CM5_CENTOS_REPO = ( '[cloudera-manager]' '\nname=Cloudera Manager' '\nbaseurl=http://archive.cloudera.com/cm5/redhat/6' '/x86_64/cm/5.7.0/' '\ngpgkey = http://archive.cloudera.com/cm5/redhat/6' '/x86_64/cm/RPM-GPG-KEY-cloudera' '\ngpgcheck = 1') KEY_TRUSTEE_UBUNTU_REPO_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/ubuntu/trusty/amd64/navigator-' 'keytrustee/cloudera.list') DEFAULT_KEY_TRUSTEE_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/ubuntu/trusty/amd64/navigator-' 'keytrustee/archive.key') KEY_TRUSTEE_CENTOS_REPO_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/redhat/6/x86_64/navigator-' 'keytrustee/navigator-keytrustee5.repo') DEFAULT_SWIFT_LIB_URL = ( 'https://repository.cloudera.com/artifactory/repo/org' '/apache/hadoop/hadoop-openstack/2.6.0-cdh5.7.0' '/hadoop-openstack-2.6.0-cdh5.7.0.jar') SWIFT_LIB_URL = p.Config( 'Hadoop OpenStack library URL', 'general', 'cluster', priority=1, default_value=DEFAULT_SWIFT_LIB_URL, description=("Library that adds Swift support to CDH. The file" " will be downloaded by VMs.")) HIVE_SERVER2_SENTRY_SAFETY_VALVE = f.get_file_text( path_to_config + 'hive-server2-sentry-safety.xml') HIVE_METASTORE_SENTRY_SAFETY_VALVE = f.get_file_text( path_to_config + 'hive-metastore-sentry-safety.xml') SENTRY_IMPALA_CLIENT_SAFETY_VALVE = f.get_file_text( path_to_config + 'sentry-impala-client-safety.xml') def __init__(self): super(ConfigHelperV570, self).__init__() self.priority_one_confs = self._load_json( self.path_to_config + 'priority-one-confs.json') self._init_all_ng_plugin_configs() sahara-8.0.0/sahara/plugins/cdh/v5_7_0/images.py0000666000175100017510000000277313245514472021320 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import images from sahara.plugins import utils as plugin_utils _validator = images.SaharaImageValidator.from_yaml( 'plugins/cdh/v5_7_0/resources/images/image.yaml', resource_roots=['plugins/cdh/v5_7_0/resources/images']) def get_image_arguments(): return _validator.get_argument_list() def pack_image(remote, test_only=False, image_arguments=None): _validator.validate(remote, test_only=test_only, image_arguments=image_arguments) def validate_images(cluster, test_only=False, image_arguments=None): image_arguments = get_image_arguments() if not test_only: instances = plugin_utils.get_instances(cluster) else: instances = plugin_utils.get_instances(cluster)[0] for instance in instances: with instance.remote() as r: _validator.validate(r, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/cdh/v5_7_0/cloudera_utils.py0000666000175100017510000000203113245514472023054 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import cloudera_utils as cu from sahara.plugins.cdh.v5_7_0 import config_helper from sahara.plugins.cdh.v5_7_0 import plugin_utils as pu from sahara.plugins.cdh.v5_7_0 import validation class ClouderaUtilsV570(cu.ClouderaUtils): def __init__(self): cu.ClouderaUtils.__init__(self) self.pu = pu.PluginUtilsV570() self.validator = validation.ValidatorV570 self.c_helper = config_helper.ConfigHelperV570() sahara-8.0.0/sahara/plugins/cdh/v5_7_0/__init__.py0000666000175100017510000000000013245514472021567 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_7_0/plugin_utils.py0000666000175100017510000000146013245514472022561 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import plugin_utils as pu from sahara.plugins.cdh.v5_7_0 import config_helper class PluginUtilsV570(pu.AbstractPluginUtils): def __init__(self): self.c_helper = config_helper.ConfigHelperV570() sahara-8.0.0/sahara/plugins/cdh/v5_7_0/edp_engine.py0000666000175100017510000000366413245514472022150 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import confighints_helper as ch_helper from sahara.plugins.cdh import edp_engine from sahara.plugins.cdh.v5_7_0 import cloudera_utils as cu from sahara.service.edp.oozie import engine as oozie_engine from sahara.utils import edp class EdpOozieEngine(edp_engine.EdpOozieEngine): def __init__(self, cluster): super(EdpOozieEngine, self).__init__(cluster) self.cloudera_utils = cu.ClouderaUtilsV570() @staticmethod def get_possible_job_config(job_type): if edp.compare_job_type(job_type, edp.JOB_TYPE_HIVE): return {'job_config': ch_helper.get_possible_hive_config_from( 'plugins/cdh/v5_7_0/resources/hive-site.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_MAPREDUCE_STREAMING): return {'job_config': ch_helper.get_possible_mapreduce_config_from( 'plugins/cdh/v5_7_0/resources/mapred-site.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_PIG): return {'job_config': ch_helper.get_possible_pig_config_from( 'plugins/cdh/v5_7_0/resources/mapred-site.xml')} return oozie_engine.OozieJobEngine.get_possible_job_config(job_type) class EdpSparkEngine(edp_engine.EdpSparkEngine): edp_base_version = "5.7.0" sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/0000775000175100017510000000000013245515027021475 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/sentry-sentry_server.json0000666000175100017510000004615313245514472026642 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_sentry_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_sentry_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Enables the health test that the Sentry Server's process state is consistent with the role configuration", "display_name": "Sentry Server Process Health Test", "name": "sentry_server_scm_health_enabled", "value": "true" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The minimum log level for Sentry Server logs", "display_name": "Sentry Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_sentry_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_sentry_server_unexpected_exits", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Sentry Server logs. Typically used by log4j or logback.", "display_name": "Sentry Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_sentry_server_scm_health", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_sentry_server_swap_memory_usage", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Log Directory", "name": "role_config_suppression_sentry_server_log_dir", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Sentry Server in Bytes", "name": "sentry_server_java_heapsize", "value": "1073741824" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Sentry Server", "name": "sentry_server_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Sentry Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "sentry_env_safety_valve", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "RPC port number of Sentry Server.", "display_name": "Sentry Server RPC Port", "name": "sentry_service_server_rpc_port", "value": "8038" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_sentry_server_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The maximum size, in megabytes, per log file for Sentry Server logs. Typically used by log4j or logback.", "display_name": "Sentry Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "When computing the overall Sentry Server health, consider the host's health.", "display_name": "Sentry Server Host Health Test", "name": "sentry_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Sentry Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Sentry Server", "name": "role_config_suppression_sentry_server_java_opts", "value": "false" }, { "desc": "Directory where Sentry Server will place its log files.", "display_name": "Sentry Server Log Directory", "name": "sentry_server_log_dir", "value": "/var/log/sentry" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_sentry_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "sentry_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Sentry Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/solr-service.json0000666000175100017510000006633213245514472025024 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode", "name": "service_config_suppression_zookeeper_znode", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_solr_https_keystore_file", "value": "false" }, { "desc": "Whether to suppress the results of the Solr Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Solr Server Health", "name": "service_health_suppression_solr_solr_servers_healthy", "value": "false" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access.", "display_name": "Sentry User to Group Mapping Class", "name": "solr_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "The health test thresholds of the overall Solr Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Solr Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Solr Servers falls below the critical threshold.", "display_name": "Healthy Solr Server Monitoring Thresholds", "name": "solr_solr_servers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/solr/audit" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_solr_https_keystore_password", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Enable the background watchdog thread that can kill Catalina process if Solr is not responsive.", "display_name": "Enable Solrd Watchdog", "name": "solrd_enable_watchdog", "value": "true" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "solr" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "solr_hdfs_site_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "The password for the Solr JKS keystore file.", "display_name": "Solr TLS/SSL Server JKS Keystore File Password", "name": "solr_https_keystore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the LDAP TLS Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP TLS Validator", "name": "service_config_suppression_solr_ldap_tls_validator", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "solr" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. Sentry is supported only on Search 1.1 or later and CDH 5 or later deployments and requires authentication to be turned on for Solr..", "display_name": "Enable Sentry Authorization", "name": "solr_sentry_enabled", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Solr events:\n

\n\n
    \n
  • operation: the Solr operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_solr_sentry_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP Secure URI and Start TLS Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP Secure URI and Start TLS Validator", "name": "service_config_suppression_solr_ldaps_or_tls_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_solr_ldap_basedn", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Solr might connect to. This is used when Solr is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Solr TLS/SSL Certificate Trust Store File", "name": "solr_https_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_sentry_solr_provider_resource", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "solr_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_solr_https_truststore_file", "value": "false" }, { "desc": "This parameter is useful when authenticating against a non Active Directory server such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server will look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "solr_ldap_basedn", "value": null }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "solr_ldap_uri", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_ldap_domain", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_solr_core_site_safety_valve", "value": "false" }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "solr_ldap_enable_starttls", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "solr_sentry_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_solr_https_truststore_password", "value": "false" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Solr events:\n

\n\n
    \n
  • operation: the Solr operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Solr Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Solr Server Count Validator", "name": "service_config_suppression_solr_server_count_validator", "value": "false" }, { "desc": "ZooKeeper znode used to store information about this Solr service.", "display_name": "ZooKeeper Znode", "name": "zookeeper_znode", "value": "/solr" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_solr_ldap_uri", "value": "false" }, { "desc": "Name of the HDFS service that this Search service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Encrypt communication between clients and Solr using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)). Additional manual steps must be performed; see Enabling TLS/SSL for Solr.", "display_name": "Enable TLS/SSL for Solr", "name": "solr_use_ssl", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "solr" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "ldap_domain", "value": null }, { "desc": "If Solr does not respond on its web URL within this time interval, the Catalina process is killed.", "display_name": "Solrd Watchdog Timeout", "name": "solrd_watchdog_timeout", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Data Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Data Directory", "name": "service_config_suppression_hdfs_data_dir", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Solr is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Solr TLS/SSL Server JKS Keystore File Location", "name": "solr_https_keystore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "service_config_suppression_solr_hdfs_site_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_solr_env_safety_valve", "value": "false" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "solr_enable_ldap_auth", "value": "false" }, { "desc": "The password for the Solr TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Solr TLS/SSL Certificate Trust Store Password", "name": "solr_https_truststore_password", "value": null }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "sentry_solr_provider_resource", "value": "/user/solr/sentry/sentry-provider.ini" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "HDFS directory used for storage by this Solr service.", "display_name": "HDFS Data Directory", "name": "hdfs_data_dir", "value": "/solr" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Choose the authentication mechanism used by Solr.", "display_name": "Solr Secure Authentication", "name": "solr_security_authentication", "value": "simple" }, { "desc": "Name of the ZooKeeper service that this Search service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "solr_core_site_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/flume-agent.json0000666000175100017510000012217113245514472024605 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_flume_agent_log_directory_free_space", "value": "false" }, { "desc": "Enables the health test that the Agent's process state is consistent with the role configuration", "display_name": "Agent Process Health Test", "name": "flume_agent_scm_health_enabled", "value": "true" }, { "desc": "When computing the overall Agent health, consider the host's health.", "display_name": "Agent Host Health Test", "name": "flume_agent_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Name parameter.", "display_name": "Suppress Parameter Validation: Agent Name", "name": "role_config_suppression_agent_name", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_flume_agent_host_health", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Agent Environment Advanced Configuration Snippet (Safety Valve)", "name": "AGENT_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Agent Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Agent in Bytes", "name": "agent_java_heapsize", "value": "1073741824" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Text that goes verbatim into grok-dictionary.conf file used by the Flume-NG Solr sink.", "display_name": "Grok Dictionary File", "name": "agent_grok_dictionary_conf_file", "value": "USERNAME [a-zA-Z0-9._-]+\nUSER %{USERNAME}\nINT (?:[+-]?(?:[0-9]+))\nBASE10NUM (?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))\nNUMBER (?:%{BASE10NUM})\nBASE16NUM (?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))\nUUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}\n\n# Networking\nMAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})\nCISCOMAC (?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})\nWINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})\nCOMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})\nIP (?/(?>[\\w_%!$@:.,-]+|\\\\.)*)+\n#UNIXPATH (?/dev/pts/%{NONNEGINT})\nBSDTTY (?>/dev/tty[pq][a-z0-9])\nTTY (?:%{BSDTTY}|%{LINUXTTY})\nWINPATH (?>[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+\nURIPROTO [A-Za-z]+(\\+[A-Za-z+]+)?\nURIHOST %{IPORHOST}(?::%{POSINT:port})?\n# uripath comes loosely from RFC1738, but mostly from what Firefox\n# doesn't turn into %XX\nURIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=#%_\\-]*)+\n#URIPARAM \\?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?\nURIPARAM \\?[A-Za-z0-9$.+!*'|(){},~#%&/=:;_?\\-\\[\\]]*\nURIPATHPARAM %{URIPATH}(?:%{URIPARAM})?\nURI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?\n\n# Months: January, Feb, 3, 03, 12, December\nMONTH \\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b\nMONTHNUM (?:0?[1-9]|1[0-2])\nMONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\n\n# Days: Monday, Tue, Thu, etc...\nDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)\n\n# Years?\nYEAR (?>\\d\\d){1,2}\n# Time: HH:MM:SS\n#TIME \\d{2}:\\d{2}(?::\\d{2}(?:\\.\\d+)?)?\n# I'm still on the fence about using grok to perform the time match,\n# since it's probably slower.\n# TIME %{POSINT<24}:%{POSINT<60}(?::%{POSINT<60}(?:\\.%{POSINT})?)?\nHOUR (?:2[0123]|[01]?[0-9])\nMINUTE (?:[0-5][0-9])\n# '60' is a leap second in most time standards and thus is valid.\nSECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\nTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])\n# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)\nDATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}\nDATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}\nISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))\nISO8601_SECOND (?:%{SECOND}|60)\nTIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?\nDATE %{DATE_US}|%{DATE_EU}\nDATESTAMP %{DATE}[- ]%{TIME}\nTZ (?:[PMCE][SD]T)\nDATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}\nDATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}\n\n# Syslog Dates: Month Day HH:MM:SS\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}\nPROG (?:[\\w._/%-]+)\nSYSLOGPROG %{PROG:program}(?:\\[%{POSINT:pid}\\])?\nSYSLOGHOST %{IPORHOST}\nSYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>\nHTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}\n\n# Shortcuts\nQS %{QUOTEDSTRING}\n\n# Log formats\nSYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:\nCOMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}\n\n# Log Levels\nLOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" }, { "desc": "The minimum log level for Agent logs", "display_name": "Agent Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Used to select an agent configuration to use from flume.conf. Multiple agents may share the same agent name, in which case they will be assigned the same agent configuration.", "display_name": "Agent Name", "name": "agent_name", "value": "tier1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Text that goes verbatim into custom-mimetypes.xml file used by the Flume-NG Solr sink.", "display_name": "Custom Mime-types File", "name": "agent_custom_mimetypes_file", "value": "\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Grok Dictionary File parameter.", "display_name": "Suppress Parameter Validation: Grok Dictionary File", "name": "role_config_suppression_agent_grok_dictionary_conf_file", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "agent_web_metric_collection_enabled", "value": "true" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "agent_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_flume_agent_web_metric_collection", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Agent logs. Typically used by log4j or logback.", "display_name": "Agent Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Home Directory parameter.", "display_name": "Suppress Parameter Validation: Flume Home Directory", "name": "role_config_suppression_agent_home_dir", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_flume_agent_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_flume_agent_swap_memory_usage", "value": "false" }, { "desc": "Disables import of ZooKeeper configuration from the HBase classpath. This prevents zoo.cfg from overriding hbase-site.xml for Zookeeper quorum information. This option is only supported on CDH 4.4 or later deployments.", "display_name": "HBase sink prefer hbase-site.xml over Zookeeper config", "name": "agent_disable_zoo_cfg", "value": "true" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_flume_agent_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_flume_agent_file_descriptor", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Agent Log Directory parameter.", "display_name": "Suppress Parameter Validation: Flume Agent Log Directory", "name": "role_config_suppression_flume_agent_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Morphlines File parameter.", "display_name": "Suppress Parameter Validation: Morphlines File", "name": "role_config_suppression_agent_morphlines_conf_file", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here. Note that Flume agent only uses options that start with -D and -X (including -XX).", "display_name": "Java Configuration Options for Flume Agent", "name": "flume_agent_java_opts", "value": "" }, { "desc": "Text that goes into morphlines.conf file used by the Flume-NG Solr sink. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service.", "display_name": "Morphlines File", "name": "agent_morphlines_conf_file", "value": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Application configuration file in HOCON format (Human-Optimized Config Object Notation). \n# HOCON syntax is defined at http://github.com/typesafehub/config/blob/master/HOCON.md\n# and also used by Akka (http://www.akka.io) and Play (http://www.playframework.org/).\n# For more examples see http://doc.akka.io/docs/akka/2.1.2/general/configuration.html\n\n# morphline.conf example file\n# this is a comment\n\n# Specify server locations in a SOLR_LOCATOR variable; used later in variable substitutions:\nSOLR_LOCATOR : {\n # Name of solr collection\n collection : collection1\n \n # ZooKeeper ensemble\n zkHost : \"$ZK_HOST\"\n \n # Relative or absolute path to a directory containing conf/solrconfig.xml and conf/schema.xml\n # If this path is uncommented it takes precedence over the configuration stored in ZooKeeper. \n # solrHomeDir : \"example/solr/collection1\"\n \n # The maximum number of documents to send to Solr per network batch (throughput knob)\n # batchSize : 100\n}\n\n# Specify an array of one or more morphlines, each of which defines an ETL \n# transformation chain. A morphline consists of one or more (potentially \n# nested) commands. A morphline is a way to consume records (e.g. Flume events, \n# HDFS files or blocks), turn them into a stream of records, and pipe the stream \n# of records through a set of easily configurable transformations on it's way to \n# Solr (or a MapReduceIndexerTool RecordWriter that feeds via a Reducer into Solr).\nmorphlines : [\n {\n # Name used to identify a morphline. E.g. used if there are multiple morphlines in a \n # morphline config file\n id : morphline1 \n \n # Import all morphline commands in these java packages and their subpackages.\n # Other commands that may be present on the classpath are not visible to this morphline.\n importCommands : [\"org.kitesdk.**\", \"org.apache.solr.**\"]\n \n commands : [ \n { \n # Parse Avro container file and emit a record for each avro object\n readAvroContainer {\n # Optionally, require the input record to match one of these MIME types:\n # supportedMimeTypes : [avro/binary]\n \n # Optionally, use a custom Avro schema in JSON format inline:\n # schemaString : \"\"\"\"\"\"\n \n # Optionally, use a custom Avro schema file in JSON format:\n # schemaFile : /path/to/syslog.avsc\n }\n } \n \n { \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # extractAvroPaths is a command that uses zero or more avro path expressions to extract \n # values from an Avro object. Each expression consists of a record output field name (on \n # the left side of the colon ':') as well as zero or more path steps (on the right hand \n # side), each path step separated by a '/' slash. Avro arrays are traversed with the '[]'\n # notation.\n #\n # The result of a path expression is a list of objects, each of which is added to the \n # given record output field.\n # \n # The path language supports all Avro concepts, including nested structures, records, \n # arrays, maps, unions, etc, as well as a flatten option that collects the primitives in \n # a subtree into a flat list.\n extractAvroPaths {\n flatten : false\n paths : { \n id : /id \n text : /text \n user_friends_count : /user_friends_count\n user_location : /user_location\n user_description : /user_description\n user_statuses_count : /user_statuses_count\n user_followers_count : /user_followers_count\n user_name : /user_name\n user_screen_name : /user_screen_name\n created_at : /created_at\n retweet_count : /retweet_count\n retweeted : /retweeted\n in_reply_to_user_id : /in_reply_to_user_id\n source : /source\n in_reply_to_status_id : /in_reply_to_status_id\n media_url_https : /media_url_https\n expanded_url : /expanded_url\n }\n }\n }\n \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # convert timestamp field to native Solr timestamp format\n # e.g. 2012-09-06T07:14:34Z to 2012-09-06T07:14:34.000Z\n {\n convertTimestamp {\n field : created_at\n inputFormats : [\"yyyy-MM-dd'T'HH:mm:ss'Z'\", \"yyyy-MM-dd\"]\n inputTimezone : America/Los_Angeles\n# outputFormat : \"yyyy-MM-dd'T'HH:mm:ss.SSSZ\" \n outputTimezone : UTC\n }\n }\n \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # Command that sanitizes record fields that are unknown to Solr schema.xml by either \n # deleting them (renameToPrefix is absent or a zero length string), or by moving them to a\n # field prefixed with the given renameToPrefix (e.g. renameToPrefix = \"ignored_\" to use \n # typical dynamic Solr fields).\n #\n # Recall that Solr throws an exception on any attempt to load a document that contains a \n # field that isn't specified in schema.xml.\n {\n sanitizeUnknownSolrFields {\n # Location from which to fetch Solr schema\n solrLocator : ${SOLR_LOCATOR}\n \n # renameToPrefix : \"ignored_\"\n }\n } \n \n # log the record at DEBUG level to SLF4J\n { logDebug { format : \"output record: {}\", args : [\"@{}\"] } } \n \n # load the record into a SolrServer or MapReduce SolrOutputFormat.\n { \n loadSolr {\n solrLocator : ${SOLR_LOCATOR}\n }\n }\n ]\n }\n]\n" }, { "desc": "Verbatim contents of flume.conf. Multiple agents may be configured from the same configuration file; the Agent Name setting can be overridden to select which agent configuration to use for each agent.

To integrate with a secured cluster, you can use the substitution strings \"$KERBEROS_PRINCIPAL\" and \"$KERBEROS_KEYTAB\", which will be replaced by the principal name and the keytab path respectively.", "display_name": "Configuration File", "name": "agent_config_file", "value": "# Please paste flume.conf here. Example:\n\n# Sources, channels, and sinks are defined per\n# agent name, in this case 'tier1'.\ntier1.sources = source1\ntier1.channels = channel1\ntier1.sinks = sink1\n\n# For each source, channel, and sink, set\n# standard properties.\ntier1.sources.source1.type = netcat\ntier1.sources.source1.bind = 127.0.0.1\ntier1.sources.source1.port = 9999\ntier1.sources.source1.channels = channel1\ntier1.channels.channel1.type = memory\ntier1.sinks.sink1.type = logger\ntier1.sinks.sink1.channel = channel1\n\n# Other properties are specific to each type of\n# source, channel, or sink. In this case, we\n# specify the capacity of the memory channel.\ntier1.channels.channel1.capacity = 100\n" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_flume_agent_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Flume Agent parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Flume Agent", "name": "role_config_suppression_flume_agent_java_opts", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Directory where Flume Agent will place its log files.", "display_name": "Flume Agent Log Directory", "name": "flume_agent_log_dir", "value": "/var/log/flume-ng" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Custom Mime-types File parameter.", "display_name": "Suppress Parameter Validation: Custom Mime-types File", "name": "role_config_suppression_agent_custom_mimetypes_file", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "flume_agent_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "List of Flume plugin directories. This overrides the default Flume plugin directory.", "display_name": "Plugin directories", "name": "agent_plugin_dirs", "value": "/usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d" }, { "desc": "The maximum size, in megabytes, per log file for Agent logs. Typically used by log4j or logback.", "display_name": "Agent Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Plugin directories parameter.", "display_name": "Suppress Parameter Validation: Plugin directories", "name": "role_config_suppression_agent_plugin_dirs", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Configuration File parameter.", "display_name": "Suppress Parameter Validation: Configuration File", "name": "role_config_suppression_agent_config_file", "value": "false" }, { "desc": "Home directory for Flume user. The File Channel uses paths for checkpoint and data directories that are within the user home.", "display_name": "Flume Home Directory", "name": "agent_home_dir", "value": "/var/lib/flume-ng" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Agent Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The port on which the Flume web server listens for requests.", "display_name": "HTTP Port", "name": "agent_http_port", "value": "41414" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Agent Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_agent_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/sentry-impala-client-safety.xml0000666000175100017510000000063613245514472027563 0ustar zuulzuul00000000000000 sentry.service.client.server.rpc-port 3893 sentry.service.client.server.rpc-address hostname sentry.service.client.server.rpc-connection-timeout 200000 sentry.service.security.mode none sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-failovercontroller.json0000666000175100017510000005556213245514472027247 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hdfs_failovercontroller_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Failover Controller", "name": "failover_controller_java_opts", "value": "" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Failover Controller in Bytes", "name": "failover_controller_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Failover Controller parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Failover Controller", "name": "role_config_suppression_failover_controller_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Log Directory parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Log Directory", "name": "role_config_suppression_failover_controller_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hdfs_failovercontroller_log_directory_free_space", "value": "false" }, { "desc": "When computing the overall Failover Controller health, consider the host's health.", "display_name": "Failover Controller Host Health Test", "name": "failovercontroller_host_health_enabled", "value": "true" }, { "desc": "The minimum log level for Failover Controller logs", "display_name": "Failover Controller Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_fc_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_failovercontroller_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hdfs_failovercontroller_scm_health", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Failover Controller logs. Typically used by log4j or logback.", "display_name": "Failover Controller Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The RPC timeout for the HA health monitor.", "display_name": "HA Health Monitor RPC Timeout", "name": "ha_health_monitor_rpc_timeout_ms", "value": "45000" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Enables the health test that the Failover Controller's process state is consistent with the role configuration", "display_name": "Failover Controller Process Health Test", "name": "failovercontroller_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hdfs_failovercontroller_file_descriptor", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hdfs_failovercontroller_host_health", "value": "false" }, { "desc": "Directory where Failover Controller will place its log files.", "display_name": "Failover Controller Log Directory", "name": "failover_controller_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "fc_config_safety_valve", "value": null }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Failover Controller Environment Advanced Configuration Snippet (Safety Valve)", "name": "FAILOVERCONTROLLER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hdfs_failovercontroller_swap_memory_usage", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Failover Controller logs. Typically used by log4j or logback.", "display_name": "Failover Controller Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Failover Controller Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "failovercontroller_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hdfs_failovercontroller_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/oozie-oozie_server.json0000666000175100017510000011402313245514472026234 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action From Address parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action From Address", "name": "role_config_suppression_oozie_email_from_address", "value": "false" }, { "desc": "The SMTP server port to use for Oozie email action", "display_name": "Oozie Email Action SMTP Port", "name": "oozie_email_smtp_prt", "value": "25" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_oozie_https_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie ActionService Executor Extension Classes parameter.", "display_name": "Suppress Parameter Validation: Oozie ActionService Executor Extension Classes", "name": "role_config_suppression_oozie_executor_extension_classes", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "If true, enables the Oozie Server web console. ExtJS 2.2 zip archive must be extracted to /var/lib/oozie on the same host as the Oozie Server.", "display_name": "Enable Oozie Server Web Console", "name": "oozie_web_console", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Oozie Server in Bytes", "name": "oozie_java_heapsize", "value": "1073741824" }, { "desc": "Completed workflow jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Workflow Jobs", "name": "purgeservice_older_than", "value": "30" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Completed coordinator jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Coordinator Jobs", "name": "purgeservice_coord_older_than", "value": "7" }, { "desc": "If true, enables version check for Oozie Server and installed shared libraries.", "display_name": "Enable Oozie Server Shared Libraries Version Check", "name": "oozie_server_shared_lib_version_check_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Name parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Name", "name": "role_config_suppression_oozie_database_name", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_oozie_server_file_descriptor", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_oozie_https_keystore_file", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Oozie Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "OOZIE_SERVER_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Username for connecting to the database used by Oozie Server. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database User", "name": "oozie_database_user", "value": "sa" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Oozie might connect to. This is used when Oozie is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Oozie TLS/SSL Certificate Trust Store File", "name": "oozie_https_truststore_file", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_oozie_https_keystore_password", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Enables the health test that the Oozie Server's process state is consistent with the role configuration", "display_name": "Oozie Server Process Health Test", "name": "oozie_server_scm_health_enabled", "value": "true" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Comma-separated list of Oozie plug-ins to be activated. If one plugin cannot be loaded, all the plugins are ignored.", "display_name": "Oozie Server Plugins", "name": "oozie_plugins_list", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Authentication Username parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Authentication Username", "name": "role_config_suppression_oozie_email_smtp_username", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie SchemaService Workflow Extension Schemas parameter.", "display_name": "Suppress Parameter Validation: Oozie SchemaService Workflow Extension Schemas", "name": "role_config_suppression_oozie_workflow_extension_schemas", "value": "false" }, { "desc": "When computing the overall Oozie Server health, consider the host's health.", "display_name": "Oozie Server Host Health Test", "name": "oozie_server_host_health_enabled", "value": "true" }, { "desc": "SMTP username for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Username", "name": "oozie_email_smtp_username", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_oozie_server_role_env_safety_valve", "value": "false" }, { "desc": "Number of threads used for executing callables", "display_name": "Number Threads For Executing Callables", "name": "oozie_service_callablequeueservice_threads", "value": "50" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "oozie_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only. A string to be inserted into oozie-site.xml for this role only.", "display_name": "Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml", "name": "oozie_config_safety_valve", "value": null }, { "desc": "The admin port Oozie server runs.", "display_name": "Oozie Admin Port", "name": "oozie_admin_port", "value": "11001" }, { "desc": "Directory where Oozie Server will place its log files.", "display_name": "Oozie Server Log Directory", "name": "oozie_log_dir", "value": "/var/log/oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database User parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database User", "name": "role_config_suppression_oozie_database_user", "value": "false" }, { "desc": "The SMTP server host to use for Oozie email action", "display_name": "Oozie Email Action SMTP Host", "name": "oozie_email_smtp_host", "value": "localhost" }, { "desc": "Comma-separated list of SchemaService workflow extension schemas for additional action types. From CDH 5.4 and higher, this parameter is used only to specify additional schemas for workflows. All schemas included in that release will be added automatically and do not need to be specified.", "display_name": "Oozie SchemaService Workflow Extension Schemas", "name": "oozie_workflow_extension_schemas", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Host parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Host", "name": "role_config_suppression_oozie_email_smtp_host", "value": "false" }, { "desc": "Directory where the Oozie Server places its data. Only applicable when using Derby as the database type.", "display_name": "Oozie Server Data Directory", "name": "oozie_data_dir", "value": "/var/lib/oozie/data" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Oozie is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Oozie TLS/SSL Server JKS Keystore File Location", "name": "oozie_https_keystore_file", "value": "/var/lib/oozie/.keystore" }, { "desc": "Completed bundle jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Bundle Jobs", "name": "purgeservice_bundle_older_than", "value": "7" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_oozie_server_host_health", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Maximum concurrency for a given callable type. Each command is a callable type: submit, start, run, etc. Each action type is a callable type: MapReduce, SSH, sub-workflow, etc. All commands that use action executors (action-start, action-end. etc.) use the action type as the callable type.", "display_name": "Maximum concurrency for a given callable type", "name": "oozie_service_callablequeueservice_callable_concurrency", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Data Directory parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Data Directory", "name": "role_config_suppression_oozie_data_dir", "value": "false" }, { "desc": "The password for the Oozie TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Oozie TLS/SSL Certificate Trust Store Password", "name": "oozie_https_truststore_password", "value": null }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_oozie_server_web_metric_collection", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Oozie Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "SMTP password for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Password", "name": "oozie_email_smtp_password", "value": null }, { "desc": "The health test thresholds of the weighted average size of the Oozie Server callable queue over a recent period. See also Oozie Server Callable Queue Monitoring Period.", "display_name": "Oozie Server Callable Queue Monitoring Threshold", "name": "oozie_server_callable_queue_threshold", "value": "{\"critical\":\"95.0\",\"warning\":\"80.0\"}" }, { "desc": "The from address to be used for mailing all emails for Oozie email action", "display_name": "Oozie Email Action From Address", "name": "oozie_email_from_address", "value": "oozie@localhost" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_oozie_server_swap_memory_usage", "value": "false" }, { "desc": "Port of the Oozie Server when using TLS/SSL.", "display_name": "Oozie HTTPS Port", "name": "oozie_https_port", "value": "11443" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Log Directory", "name": "role_config_suppression_oozie_log_dir", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "oozie_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Authentication Password parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Authentication Password", "name": "role_config_suppression_oozie_email_smtp_password", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Oozie Server", "name": "oozie_java_opts", "value": "" }, { "desc": "The minimum log level for Oozie Server logs", "display_name": "Oozie Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "oozie_server_pause_duration_window", "value": "5" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Workflow Status metrics collection interval.", "display_name": "Workflow Status Metrics Collection Interval", "name": "oozie_job_metric_collection_interval", "value": "1" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "oozie_server_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Password parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Password", "name": "role_config_suppression_oozie_database_password", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_oozie_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "The password for the Oozie JKS keystore file.", "display_name": "Oozie TLS/SSL Server JKS Keystore File Password", "name": "oozie_https_keystore_password", "value": null }, { "desc": "Name of the database used by Oozie Server.", "display_name": "Oozie Server Database Name", "name": "oozie_database_name", "value": "oozie" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Enable SMTP authentication for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Enabled", "name": "oozie_email_smtp_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_oozie_https_truststore_password", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to use the Codehale-based metrics for instrumentation. Enabling this disables the 'instrumentation' REST endpoint and enables the 'metrics' REST endpoint (<hostname:port>/v2/admin/metrics).", "display_name": "Enable The Metrics Instrumentation Service", "name": "oozie_use_metric_instrumentation", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Host parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Host", "name": "role_config_suppression_oozie_database_host", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Comma-separated list of ActionService executor extension classes. Only action types with associated executors can be used in workflows. For CDH 5.4 and higher, this parameter is used only to specify additional classes for workflows. All executor extension classes included in that release will be added automatically and do not need to be specified.", "display_name": "Oozie ActionService Executor Extension Classes", "name": "oozie_executor_extension_classes", "value": "" }, { "desc": "Maximum callable queue size", "display_name": "Maximum Callable Queue Size", "name": "oozie_service_callablequeueservice_queue_size", "value": "10000" }, { "desc": "Whether to purge completed workflows and their corresponding coordinator actions of long-running coordinator jobs if the completed workflow jobs are older than the value specified in oozie.service.PurgeService.older.than.", "display_name": "Enable Purge for Long-Running Coordinator Jobs", "name": "purgeservice_purge_old_coord_action", "value": "true" }, { "desc": "The maximum number of rolled log files to keep for Oozie Server logs. Typically used by log4j or logback.", "display_name": "Oozie Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "720" }, { "desc": "Port of Oozie Server", "display_name": "Oozie HTTP Port", "name": "oozie_http_port", "value": "11000" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_oozie_server_pause_duration", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml", "name": "role_config_suppression_oozie_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Type of the database used by Oozie Server.", "display_name": "Oozie Server Database Type", "name": "oozie_database_type", "value": "derby" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "oozie_server_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Oozie Server Shared Library Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Oozie Server Shared Library Check", "name": "role_health_suppression_oozie_server_shared_lib_version_health", "value": "false" }, { "desc": "The period over which to compute the moving average of the callable queue size.", "display_name": "Oozie Server Callable Queue Monitoring Period", "name": "oozie_server_callable_queue_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Plugins parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Plugins", "name": "role_config_suppression_oozie_plugins_list", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_oozie_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_oozie_server_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Oozie Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Oozie Server", "name": "role_config_suppression_oozie_java_opts", "value": "false" }, { "desc": "Password for connecting to the database used by Oozie Server. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database Password", "name": "oozie_database_password", "value": "" }, { "desc": "Whether to suppress the results of the Callable Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Callable Queue Size", "name": "role_health_suppression_oozie_server_callablequeue_size_health", "value": "false" }, { "desc": "Hostname of the database used by Oozie Server. If the port is non-default for your database type, use host:port notation. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database Host", "name": "oozie_database_host", "value": "localhost" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_oozie_server_unexpected_exits", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/cdh_config.py0000666000175100017510000000757113245514472024151 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json import six from sahara.plugins.cdh.client import api_client # -- cm config -- cm_address = 'localhost' cm_port = 7180 cm_username = 'admin' cm_password = 'admin' hdfs_service_name = 'hdfs01' yarn_service_name = 'yarn01' oozie_service_name = 'oozie01' hive_service_name = 'hive01' hue_service_name = 'hue01' spark_service_name = 'spark_on_yarn01' zookeeper_service_name = 'zookeeper01' hbase_service_name = 'hbase01' flume_service_name = 'flume01' sqoop_service_name = 'sqoop01' solr_service_name = 'solr01' ks_indexer_service_name = 'ks_indexer01' impala_service_name = 'impala01' sentry_service_name = 'sentry01' kms_service_name = 'kms01' def get_cm_api(): return api_client.ApiResource(cm_address, server_port=cm_port, username=cm_username, password=cm_password) def get_cluster(api): return api.get_all_clusters()[0] def process_service(service, service_name): for role_cfgs in service.get_all_role_config_groups(): role_cm_cfg = role_cfgs.get_config(view='full') role_cfg = parse_config(role_cm_cfg) role_name = role_cfgs.roleType.lower() write_cfg(role_cfg, '%s-%s.json' % (service_name, role_name)) service_cm_cfg = service.get_config(view='full')[0] service_cfg = parse_config(service_cm_cfg) write_cfg(service_cfg, '%s-service.json' % service_name) def parse_config(config): cfg = [] for name, value in six.iteritems(config): p = { 'name': value.name, 'value': value.default, 'display_name': value.displayName, 'desc': value.description } cfg.append(p) return cfg def write_cfg(cfg, file_name): to_write = json.dumps(cfg, sort_keys=True, indent=4, separators=(',', ': ')) with open(file_name, 'w') as f: f.write(to_write) def main(): client = get_cm_api() cluster = get_cluster(client) hdfs = cluster.get_service(hdfs_service_name) process_service(hdfs, 'hdfs') yarn = cluster.get_service(yarn_service_name) process_service(yarn, 'yarn') oozie = cluster.get_service(oozie_service_name) process_service(oozie, 'oozie') hive = cluster.get_service(hive_service_name) process_service(hive, 'hive') hue = cluster.get_service(hue_service_name) process_service(hue, 'hue') spark = cluster.get_service(spark_service_name) process_service(spark, 'spark') zookeeper = cluster.get_service(zookeeper_service_name) process_service(zookeeper, 'zookeeper') hbase = cluster.get_service(hbase_service_name) process_service(hbase, 'hbase') flume = cluster.get_service(flume_service_name) process_service(flume, 'flume') sqoop = cluster.get_service(sqoop_service_name) process_service(sqoop, 'sqoop') solr = cluster.get_service(solr_service_name) process_service(solr, 'solr') ks_indexer = cluster.get_service(ks_indexer_service_name) process_service(ks_indexer, 'ks_indexer') impala = cluster.get_service(impala_service_name) process_service(impala, 'impala') sentry = cluster.get_service(sentry_service_name) process_service(sentry, 'sentry') kms = cluster.get_service(kms_service_name) process_service(kms, 'kms') if __name__ == '__main__': main() sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/impala-statestore.json0000666000175100017510000005527413245514472026050 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Port where StateStore debug web server runs.", "display_name": "StateStore HTTP Server Port", "name": "statestore_webserver_port", "value": "25010" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Log Directory parameter.", "display_name": "Suppress Parameter Validation: StateStore Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Directory where a StateStore core dump is placed.", "display_name": "StateStore Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/statestore" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Port where StateStoreService is exported.", "display_name": "StateStore Service Port", "name": "state_store_port", "value": "24000" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_statestore_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Statestore Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: StateStore Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "Password for Statestore webserver authentication.", "display_name": "Statestore Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_statestore_cmd_args_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when StateStore Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "StateStore Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "statestore_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The minimum log level for Impala StateStore logs", "display_name": "Impala StateStore Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_statestore_scm_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Maximum number of tasks allowed to be pending at the thread manager underlying the StateStore Thrift server (0 allows infinitely many pending tasks)", "display_name": "Maximum StateStore Pending Tasks", "name": "state_store_pending_task_count_max", "value": "0" }, { "desc": "Directory where StateStore will place its log files.", "display_name": "StateStore Log Directory", "name": "log_dir", "value": "/var/log/statestore" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Statestore Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_statestore_log_directory_free_space", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "statestore_web_metric_collection_enabled", "value": "true" }, { "desc": "Number of worker threads for the thread manager underlying the StateStore Thrift server.", "display_name": "StateStore Worker Threads", "name": "state_store_num_server_worker_threads", "value": "4" }, { "desc": "Enable/Disable StateStore web server. This web server contains useful information about StateStore daemon.", "display_name": "Enable StateStore Web Server", "name": "statestore_enable_webserver", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_statestore_file_descriptor", "value": "false" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "StateStore Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala StateStore Environment Advanced Configuration Snippet (Safety Valve)", "name": "STATESTORE_role_env_safety_valve", "value": null }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "StateStore Maximum Log Files", "name": "max_log_files", "value": "10" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_statestore_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "statestore_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "The password for the private key in the StateStore Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "StateStore Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "statestore_startup_tolerance", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_statestore_web_metric_collection", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above. ", "display_name": "StateStore Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when StateStore Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "StateStore Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_statestore_unexpected_exits", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_statestore_host_health", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Impala StateStore logs. Typically used by log4j or logback.", "display_name": "Impala StateStore Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Enables the health test that the Impala StateStore's process state is consistent with the role configuration", "display_name": "Impala StateStore Process Health Test", "name": "statestore_scm_health_enabled", "value": "true" }, { "desc": "Username for Statestore webserver authentication.", "display_name": "Statestore Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "When computing the overall Impala StateStore health, consider the host's health.", "display_name": "Impala StateStore Host Health Test", "name": "statestore_host_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to StateStore command line flags.", "display_name": "Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "statestore_cmd_args_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala StateStore Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala StateStore Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_statestore_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_statestore_memory_rss_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/kafka-kafka_mirror_maker.json0000666000175100017510000007023613245514472027306 0ustar zuulzuul00000000000000[ { "desc": "The maximum size, in megabytes, per log file for Kafka MirrorMaker logs. Typically used by log4j or logback.", "display_name": "Kafka MirrorMaker Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Stop the entire mirror maker when a send failure occurs.", "display_name": "Abort on Send Failure", "name": "abort.on.send.failure", "value": "true" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Maximum number of bytes that can be buffered between producer and consumer. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Queue Size", "name": "queue.byte.size", "value": "100000000" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kafka_kafka_mirror_maker_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kafka_mirror_maker_role_env_safety_valve", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Only required if source Kafka cluster requires client authentication.", "display_name": "Source Kafka Cluster's Client Auth", "name": "source.ssl.client.auth", "value": "false" }, { "desc": "When computing the overall Kafka MirrorMaker health, consider the host's health.", "display_name": "Kafka MirrorMaker Host Health Test", "name": "kafka_mirror_maker_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties", "name": "role_config_suppression_ssl_client.properties_role_safety_valve", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Kafka MirrorMaker logs. Typically used by log4j or logback.", "display_name": "Kafka MirrorMaker Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topic Blacklist parameter.", "display_name": "Suppress Parameter Validation: Topic Blacklist", "name": "role_config_suppression_blacklist", "value": "false" }, { "desc": "The minimum log level for Kafka MirrorMaker logs", "display_name": "Kafka MirrorMaker Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Offset commit interval in milliseconds.", "display_name": "Offset Commit Interval", "name": "offset.commit.interval.ms", "value": "60000" }, { "desc": "For advanced use only. A string to be inserted into mirror_maker_producers.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties", "name": "mirror_maker_producers.properties_role_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kafka_kafka_mirror_maker_host_health", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kafka_mirror_maker_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties", "name": "role_config_suppression_ssl_server.properties_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The password for the Kafka MirrorMaker JKS keystore file.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Destination Broker List parameter.", "display_name": "Suppress Parameter Validation: Destination Broker List", "name": "role_config_suppression_bootstrap.servers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Number of producer instances. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Number of Producers", "name": "num.producers", "value": "1" }, { "desc": "For advanced use only. A string to be inserted into ssl_client.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties", "name": "ssl_client.properties_role_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_MIRROR_MAKER_role_env_safety_valve", "value": null }, { "desc": "Name of the consumer group used by MirrorMaker. When multiple role instances are configured with the same topics and same group ID, the role instances load-balance replication for the topics. When multiple role instances are configured with the same topics but different group ID, each role instance replicates all the events for those topics - this can be used to replicate the source cluster into multiple destination clusters.", "display_name": "Consumer Group ID", "name": "group.id", "value": "cloudera_mirrormaker" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kafka_kafka_mirror_maker_swap_memory_usage", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Consumer Group ID parameter.", "display_name": "Suppress Parameter Validation: Consumer Group ID", "name": "role_config_suppression_group.id", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Consumer Rebalance Listener Arguments parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Consumer Rebalance Listener Arguments", "name": "role_config_suppression_consumer.rebalance.listener.args", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka MirrorMaker might connect to. This is used when Kafka MirrorMaker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Kafka MirrorMaker TLS/SSL Certificate Trust Store File", "name": "ssl_client_truststore_location", "value": "" }, { "desc": "Enables the health test that the Kafka MirrorMaker's process state is consistent with the role configuration", "display_name": "Kafka MirrorMaker Process Health Test", "name": "kafka_mirror_maker_scm_health_enabled", "value": "true" }, { "desc": "Number of messages that are buffered between producer and consumer. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Message Queue Size", "name": "queue.size", "value": "10000" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "A consumer rebalance listener of type ConsumerRebalanceListener to be invoked when MirrorMaker's consumer rebalances.", "display_name": "MirrorMaker Consumer Rebalance Listener", "name": "consumer.rebalance.listener", "value": "" }, { "desc": "Run with MirrorMaker settings that eliminate potential loss of data. This impacts performance, but is highly recommended. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Avoid Data Loss", "name": "no.data.loss", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties", "name": "role_config_suppression_mirror_maker_producers.properties_role_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into mirror_maker_consumers.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties", "name": "mirror_maker_consumers.properties_role_safety_valve", "value": null }, { "desc": "Protocol to be used for communication with source kafka cluster.", "display_name": "Source Kafka Cluster's Security Protocol", "name": "source.security.protocol", "value": "PLAINTEXT" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Consumer Rebalance Listener parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Consumer Rebalance Listener", "name": "role_config_suppression_consumer.rebalance.listener", "value": "false" }, { "desc": "Protocol to be used for communication with destination kafka cluster.", "display_name": "Destination Kafka Cluster's Security Protocol", "name": "destination.security.protocol", "value": "PLAINTEXT" }, { "desc": "Regular expression that represents a set of topics to mirror. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used.", "display_name": "Topic Whitelist", "name": "whitelist", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kafka_kafka_mirror_maker_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topic Whitelist parameter.", "display_name": "Suppress Parameter Validation: Topic Whitelist", "name": "role_config_suppression_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Port for JMX.", "display_name": "JMX Port", "name": "jmx_port", "value": "9394" }, { "desc": "Arguments used by MirrorMaker message handler.", "display_name": "MirrorMaker Message Handler Arguments", "name": "message.handler.args", "value": "" }, { "desc": "List of brokers on destination cluster. This should be more than one, for high availability, but there's no need to list all brokers.", "display_name": "Destination Broker List", "name": "bootstrap.servers", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Source Broker List parameter.", "display_name": "Suppress Parameter Validation: Source Broker List", "name": "role_config_suppression_source.bootstrap.servers", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kafka_kafka_mirror_maker_file_descriptor", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka MirrorMaker is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Message Handler parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Message Handler", "name": "role_config_suppression_message.handler", "value": "false" }, { "desc": "Only required if destination Kafka cluster requires client authentication.", "display_name": "Destination Kafka Cluster's Client Auth", "name": "destination.ssl.client.auth", "value": "false" }, { "desc": "A MirrorMaker message handler of type MirrorMakerMessageHandler that will process every record in-between producer and consumer.", "display_name": "MirrorMaker Message Handler", "name": "message.handler", "value": "" }, { "desc": "List of brokers on source cluster. This should be more than one, for high availability, but there's no need to list all brokers.", "display_name": "Source Broker List", "name": "source.bootstrap.servers", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties", "name": "role_config_suppression_mirror_maker_consumers.properties_role_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "For advanced use only. A string to be inserted into ssl_server.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties", "name": "ssl_server.properties_role_safety_valve", "value": null }, { "desc": "Number of consumer threads.", "display_name": "Number of Consumer Threads", "name": "num.streams", "value": "1" }, { "desc": "The password that protects the private key contained in the JKS keystore used when Kafka MirrorMaker is acting as a TLS/SSL server.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": "" }, { "desc": "Encrypt communication between clients and Kafka MirrorMaker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Kafka MirrorMaker", "name": "ssl_enabled", "value": "false" }, { "desc": "Regular expression that represents a set of topics to avoid mirroring. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Topic Blacklist", "name": "blacklist", "value": "" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Arguments used by MirrorMaker consumer rebalance listener.", "display_name": "MirrorMaker Consumer Rebalance Listener Arguments", "name": "consumer.rebalance.listener.args", "value": "" }, { "desc": "The log directory for log files of the role Kafka MirrorMaker.", "display_name": "Kafka MirrorMaker Log Directory", "name": "log_dir", "value": "/var/log/kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Message Handler Arguments parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Message Handler Arguments", "name": "role_config_suppression_message.handler.args", "value": "false" }, { "desc": "The password for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Kafka MirrorMaker TLS/SSL Certificate Trust Store Password", "name": "ssl_client_truststore_password", "value": "" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hue-kt_renewer.json0000666000175100017510000003322113245514472025322 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve)", "name": "KT_RENEWER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kt_renewer_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kt_renewer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kt_renewer_scm_health", "value": "false" }, { "desc": "When computing the overall Kerberos Ticket Renewer health, consider the host's health.", "display_name": "Kerberos Ticket Renewer Host Health Test", "name": "kt_renewer_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kt_renewer_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Interval in seconds with which Hue's Kerberos ticket will get renewed.", "display_name": "Hue Keytab Renewal Interval", "name": "keytab_reinit_frequency", "value": "3600" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kt_renewer_file_descriptor", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Directory where Kerberos Ticket Renewer will place its log files.", "display_name": "Kerberos Ticket Renewer Log Directory", "name": "kt_renewer_log_dir", "value": "/var/log/hue" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Ticket Renewer Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kerberos Ticket Renewer Log Directory", "name": "role_config_suppression_kt_renewer_log_dir", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Enables the health test that the Kerberos Ticket Renewer's process state is consistent with the role configuration", "display_name": "Kerberos Ticket Renewer Process Health Test", "name": "kt_renewer_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kt_renewer_host_health", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_kt_renewer_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_kt_renewer_log_directory_free_space", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kt_renewer_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/solr-solr_server.json0000666000175100017510000006340713245514472025731 0ustar zuulzuul00000000000000[ { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "solr_server_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Solr Server Log Directory", "name": "role_config_suppression_solr_log_dir", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "solr_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Admin port of the Solr Server.", "display_name": "Solr Admin Port", "name": "solr_admin_port", "value": "8984" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When computing the overall Solr Server health, consider the host's health.", "display_name": "Solr Server Host Health Test", "name": "solr_server_host_health_enabled", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "HTTP port of Solr Server.", "display_name": "Solr HTTP Port", "name": "solr_http_port", "value": "8983" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Number of slabs per block cache. The size of the cache is 8 KB (the block size) times the number of blocks per slab times the number of slabs.", "display_name": "HDFS Block Cache Number of Slabs", "name": "solr_hdfs_blockcache_slab_count", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Data Directory parameter.", "display_name": "Suppress Parameter Validation: Solr Data Directory", "name": "role_config_suppression_solr_data_dir", "value": "false" }, { "desc": "The minimum log level for Solr Server logs", "display_name": "Solr Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_solr_server_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "HTTPS port of Solr Server.", "display_name": "Solr HTTPS port", "name": "solr_https_port", "value": "8985" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of Solr Server in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of Solr Server in Bytes", "name": "role_config_suppression_solr_java_heapsize", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Directory on local file system where Solr Server keeps the configurations for collections.", "display_name": "Solr Data Directory", "name": "solr_data_dir", "value": "/var/lib/solr" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Solr Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Solr Server", "name": "role_config_suppression_solr_java_opts", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Solr Server logs. Typically used by log4j or logback.", "display_name": "Solr Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Enables the health test that the Solr Server's process state is consistent with the role configuration", "display_name": "Solr Server Process Health Test", "name": "solr_server_scm_health_enabled", "value": "true" }, { "desc": "Address of the load balancer, specified in host:port format.", "display_name": "Solr Load Balancer", "name": "solr_load_balancer", "value": null }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "solr_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Number of blocks per cache slab. The size of the cache is 8 KB (the block size) times the number of blocks per slab times the number of slabs.", "display_name": "HDFS Block Cache Blocks per Slab", "name": "solr_hdfs_blockcache_blocksperbank", "value": "16384" }, { "desc": "Whether to suppress the results of the Solr Server API Liveness heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Solr Server API Liveness", "name": "role_health_suppression_solr_core_status_collection_health", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "solr_server_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Solr Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SOLR_SERVER_role_env_safety_valve", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_solr_server_gc_duration", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_solr_server_unexpected_exits", "value": "false" }, { "desc": "Enable caching of HDFS blocks in Solr. There is one block cache per Solr collection. configured to use off-heap memory, Maximum Off-Heap Memory must be set high enough to account for all block caches.", "display_name": "HDFS Block Cache", "name": "solr_hdfs_blockcache_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_solr_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Solr Server in Bytes", "name": "solr_java_heapsize", "value": "1073741824" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather status of Solr Cores from the Solr Server with a simple API request.", "display_name": "Solr Server API Liveness", "name": "solr_core_status_collection_health_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_solr_server_file_descriptor", "value": "false" }, { "desc": "The maximum number of request processing threads to be created by Solr server, which determines the maximum number of simultaneous requests that can be handled.", "display_name": "Solr Max Connector Threads", "name": "solr_max_connector_thread", "value": "10000" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_solr_server_swap_memory_usage", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Solr Server logs. Typically used by log4j or logback.", "display_name": "Solr Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Directory where Solr Server will place its log files.", "display_name": "Solr Server Log Directory", "name": "solr_log_dir", "value": "/var/log/solr" }, { "desc": "Maximum amount of off-heap memory in bytes that may be allocated by the Java process. Passed to Java -XX:MaxDirectMemorySize. If unset, defaults to the size of the heap.", "display_name": "Java Direct Memory Size of Solr Server in Bytes", "name": "solr_java_direct_memory_size", "value": "1073741824" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Solr Server", "name": "solr_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_solr_server_scm_health", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Solr Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "solr_server_gc_duration_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Solr Load Balancer", "name": "role_config_suppression_solr_load_balancer", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_solr_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Use off-heap memory when caching HDFS blocks in Solr.", "display_name": "HDFS Block Cache Off-Heap Memory", "name": "solr_hdfs_blockcache_direct_memory_allocation", "value": "true" }, { "desc": "The health test thresholds on the duration of the Solr Server API request.", "display_name": "Solr Server API Liveness Request Duration", "name": "solr_core_status_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_solr_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_solr_server_web_metric_collection", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hbase-hbasethriftserver.json0000666000175100017510000007251513245514472027221 0ustar zuulzuul00000000000000[ { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Log Directory", "name": "role_config_suppression_hbase_thriftserver_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Bind Address parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Bind Address", "name": "role_config_suppression_hbase_thriftserver_bindaddress", "value": "false" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase Thrift Server over HTTP is acting as a TLS/SSL server.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password", "name": "hbase_thriftserver_http_keystore_keypassword", "value": null }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbasethriftserver_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The host name or IP address of the DNS name server which an HBase Thrift Server should use to determine the host name used for communication and display purposes.", "display_name": "HBase Thrift Server DNS Name Server", "name": "hbase_thriftserver_dns_nameserver", "value": null }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The port that HBase Thrift Server binds to.", "display_name": "HBase Thrift Server Port", "name": "hbase_thriftserver_port", "value": "9090" }, { "desc": "Use the TCompactProtocol instead of the default TBinaryProtocol. TCompactProtocol is a binary protocol that is more compact than the default and typically more efficient.", "display_name": "Enable HBase Thrift Server Compact Protocol", "name": "hbase_thriftserver_compact", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase Thrift Server in Bytes", "name": "hbase_thriftserver_java_heapsize", "value": "1073741824" }, { "desc": "The password for the HBase Thrift Server over HTTP JKS keystore file.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password", "name": "hbase_thriftserver_http_keystore_password", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_thrift_server_scm_health", "value": "false" }, { "desc": "The minimum log level for HBase Thrift Server logs", "display_name": "HBase Thrift Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase Thrift Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase Thrift Server", "name": "role_config_suppression_hbase_thriftserver_java_opts", "value": "false" }, { "desc": "Use framed transport. When using the THsHaServer or TNonblockingServer, framed transport is always used irrespective of this configuration value.", "display_name": "Enable HBase Thrift Server Framed Transport", "name": "hbase_thriftserver_framed", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server DNS Name Server", "name": "role_config_suppression_hbase_thriftserver_dns_nameserver", "value": "false" }, { "desc": "Encrypt communication between clients and HBase Thrift Server over HTTP using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HBase Thrift Server over HTTP", "name": "hbase_thriftserver_http_use_ssl", "value": "false" }, { "desc": "When computing the overall HBase Thrift Server health, consider the host's health.", "display_name": "HBase Thrift Server Host Health Test", "name": "hbasethriftserver_host_health_enabled", "value": "true" }, { "desc": "The \"core size\" of the thread pool. New threads are created on every connection until this many threads are created.", "display_name": "HBase Thrift Server Min Worker Threads", "name": "hbase_thriftserver_min_worker_threads", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_thrift_server_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_thriftserver_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server DNS Network Interface", "name": "role_config_suppression_hbase_thriftserver_dns_interface", "value": "false" }, { "desc": "If true, HBase Thrift Server Web UI will bind to a wildcard address (0.0.0.0). Otherwise it will bind to a host name. Only available in CDH 4.3 and later.", "display_name": "HBase Thrift Server Web UI Bind to Wildcard Address", "name": "hbase_thriftserver_info_bind_to_wildcard", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_thrift_server_unexpected_exits", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HBase Thrift Server logs. Typically used by log4j or logback.", "display_name": "HBase Thrift Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASETHRIFTSERVER_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_thrift_server_host_health", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Enables the health test that the HBase Thrift Server's process state is consistent with the role configuration", "display_name": "HBase Thrift Server Process Health Test", "name": "hbasethriftserver_scm_health_enabled", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbasethriftserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_thrift_server_file_descriptor", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Address to bind the HBase Thrift Server to. When using the THsHaServer or the TNonblockingServer, always binds to 0.0.0.0 irrespective of this configuration value.", "display_name": "HBase Thrift Server Bind Address", "name": "hbase_thriftserver_bindaddress", "value": "0.0.0.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The name of the DNS network interface from which an HBase Thrift Server should report its IP address.", "display_name": "HBase Thrift Server DNS Network Interface", "name": "hbase_thriftserver_dns_interface", "value": null }, { "desc": "Directory where HBase Thrift Server will place its log files.", "display_name": "HBase Thrift Server Log Directory", "name": "hbase_thriftserver_log_dir", "value": "/var/log/hbase" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The port that HBase Thrift Server Web UI binds to.", "display_name": "HBase Thrift Server Web UI Port", "name": "hbase_thriftserver_info_port", "value": "9095" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_hbase_thriftserver_http_keystore_keypassword", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase Thrift Server", "name": "hbase_thriftserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The maximum size, in megabytes, per log file for HBase Thrift Server logs. Typically used by log4j or logback.", "display_name": "HBase Thrift Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_thrift_server_log_directory_free_space", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_hbase_thriftserver_http_keystore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_thriftserver_config_safety_valve", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase Thrift Server over HTTP is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location", "name": "hbase_thriftserver_http_keystore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_hbase_thriftserver_http_keystore_password", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_thrift_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Type of HBase Thrift Server.", "display_name": "HBase Thrift Server Type", "name": "hbase_thriftserver_type", "value": "threadpool" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-journalnode.json0000666000175100017510000007211713245514472025647 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "jn_config_safety_valve", "value": null }, { "desc": "Enables the health check that verifies the active NameNode's sync status to the JournalNode", "display_name": "Active NameNode Sync Status Health Check", "name": "journalnode_sync_status_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_journal_node_host_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Enables the health test that the JournalNode's process state is consistent with the role configuration", "display_name": "JournalNode Process Health Test", "name": "journalnode_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_journal_node_unexpected_exits", "value": "false" }, { "desc": "Directory where JournalNode will place its log files.", "display_name": "JournalNode Log Directory", "name": "journalnode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: JournalNode Log Directory", "name": "role_config_suppression_journalnode_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Fsync Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Fsync Latency", "name": "role_health_suppression_journal_node_fsync_latency", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_journal_node_web_metric_collection", "value": "false" }, { "desc": "The minimum log level for JournalNode logs", "display_name": "JournalNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Port for the JournalNode HTTP web UI. Combined with the JournalNode hostname to build its HTTP address.", "display_name": "JournalNode HTTP Port", "name": "dfs_journalnode_http_port", "value": "8480" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "journalnode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The maximum number of rolled log files to keep for JournalNode logs. Typically used by log4j or logback.", "display_name": "JournalNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Sync Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sync Status", "name": "role_health_suppression_journal_node_sync_status", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of JournalNode in Bytes", "name": "journalNode_java_heapsize", "value": "536870912" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Edits Directory parameter.", "display_name": "Suppress Parameter Validation: JournalNode Edits Directory", "name": "role_config_suppression_dfs_journalnode_edits_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_journal_node_file_descriptor", "value": "false" }, { "desc": "Port for the JournalNode's RPC. Combined with the JournalNode's hostname to build its RPC address.", "display_name": "JournalNode RPC Port", "name": "dfs_journalnode_rpc_port", "value": "8485" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_journal_node_gc_duration", "value": "false" }, { "desc": "The base port where the secure JournalNode web UI listens. Combined with the JournalNode's hostname to build its secure web UI address.", "display_name": "Secure JournalNode Web UI Port (TLS/SSL)", "name": "dfs_journalnode_https_port", "value": "8481" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory.", "display_name": "JournalNode Edits Directory Free Space Monitoring Absolute Thresholds", "name": "journalnode_edits_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JournalNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_journalnode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_journal_node_scm_health", "value": "false" }, { "desc": "Directory on the local file system where NameNode edits are written.", "display_name": "JournalNode Edits Directory", "name": "dfs_journalnode_edits_dir", "value": null }, { "desc": "Whether to suppress the results of the JournalNode Edits Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JournalNode Edits Directory Free Space", "name": "role_health_suppression_journal_node_edits_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JournalNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The amount of time at JournalNode startup allowed for the active NameNode to get in sync with the JournalNode.", "display_name": "Active NameNode Sync Status Startup Tolerance", "name": "journalnode_sync_status_startup_tolerance", "value": "180" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_journal_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for JournalNode", "name": "journalNode_java_opts", "value": "" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_journal_node_log_directory_free_space", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "journalnode_web_metric_collection_enabled", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "journalnode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "journalnode_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The maximum size, in megabytes, per log file for JournalNode logs. Typically used by log4j or logback.", "display_name": "JournalNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "If enabled, the JournalNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind JournalNode to Wildcard Address", "name": "journalnode_bind_wildcard", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JournalNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for JournalNode", "name": "role_config_suppression_journalnode_java_opts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "JournalNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a JournalNode Edits Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "JournalNode Edits Directory Free Space Monitoring Percentage Thresholds", "name": "journalnode_edits_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_jn_config_safety_valve", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "journalnode_gc_duration_window", "value": "5" }, { "desc": "When computing the overall JournalNode health, consider the host's health.", "display_name": "JournalNode Host Health Test", "name": "journalnode_host_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "JournalNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "JOURNALNODE_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_journal_node_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for JournalNode fsync latency.", "display_name": "JournalNode Fsync Latency Thresholds", "name": "journalnode_fsync_latency_thresholds", "value": "{\"critical\":\"3000.0\",\"warning\":\"1000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/yarn-service.json0000666000175100017510000011206013245514472025004 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "service_config_suppression_yarn_service_replication_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_yarn_core_site_safety_valve", "value": "false" }, { "desc": "Password for the server keystore file used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "How long to keep aggregation logs before deleting them.", "display_name": "Log Aggregation Retention Period", "name": "yarn_log_aggregation_retain_seconds", "value": "604800" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use.", "display_name": "Fair Scheduler Configuration Rules (Deployed)", "name": "yarn_fs_schedule_rules", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the NodeManager Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NodeManager Count Validator", "name": "service_config_suppression_nodemanager_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_yarn_service_env_safety_valve", "value": "false" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. Typically edited using the Rules configuration UI.", "display_name": "Fair Scheduler Configuration Rules (Staged)", "name": "yarn_fs_schedule_rules_draft", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_yarn_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "yarn" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the ResourceManager Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: ResourceManager Count Validator", "name": "service_config_suppression_resourcemanager_count_validator", "value": "false" }, { "desc": "Enables Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Note: This is effective only if Kerberos is enabled for the HDFS service.", "display_name": "Enable Kerberos Authentication for HTTP Web-Consoles", "name": "hadoop_secure_web_ui", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "yarn_hadoop_policy_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "service_config_suppression_yarn_service_config_safety_valve", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the Llama (AM for Impala) user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Llama Proxy User Groups", "name": "llama_proxy_user_groups_list", "value": "*" }, { "desc": "Cluster ID used when ResourceManager is Highly Available.", "display_name": "RM-HA Cluster ID", "name": "yarn_rm_ha_cluster_id", "value": "yarnRM" }, { "desc": "Whether to suppress the results of the YARN Container Usage Aggregation heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: YARN Container Usage Aggregation", "name": "service_health_suppression_yarn_usage_aggregation_health", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "yarn_ssl_server_safety_valve", "value": null }, { "desc": "Name of the HDFS service that this YARN service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "When set to true, pools specified in applications but not explicitly configured, are created at runtime with default settings. When set to false, applications specifying pools not explicitly configured run in a pool named default. This setting applies when an application explicitly specifies a pool and when the application runs in a pool named with the username associated with the application.", "display_name": "Allow Undeclared Pools", "name": "yarn_scheduler_fair_allow_undeclared_pools", "value": "true" }, { "desc": "The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory.", "display_name": "System User's Home Directory", "name": "hdfs_user_home_dir", "value": "/var/lib/hadoop-yarn" }, { "desc": "For advanced use only, a string to be inserted into mapred-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)", "name": "yarn_service_mapred_safety_valve", "value": null }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted shuffle and encrypted web UIs. Applies to all configurations of daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Name of the ZooKeeper service that this YARN service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "When computing the overall YARN service health, whether to consider the active ResourceManager's health.", "display_name": "Active ResourceManager Role Health Check", "name": "yarn_resourcemanagers_health_enabled", "value": "true" }, { "desc": "Controls the aggregate metrics generated for YARN applications. The structure is a JSON list of the attributes to aggregate and the entities to aggregate to. For example, if the attributeName is 'maps_completed' and the aggregationTargets is ['USER'] then the Service Monitor will create the metric 'yarn_application_maps_completed_rate' and, every ten minutes, will record the total maps completed for each user across all their YARN applications. By default it will also record the number of applications submitted ('apps_submitted_rate') for both users and pool. For a full list of the supported attributes see the YARN search page. Note that the valid aggregation targets are USER, YARN_POOL, and YARN (the service), and that these aggregate metrics can be viewed on both the reports and charts search pages.", "display_name": "YARN Application Aggregates", "name": "yarn_application_aggregates", "value": "[\n {\n \"attributeName\": \"maps_total\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"reduces_total\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"mb_millis_maps\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"mb_millis_reduces\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"vcores_millis_maps\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"vcores_millis_reduces\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"file_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"file_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cm_cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"application_duration\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n }\n]\n" }, { "desc": "UNIX user that containers run as when Linux-container-executor is used in nonsecure mode.", "display_name": "UNIX User for Nonsecure Mode with Linux Container Executor", "name": "yarn_nodemanager_linux_container_executor_nonsecure_mode_local_user", "value": "nobody" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin ACL parameter.", "display_name": "Suppress Parameter Validation: Admin ACL", "name": "service_config_suppression_yarn_admin_acl", "value": "false" }, { "desc": "Whether to suppress the results of the NodeManager Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Health", "name": "service_health_suppression_yarn_node_managers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the JobHistory Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: JobHistory Server Count Validator", "name": "service_config_suppression_jobhistory_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "service_config_suppression_yarn_ssl_client_safety_valve", "value": "false" }, { "desc": "Whether YARN creates a cgroup per container, thereby isolating the CPU usage of containers. When set, yarn.nodemanager.linux-container-executor.resources-handler.class is configured to org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler. The host (in Cloudera Manager) must have cgroups enabled. The number of shares allocated to all YARN containers is configured by adjusting the CPU shares value of the Node Manager in the Resource Management configuration group.", "display_name": "Use CGroups for Resource Management", "name": "yarn_service_cgroups", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. Typically edited using the Pools configuration UI.", "display_name": "Fair Scheduler Allocations (Staged)", "name": "yarn_fs_scheduled_allocations_draft", "value": null }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules.", "display_name": "Fair Scheduler Allocations (Deployed)", "name": "yarn_fs_scheduled_allocations", "value": "{\"defaultFairSharePreemptionThreshold\":null,\"defaultFairSharePreemptionTimeout\":null,\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":null,\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":null,\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null}],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null}],\"userMaxAppsDefault\":null,\"users\":[]}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Client Truststore File Location parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Client Truststore File Location", "name": "service_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Password for the TLS/SSL client truststore. Overrides the cluster-wide default truststore password set in HDFS.", "display_name": "TLS/SSL Client Truststore File Password", "name": "ssl_client_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Application Classpath parameter.", "display_name": "Suppress Parameter Validation: YARN Application Classpath", "name": "service_config_suppression_yarn_application_classpath", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service MapReduce Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_yarn_service_mapred_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into mapred-site.xml. Applies to all HDFS Replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "mapreduce_service_replication_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "service_config_suppression_mapreduce_service_replication_config_safety_valve", "value": "false" }, { "desc": "Entries to add to the classpaths of YARN applications.", "display_name": "YARN Application Classpath", "name": "yarn_application_classpath", "value": "$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*" }, { "desc": "Whether YARN uses the Linux Container Executor both in secure (Kerberos) and insecure (not Kerberos) environments. Cgroups enforcement only works when the Linux Container Executor is used.", "display_name": "Always Use Linux Container Executor", "name": "yarn_service_lce_always", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "yarn_core_site_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the UNIX User for Nonsecure Mode with Linux Container Executor parameter.", "display_name": "Suppress Parameter Validation: UNIX User for Nonsecure Mode with Linux Container Executor", "name": "service_config_suppression_yarn_nodemanager_linux_container_executor_nonsecure_mode_local_user", "value": "false" }, { "desc": "Whether users and groups specified in Admin ACL should be checked for authorization to perform admin operations.", "display_name": "Enable ResourceManager ACLs", "name": "yarn_acl_enable", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into yarn-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_service_config_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into ssl-client.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "yarn_ssl_client_safety_valve", "value": null }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Client Truststore File Password parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Client Truststore File Password", "name": "service_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "service_config_suppression_yarn_hadoop_policy_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Application Aggregates parameter.", "display_name": "Suppress Parameter Validation: YARN Application Aggregates", "name": "service_config_suppression_yarn_application_aggregates", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "yarn_service_env_safety_valve", "value": null }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Path to the truststore file used when roles of this service act as TLS/SSL clients. Overrides the cluster-wide default truststore location set in HDFS. This truststore must be in JKS format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. The contents of the truststore can be modified without restarting any roles. By default, changes to its contents are picked up within ten seconds. If not set, the default Java truststore is used to verify certificates.", "display_name": "TLS/SSL Client Truststore File Location", "name": "ssl_client_truststore_location", "value": null }, { "desc": "When computing the overall YARN health, consider JobHistory Server's health", "display_name": "JobHistory Server Role Health Test", "name": "yarn_jobhistoryserver_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User's Home Directory parameter.", "display_name": "Suppress Parameter Validation: System User's Home Directory", "name": "service_config_suppression_hdfs_user_home_dir", "value": "false" }, { "desc": "Whether to suppress the results of the ResourceManager Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ResourceManager Health", "name": "service_health_suppression_yarn_resourcemanagers_health", "value": "false" }, { "desc": "The user that this service's processes should run as. (Except the Job History Server, which has its own user)", "display_name": "System User", "name": "process_username", "value": "yarn" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "When computing the overall YARN service health, whether to consider the health of the standby ResourceManager.", "display_name": "Standby ResourceManager Health Check", "name": "yarn_standby_resourcemanager_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "The tolerance window used in YARN service tests that depend on detection of the active ResourceManager.", "display_name": "Active ResourceManager Detection Window", "name": "yarn_active_resourcemanager_detecton_window", "value": "3" }, { "desc": "The health test thresholds of the overall NodeManager health. The check returns \"Concerning\" health if the percentage of \"Healthy\" NodeManagers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" NodeManagers falls below the critical threshold.", "display_name": "Healthy NodeManager Monitoring Thresholds", "name": "yarn_nodemanagers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Llama Proxy User Hosts", "name": "service_config_suppression_llama_proxy_user_hosts_list", "value": "false" }, { "desc": "Controls which applications non-admin users can see in the applications list view", "display_name": "Non-Admin Users Applications List Visibility Settings", "name": "user_application_list_settings", "value": "ALL" }, { "desc": "For advanced use only, a string to be inserted into yarn-site.xml. Applies to all HDFS Replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_service_replication_config_safety_valve", "value": null }, { "desc": "ACL that determines which users and groups can submit and kill applications in any pool, and can issue commands on ResourceManager roles.", "display_name": "Admin ACL", "name": "yarn_admin_acl", "value": "*" }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "The group that this service's processes should run as. (Except the Job History Server, which has its own group.)", "display_name": "System Group", "name": "process_groupname", "value": "hadoop" }, { "desc": "Comma-delimited list of hosts where you want to allow the Llama (AM for Impala) user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Llama Proxy User Hosts", "name": "llama_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to enable log aggregation", "display_name": "Enable Log Aggregation", "name": "yarn_log_aggregation_enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Llama Proxy User Groups", "name": "service_config_suppression_llama_proxy_user_groups_list", "value": "false" }, { "desc": "Controls which applications an admin user can see in the applications list view", "display_name": "Admin Users Applications List Visibility Settings", "name": "admin_application_list_settings", "value": "ALL" }, { "desc": "Whether to suppress the results of the JobHistory Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JobHistory Server Health", "name": "service_health_suppression_yarn_jobhistory_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RM-HA Cluster ID parameter.", "display_name": "Suppress Parameter Validation: RM-HA Cluster ID", "name": "service_config_suppression_yarn_rm_ha_cluster_id", "value": "false" }, { "desc": "The amount of time after ResourceManager(s) start that the lack of an active ResourceManager will be tolerated. This is an advanced option that does not often need to be changed.", "display_name": "ResourceManager Activation Startup Tolerance", "name": "yarn_resourcemanager_activation_startup_tolerance", "value": "180" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hive-hiveserver2.json0000666000175100017510000012153013245514472025574 0ustar zuulzuul00000000000000[ { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hive/metrics-hivemetastore/metrics.log. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Location", "name": "hive_metrics_sample_file_location", "value": null }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_hs2_config_safety_valve", "value": null }, { "desc": "When the average output file size of a job is less than the value of this property, Hive will start an additional map-only job to merge the output files into bigger files. This is only done for map-only jobs if hive.merge.mapfiles is true, for map-reduce jobs if hive.merge.mapredfiles is true, and for Spark jobs if hive.merge.sparkfiles is true.", "display_name": "Small File Average Size Merge Threshold", "name": "hiveserver2_merge_smallfiles_avgsize", "value": "16777216" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Hive Downloaded Resources Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Downloaded Resources Directory Free Space", "name": "role_health_suppression_hiveserver2_downloaded_resources_directory_free_space", "value": "false" }, { "desc": "Port on which HiveServer2 will listen for connections.", "display_name": "HiveServer2 Port", "name": "hs2_thrift_address_port", "value": "10000" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hiveserver2_log_directory_free_space", "value": "false" }, { "desc": "Enabled the Calcite-based Cost-Based Optimizer for HiveServer2.", "display_name": "Enable Cost-Based Optimizer for Hive", "name": "hiveserver2_enable_cbo", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Downloaded Resources Directory.", "display_name": "Hive Downloaded Resources Directory Free Space Monitoring Absolute Thresholds", "name": "hiveserver2_downloaded_resources_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Local Directory where Hive stores jars and data when performing a MapJoin optimization. If not specified, Hive uses a default location.", "display_name": "Hive Local Scratch Directory", "name": "hiveserver2_exec_local_scratchdir", "value": null }, { "desc": "Remove extra map-reduce jobs if the data is already clustered by the same key, eliminating the need to repartition the dataset again.", "display_name": "Enable ReduceDeDuplication Optimization", "name": "hiveserver2_optimize_reducededuplication", "value": "true" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Above this size, queries are converted to fetch tasks.", "display_name": "Fetch Task Query Conversion Threshold", "name": "hiveserver2_fetch_task_conversion_threshold", "value": "268435456" }, { "desc": "Ratio between 0.0 and 1.0 of entries in the vectorized group-by aggregation hash that is flushed when the memory threshold is exceeded.", "display_name": "Vectorized GroupBy Flush Ratio", "name": "hiveserver2_vectorized_groupby_flush_ratio", "value": "0.1" }, { "desc": "Merge small files at the end of a Spark job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Spark Job", "name": "hiveserver2_merge_sparkfiles", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hiveserver2_file_descriptor", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive HDFS Scratch Directory parameter.", "display_name": "Suppress Parameter Validation: Hive HDFS Scratch Directory", "name": "role_config_suppression_hiveserver2_exec_scratchdir", "value": "false" }, { "desc": "Number of Spark executors assigned to each application. This should not be set when Dynamic Executor Allocation is enabled.", "display_name": "Spark Executors Per Application", "name": "hiveserver2_spark_executor_instances", "value": null }, { "desc": "Merge small files at the end of a map-only job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Map-Only Job", "name": "hiveserver2_merge_mapfiles", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hiveserver2_unexpected_exits", "value": "false" }, { "desc": "Operation will be closed when not accessed for this duration of time, in milliseconds; disable by setting to zero. For a positive value, checked for operations in terminal state only (FINISHED, CANCELED, CLOSED, ERROR). For a negative value, checked for all of the operations regardless of state.", "display_name": "Idle Operation Timeout", "name": "hiveserver2_idle_operation_timeout", "value": "259200000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_hs2_env_safety_valve", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HiveServer2 parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HiveServer2", "name": "role_config_suppression_hiveserver2_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hiveserver2_host_health", "value": "false" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_hiveserver2_pause_duration", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for HiveServer2 logs. Typically used by log4j or logback.", "display_name": "HiveServer2 Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "hiveserver2_pause_duration_window", "value": "5" }, { "desc": "HiveServer2 will impersonate the beeline client user when talking to other services such as MapReduce and HDFS.", "display_name": "HiveServer2 Enable Impersonation", "name": "hiveserver2_enable_impersonation", "value": "true" }, { "desc": "Maximum size of each Spark driver's Java heap memory when Hive is running on Spark.", "display_name": "Spark Driver Maximum Java Heap Size", "name": "hiveserver2_spark_driver_memory", "value": "268435456" }, { "desc": "Initial number of executors used by the application at any given time. This is required if the dynamic executor allocation feature is enabled.", "display_name": "Initial Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_initial_executors", "value": "1" }, { "desc": "When the number of ReduceSink operators after merging is less than this number, the ReduceDeDuplication optimization will be disabled.", "display_name": "Mininum Reducers for ReduceDeDuplication Optimization", "name": "hiveserver2_optimize_reducededuplication_min_reducer", "value": "4" }, { "desc": "Enable optimization that vectorizes query execution by streamlining operations by processing a block of 1024 rows at a time.", "display_name": "Enable Vectorization Optimization", "name": "hiveserver2_vectorized_enabled", "value": "true" }, { "desc": "The check interval for session/operation timeout, in milliseconds, which can be disabled by setting to zero or a negative value. ", "display_name": "Session Check Interval", "name": "hiveserver2_session_check_interval", "value": "3600000" }, { "desc": "Whether column stats for a table are fetched during explain.", "display_name": "Load Column Statistics", "name": "hiveserver2_stats_fetch_column_stats", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Enable optimization that checks if a query can be answered using statistics. If so, answers the query using only statistics stored in metastore.", "display_name": "Enable Stats Optimization", "name": "hiveserver2_compute_query_using_stats", "value": "false" }, { "desc": "Session will be closed when not accessed for this duration of time, in milliseconds; disable by setting to zero or a negative value.", "display_name": "Idle Session Timeout", "name": "hiveserver2_idle_session_timeout", "value": "604800000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Minimum number of worker threads in HiveServer2's thread pool", "display_name": "Min HiveServer2 Threads", "name": "hiveserver2_min_threads", "value": "5" }, { "desc": "The desired file size after merging. This should be larger than hive.merge.smallfiles.avgsize.", "display_name": "Desired File Size After Merging", "name": "hiveserver2_merge_size_per_task", "value": "268435456" }, { "desc": "Some select queries can be converted to a single FETCH task instead of a MapReduce task, minimizing latency. A value of none disables all conversion, minimal converts simple queries such as SELECT * and filter on partition columns, and more converts SELECT queries including FILTERS.", "display_name": "Fetch Task Query Conversion", "name": "hiveserver2_fetch_task_conversion", "value": "minimal" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Upper bound on the number of executors used by the application at any given time. This is used by dynamic executor allocation.", "display_name": "Upper Bound on Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_max_executors", "value": "2147483647" }, { "desc": "Whether to try sorted merge bucket (SMB) join.", "display_name": "Hive Optimize Sorted Merge Bucket Join", "name": "hiveserver2_optimize_bucketmapjoin_sortedmerge", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_hs2_config_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Enables the health test that the HiveServer2's process state is consistent with the role configuration", "display_name": "HiveServer2 Process Health Test", "name": "hiveserver2_scm_health_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "When dynamic partition is enabled, reducers keep only one record writer at all times, which lowers the memory pressure on reducers.", "display_name": "Enable Sorted Dynamic Partition Optimizer", "name": "hiveserver2_optimize_sort_dynamic_partition", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HiveServer2", "name": "hiveserver2_java_opts", "value": "-XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Number of cores per Spark executor.", "display_name": "Spark Executor Cores", "name": "hiveserver2_spark_executor_cores", "value": "1" }, { "desc": "This is the amount of extra off-heap memory that can be requested from YARN, per executor process. This, together with spark.executor.memory, is the total memory that YARN can use to create JVM for an executor process.", "display_name": "Spark Executor Memory Overhead", "name": "hiveserver2_spark_yarn_executor_memory_overhead", "value": "26" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Local directory where Hive stores jars downloaded for remote file systems (HDFS). If not specified, Hive uses a default location.", "display_name": "Hive Downloaded Resources Directory", "name": "hiveserver2_downloaded_resources_dir", "value": null }, { "desc": "This is the amount of extra off-heap memory that can be requested from YARN, per driver. This, together with spark.driver.memory, is the total memory that YARN can use to create JVM for a driver process.", "display_name": "Spark Driver Memory Overhead", "name": "hiveserver2_spark_yarn_driver_memory_overhead", "value": "26" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Downloaded Resources Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Downloaded Resources Directory", "name": "role_config_suppression_hiveserver2_downloaded_resources_dir", "value": "false" }, { "desc": "Lower bound on the number of executors used by the application at any given time. This is used by dynamic executor allocation.", "display_name": "Lower Bound on Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_min_executors", "value": "1" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "hiveserver2_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "When enabled, Spark will add and remove executors dynamically to Hive jobs. This is done based on the workload.", "display_name": "Enable Dynamic Executor Allocation", "name": "hiveserver2_spark_dynamic_allocation_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hiveserver2_swap_memory_usage", "value": "false" }, { "desc": "If Hive auto convert join is on, and the sum of the size for n-1 of the tables/partitions for a n-way join is smaller than the specified size, the join is directly converted to a MapJoin (there is no conditional task).", "display_name": "Hive Auto Convert Join Noconditional Size", "name": "hiveserver2_auto_convert_join_noconditionaltask_size", "value": "20971520" }, { "desc": "Directory in HDFS where Hive writes intermediate data between MapReduce jobs. If not specified, Hive uses a default location.", "display_name": "Hive HDFS Scratch Directory", "name": "hiveserver2_exec_scratchdir", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hiveserver2_scm_health", "value": "false" }, { "desc": "The minimum log level for HiveServer2 logs", "display_name": "HiveServer2 Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Local Scratch Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Local Scratch Directory", "name": "role_config_suppression_hiveserver2_exec_local_scratchdir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Local Scratch Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Hive Local Scratch Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Hive Local Scratch Directory Free Space Monitoring Percentage Thresholds", "name": "hiveserver2_exec_local_scratch_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hive_metrics_sample_file_location", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Logging Frequency", "name": "hive_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Controls whether the Hive metrics subsystem is enabled for the role.", "display_name": "Enable Metrics Subsystem", "name": "hive_metrics_enabled", "value": "true" }, { "desc": "Maximum number of worker threads in HiveServer2's thread pool", "display_name": "Max HiveServer2 Threads", "name": "hiveserver2_max_threads", "value": "100" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Local Scratch Directory.", "display_name": "Hive Local Scratch Directory Free Space Monitoring Absolute Thresholds", "name": "hiveserver2_exec_local_scratch_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to vectorize the reduce side of query execution.", "display_name": "Enable Reduce-Side Vectorization", "name": "hiveserver2_vectorized_reduce_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Load Balancer parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Load Balancer", "name": "role_config_suppression_hiverserver2_load_balancer", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When enabled, HiveServer2 logs EXPLAIN EXTENDED output for every query at INFO log4j level.", "display_name": "Enable Explain Logging", "name": "hiveserver2_enable_explain_output", "value": "false" }, { "desc": "The number of rows with the same key value to be cached in memory per SMB-joined table.", "display_name": "Hive SMB Join Cache Rows", "name": "hiveserver2_smbjoin_cache_rows", "value": "10000" }, { "desc": "Whether to use the indexing optimization for all queries.", "display_name": "Enable Automatic Use of Indexes", "name": "hiveserver2_optimize_index_filter", "value": "true" }, { "desc": "In vectorized group-by, the number of row entries added to the hash table before re-checking average variable size for memory usage estimation.", "display_name": "Vectorized GroupBy Check Interval", "name": "hiveserver2_vectorized_groupby_checkinterval", "value": "4096" }, { "desc": "Merge small files at the end of a map-reduce job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Map-Reduce Job", "name": "hiveserver2_merge_mapredfiles", "value": "false" }, { "desc": "Enable optimization that converts common join into MapJoin based on input file size.", "display_name": "Enable MapJoin Optimization", "name": "hiveserver2_enable_mapjoin", "value": "true" }, { "desc": "Address of the load balancer used for HiveServer2 roles, specified in host:port format. If port is not specified, the port used by HiveServer2 is used. Note: Changing this property regenerates Kerberos keytabs for all HiveServer2 roles.", "display_name": "HiveServer2 Load Balancer", "name": "hiverserver2_load_balancer", "value": null }, { "desc": "The maximum percentage of heap to be used for hash in ReduceSink operator for Top-K selection. 0 means the optimization is disabled. Accepted values are between 0 and 1.", "display_name": "Maximum ReduceSink Top-K Memory Usage", "name": "hiveserver2_limit_pushdown_memory_usage", "value": "0.1" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hiveserver2_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "When computing the overall HiveServer2 health, consider the host's health.", "display_name": "HiveServer2 Host Health Test", "name": "hiveserver2_host_health_enabled", "value": "true" }, { "desc": "The maximum number of rolled log files to keep for HiveServer2 logs. Typically used by log4j or logback.", "display_name": "HiveServer2 Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Enable map-side partial aggregation, which cause the mapper to generate fewer rows. This reduces the data to be sorted and distributed to reducers.", "display_name": "Enable Map-Side Aggregation", "name": "hiveserver2_map_aggr", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Log Directory parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Log Directory", "name": "role_config_suppression_hive_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Portion of total memory used in map-side partial aggregation. When exceeded, the partially aggregated results will be flushed from the map task to the reducers.", "display_name": "Ratio of Memory Usage for Map-Side Aggregation", "name": "hiveserver2_map_aggr_hash_memory_ratio", "value": "0.5" }, { "desc": "Maximum size of each Spark executor's Java heap memory when Hive is running on Spark.", "display_name": "Spark Executor Maximum Java Heap Size", "name": "hiveserver2_spark_executor_memory", "value": "268435456" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hiveserver2_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Downloaded Resources Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Hive Downloaded Resources Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Hive Downloaded Resources Directory Free Space Monitoring Percentage Thresholds", "name": "hiveserver2_downloaded_resources_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HiveServer2 Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_hs2_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Executor Cores parameter.", "display_name": "Suppress Parameter Validation: Spark Executor Cores", "name": "role_config_suppression_hiveserver2_spark_executor_cores", "value": "false" }, { "desc": "Whether to suppress the results of the Hive Local Scratch Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Local Scratch Directory Free Space", "name": "role_health_suppression_hiveserver2_exec_local_scratch_directory_free_space", "value": "false" }, { "desc": "Directory where HiveServer2 will place its log files.", "display_name": "HiveServer2 Log Directory", "name": "hive_log_dir", "value": "/var/log/hive" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HiveServer2 in Bytes", "name": "hiveserver2_java_heapsize", "value": "4294967296" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hue-service.json0000666000175100017510000011325613245514472024624 0ustar zuulzuul00000000000000[ { "desc": "Base filter for searching for groups. For Active Directory, this is typically '(objectClass=group)'.", "display_name": "LDAP Group Filter", "name": "group_filter", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Whitelist parameter.", "display_name": "Suppress Parameter Validation: Whitelist", "name": "service_config_suppression_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Enable debug output in HTTP Internal Server Error (status 500) responses. Debugging information may contain sensitive data. If Enable Django Debug Mode is set, this is automatically enabled.", "display_name": "Enable Debugging of Internal Server Error Responses", "name": "http_500_debug_enable", "value": "false" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hue/audit" }, { "desc": "Whether to suppress configuration warnings produced by the Load Balancer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Load Balancer Count Validator", "name": "service_config_suppression_hue_load_balancer_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Blacklist parameter.", "display_name": "Suppress Parameter Validation: Blacklist", "name": "service_config_suppression_blacklist", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hue" }, { "desc": "Comma-separated list of regular expressions, which match 'host:port' of requested proxy target.", "display_name": "Whitelist", "name": "whitelist", "value": "(localhost|127\\.0\\.0\\.1):(50030|50070|50060|50075)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Hostname parameter.", "display_name": "Suppress Parameter Validation: Hue Database Hostname", "name": "service_config_suppression_database_host", "value": "false" }, { "desc": "The PAM service name to use when authenticating over desktop.auth.backend.PamBackend. This is typically the name of a file under /etc/pam.d/ on the Hue host.", "display_name": "PAM Backend Service Name", "name": "pam_auth_service", "value": "login" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "The health test thresholds of the overall Kerberos Ticket Renewer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Kerberos Ticket Renewers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Kerberos Ticket Renewers falls below the critical threshold.", "display_name": "Healthy Kerberos Ticket Renewer Monitoring Thresholds", "name": "hue_kt_renewers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress the results of the Kerberos Ticket Renewer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Kerberos Ticket Renewer Health", "name": "service_health_suppression_hue_kt_renewers_healthy", "value": "false" }, { "desc": "The username attribute in the LDAP schema. For Active Directory, this is typically 'sAMAccountName'.", "display_name": "LDAP Username Attribute", "name": "user_name_attr", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Filter parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Filter", "name": "service_config_suppression_group_filter", "value": "false" }, { "desc": "Whether to suppress the results of the Hue Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hue Server Health", "name": "service_health_suppression_hue_hue_servers_healthy", "value": "false" }, { "desc": "In debug mode, Django displays a detailed traceback when an exception occurs. Debugging information may contain sensitive data. Django remembers every SQL query it executes in debug mode, which will rapidly consume memory.", "display_name": "Enable Django Debug Mode", "name": "django_debug_enable", "value": "false" }, { "desc": "Whether to suppress the results of the Load Balancer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Load Balancer Health", "name": "service_health_suppression_hue_load_balancer_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hue Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hue_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Database Directory", "name": "service_config_suppression_database_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Search Base parameter.", "display_name": "Suppress Parameter Validation: LDAP Search Base", "name": "service_config_suppression_base_dn", "value": "false" }, { "desc": "Name of the Solr service that this Hue service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default User Group parameter.", "display_name": "Suppress Parameter Validation: Default User Group", "name": "service_config_suppression_default_user_group", "value": "false" }, { "desc": "HTTPFS role or Namenode (if webhdfs is enabled) that hue can use to communicate with HDFS.", "display_name": "HDFS Web Interface Role", "name": "hue_webhdfs", "value": null }, { "desc": "Name of the ZooKeeper service that this Hue service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_hue_sentry_safety_valve", "value": "false" }, { "desc": "Class that defines extra accessor methods for user objects.", "display_name": "User Augmentor", "name": "user_augmentor", "value": "desktop.auth.backend.DefaultUserAugmentor" }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "use_start_tls", "value": "true" }, { "desc": "When you enable anonymous usage data collection Hue tracks anonymised pages and application versions in order to gather information about each application's usage levels. The data collected does not include any hostnames or IDs. Data collection option is available on CDH 4.4 and later deployments.", "display_name": "Enable Usage Data Collection", "name": "usage_data_collection_enable", "value": "true" }, { "desc": "Time zone name.", "display_name": "Time Zone", "name": "time_zone", "value": "America/Los_Angeles" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Web Interface Role parameter.", "display_name": "Suppress Parameter Validation: HDFS Web Interface Role", "name": "service_config_suppression_hue_webhdfs", "value": "false" }, { "desc": "The password of the bind user.", "display_name": "LDAP Bind Password", "name": "bind_password", "value": null }, { "desc": "Name of the Sqoop service that this Hue service instance depends on", "display_name": "Sqoop Service", "name": "sqoop_service", "value": null }, { "desc": "Type of database used for Hue", "display_name": "Hue Database Type", "name": "database_type", "value": "sqlite3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Temporary Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Temporary Directory", "name": "service_config_suppression_hdfs_tmp_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Username Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Username Attribute", "name": "service_config_suppression_user_name_attr", "value": "false" }, { "desc": "Port on host where the Hue database is running. Not necessary for SQLite3.", "display_name": "Hue Database Port", "name": "database_port", "value": "3306" }, { "desc": "For advanced use only, a string to be inserted into hue_safety_valve.ini. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", "name": "hue_service_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Password parameter.", "display_name": "Suppress Parameter Validation: Hue Database Password", "name": "service_config_suppression_database_password", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Hue events:\n

\n\n
    \n
  • operation: the Hue operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Username parameter.", "display_name": "Suppress Parameter Validation: Hue Database Username", "name": "service_config_suppression_database_user", "value": "false" }, { "desc": "Comma-separated list of regular expressions, which match any prefix of 'host:port/path' of requested proxy target. This does not support matching GET parameters.", "display_name": "Blacklist", "name": "blacklist", "value": "()" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Database Dump File parameter.", "display_name": "Suppress Parameter Validation: Database Dump File", "name": "service_config_suppression_database_dump_file", "value": "false" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "ldap_url", "value": null }, { "desc": "Default encoding for site data.", "display_name": "Default Site Encoding", "name": "default_site_encoding", "value": "utf" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Server CA Certificate parameter.", "display_name": "Suppress Parameter Validation: LDAP Server CA Certificate", "name": "service_config_suppression_ldap_cert", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the PAM Backend Service Name parameter.", "display_name": "Suppress Parameter Validation: PAM Backend Service Name", "name": "service_config_suppression_pam_auth_service", "value": "false" }, { "desc": "The health test thresholds of the overall Load Balancer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Load Balancers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Load Balancers falls below the critical threshold.", "display_name": "Healthy Load Balancer Monitoring Thresholds", "name": "hue_load_balancer_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Thrift Usage Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Thrift Usage Validator", "name": "service_config_suppression_hue_hbase_thrift_server_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_ldap_url", "value": "false" }, { "desc": "The group name attribute in the LDAP schema. For Active Directory, this is typically 'cn'.", "display_name": "LDAP Group Name Attribute", "name": "group_name_attr", "value": null }, { "desc": "The distinguished name to use as a search base for finding users and groups. This should be similar to 'dc=hadoop,dc=mycompany,dc=com'.", "display_name": "LDAP Search Base", "name": "base_dn", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Username Pattern parameter.", "display_name": "Suppress Parameter Validation: LDAP Username Pattern", "name": "service_config_suppression_ldap_username_pattern", "value": "false" }, { "desc": "The base filter for searching for users. For Active Directory, this is typically '(objectClass=user)'.", "display_name": "LDAP User Filter", "name": "user_filter", "value": null }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "nt_domain", "value": null }, { "desc": "HDFS directory used for storing temporary files.", "display_name": "HDFS Temporary Directory", "name": "hdfs_tmp_dir", "value": "/tmp" }, { "desc": "Password for Hue database. Not necessary for SQLite3.", "display_name": "Hue Database Password", "name": "database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the User Augmentor parameter.", "display_name": "Suppress Parameter Validation: User Augmentor", "name": "service_config_suppression_user_augmentor", "value": "false" }, { "desc": "File where the database gets dumped to or loaded from.", "display_name": "Database Dump File", "name": "database_dump_file", "value": "/tmp/hue_database_dump.json" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS HTTPFS Usage Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS HTTPFS Usage Validator", "name": "service_config_suppression_hdfs_httpfs_present_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Name Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Name Attribute", "name": "service_config_suppression_group_name_attr", "value": "false" }, { "desc": "When computing the overall HUE health, consider Beeswax Server's health", "display_name": "Beeswax Server Role Health Test", "name": "hue_beeswax_server_health_enabled", "value": "true" }, { "desc": "Create users in Hue when they try to login with their LDAP credentials. For use when using LdapBackend for Hue authentication.", "display_name": "Create LDAP users on login", "name": "create_users_on_login", "value": "true" }, { "desc": "The name of a default group that users will be added to at creation time.", "display_name": "Default User Group", "name": "default_user_group", "value": null }, { "desc": "Name of host where the Hue database is running. Not necessary for SQLite3.", "display_name": "Hue Database Hostname", "name": "database_host", "value": "localhost" }, { "desc": "Thrift server to use for HBase app.", "display_name": "HBase Thrift Server", "name": "hue_hbase_thrift", "value": null }, { "desc": "Number of threads used by the Hue web server.", "display_name": "Hue Web Server Threads", "name": "cherrypy_server_threads", "value": "50" }, { "desc": "Mode of authenticating login credentials. Select desktop.auth.backend.LdapBackend to use LDAP to authenticate login credentials. LDAP requires you to also set the LDAP URL, Active Directory Domain, and optionally LDAP certificate if you are using secure LDAP. Select desktop.auth.backend.PamBackend to use PAM to authenticate login credentials.", "display_name": "Authentication Backend", "name": "auth_backend", "value": "desktop.auth.backend.AllowFirstUserDjangoBackend" }, { "desc": "Distinguished name of the user to bind as. This is used to connect to LDAP/AD for searching user and group information. This may be left blank if the LDAP server supports anonymous binds.", "display_name": "LDAP Bind User Distinguished Name", "name": "bind_dn", "value": null }, { "desc": "Whether to suppress the results of the Beeswax Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Beeswax Server Health", "name": "service_health_suppression_hue_beeswax_server_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Time Zone parameter.", "display_name": "Suppress Parameter Validation: Time Zone", "name": "service_config_suppression_time_zone", "value": "false" }, { "desc": "The username to use to log into the Hue database. Not necessary for SQLite3.", "display_name": "Hue Database Username", "name": "database_user", "value": "hue" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "hue_sentry_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hue_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_nt_domain", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Search Bind Authentication connects to the LDAP server using credentials provided in the 'bind_dn' and 'bind_password' configurations. If these configurations are not set, then an anonymous search is performed.", "display_name": "Use Search Bind Authentication", "name": "search_bind_authentication", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hue Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hue Server Count Validator", "name": "service_config_suppression_hue_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Site Encoding parameter.", "display_name": "Suppress Parameter Validation: Default Site Encoding", "name": "service_config_suppression_default_site_encoding", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Bind Password parameter.", "display_name": "Suppress Parameter Validation: LDAP Bind Password", "name": "service_config_suppression_bind_password", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "The attribute of the group object that identifies the members of the group. For Active Directory, this is typically 'member'.", "display_name": "LDAP Group Membership Attribute", "name": "group_member_attr", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Name parameter.", "display_name": "Suppress Parameter Validation: Hue Database Name", "name": "service_config_suppression_database_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP User Filter parameter.", "display_name": "Suppress Parameter Validation: LDAP User Filter", "name": "service_config_suppression_user_filter", "value": "false" }, { "desc": "Name of the Sentry service that this Hue service instance depends on", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Name of the HBase service that this Hue service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Name of the Hive service that this Hue service instance depends on", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Bind User Distinguished Name parameter.", "display_name": "Suppress Parameter Validation: LDAP Bind User Distinguished Name", "name": "service_config_suppression_bind_dn", "value": "false" }, { "desc": "The location on disk of the certificate, in .pem format, used to confirm the authenticity of the LDAP server certificate. This is the Certificate Authority (CA) certificate, and it was used to sign the LDAP server certificate. If not set, all certificates are trusted, which means that an attacker could potentially intercept otherwise encrypted usernames and passwords.", "display_name": "LDAP Server CA Certificate", "name": "ldap_cert", "value": null }, { "desc": "Name of the Oozie service that this Hue service instance depends on", "display_name": "Oozie Service", "name": "oozie_service", "value": null }, { "desc": "Name of the Impala service that this Hue service instance depends on", "display_name": "Impala Service", "name": "impala_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "LDAP Username Pattern for use with non-Active Directory LDAP implementations. Must contain the special '<username>' string for replacement during authentication.", "display_name": "LDAP Username Pattern", "name": "ldap_username_pattern", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Membership Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Membership Attribute", "name": "service_config_suppression_group_member_attr", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Kerberos Ticket Renewer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kerberos Ticket Renewer Count Validator", "name": "service_config_suppression_kt_renewer_count_validator", "value": "false" }, { "desc": "Name of Hue database.", "display_name": "Hue Database Name", "name": "database_name", "value": "hue" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hue" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini parameter.", "display_name": "Suppress Parameter Validation: Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", "name": "service_config_suppression_hue_service_safety_valve", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hue" }, { "desc": "The health test thresholds of the overall Hue Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Hue Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Hue Servers falls below the critical threshold.", "display_name": "Healthy Hue Server Monitoring Thresholds", "name": "hue_hue_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "If the database is SQLite3, this is the filename of the database to use, and the directory of this file must be writable by the 'hue' user.", "display_name": "Hue Database Directory", "name": "database_dir", "value": "/var/lib/hue/desktop.db" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/spark-spark_yarn_history_server.json0000666000175100017510000004525013245514472031041 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf parameter.", "display_name": "Suppress Parameter Validation: History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf", "name": "role_config_suppression_spark-history-server.conf_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: History Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_file_descriptor", "value": "false" }, { "desc": "Enables the health test that the History Server's process state is consistent with the role configuration", "display_name": "History Server Process Health Test", "name": "spark_yarn_history_server_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "spark_yarn_history_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_scm_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "How often the History Server will clean up event log files.", "display_name": "Event Log Cleaner Interval", "name": "event_log_cleaner_interval", "value": "86400" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Max number of application UIs to keep in the History Server's memory. All applications will still be available, but may take longer to load if they're not in memory.", "display_name": "Retained App Count", "name": "history_server_retained_apps", "value": "50" }, { "desc": "Specifies the maximum age of the event logs.", "display_name": "Maximum Event Log Age", "name": "event_log_cleaner_max_age", "value": "604800" }, { "desc": "The minimum log level for History Server logs", "display_name": "History Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into spark-conf/spark-env.sh for this role only.", "display_name": "History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_role_safety_valve", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "History Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SPARK_YARN_HISTORY_SERVER_role_env_safety_valve", "value": null }, { "desc": "The maximum number of rolled log files to keep for History Server logs. Typically used by log4j or logback.", "display_name": "History Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The port of the history server WebUI", "display_name": "History Server WebUI Port", "name": "history_server_web_port", "value": "18088" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes.", "display_name": "Java Heap Size of History Server in Bytes", "name": "history_server_max_heapsize", "value": "536870912" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: History Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_spark_yarn_history_server_role_env_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: History Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "role_config_suppression_spark-conf/spark-env.sh_role_safety_valve", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_unexpected_exits", "value": "false" }, { "desc": "The log directory for log files of the role History Server.", "display_name": "History Server Log Directory", "name": "log_dir", "value": "/var/log/spark" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "How often to poll HDFS for new applications.", "display_name": "HDFS Polling Interval", "name": "history_server_fs_poll_interval", "value": "10" }, { "desc": "The maximum size, in megabytes, per log file for History Server logs. Typically used by log4j or logback.", "display_name": "History Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "History Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Specifies whether the History Server should periodically clean up event logs from storage.", "display_name": "Enable Event Log Cleaner", "name": "event_log_cleaner_enabled", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into spark-history-server.conf for this role only.", "display_name": "History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf", "name": "spark-history-server.conf_role_safety_valve", "value": null }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When computing the overall History Server health, consider the host's health.", "display_name": "History Server Host Health Test", "name": "spark_yarn_history_server_host_health_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/yarn-nodemanager.json0000666000175100017510000013565213245514472025640 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "nodemanager_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Enables the health test that the NodeManager's process state is consistent with the role configuration", "display_name": "NodeManager Process Health Test", "name": "nodemanager_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_node_manager_unexpected_exits", "value": "false" }, { "desc": "Path (rooted in the cgroups hierarchy on the machine) where to place YARN-managed cgroups.", "display_name": "CGroups Hierarchy", "name": "linux_container_executor_cgroups_hierarchy", "value": "/hadoop-yarn" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Maximum allowed threads for serving shuffle connections. Set to zero to indicate the default of 2 times the number of available processors.", "display_name": "Max Shuffle Threads", "name": "mapreduce_shuffle_max_threads", "value": "80" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Recovery Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Recovery Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Recovery Directory Free Space Monitoring Percentage Thresholds", "name": "nodemanager_recovery_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Enables the health check that verifies the NodeManager is connected to the ResourceManager.", "display_name": "NodeManager Connectivity Health Check", "name": "nodemanager_connectivity_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Containers Environment Variables Whitelist parameter.", "display_name": "Suppress Parameter Validation: Containers Environment Variables Whitelist ", "name": "role_config_suppression_yarn_nodemanager_env_whitelist", "value": "false" }, { "desc": "List of users banned from running containers.", "display_name": "Banned System Users", "name": "container_executor_banned_users", "value": "hdfs,yarn,mapred,bin" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "nodemanager_mapred_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_node_manager_file_descriptor", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The system group that owns the container-executor binary. This does not need to be changed unless the ownership of the binary is explicitly changed.", "display_name": "Container Executor Group", "name": "container_executor_group", "value": "yarn" }, { "desc": "Target size of localizer cache in MB, per local directory.", "display_name": "Localizer Cache Target Size", "name": "yarn_nodemanager_localizer_cache_target_size_mb", "value": "10240" }, { "desc": "The HTTPS port of the NodeManager web application.", "display_name": "NodeManager Web Application HTTPS Port (TLS/SSL)", "name": "nodemanager_webserver_https_port", "value": "8044" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Banned System Users parameter.", "display_name": "Suppress Parameter Validation: Banned System Users", "name": "role_config_suppression_container_executor_banned_users", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Log Directory parameter.", "display_name": "Suppress Parameter Validation: NodeManager Log Directory", "name": "role_config_suppression_node_manager_log_dir", "value": "false" }, { "desc": "The minimum fraction of number of disks to be healthy for the NodeManager to launch new containers. This correspond to both local dirs and log dirs, i.e., if there are fewer number of healthy local dirs (or log dirs) available, then new containers will not be launched on this node.", "display_name": "Disk Health Checker Minimum Health Disks Fraction", "name": "yarn_nodemanager_disk_health_checker_min_healthy_disks", "value": "0.25" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed System Users parameter.", "display_name": "Suppress Parameter Validation: Allowed System Users", "name": "role_config_suppression_container_executor_allowed_system_users", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Number of threads to use for localization fetching.", "display_name": "Localizer Fetch Thread Count", "name": "yarn_nodemanager_localizer_fetch_thread_count", "value": "4" }, { "desc": "Whether to suppress the results of the NodeManager Container Log Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Container Log Directories Free Space", "name": "role_health_suppression_nodemanager_log_directories_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_node_manager_gc_duration", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "If enabled, adds 'org.apache.hadoop.mapred.ShuffleHandler' to the NodeManager auxiliary services. This is required for MapReduce applications.", "display_name": "Enable Shuffle Auxiliary Service", "name": "mapreduce_aux_service", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The maximum size, in megabytes, per log file for NodeManager logs. Typically used by log4j or logback.", "display_name": "NodeManager Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "nodemanager_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the NodeManager Recovery Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Recovery Directory Free Space", "name": "role_health_suppression_nodemanager_recovery_directory_free_space", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The amount of time to wait for the NodeManager to fully start up and connect to the ResourceManager before enforcing the connectivity check.", "display_name": "NodeManager Connectivity Tolerance at Startup", "name": "nodemanager_connectivity_tolerance_seconds", "value": "180" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "nodemanager_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Amount of physical memory, in MiB, that can be allocated for containers.", "display_name": "Container Memory", "name": "yarn_nodemanager_resource_memory_mb", "value": "8192" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_node_manager_heap_dump_directory_free_space", "value": "false" }, { "desc": "The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}", "display_name": "Remote App Log Directory Suffix", "name": "yarn_nodemanager_remote_app_log_dir_suffix", "value": "logs" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "nodemanager_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Container Executor Group parameter.", "display_name": "Suppress Parameter Validation: Container Executor Group", "name": "role_config_suppression_container_executor_group", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Healthchecker Script Path parameter.", "display_name": "Suppress Parameter Validation: Healthchecker Script Path", "name": "role_config_suppression_mapred_healthchecker_script_path", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_nodemanager_config_safety_valve", "value": "false" }, { "desc": "HDFS directory where application logs are stored when an application completes.", "display_name": "Remote App Log Directory", "name": "yarn_nodemanager_remote_app_log_dir", "value": "/tmp/logs" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_node_manager_web_metric_collection", "value": "false" }, { "desc": "When computing the overall NodeManager health, consider the host's health.", "display_name": "NodeManager Host Health Test", "name": "nodemanager_host_health_enabled", "value": "true" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "nodemanager_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the ResourceManager Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ResourceManager Connectivity", "name": "role_health_suppression_node_manager_connectivity", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "nodemanager_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Local Directories.", "display_name": "NodeManager Local Directories Free Space Monitoring Absolute Thresholds", "name": "nodemanager_local_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NodeManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The minimum Linux user ID allowed. Used to prevent other super users.", "display_name": "Minimum User ID", "name": "container_executor_min_user_id", "value": "1000" }, { "desc": "Absolute path to the script which is periodically run by the node health monitoring service to determine if the node is healthy or not. If the value of this key is empty or the file does not exist in the location configured here, the node health monitoring service is not started.", "display_name": "Healthchecker Script Path", "name": "mapred_healthchecker_script_path", "value": "" }, { "desc": "Enables the health check that verifies the NodeManager is seen as healthy by the ResourceManager.", "display_name": "NodeManager Health Checker Health Check", "name": "nodemanager_health_checker_health_enabled", "value": "true" }, { "desc": "Address where the localizer IPC is.", "display_name": "Localizer Port", "name": "yarn_nodemanager_localizer_address", "value": "8040" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Remote App Log Directory parameter.", "display_name": "Suppress Parameter Validation: Remote App Log Directory", "name": "role_config_suppression_yarn_nodemanager_remote_app_log_dir", "value": "false" }, { "desc": "Number of virtual CPU cores that can be allocated for containers. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores", "name": "yarn_nodemanager_resource_cpu_vcores", "value": "8" }, { "desc": "Frequency, in milliseconds, of running disk health checker.", "display_name": "Disk Health Checker Frequency", "name": "yarn_nodemanager_disk_health_checker_interval_ms", "value": "120000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Container Log Directories parameter.", "display_name": "Suppress Parameter Validation: NodeManager Container Log Directories", "name": "role_config_suppression_yarn_nodemanager_log_dirs", "value": "false" }, { "desc": "Comma-separated list of arguments which are to be passed to node health script when it is being launched.", "display_name": "Healthchecker Script Arguments", "name": "mapred_healthchecker_script_args", "value": "" }, { "desc": "The minimum log level for NodeManager logs", "display_name": "NodeManager Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the NodeManager Local Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Local Directories Free Space", "name": "role_health_suppression_nodemanager_local_data_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Local Directories parameter.", "display_name": "Suppress Parameter Validation: NodeManager Local Directories", "name": "role_config_suppression_yarn_nodemanager_local_dirs", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_nodemanager_mapred_safety_valve", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Recovery Directory.", "display_name": "NodeManager Recovery Directory Free Space Monitoring Absolute Thresholds", "name": "nodemanager_recovery_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NodeManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "NODEMANAGER_role_env_safety_valve", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Number of threads to handle localization requests.", "display_name": "Localizer Client Thread Count", "name": "yarn_nodemanager_localizer_client_thread_count", "value": "5" }, { "desc": "List of directories on the local filesystem where a NodeManager stores container log files.", "display_name": "NodeManager Container Log Directories", "name": "yarn_nodemanager_log_dirs", "value": "/var/log/hadoop-yarn/container" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NodeManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Local Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Local Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Local Directories Free Space Monitoring Percentage Thresholds", "name": "nodemanager_local_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The local filesystem directory in which the NodeManager stores state when recovery is enabled. Recovery is enabled by default.", "display_name": "NodeManager Recovery Directory", "name": "yarn_nodemanager_recovery_dir", "value": "/var/lib/hadoop-yarn/yarn-nm-recovery" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_node_manager_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The address of the NodeManager IPC.", "display_name": "NodeManager IPC Address", "name": "yarn_nodemanager_address", "value": "8041" }, { "desc": "The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the NodeManager will check for full disk. This applies to local dirs and log dirs.", "display_name": "Dish Health Checker Max Disk Utilization Percent", "name": "yarn_nodemanager_disk_health_checker_max_disk_utilization_per_disk_percentage", "value": "90.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The HTTP Port of the NodeManager web application.", "display_name": "NodeManager Web Application HTTP Port", "name": "nodemanager_webserver_port", "value": "8042" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the NodeManager Health Checker heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Health Checker", "name": "role_health_suppression_node_manager_health_checker", "value": "false" }, { "desc": "Number of threads container manager uses.", "display_name": "Container Manager Thread Count", "name": "yarn_nodemanager_container_manager_thread_count", "value": "20" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Containers Environment Variable parameter.", "display_name": "Suppress Parameter Validation: Containers Environment Variable", "name": "role_config_suppression_yarn_nodemanager_admin_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Single User Mode Overrides Validator configuration validator.", "display_name": "Suppress Configuration Validator: Single User Mode Overrides Validator", "name": "role_config_suppression_single_user_mode_override_validator", "value": "false" }, { "desc": "Address where the localizer IPC is.", "display_name": "Localizer Cache Cleanup Interval", "name": "yarn_nodemanager_localizer_cache_cleanup_interval_ms", "value": "600000" }, { "desc": "List of users explicitly whitelisted to be allowed to run containers. Users with IDs lower than the \"Minimum User Id\" setting may be whitelisted by using this setting.", "display_name": "Allowed System Users", "name": "container_executor_allowed_system_users", "value": "nobody,impala,hive,llama" }, { "desc": "List of directories on the local filesystem where a NodeManager stores intermediate data files.", "display_name": "NodeManager Local Directories", "name": "yarn_nodemanager_local_dirs", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Remote App Log Directory Suffix parameter.", "display_name": "Suppress Parameter Validation: Remote App Log Directory Suffix", "name": "role_config_suppression_yarn_nodemanager_remote_app_log_dir_suffix", "value": "false" }, { "desc": "Number of threads used in cleanup.", "display_name": "Cleanup Thread Count", "name": "yarn_nodemanager_delete_thread_count", "value": "4" }, { "desc": "Environment variables that should be forwarded from the NodeManager's environment to the container's.", "display_name": "Containers Environment Variable", "name": "yarn_nodemanager_admin_env", "value": "MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX" }, { "desc": "The maximum number of rolled log files to keep for NodeManager logs. Typically used by log4j or logback.", "display_name": "NodeManager Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_node_manager_scm_health", "value": "false" }, { "desc": "Environment variables that containers may override rather than use NodeManager's default.", "display_name": "Containers Environment Variables Whitelist ", "name": "yarn_nodemanager_env_whitelist", "value": "JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME" }, { "desc": "Directory where NodeManager will place its log files.", "display_name": "NodeManager Log Directory", "name": "node_manager_log_dir", "value": "/var/log/hadoop-yarn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NodeManager in Bytes", "name": "node_manager_java_heapsize", "value": "1073741824" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Container Log Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Container Log Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Container Log Directories Free Space Monitoring Percentage Thresholds", "name": "nodemanager_log_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_node_manager_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Recovery Directory parameter.", "display_name": "Suppress Parameter Validation: NodeManager Recovery Directory", "name": "role_config_suppression_yarn_nodemanager_recovery_dir", "value": "false" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Healthchecker Script Arguments parameter.", "display_name": "Suppress Parameter Validation: Healthchecker Script Arguments", "name": "role_config_suppression_mapred_healthchecker_script_args", "value": "false" }, { "desc": "Heartbeat interval to ResourceManager", "display_name": "Heartbeat Interval", "name": "yarn_nodemanager_heartbeat_interval_ms", "value": "1000" }, { "desc": "The minimum space that must be available on a disk for it to be used. This applies to local dirs and log dirs.", "display_name": "Disk Health Checker Minimum Health Disk Space", "name": "yarn_nodemanager_disk_health_checker_min_free_space_per_disk_mb", "value": "0" }, { "desc": "Time in seconds to retain user logs. Only applicable if log aggregation is disabled.", "display_name": "Log Retain Duration", "name": "yarn_nodemanager_log_retain_seconds", "value": "10800" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Container Log Directories.", "display_name": "NodeManager Container Log Directories Free Space Monitoring Absolute Thresholds", "name": "nodemanager_log_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_node_manager_host_health", "value": "false" }, { "desc": "Number of seconds after an application finishes before the NodeManager's DeletionService will delete the application's localized file and log directory. To diagnose YARN application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories.", "display_name": "Localized Dir Deletion Delay", "name": "yarn_nodemanager_delete_debug_delay_sec", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the CGroups Hierarchy parameter.", "display_name": "Suppress Parameter Validation: CGroups Hierarchy", "name": "role_config_suppression_linux_container_executor_cgroups_hierarchy", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NodeManager parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NodeManager", "name": "role_config_suppression_node_manager_java_opts", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NodeManager", "name": "node_manager_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NodeManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_nodemanager_role_env_safety_valve", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-balancer.json0000666000175100017510000001537713245514472025103 0ustar zuulzuul00000000000000[ { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Balancer", "name": "balancer_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_balancer_config_safety_valve", "value": "false" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The percentage deviation from average utilization, after which a node will be rebalanced. (for example, '10.0' for 10%)", "display_name": "Rebalancing Threshold", "name": "rebalancer_threshold", "value": "10.0" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "balancer_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Balancer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Balancer", "name": "role_config_suppression_balancer_java_opts", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Balancer in Bytes", "name": "balancer_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The policy that should be used to rebalance HDFS storage. The default DataNode policy balances the storage at the DataNode level. This is similar to the balancing policy from prior releases. The BlockPool policy balances the storage at the block pool level as well as at the DataNode level. The BlockPool policy is relevant only to a Federated HDFS service.", "display_name": "Rebalancing Policy", "name": "rebalancing_policy", "value": "DataNode" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/sentry-service.json0000666000175100017510000004424013245514472025363 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Name parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Name", "name": "service_config_suppression_sentry_server_database_name", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/sentry/audit" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Host parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Host", "name": "service_config_suppression_sentry_server_database_host", "value": "false" }, { "desc": "Name of Sentry Server database.", "display_name": "Sentry Server Database Name", "name": "sentry_server_database_name", "value": "sentry" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "sentry" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Sentry Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Sentry Server Count Validator", "name": "service_config_suppression_sentry_server_count_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "sentry" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Sentry events:\n

\n\n
    \n
  • operation: the Sentry operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "The health test thresholds of the overall Sentry Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Sentry Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Sentry Servers falls below the critical threshold.", "display_name": "Healthy Sentry Server Monitoring Thresholds", "name": "sentry_sentry_server_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "List of users allowed to connect to the Sentry Server. These are usually service users such as hive and impala, and the list does not usually need to include end users.", "display_name": "Allowed Connecting Users", "name": "sentry_service_allow_connect", "value": "hive,impala,hue,hdfs" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database User parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database User", "name": "service_config_suppression_sentry_server_database_user", "value": "false" }, { "desc": "Host name of Sentry Server database.", "display_name": "Sentry Server Database Host", "name": "sentry_server_database_host", "value": "localhost" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin Groups parameter.", "display_name": "Suppress Parameter Validation: Admin Groups", "name": "service_config_suppression_sentry_service_admin_group", "value": "false" }, { "desc": "If an end user is in one of these admin groups, that user has administrative privileges on the Sentry Server.", "display_name": "Admin Groups", "name": "sentry_service_admin_group", "value": "hive,impala,hue" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Connecting Users parameter.", "display_name": "Suppress Parameter Validation: Allowed Connecting Users", "name": "service_config_suppression_sentry_service_allow_connect", "value": "false" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Sentry events:\n

\n\n
    \n
  • operation: the Sentry operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "Password for Sentry Server database.", "display_name": "Sentry Server Database Password", "name": "sentry_server_database_password", "value": "" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Name of the HDFS service that this Sentry service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Port number of Sentry Server database.", "display_name": "Sentry Server Database Port", "name": "sentry_server_database_port", "value": "3306" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Password parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Password", "name": "service_config_suppression_sentry_server_database_password", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "sentry" }, { "desc": "User for Sentry Server database.", "display_name": "Sentry Server Database User", "name": "sentry_server_database_user", "value": "sentry" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "sentry_server_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_sentry_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sentry Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "SENTRY_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_sentry_server_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Sentry service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Type of Sentry Server database.", "display_name": "Sentry Server Database Type", "name": "sentry_server_database_type", "value": "mysql" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Sentry Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sentry Server Health", "name": "service_health_suppression_sentry_sentry_servers_healthy", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/impala-catalogserver.json0000666000175100017510000007475013245514472026514 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Directory where the Catalog Server core dump is placed.", "display_name": "Catalog Server Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/catalogd" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_catalogserver_log_directory_free_space", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "When computing the overall Impala Catalog Server health, consider the host's health.", "display_name": "Impala Catalog Server Host Health Test", "name": "catalogserver_host_health_enabled", "value": "true" }, { "desc": "Port where Catalog Server debug web server runs.", "display_name": "Catalog Server HTTP Server Port", "name": "catalogserver_webserver_port", "value": "25020" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "Password for Catalog Server web server authentication.", "display_name": "Catalog Server Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Catalog Server command line flags. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_cmd_args_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server HBase Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server HBase Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hbase_conf_safety_valve", "value": "false" }, { "desc": "The minimum log level for Impala Catalog Server logs", "display_name": "Impala Catalog Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Catalog Server", "name": "catalogd_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Hive Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Hive Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hive_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "CATALOGSERVER_role_env_safety_valve", "value": null }, { "desc": "Enables the health test that verifies the Catalog Server is connected to the StateStore", "display_name": "Catalog Server Connectivity Health Test", "name": "catalogserver_connectivity_health_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Timeout for requests to the Hive Metastore Server from Catalog Server. Consider increasing this if you have tables with a lot of metadata and see timeout errors.", "display_name": "Catalog Server Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "3600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "Directory where Catalog Server will place its log files.", "display_name": "Catalog Server Log Directory", "name": "log_dir", "value": "/var/log/catalogd" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "Catalog Server Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "The amount of time to wait for the Catalog Server to fully start up and connect to the StateStore before enforcing the connectivity check.", "display_name": "Catalog Server Connectivity Tolerance at Startup", "name": "catalogserver_connectivity_tolerance", "value": "180" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_cmd_args_safety_valve", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_catalogserver_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_catalogserver_web_metric_collection", "value": "false" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "Catalog Server Maximum Log Files", "name": "max_log_files", "value": "10" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "catalogserver_startup_tolerance", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Catalog Server Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "The password for the private key in the Catalog Server Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Catalog Server Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "Enables the health test that the Impala Catalog Server's process state is consistent with the role configuration", "display_name": "Impala Catalog Server Process Health Test", "name": "catalogserver_scm_health_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml for this role only.", "display_name": "Catalog Server HDFS Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hdfs_site_conf_safety_valve", "value": null }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_catalogserver_memory_rss_health", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_catalogserver_heap_dump_directory_free_space", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above.", "display_name": "Catalog Server Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Catalog Server Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Catalog Server in Bytes", "name": "catalogd_embedded_jvm_heapsize", "value": "4294967296" }, { "desc": "Enable/Disable Catalog Server web server. This web server contains useful information about Catalog Server daemon.", "display_name": "Enable Catalog Server Web Server", "name": "catalogd_enable_webserver", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml for this role only.", "display_name": "Catalog Server Hive Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hive_conf_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Catalog Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Catalog Server", "name": "role_config_suppression_catalogd_java_opts", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_catalogserver_swap_memory_usage", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "catalogserver_web_metric_collection_enabled", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Impala Catalog Server logs. Typically used by log4j or logback.", "display_name": "Impala Catalog Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Username for Catalog Server web server authentication.", "display_name": "Catalog Server Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_catalogserver_file_descriptor", "value": "false" }, { "desc": "Port where Catalog Server is exported.", "display_name": "Catalog Server Service Port", "name": "catalog_service_port", "value": "26000" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the StateStore Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: StateStore Connectivity", "name": "role_health_suppression_catalogserver_connectivity", "value": "false" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server HDFS Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server HDFS Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hdfs_site_conf_safety_valve", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml for this role only.", "display_name": "Catalog Server HBase Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hbase_conf_safety_valve", "value": null }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "catalogserver_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "If true, loads catalog metadata in the background. If false, metadata is loaded lazily (on access). Only effective in CDH 5 and Impala 1.2.4 and higher.", "display_name": "Load Catalog in Background", "name": "load_catalog_in_background", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_catalogserver_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_catalogserver_host_health", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "catalogserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-nfsgateway.json0000666000175100017510000006524713245514472025505 0ustar zuulzuul00000000000000[ { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NFS Gateway Environment Advanced Configuration Snippet (Safety Valve)", "name": "NFSGATEWAY_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_nfsgateway_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_nfsgateway_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_nfsgateway_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Log Directory parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Log Directory", "name": "role_config_suppression_nfsgateway_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "nfsgateway_config_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NFS Gateway in Bytes", "name": "nfsgateway_java_heapsize", "value": "268435456" }, { "desc": "By default, NFS Gateway exported directories can be mounted by any client. For better access control, update this property with a list of host names and access privileges separated by whitespace characters. Host name format can be a single host, a Java regular expression, or an IPv4 address. The access privilege uses rw to specify readwrite and ro to specify readonly access. If the access privilege is not provided, the default is read-only. Examples of host name format and access privilege: \"192.168.0.0/22 rw\", \"host.*.example.com\", \"host1.test.org ro\".", "display_name": "Allowed Hosts and Privileges", "name": "dfs_nfs_exports_allowed_hosts", "value": "* rw" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_nfsgateway_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "The minimum log level for NFS Gateway logs", "display_name": "NFS Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The port number of the system portmap or rpcbind service. This configuration is used by Cloudera Manager to verify if the system portmap or rpcbind service is running before starting NFS Gateway role. Cloudera Manager does not manage the system portmap or rpcbind service.", "display_name": "Portmap (or Rpcbind) Port", "name": "nfs3_portmap_port", "value": "111" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where NFS Gateway will place its log files.", "display_name": "NFS Gateway Log Directory", "name": "nfsgateway_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NFS Gateway parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NFS Gateway", "name": "role_config_suppression_nfsgateway_java_opts", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_nfsgateway_swap_memory_usage", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for NFS Gateway logs. Typically used by log4j or logback.", "display_name": "NFS Gateway Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory.", "display_name": "Temporary Dump Directory Free Space Monitoring Absolute Thresholds", "name": "nfsgateway_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The port number of the mount daemon implemented inside the NFS Gateway server role.", "display_name": "NFS Gateway MountD Port", "name": "nfs3_mountd_port", "value": "4242" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Temporary Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Temporary Dump Directory", "name": "role_config_suppression_dfs_nfs3_dump_dir", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_nfsgateway_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_nfsgateway_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Temporary Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Temporary Dump Directory Free Space", "name": "role_health_suppression_nfsgateway_dump_directory_free_space", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_nfsgateway_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Temporary Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Temporary Dump Directory Free Space Monitoring Percentage Thresholds", "name": "nfsgateway_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NFS Gateway", "name": "nfsgateway_java_opts", "value": "" }, { "desc": "The maximum size, in megabytes, per log file for NFS Gateway logs. Typically used by log4j or logback.", "display_name": "NFS Gateway Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "nfsgateway_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The NFS Gateway server port.", "display_name": "NFS Gateway Server Port", "name": "nfs3_server_port", "value": "2049" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NFS Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Hosts and Privileges parameter.", "display_name": "Suppress Parameter Validation: Allowed Hosts and Privileges", "name": "role_config_suppression_dfs_nfs_exports_allowed_hosts", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When computing the overall NFS Gateway health, consider the host's health.", "display_name": "NFS Gateway Host Health Test", "name": "nfsgateway_host_health_enabled", "value": "true" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "NFS clients often reorder writes. As a result, sequential writes can arrive at the NFS Gateway in random order. This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. Please make sure this directory has enough space. For example, if the application uploads 10 files with each having 100MB, it is recommended that this directory have roughly 1GB of space in case write reorder happens (in the worst case) to every file.", "display_name": "Temporary Dump Directory", "name": "dfs_nfs3_dump_dir", "value": "/tmp/.hdfs-nfs" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_nfsgateway_role_env_safety_valve", "value": "false" }, { "desc": "Enables the health test that the NFS Gateway's process state is consistent with the role configuration", "display_name": "NFS Gateway Process Health Test", "name": "nfsgateway_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hive-server2-sentry-safety.xml0000666000175100017510000000065513245514472027366 0ustar zuulzuul00000000000000 hive.security.authorization.task.factory org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl hive.server2.session.hook org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook hive.sentry.conf.url file:///{{CMF_CONF_DIR}}/sentry-site.xml sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hive-site.xml0000666000175100017510000000303213245514472024117 0ustar zuulzuul00000000000000 hive.metastore.local false hive.metastore.uris thrift://test-master-001.novalocal:9083 hive.metastore.client.socket.timeout 300 hive.metastore.warehouse.dir /user/hive/warehouse hive.warehouse.subdir.inherit.perms true mapred.reduce.tasks -1 hive.exec.reducers.bytes.per.reducer 1073741824 hive.exec.copyfile.maxsize 33554432 hive.exec.reducers.max 999 hive.metastore.execute.setugi true hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.MemoryTokenStore hive.server2.enable.doAs true fs.hdfs.impl.disable.cache true hive.server2.use.SSL false sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hive-gateway.json0000666000175100017510000001221013245514472024763 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hive_client_java_opts", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hive" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_client_config_safety_valve", "value": "false" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hive_client_java_heapsize", "value": "2147483648" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hive-env.sh", "display_name": "Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh", "name": "hive_client_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh parameter.", "display_name": "Suppress Parameter Validation: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh", "name": "role_config_suppression_hive_client_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for hive-site.xml.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hive_client_java_opts", "value": "-XX:MaxPermSize=512M -Djava.net.preferIPv4Stack=true" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Timeout for requests to the Hive Metastore Server. Consider increasing this if you have tables with a lot of metadata and see timeout errors. Used by most Hive Metastore clients such as Hive CLI and HiveServer2, but not by Impala. Impala has a separately configured timeout.", "display_name": "Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "300" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/ks_indexer-service.json0000666000175100017510000004671113245514472026177 0ustar zuulzuul00000000000000[ { "desc": "The password for the HBase Indexer TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HBase Indexer TLS/SSL Certificate Trust Store Password", "name": "keystore_indexer_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_keystore_indexer_sentry_safety_valve", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HBase Indexer might connect to. This is used when HBase Indexer is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HBase Indexer TLS/SSL Certificate Trust Store File", "name": "keystore_indexer_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "ks_indexer_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_keystore_indexer_provider_resource", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "keystore_indexer_sentry_safety_valve", "value": null }, { "desc": "Name of the HBase service that this Key-Value Store Indexer service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. Sentry is supported only on CDH 5.4 or later deployments and requires authentication to be turned on for HBase Indexer..", "display_name": "Enable Sentry Authorization using Policy Files", "name": "keystore_indexer_sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Custom Mime-types File parameter.", "display_name": "Suppress Parameter Validation: Custom Mime-types File", "name": "service_config_suppression_custom_mimetypes_file", "value": "false" }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "keystore_indexer_provider_resource", "value": "/user/hbaseindexer/sentry/sentry-provider.ini" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Sentry Validator", "name": "service_config_suppression_keystore_indexer_sentry_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Lily HBase Indexer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Lily HBase Indexer Count Validator", "name": "service_config_suppression_hbase_indexer_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Grok Dictionary File parameter.", "display_name": "Suppress Parameter Validation: Grok Dictionary File", "name": "service_config_suppression_grok_dictionary_conf_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_keystore_indexer_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_ks_indexer_env_safety_valve", "value": "false" }, { "desc": "Text that goes verbatim into grok-dictionary.conf file used by HBase Indexers.", "display_name": "Grok Dictionary File", "name": "grok_dictionary_conf_file", "value": "USERNAME [a-zA-Z0-9._-]+\nUSER %{USERNAME}\nINT (?:[+-]?(?:[0-9]+))\nBASE10NUM (?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))\nNUMBER (?:%{BASE10NUM})\nBASE16NUM (?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))\nUUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}\n\n# Networking\nMAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})\nCISCOMAC (?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})\nWINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})\nCOMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})\nIP (?/(?>[\\w_%!$@:.,-]+|\\\\.)*)+\n#UNIXPATH (?/dev/pts/%{NONNEGINT})\nBSDTTY (?>/dev/tty[pq][a-z0-9])\nTTY (?:%{BSDTTY}|%{LINUXTTY})\nWINPATH (?>[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+\nURIPROTO [A-Za-z]+(\\+[A-Za-z+]+)?\nURIHOST %{IPORHOST}(?::%{POSINT:port})?\n# uripath comes loosely from RFC1738, but mostly from what Firefox\n# doesn't turn into %XX\nURIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=#%_\\-]*)+\n#URIPARAM \\?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?\nURIPARAM \\?[A-Za-z0-9$.+!*'|(){},~#%&/=:;_?\\-\\[\\]]*\nURIPATHPARAM %{URIPATH}(?:%{URIPARAM})?\nURI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?\n\n# Months: January, Feb, 3, 03, 12, December\nMONTH \\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b\nMONTHNUM (?:0?[1-9]|1[0-2])\nMONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\n\n# Days: Monday, Tue, Thu, etc...\nDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)\n\n# Years?\nYEAR (?>\\d\\d){1,2}\n# Time: HH:MM:SS\n#TIME \\d{2}:\\d{2}(?::\\d{2}(?:\\.\\d+)?)?\n# I'm still on the fence about using grok to perform the time match,\n# since it's probably slower.\n# TIME %{POSINT<24}:%{POSINT<60}(?::%{POSINT<60}(?:\\.%{POSINT})?)?\nHOUR (?:2[0123]|[01]?[0-9])\nMINUTE (?:[0-5][0-9])\n# '60' is a leap second in most time standards and thus is valid.\nSECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\nTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])\n# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)\nDATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}\nDATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}\nISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))\nISO8601_SECOND (?:%{SECOND}|60)\nTIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?\nDATE %{DATE_US}|%{DATE_EU}\nDATESTAMP %{DATE}[- ]%{TIME}\nTZ (?:[PMCE][SD]T)\nDATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}\nDATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}\n\n# Syslog Dates: Month Day HH:MM:SS\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}\nPROG (?:[\\w._/%-]+)\nSYSLOGPROG %{PROG:program}(?:\\[%{POSINT:pid}\\])?\nSYSLOGHOST %{IPORHOST}\nSYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>\nHTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}\n\n# Shortcuts\nQS %{QUOTEDSTRING}\n\n# Log formats\nSYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:\nCOMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}\n\n# Log Levels\nLOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_keystore_indexer_truststore_password", "value": "false" }, { "desc": "The health test thresholds of the overall Lily HBase Indexer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Lily HBase Indexers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Lily HBase Indexers falls below the critical threshold.", "display_name": "Healthy Lily HBase Indexer Monitoring Thresholds", "name": "ks_indexer_indexers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access.", "display_name": "Sentry User to Group Mapping Class", "name": "keystore_indexer_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Text that goes verbatim into custom-mimetypes.xml file used by HBase Indexers.", "display_name": "Custom Mime-types File", "name": "custom_mimetypes_file", "value": "\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "desc": "Authentication mechanism used by HBase Indexer.", "display_name": "HBase Indexer Secure Authentication", "name": "hbase_indexer_security_authentication", "value": "simple" }, { "desc": "Text that goes into morphlines.conf file used by HBase Indexers. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service.", "display_name": "Morphlines File", "name": "morphlines_conf_file", "value": "SOLR_LOCATOR : {\n # Name of solr collection\n collection : collection\n \n # ZooKeeper ensemble\n zkHost : \"$ZK_HOST\" \n}\n\n\nmorphlines : [\n{\nid : morphline\nimportCommands : [\"org.kitesdk.**\", \"com.ngdata.**\"]\n\ncommands : [ \n {\n extractHBaseCells {\n mappings : [\n {\n inputColumn : \"data:*\"\n outputField : \"data\" \n type : string \n source : value\n }\n ]\n }\n }\n\n\n { logDebug { format : \"output record: {}\", args : [\"@{}\"] } }\n]\n}\n]\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Morphlines File parameter.", "display_name": "Suppress Parameter Validation: Morphlines File", "name": "service_config_suppression_morphlines_conf_file", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Name of the Solr service that this Key-Value Store Indexer service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the Lily HBase Indexer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Lily HBase Indexer Health", "name": "service_health_suppression_ks_indexer_hbase_indexers_healthy", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/zookeeper-service.json0000666000175100017510000003121713245514472026042 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Enable Kerberos authentication for ZooKeeper.", "display_name": "Enable Kerberos Authentication", "name": "enableSecurity", "value": "false" }, { "desc": "Amount of time, in ticks, to allow followers to connect and sync to a leader. Increase this value as needed, if the amount of data managed by ZooKeeper is large.", "display_name": "Initialization Limit", "name": "initLimit", "value": "10" }, { "desc": "The health test thresholds of the overall Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Servers falls below the critical threshold.", "display_name": "Healthy Server Monitoring Thresholds", "name": "zookeeper_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The number of snapshot files and corresponding transaction logs to keep when running the Cleanup command.", "display_name": "Cleanup Retain Count", "name": "cleanupRetainCount", "value": "5" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "zookeeper" }, { "desc": "The length of time, in milliseconds, of a single tick, which is the basic time unit used by ZooKeeper. A tick is used to regulate heartbeats and timeouts.", "display_name": "Tick Time", "name": "tickTime", "value": "2000" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "zookeeper_env_safety_valve", "value": null }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Whether the leader accepts client connections.", "display_name": "Leader Serves", "name": "leaderServes", "value": "yes" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "zookeeper" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Canary Root Znode Path parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Canary Root Znode Path", "name": "service_config_suppression_zookeeper_canary_root_path", "value": "false" }, { "desc": "The time interval in hours for which the purge task has to be triggered. Set to a positive integer (1 and above) to enable the auto purging. Defaults to 24.", "display_name": "Auto Purge Time Interval", "name": "autopurgeInterval", "value": "24" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Configures the path of the root znode under which all canary updates are performed", "display_name": "ZooKeeper Canary Root Znode Path", "name": "zookeeper_canary_root_path", "value": "/cloudera_manager_zookeeper_canary" }, { "desc": "When enabled, ZooKeeper auto purge feature retains this many most recent snapshots and the corresponding transaction logs in the dataDir and dataLogDir respectively and deletes the rest. Defaults to 5. Minimum value is 3.", "display_name": "Auto Purge Snapshots Retain Count", "name": "autopurgeSnapRetainCount", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the ZooKeeper Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: ZooKeeper Server Count Validator", "name": "service_config_suppression_zookeeper_odd_number_of_servers_validator", "value": "false" }, { "desc": "The maximum size of the data that can be stored in a znode in bytes.", "display_name": "Jute Max Buffer", "name": "jute_maxbuffer", "value": "4194304" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Configures the timeout used by the canary for ZooKeeper operations", "display_name": "ZooKeeper Canary Operation Timeout", "name": "zookeeper_canary_operation_timeout", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Server Count Validator", "name": "service_config_suppression_server_count_validator", "value": "false" }, { "desc": "Automatically create data directories at startup, if they do not exist. Enabling this configuration should be used with care as it will suppress any errors in setup of data directories.", "display_name": "Enable auto-creation of data directories", "name": "zookeeper_datadir_autocreate", "value": "false" }, { "desc": "Whether to suppress the results of the ZooKeeper Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ZooKeeper Canary", "name": "service_health_suppression_zookeeper_canary_health", "value": "false" }, { "desc": "Configures the timeout used by the canary sessions with ZooKeeper servers", "display_name": "ZooKeeper Canary Session Timeout", "name": "zookeeper_canary_session_timeout", "value": "30000" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "zookeeper" }, { "desc": "Whether to suppress the results of the Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Server Health", "name": "service_health_suppression_zookeeper_servers_healthy", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Configures the timeout used by the canary for connection establishment with ZooKeeper servers", "display_name": "ZooKeeper Canary Connection Timeout", "name": "zookeeper_canary_connection_timeout", "value": "10000" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_zookeeper_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Amount of time, in ticks, to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they are dropped.", "display_name": "Synchronization Limit", "name": "syncLimit", "value": "5" }, { "desc": "Enables the health check that a client can connect to ZooKeeper and perform basic operations", "display_name": "ZooKeeper Canary Health Check", "name": "zookeeper_canary_health_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/priority-one-confs.json0000666000175100017510000000320513245514472026143 0ustar zuulzuul00000000000000[ "dfs_block_size", "dfs_umaskmode", "dfs_webhdfs_enabled", "dfs_permissions", "dfs_replication", "io_compression_codecs", "io_sort_mb", "dfs_datanode_du_reserved", "dfs_datanode_failed_volumes_tolerated", "dfs_name_dir_restore", "fs_trash_interval", "dfs_safemode_min_datanodes", "dfs_safemode_extension", "dfs_access_time_precision", "yarn_acl_enable", "yarn_admin_acl", "yarn_log_aggregation_enable", "yarn_log_aggregation_retain_seconds", "mapreduce_jobhistory_max_age_ms", "mapreduce_jobhistory_cleaner_interval", "mapreduce_reduce_memory_mb", "mapreduce_reduce_java_opts", "mapreduce_map_memory_mb", "mapreduce_map_java_opts", "yarn_nodemanager_container_manager_thread_count", "yarn_nodemanager_delete_thread_count", "yarn_nodemanager_heartbeat_interval_ms", "yarn_nodemanager_localizer_cache_cleanup_interval_ms", "yarn_nodemanager_localizer_client_thread_count", "yarn_nodemanager_localizer_cache_target_size_mb", "yarn_nodemanager_localizer_fetch_thread_count", "yarn_nodemanager_log_retain_seconds", "yarn_nodemanager_resource_memory_mb", "yarn_resourcemanager_client_thread_count", "yarn_resourcemanager_scheduler_client_thread_count", "yarn_resourcemanager_admin_client_thread_count", "yarn_resourcemanager_amliveliness_monitor_interval_ms", "yarn_am_liveness_monitor_expiry_interval_ms", "yarn_resourcemanager_am_max_retries", "yarn_scheduler_minimum_allocation_mb", "yarn_scheduler_maximum_allocation_mb", "yarn_app_mapreduce_am_command_opts", "yarn_app_mapreduce_am_resource_mb" ] sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hive-service.json0000666000175100017510000014232413245514472024774 0ustar zuulzuul00000000000000[ { "desc": "The directory in which Hive on Spark lineage log files are written.", "display_name": "Hive on Spark Lineage Log Directory", "name": "lineage_event_log_dir", "value": "/var/log/hive/lineage" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Replication Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hive_replication_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Auxiliary JARs Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Auxiliary JARs Directory", "name": "service_config_suppression_hive_aux_jars_path_dir", "value": "false" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hive/audit" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Bypass Sentry Authorization Users parameter.", "display_name": "Suppress Parameter Validation: Bypass Sentry Authorization Users", "name": "service_config_suppression_sentry_metastore_service_users", "value": "false" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "hiveserver2_ldap_uri", "value": null }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hive" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_hive_sentry_provider_resource", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Access Control and Proxy User Groups Override parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Access Control and Proxy User Groups Override", "name": "service_config_suppression_hive_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "service_config_suppression_hive_service_replication_config_safety_valve", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Hive on Spark lineage logs. Typically used by log4j or logback.", "display_name": "Hive on Spark Maximum Lineage Log File Size", "name": "max_lineage_log_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Derby Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Derby Validator", "name": "service_config_suppression_hive_derby_validator", "value": "false" }, { "desc": "Name of Hive Metastore database", "display_name": "Hive Metastore Database Name", "name": "hive_metastore_database_name", "value": "metastore" }, { "desc": "This configuration overrides the value set for Hive Proxy User Groups configuration in HDFS service for use by Hive Metastore Server. Specify a comma-delimited list of groups that you want to allow access to Hive Metastore metadata and allow the Hive user to impersonate. A value of '*' allows all groups. The default value of empty inherits the value set for Hive Proxy User Groups configuration in the HDFS service.", "display_name": "Hive Metastore Access Control and Proxy User Groups Override", "name": "hive_proxy_user_groups_list", "value": null }, { "desc": "Perform DataNucleus validation of metadata during startup. Note: when enabled, Hive will log DataNucleus warnings even though Hive will function normally.", "display_name": "Hive Metastore Database DataNucleus Metadata Validation", "name": "hive_metastore_database_datanucleus_metadata_validation", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_hive_core_site_safety_valve", "value": "false" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Hive events:\n

\n\n
    \n
  • userName: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • operation: the Hive operation being performed.
  • \n
  • databaseName: the databaseName for the operation.
  • \n
  • tableName: the tableName for the operation.
  • \n
\n\n

\nThe default Hive audit event filter discards HDFS directory events generated by\nHive jobs that reference the /tmp directory.\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default Hive audit event filter discards HDFS directory events \",\n \"generated by Hive jobs that reference the /tmp directory.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"operation\", \"match\" : \"QUERY\" },\n { \"name\" : \"objectType\", \"match\" : \"DFS_DIR\"},\n { \"name\" : \"resourcePath\", \"match\" : \"/tmp/hive-(?:.+)?/hive_(?:.+)?/-mr-.*\" }\n ]\n }\n ]\n}\n" }, { "desc": "The health test thresholds of the overall HiveServer2 health. The check returns \"Concerning\" health if the percentage of \"Healthy\" HiveServer2s falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" HiveServer2s falls below the critical threshold.", "display_name": "Healthy HiveServer2 Monitoring Thresholds", "name": "hive_hiveserver2s_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Warehouse Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Warehouse Directory", "name": "service_config_suppression_hive_warehouse_directory", "value": "false" }, { "desc": "Size per reducer. If the input size is 10GiB and this is set to 1GiB, Hive will use 10 reducers.", "display_name": "Hive Bytes Per Reducer", "name": "hive_bytes_per_reducer", "value": "67108864" }, { "desc": "Let the table directories inherit the permission of the Warehouse or Database directory instead of being created with the permissions derived from dfs umask. This allows Impala to insert into tables created via Hive.", "display_name": "Hive Warehouse Subdirectories Inherit Permissions", "name": "hive_warehouse_subdir_inherit_perms", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml. Applies to all Hive Replication jobs.", "display_name": "Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_service_replication_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hive_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Bypass Metastore Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Bypass Metastore Validator", "name": "service_config_suppression_hive_bypass_metastore_validator", "value": "false" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "hiveserver2_enable_ldap_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Sentry Validator", "name": "service_config_suppression_hive_sentry_validator", "value": "false" }, { "desc": "This parameter is useful when authenticating against a non Active Directory server such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server will look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "hiveserver2_ldap_basedn", "value": null }, { "desc": "Name of the ZooKeeper service that this Hive service instance depends on.", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "List of users that are allowed to bypass Sentry Authorization in the Hive metastore. These are usually service users that already ensure that all activity has been authorized, such as hive and impala. Only applies when Hive is using Sentry Service.", "display_name": "Bypass Sentry Authorization Users", "name": "sentry_metastore_service_users", "value": "hive,impala,hue,hdfs" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of Hive replication jobs.", "display_name": "Hive Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_replication_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Name for Sentry Authorization parameter.", "display_name": "Suppress Parameter Validation: Server Name for Sentry Authorization", "name": "service_config_suppression_hive_sentry_server", "value": "false" }, { "desc": "Default number of reduce tasks per job. Usually set to a prime number close to the number of available hosts. Ignored when mapred.job.tracker is \"local\". Hadoop sets this to 1 by default, while Hive uses -1 as the default. When set to -1, Hive will automatically determine an appropriate number of reducers for each job.", "display_name": "Hive Reduce Tasks", "name": "hive_reduce_tasks", "value": "-1" }, { "desc": "Cloudera does not support Hive on Spark in CDH 5.4, 5.5, or 5.6. Enable Hive to use Spark for execution even though it is not supported. For evaluation purposes only. This configuration only takes effect when Hive is configured with a Spark on YARN Service. See Configuring Hive on Spark for more information about using Hive on Spark.", "display_name": "Enable Hive on Spark (Unsupported)", "name": "enable_hive_on_spark", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_hiveserver2_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Instead of talking to Hive Metastore Server for Metastore information, Hive clients will talk directly to the Metastore database.", "display_name": "Bypass Hive Metastore Server", "name": "hive_bypass_metastore_server", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Password parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Password", "name": "service_config_suppression_hive_metastore_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "service_config_suppression_hive_service_config_safety_valve", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userNname, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Hive events:\n

\n\n
    \n
  • operation: the Hive operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • databaseName: the database affected by the operation.
  • \n
  • tableName: the table or view affected by the operation.
  • \n
  • objectType: the type of object affected by the operation.
  • \n
  • resourcePath: the path of the resource affected by the operation.
  • \n
\n\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database User parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database User", "name": "service_config_suppression_hive_metastore_database_user", "value": "false" }, { "desc": "Automatically create or upgrade tables in the Hive Metastore database when needed. Consider setting this to false and managing the schema manually.", "display_name": "Auto Create and Upgrade Hive Metastore Database Schema", "name": "hive_metastore_database_auto_create_schema", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Derby Path parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Derby Path", "name": "service_config_suppression_hive_metastore_derby_path", "value": "false" }, { "desc": "Allows URIs when defining privileges in per-database policy files. Warning: Typically, this configuration should be disabled. Enabling it would allow database policy file owner (which is generally not Hive admin user) to grant load privileges to any directory with read access to Hive admin user, including databases controlled by other database policy files.", "display_name": "Allow URIs in Database Policy File", "name": "sentry_allow_uri_db_policyfile", "value": "false" }, { "desc": "The server name used when defining privilege rules in Sentry authorization. Sentry uses this name as an alias for the Hive service. It does not correspond to any physical server name.", "display_name": "Server Name for Sentry Authorization", "name": "hive_sentry_server", "value": "server1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_hiveserver2_keystore_password", "value": "false" }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "Hive warehouse directory is the location in HDFS where Hive's tables are stored. Note that Hive's default value for its warehouse directory is '/user/hive/warehouse'.", "display_name": "Hive Warehouse Directory", "name": "hive_warehouse_directory", "value": "/user/hive/warehouse" }, { "desc": "User for Hive Metastore database", "display_name": "Hive Metastore Database User", "name": "hive_metastore_database_user", "value": "hive" }, { "desc": "The health test thresholds of the overall Hive Metastore Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Hive Metastore Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Hive Metastore Servers falls below the critical threshold.", "display_name": "Healthy Hive Metastore Server Monitoring Thresholds", "name": "hive_hivemetastores_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress the results of the WebHCat Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: WebHCat Server Health", "name": "service_health_suppression_hive_webhcats_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Host parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Host", "name": "service_config_suppression_hive_metastore_database_host", "value": "false" }, { "desc": "Host name of Hive Metastore database", "display_name": "Hive Metastore Database Host", "name": "hive_metastore_database_host", "value": "localhost" }, { "desc": "Whether to suppress the results of the Hive Metastore Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Metastore Server Health", "name": "service_health_suppression_hive_hivemetastores_healthy", "value": "false" }, { "desc": "Whether Hive Metastore should try to use direct SQL queries instead of DataNucleus for certain read paths. This can improve metastore performance when fetching many partitions by orders of magnitude. In case of failure, execution will fall back to DataNucleus. This configuration is not supported and is disabled when Hive service is configured with Postgres.", "display_name": "Enable Direct SQL", "name": "hive_metastore_database_datanucleus_try_direct_sql", "value": "false" }, { "desc": "In unsecure mode, setting this property to true will cause the Metastore Server to execute DFS operations using the client's reported user and group permissions. Cloudera Manager will set this for all clients and servers.", "display_name": "Set User and Group Information", "name": "hive_set_ugi", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Client TLS/SSL In Use With LDAP Authentication Validator configuration validator.", "display_name": "Suppress Configuration Validator: Client TLS/SSL In Use With LDAP Authentication Validator", "name": "service_config_suppression_hive_client_ssl_recommended_with_ldap_auth_validator", "value": "false" }, { "desc": "Prevent Metastore operations in the event of schema version incompatibility. Consider setting this to true to reduce probability of schema corruption during Metastore operations. Note that setting this property to true will also set datanucleus.autoCreateSchema property to false and datanucleus.fixedDatastore property to true. Any values set in Cloudera Manager for these properties will be overridden.", "display_name": "Strict Hive Metastore Schema Validation", "name": "hive_metastore_schema_verification", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Type of Hive Metastore database. Note that Derby is not recommended and Cloudera Impala does not support Derby.", "display_name": "Hive Metastore Database Type", "name": "hive_metastore_database_type", "value": "mysql" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the HiveServer2 Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HiveServer2 Count Validator", "name": "service_config_suppression_hiveserver2_count_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "hive_server2_sentry_safety_valve", "value": null }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop user to group mapping or local groups defined in the policy file. Hadoop user to group mapping may be configured in the Cloudera Manager HDFS service configuration page under the Security section.", "display_name": "Sentry User to Group Mapping Class", "name": "hive_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Whether to suppress the results of the HiveServer2 Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HiveServer2 Health", "name": "service_health_suppression_hive_hiveserver2s_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Concurrency Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Concurrency Configuration Validator", "name": "service_config_suppression_hive_concurrency_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Metastore Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Metastore Server Count Validator", "name": "service_config_suppression_hivemetastore_count_validator", "value": "false" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "hive.metastore.client.socket.timeout60" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.lineage.client.properties.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "navigator_lineage_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_hiveserver2_keystore_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the WebHCat Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: WebHCat Server Count Validator", "name": "service_config_suppression_webhcat_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_hiveserver2_ldap_basedn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_hiveserver2_ldap_domain", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "hive_core_site_safety_valve", "value": null }, { "desc": "The password for the HiveServer2 JKS keystore file.", "display_name": "HiveServer2 TLS/SSL Server JKS Keystore File Password", "name": "hiveserver2_keystore_password", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Max number of reducers to use. If the configuration parameter Hive Reduce Tasks is negative, Hive will limit the number of reducers to the value of this parameter.", "display_name": "Hive Max Reducers", "name": "hive_max_reducers", "value": "1099" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_hiveserver2_ldap_uri", "value": "false" }, { "desc": "Port number of Hive Metastore database", "display_name": "Hive Metastore Database Port", "name": "hive_metastore_database_port", "value": "3306" }, { "desc": "MapReduce jobs are run against this service.", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null }, { "desc": "Enable collection of lineage from the service's roles.", "display_name": "Enable Lineage Collection", "name": "navigator_lineage_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive on Spark Lineage Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hive on Spark Lineage Log Directory", "name": "service_config_suppression_lineage_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Proxy Groups Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Proxy Groups Validator", "name": "service_config_suppression_hive_proxy_groups_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_hiveserver2_truststore_password", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HiveServer2 is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HiveServer2 TLS/SSL Server JKS Keystore File Location", "name": "hiveserver2_keystore_path", "value": null }, { "desc": "Encrypt communication between clients and HiveServer2 using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HiveServer2", "name": "hiveserver2_enable_ssl", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using the Sentry service instead, add the Sentry service as a dependency to the Hive service. The Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry is supported only on CDH 4.4 or later deployments. Before enabling Sentry, read the requirements and configuration steps in Setting Up Hive Authorization with Sentry.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "sentry_enabled", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_service_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HiveServer2 might connect to. This is used when HiveServer2 is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HiveServer2 TLS/SSL Certificate Trust Store File", "name": "hiveserver2_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_hive_server2_sentry_safety_valve", "value": "false" }, { "desc": "Directory containing auxiliary JARs used by Hive. This should be a directory location and not a classpath containing one or more JARs. This directory must be created and managed manually on hosts that run the Hive Metastore Server, HiveServer2, or the Hive CLI. The directory location is set in the environment as HIVE_AUX_JARS_PATH and will generally override the hive.aux.jars.path property set in XML files, even if hive.aux.jars.path is set in an advanced configuration snippet.", "display_name": "Hive Auxiliary JARs Directory", "name": "hive_aux_jars_path_dir", "value": null }, { "desc": "Name of the Sentry service that this Hive service instance depends on. If selected, Hive uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Password for Hive Metastore database", "display_name": "Hive Metastore Database Password", "name": "hive_metastore_database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "service_config_suppression_navigator_lineage_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Name parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Name", "name": "service_config_suppression_hive_metastore_database_name", "value": "false" }, { "desc": "Name of the HBase service that this Hive service instance depends on.", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Directory name where Hive Metastore's database is stored (only for Derby)", "display_name": "Hive Metastore Derby Path", "name": "hive_metastore_derby_path", "value": "/var/lib/hive/cloudera_manager/derby/metastore_db" }, { "desc": "The password for the HiveServer2 TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HiveServer2 TLS/SSL Certificate Trust Store Password", "name": "hiveserver2_truststore_password", "value": null }, { "desc": "Name of the Spark on YARN service that this Hive service instance depends on. If selected, Hive jobs can use the Spark execution engine instead of MapReduce2. Requires that Hive depends on YARN. See Configuring Hive on Spark for more information about Hive on Spark. In CDH lower than 5.7, Hive on Spark also requires setting Enable Hive on Spark to true.", "display_name": "Spark On YARN Service", "name": "spark_on_yarn_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "hive_sentry_provider_resource", "value": "/user/hive/sentry/sentry-provider.ini" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hive" }, { "desc": "The health test thresholds of the overall WebHCat Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" WebHCat Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" WebHCat Servers falls below the critical threshold.", "display_name": "Healthy WebHCat Server Monitoring Thresholds", "name": "hive_webhcats_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hive" }, { "desc": "Smaller than this size, Hive uses a single-threaded copy; larger than this size, Hive uses DistCp.", "display_name": "Hive Copy Large File Size", "name": "hive_exec_copyfile_maxsize", "value": "33554432" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "hiveserver2_ldap_domain", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hbase-master.json0000666000175100017510000010330513245514472024752 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master Web UI Address parameter.", "display_name": "Suppress Parameter Validation: HBase Master Web UI Address", "name": "role_config_suppression_hbase_master_info_bindaddress", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_master_log_directory_free_space", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "master_web_metric_collection_enabled", "value": "true" }, { "desc": "The name of the DNS network interface from which an HBase Master should report its IP address.", "display_name": "HBase Master DNS Network Interface", "name": "hbase_master_dns_interface", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Log Directory parameter.", "display_name": "Suppress Parameter Validation: Master Log Directory", "name": "role_config_suppression_hbase_master_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_master_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase Master", "name": "hbase_master_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase Master DNS Name Server", "name": "role_config_suppression_hbase_master_dns_nameserver", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum time an HLog remains in the .oldlogdir directory until an HBase Master thread deletes it.", "display_name": "Maximum Time to Keep HLogs", "name": "hbase_master_logcleaner_ttl", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_master_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "master_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Coprocessor Master Classes parameter.", "display_name": "Suppress Parameter Validation: HBase Coprocessor Master Classes", "name": "role_config_suppression_hbase_coprocessor_master_classes", "value": "false" }, { "desc": "The minimum log level for Master logs", "display_name": "Master Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_master_scm_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of pooled threads to handle the recovery of the region servers in the master.", "display_name": "RegionServer Recovery Threads", "name": "hbase_master_executor_serverops_threads", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The address for the HBase Master web UI", "display_name": "HBase Master Web UI Address", "name": "hbase_master_info_bindAddress", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Master Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_master_role_env_safety_valve", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Master logs. Typically used by log4j or logback.", "display_name": "Master Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "master_startup_tolerance", "value": "5" }, { "desc": "Number of pooled threads to handle region opening in the master.", "display_name": "Region Opening Threads", "name": "hbase_master_executor_openregion_threads", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_master_web_metric_collection", "value": "false" }, { "desc": "List of org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are loaded by default on the active HMaster process. For any implemented coprocessor methods, the listed classes will be called in order. After implementing your own MasterObserver, just put it in HBase's classpath and add the fully qualified class name here.", "display_name": "HBase Coprocessor Master Classes", "name": "hbase_coprocessor_master_classes", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_master_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The host name or IP address of the DNS name server which an HBase Master should use to determine the host name used for communication and display purposes.", "display_name": "HBase Master DNS Name Server", "name": "hbase_master_dns_nameserver", "value": null }, { "desc": "When true, HBase Master will bind to 0.0.0.0. Only available with CDH 4.3 and later.", "display_name": "HBase Master Bind to Wildcard Address", "name": "hbase_master_bind_to_wildcard_address", "value": "true" }, { "desc": "Time period in seconds to reset long-running metrics (e.g. compactions). This is an HBase specific configuration.", "display_name": "Extended Period", "name": "hbase_metrics_extended_period", "value": "3600" }, { "desc": "Number of pooled threads to handle region closing in the master.", "display_name": "Region Closing Threads", "name": "hbase_master_executor_closeregion_threads", "value": "5" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase Master DNS Network Interface", "name": "role_config_suppression_hbase_master_dns_interface", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master Log Cleaner Plugins parameter.", "display_name": "Suppress Parameter Validation: HBase Master Log Cleaner Plugins", "name": "role_config_suppression_hbase_master_logcleaner_plugins", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_master_config_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Enables the health test that the Master's process state is consistent with the role configuration", "display_name": "Master Process Health Test", "name": "master_scm_health_enabled", "value": "true" }, { "desc": "A comma-separated list of LogCleanerDelegate(s) that are used in LogsCleaner. WAL/HLog cleaner(s) are called in order, so put the log cleaner that prunes the most log files in the front. To implement your own LogCleanerDelegate, add it to HBase's classpath and add the fully-qualified class name here. You should always add the above default log cleaners in the list, unless you have a special reason not to.", "display_name": "HBase Master Log Cleaner Plugins", "name": "hbase_master_logcleaner_plugins", "value": null }, { "desc": "Directory where Master will place its log files.", "display_name": "Master Log Directory", "name": "hbase_master_log_dir", "value": "/var/log/hbase" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "master_gc_duration_window", "value": "5" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"IPC Server handler.*ClosedChannelException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"IPC Server Responder, call.*output error\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Daughter regiondir does not exist: .*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"File.*might still be open.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"File.*might still be open.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Moving table .+ state to enabled but was already enabled\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Received OPENED for region.*but region was in the state.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "When computing the overall Master health, consider the host's health.", "display_name": "Master Host Health Test", "name": "master_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Master Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_master_swap_memory_usage", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the HBase Master Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HBase Master Canary", "name": "role_health_suppression_master_canary_health", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_master_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "master_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The maximum size, in megabytes, per log file for Master logs. Typically used by log4j or logback.", "display_name": "Master Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "master_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Master Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The port for the HBase Master web UI. Set to -1 to disable the HBase Master web UI.", "display_name": "HBase Master Web UI Port", "name": "hbase_master_info_port", "value": "60010" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase Master parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase Master", "name": "role_config_suppression_hbase_master_java_opts", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase Master in Bytes", "name": "hbase_master_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_master_file_descriptor", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Number of RPC Server instances spun up on HBase Master.", "display_name": "HBase Master Handler Count", "name": "hbase_master_handler_count", "value": "25" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The port that the HBase Master binds to.", "display_name": "HBase Master Port", "name": "hbase_master_port", "value": "60000" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_master_gc_duration", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Master Environment Advanced Configuration Snippet (Safety Valve)", "name": "MASTER_role_env_safety_valve", "value": null }, { "desc": "Enables the health test that a client can connect to the HBase Master", "display_name": "HBase Master Canary Health Test", "name": "master_canary_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/kms-service.json0000666000175100017510000001370013245514472024626 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "KMS_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Key Management Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Key Management Server Count Validator", "name": "service_config_suppression_kms_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_kms_service_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "kms" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "kms" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Authentication type for the KMS. Can either be \"simple\" or \"kerberos\".", "display_name": "Authentication Type", "name": "hadoop_kms_authentication_type", "value": "simple" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/ks_indexer-hbase_indexer.json0000666000175100017510000005334013245514472027333 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only. A string to be inserted into hbase-indexer-site.xml for this role only.", "display_name": "Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml", "name": "hbase_indexer_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_indexer_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_indexer_file_descriptor", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Lily HBase Indexer in Bytes", "name": "hbase_indexer_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_indexer_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_hbase_indexer_process_username", "value": "false" }, { "desc": "Enables the health test that the Lily HBase Indexer's process state is consistent with the role configuration", "display_name": "Lily HBase Indexer Process Health Test", "name": "hbase_indexer_scm_health_enabled", "value": "true" }, { "desc": "The minimum log level for Lily HBase Indexer logs", "display_name": "Lily HBase Indexer Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbase_indexer_role_env_safety_valve", "value": "false" }, { "desc": "Kerberos principal used by the Lily HBase Indexer roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "hbase" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Lily HBase Indexer", "name": "hbase_indexer_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Lily HBase Indexer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Lily HBase Indexer", "name": "role_config_suppression_hbase_indexer_java_opts", "value": "false" }, { "desc": "HTTP port used by HBase Indexer.", "display_name": "HBase Indexer HTTP Port", "name": "hbase_indexer_http_port", "value": "11060" }, { "desc": "The maximum number of rolled log files to keep for Lily HBase Indexer logs. Typically used by log4j or logback.", "display_name": "Lily HBase Indexer Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When computing the overall Lily HBase Indexer health, consider the host's health.", "display_name": "Lily HBase Indexer Host Health Test", "name": "hbase_indexer_host_health_enabled", "value": "true" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_indexer_scm_health", "value": "false" }, { "desc": "Directory where HBase Indexer will place its log files.", "display_name": "HBase Indexer Log Directory", "name": "hbase_indexer_log_dir", "value": "/var/log/hbase-solr" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbase_indexer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASE_INDEXER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_indexer_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_indexer_unexpected_exits", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The user that the HBase Indexer process should run as.", "display_name": "System User", "name": "hbase_indexer_process_username", "value": "hbase" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The maximum size, in megabytes, per log file for Lily HBase Indexer logs. Typically used by log4j or logback.", "display_name": "Lily HBase Indexer Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_hbase_indexer_process_groupname", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer Log Directory", "name": "role_config_suppression_hbase_indexer_log_dir", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml", "name": "role_config_suppression_hbase_indexer_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The group that the HBase Indexer process should run as.", "display_name": "System Group", "name": "hbase_indexer_process_groupname", "value": "hbase" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_indexer_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hbase-hbaserestserver.json0000666000175100017510000007126613245514472026700 0ustar zuulzuul00000000000000[ { "desc": "Enables the health test that the HBase REST Server's process state is consistent with the role configuration", "display_name": "HBase REST Server Process Health Test", "name": "hbaserestserver_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server DNS Network Interface", "name": "role_config_suppression_hbase_restserver_dns_interface", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase REST Server is acting as a TLS/SSL server.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore Key Password", "name": "hbase_restserver_keystore_keypassword", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Host Address parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Host Address", "name": "role_config_suppression_hbase_restserver_host", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_rest_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_restserver_config_safety_valve", "value": "false" }, { "desc": "HBase REST Server will bind to this address.", "display_name": "HBase REST Server Host Address", "name": "hbase_restserver_host", "value": "0.0.0.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The port that HBase REST Server binds to.", "display_name": "HBase REST Server Port", "name": "hbase_restserver_port", "value": "20550" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server DNS Name Server", "name": "role_config_suppression_hbase_restserver_dns_nameserver", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HBase REST Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASERESTSERVER_role_env_safety_valve", "value": null }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_hbase_restserver_keystore_keypassword", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbaserestserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The minimum log level for HBase REST Server logs", "display_name": "HBase REST Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase REST Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore File Location", "name": "hbase_restserver_keystore_file", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The host name or IP address of the DNS name server which an HBase REST Server should use to determine the host name used for communication and display purposes.", "display_name": "HBase REST Server DNS Name Server", "name": "hbase_restserver_dns_nameserver", "value": null }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_restserver_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_rest_server_unexpected_exits", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase REST Server in Bytes", "name": "hbase_restserver_java_heapsize", "value": "1073741824" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HBase REST Server logs. Typically used by log4j or logback.", "display_name": "HBase REST Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Maximum size of the HBase REST Server thread pool. The server can process this number of concurrent requests. Setting this too high can lead to out of memory errors.", "display_name": "HBase REST Server Maximum Threads", "name": "hbase_restserver_threads_max", "value": "100" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_rest_server_file_descriptor", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The port that HBase REST Server Web UI binds to.", "display_name": "HBase REST Server Web UI Port", "name": "hbase_restserver_info_port", "value": "8085" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_rest_server_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase REST Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase REST Server", "name": "role_config_suppression_hbase_restserver_java_opts", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_hbase_restserver_keystore_password", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_rest_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "If true, HBase REST Server Web UI will bind to a wildcard address (0.0.0.0). Otherwise it will bind to a host name. Only available in CDH 4.3 and later.", "display_name": "HBase REST Server Web UI Bind to Wildcard Address", "name": "hbase_restserver_info_bind_to_wildcard", "value": "true" }, { "desc": "The password for the HBase REST Server JKS keystore file.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore File Password", "name": "hbase_restserver_keystore_password", "value": null }, { "desc": "Directory where HBase REST Server will place its log files.", "display_name": "HBase REST Server Log Directory", "name": "hbase_restserver_log_dir", "value": "/var/log/hbase" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Minimum size of the HBase REST Server thread pool. The server will maintain at least this number of threads in the pool at all times. The thread pool can grow up to the maximum size set by hbase.rest.threads.max.", "display_name": "HBase REST Server Minimum Threads", "name": "hbase_restserver_threads_min", "value": "2" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbaserestserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "When false, all HTTP methods are permitted (GET/PUT/POST/DELETE). When true, only GET is permitted.", "display_name": "Enable HBase REST Server Read Only Mode", "name": "hbase_restserver_readonly", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_rest_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Log Directory", "name": "role_config_suppression_hbase_restserver_log_dir", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The maximum size, in megabytes, per log file for HBase REST Server logs. Typically used by log4j or logback.", "display_name": "HBase REST Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HBase REST Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The name of the DNS network interface from which an HBase REST Server should report its IP address.", "display_name": "HBase REST Server DNS Network Interface", "name": "hbase_restserver_dns_interface", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Encrypt communication between clients and HBase REST Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HBase REST Server", "name": "hbase_restserver_ssl_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_hbase_restserver_keystore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When computing the overall HBase REST Server health, consider the host's health.", "display_name": "HBase REST Server Host Health Test", "name": "hbaserestserver_host_health_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase REST Server", "name": "hbase_restserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_rest_server_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/yarn-resourcemanager.json0000666000175100017510000012662213245514472026537 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_resource_manager_gc_duration", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The number of threads used to handle requests through the scheduler interface.", "display_name": "Scheduler Thread Count", "name": "yarn_resourcemanager_scheduler_client_thread_count", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_resourcemanager_mapred_safety_valve", "value": "false" }, { "desc": "When enabled, if a pool's minimum share is not met for some period of time, the Fair Scheduler preempts applications in other pools. Preemption guarantees that production applications are not starved while also allowing the cluster to be used for experimental and research applications. To minimize wasted computation, the Fair Scheduler preempts the most recently launched applications.", "display_name": "Enable Fair Scheduler Preemption", "name": "resourcemanager_fair_scheduler_preemption", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_resource_manager_scm_health", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_capacity_scheduler_configuration", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The periodic interval that the ResourceManager will check whether containers are still alive.", "display_name": "Container Monitor Interval", "name": "yarn_resourcemanager_container_liveness_monitor_interval_ms", "value": "600000" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "When set to true, the Fair Scheduler uses the username as the default pool name, in the event that a pool name is not specified. When set to false, all applications are run in a shared pool, called default.", "display_name": "Fair Scheduler User As Default Queue", "name": "resourcemanager_fair_scheduler_user_as_default_queue", "value": "true" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for ResourceManager", "name": "resource_manager_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "resourcemanager_startup_tolerance_minutes", "value": "5" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "resourcemanager_mapred_safety_valve", "value": null }, { "desc": "The maximum size, in megabytes, per log file for ResourceManager logs. Typically used by log4j or logback.", "display_name": "ResourceManager Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Maximum Attempts parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Maximum Attempts", "name": "role_config_suppression_yarn_resourcemanager_am_max_retries", "value": "false" }, { "desc": "The utilization threshold after which preemption kicks in. The utilization is computed as the maximum ratio of usage to capacity among all resources.", "display_name": "Fair Scheduler Preemption Utilization Threshold", "name": "yarn_scheduler_fair_preemption_cluster_utilization_threshold", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_fair_scheduler_configuration", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "resourcemanager_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "ResourceManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "RESOURCEMANAGER_role_env_safety_valve", "value": null }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_resource_manager_unexpected_exits", "value": "false" }, { "desc": "Enable continuous scheduling in the Fair Scheduler. When enabled, scheduling decisions are decoupled from NodeManager heartbeats, leading to faster resource allocations.", "display_name": "Enable Fair Scheduler Continuous Scheduling", "name": "yarn_scheduler_fair_continuous_scheduling_enabled", "value": "true" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "resourcemanager_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The address of the admin interface in the ResourceManager.", "display_name": "Administration Address", "name": "yarn_resourcemanager_admin_address", "value": "8033" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When computing the overall ResourceManager health, consider the host's health.", "display_name": "ResourceManager Host Health Test", "name": "resourcemanager_host_health_enabled", "value": "true" }, { "desc": "Enter an XML string that represents the Capacity Scheduler configuration.", "display_name": "Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)", "name": "resourcemanager_capacity_scheduler_configuration", "value": "\n\n \n yarn.scheduler.capacity.root.queues\n default\n \n \n yarn.scheduler.capacity.root.capacity\n 100\n \n \n yarn.scheduler.capacity.root.default.capacity\n 100\n \n\n" }, { "desc": "The address of the scheduler interface in the ResourceManager.", "display_name": "Scheduler Address", "name": "yarn_resourcemanager_scheduler_address", "value": "8030" }, { "desc": "The number of threads used to handle applications manager requests.", "display_name": "Client Thread Count", "name": "yarn_resourcemanager_client_thread_count", "value": "50" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_resource_manager_heap_dump_directory_free_space", "value": "false" }, { "desc": "The periodic interval that the ResourceManager will check whether ApplicationMasters is still alive.", "display_name": "ApplicationMaster Monitor Interval", "name": "yarn_resourcemanager_amliveliness_monitor_interval_ms", "value": "1000" }, { "desc": "For advanced use only. A string to be inserted into nodes_exclude.txt for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt", "name": "rm_hosts_exclude_safety_valve", "value": null }, { "desc": "The maximum number of completed applications that the ResourceManager keeps.", "display_name": "Max Completed Applications", "name": "yarn_resourcemanager_max_completed_applications", "value": "10000" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The timeout for the ResourceManager session with ZooKeeper. The session expires if the ZooKeeper ensemble does not hear from the ResourceManager within the specified timeout period (no heartbeat). Session expiration is managed by the ZooKeeper ensemble, not by the ResourceManager.", "display_name": "ZooKeeper Session Timeout", "name": "yarn_resourcemanager_zk_timeout_ms", "value": "60000" }, { "desc": "Enables multiple Fair Scheduler container assignments in one heartbeat, which improves cluster throughput when there are many small tasks to run.", "display_name": "Fair Scheduler Assign Multiple Tasks", "name": "resourcemanager_fair_scheduler_assign_multiple", "value": "false" }, { "desc": "Enables the health test that the ResourceManager's process state is consistent with the role configuration", "display_name": "ResourceManager Process Health Test", "name": "resourcemanager_scm_health_enabled", "value": "true" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_resource_manager_web_metric_collection", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "ResourceManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for ResourceManager parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for ResourceManager", "name": "role_config_suppression_resource_manager_java_opts", "value": "false" }, { "desc": "An XML string that will be inserted verbatim into the Fair Scheduler allocations file. For CDH5, overrides the configuration set using the Pools configuration UI. For CDH4, this is the only way to configure the Fair Scheduler for YARN.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "resourcemanager_fair_scheduler_configuration", "value": null }, { "desc": "Number of threads used to handle the ResourceManager admin interface.", "display_name": "Admin Client Thread Count", "name": "yarn_resourcemanager_admin_client_thread_count", "value": "1" }, { "desc": "If enabled, the ResourceManager binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind ResourceManager to Wildcard Address", "name": "yarn_rm_bind_wildcard", "value": "false" }, { "desc": "The class to use as the resource scheduler. FairScheduler is only supported in CDH 4.2.1 and later.", "display_name": "Scheduler Class", "name": "yarn_resourcemanager_scheduler_class", "value": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler" }, { "desc": "The minimum log level for ResourceManager logs", "display_name": "ResourceManager Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "resourcemanager_config_safety_valve", "value": null }, { "desc": "For applications that request containers on particular racks, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another rack. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. If not set, this means don't pass up any scheduling opportunities. Requires Fair Scheduler continuous scheduling to be disabled. If continuous scheduling is enabled, yarn.scheduler.fair.locality-delay-rack-ms should be used instead.", "display_name": "Fair Scheduler Rack Locality Threshold", "name": "resourcemanager_fair_scheduler_locality_threshold_rack", "value": null }, { "desc": "For applications that request containers on particular nodes, the minimum time in milliseconds the Fair Scheduler waits before accepting a placement on another node. Requires Fair Scheduler continuous scheduling to be enabled. If continuous scheduling is disabled, yarn.scheduler.fair.locality.threshold.node should be used instead.", "display_name": "Fair Scheduler Node Locality Delay", "name": "yarn_scheduler_fair_locality_delay_node_ms", "value": "2000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_role_env_safety_valve", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The smallest amount of physical memory, in MiB, that can be requested for a container. If using the Capacity or FIFO scheduler (or any scheduler, prior to CDH 5), memory requests will be rounded up to the nearest multiple of this number.", "display_name": "Container Memory Minimum", "name": "yarn_scheduler_minimum_allocation_mb", "value": "1024" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_resource_manager_log_directory_free_space", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "When enabled, ResourceManager has proxy user privileges.", "display_name": "Enable ResourceManager Proxy User Privileges", "name": "yarn_resourcemanager_proxy_user_privileges_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "resourcemanager_gc_duration_window", "value": "5" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_resourcemanager_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When enabled, any applications that were running on the cluster when the ResourceManager died will be recovered when the ResourceManager next starts. Note: If RM-HA is enabled, then this configuration is always enabled.", "display_name": "Enable ResourceManager Recovery", "name": "yarn_resourcemanager_recovery_enabled", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "resourcemanager_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The largest amount of physical memory, in MiB, that can be requested for a container.", "display_name": "Container Memory Maximum", "name": "yarn_scheduler_maximum_allocation_mb", "value": "65536" }, { "desc": "The largest number of virtual CPU cores that can be requested for a container. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores Maximum", "name": "yarn_scheduler_maximum_allocation_vcores", "value": "32" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Log Directory parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Log Directory", "name": "role_config_suppression_resource_manager_log_dir", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "When enabled, the Fair Scheduler will assign shares to individual apps based on their size, rather than providing an equal share to all apps regardless of size.", "display_name": "Fair Scheduler Size-Based Weight", "name": "resourcemanager_fair_scheduler_size_based_weight", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt", "name": "role_config_suppression_rm_hosts_allow_safety_valve", "value": "false" }, { "desc": "For applications that request containers on particular nodes, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another node. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. If not set, this means don't pass up any scheduling opportunities. Requires Fair Scheduler continuous scheduling to be disabled. If continuous scheduling is enabled, yarn.scheduler.fair.locality-delay-node-ms should be used instead.", "display_name": "Fair Scheduler Node Locality Threshold", "name": "resourcemanager_fair_scheduler_locality_threshold_node", "value": null }, { "desc": "The expiry interval to wait until an ApplicationMaster is considered dead.", "display_name": "ApplicationMaster Monitor Expiry", "name": "yarn_am_liveness_monitor_expiry_interval_ms", "value": "600000" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_resource_manager_file_descriptor", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for ResourceManager logs. Typically used by log4j or logback.", "display_name": "ResourceManager Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The smallest number of virtual CPU cores that can be requested for a container. If using the Capacity or FIFO scheduler (or any scheduler, prior to CDH 5), virtual core requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores Minimum", "name": "yarn_scheduler_minimum_allocation_vcores", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The address of the applications manager interface in the ResourceManager.", "display_name": "ResourceManager Address", "name": "yarn_resourcemanager_address", "value": "8032" }, { "desc": "The periodic interval that the ResourceManager will check whether NodeManagers are still alive.", "display_name": "NodeManager Monitor Interval", "name": "yarn_resourcemanager_nm_liveness_monitor_interval_ms", "value": "1000" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_resource_manager_swap_memory_usage", "value": "false" }, { "desc": "The HTTP port of the ResourceManager web application.", "display_name": "ResourceManager Web Application HTTP Port", "name": "resourcemanager_webserver_port", "value": "8088" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "resourcemanager_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "If using the Fair Scheduler, memory requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 5.", "display_name": "Container Memory Increment", "name": "yarn_scheduler_increment_allocation_mb", "value": "512" }, { "desc": "The maximum number of application attempts. This is a global setting for all ApplicationMasters. Each ApplicationMaster can specify its individual maximum through the API, but if the individual maximum is more than the global maximum, the ResourceManager overrides it.", "display_name": "ApplicationMaster Maximum Attempts", "name": "yarn_resourcemanager_am_max_retries", "value": "2" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The HTTPS port of the ResourceManager web application.", "display_name": "ResourceManager Web Application HTTPS Port (TLS/SSL)", "name": "resourcemanager_webserver_https_port", "value": "8090" }, { "desc": "The address of the resource tracker interface in the ResourceManager.", "display_name": "Resource Tracker Address", "name": "yarn_resourcemanager_resource_tracker_address", "value": "8031" }, { "desc": "The expiry interval to wait until a NodeManager is considered dead.", "display_name": "NodeManager Monitor Expiry", "name": "yarn_nm_liveness_monitor_expiry_interval_ms", "value": "600000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt", "name": "role_config_suppression_rm_hosts_exclude_safety_valve", "value": "false" }, { "desc": "If using the Fair Scheduler, virtual core requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 5.", "display_name": "Container Virtual CPU Cores Increment", "name": "yarn_scheduler_increment_allocation_vcores", "value": "1" }, { "desc": "Directory where ResourceManager will place its log files.", "display_name": "ResourceManager Log Directory", "name": "resource_manager_log_dir", "value": "/var/log/hadoop-yarn" }, { "desc": "Number of threads to handle resource tracker calls.", "display_name": "Resource Tracker Thread Count", "name": "yarn_resourcemanager_resource_tracker_client_thread_count", "value": "50" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_resource_manager_host_health", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of ResourceManager in Bytes", "name": "resource_manager_java_heapsize", "value": "1073741824" }, { "desc": "For applications that request containers on particular racks, the minimum time in milliseconds the Fair Scheduler waits before accepting a placement on another rack. Requires Fair Scheduler continuous scheduling to be enabled. If continuous scheduling is disabled, yarn.scheduler.fair.locality.threshold.rack should be used instead.", "display_name": "Fair Scheduler Rack Locality Delay", "name": "yarn_scheduler_fair_locality_delay_rack_ms", "value": "4000" }, { "desc": "For advanced use only. A string to be inserted into nodes_allow.txt for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt", "name": "rm_hosts_allow_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hue-hue_load_balancer.json0000666000175100017510000003772713245514472026603 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hue_load_balancer_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_ssl_certificate", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hue_load_balancer_unexpected_exits", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hue_load_balancer_scm_health", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Hue Load Balancer is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue Load Balancer TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_certificate_key", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Load Balancer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Load Balancer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hue_load_balancer_role_env_safety_valve", "value": "false" }, { "desc": "Directory where Hue Load Balancer will place its log files.", "display_name": "Hue Load Balancer Log Directory", "name": "hue_load_balancer_log_dir", "value": "/var/log/hue-httpd" }, { "desc": "For advanced use only, a string to be inserted into httpd.conf for this role only. This can only add options to the configuration, and cannot override previously defined options.", "display_name": "Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf", "name": "hue_load_balancer_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_ssl_certificate_key", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Hue Load Balancer is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue Load Balancer TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_certificate", "value": null }, { "desc": "Enables the health test that the Load Balancer's process state is consistent with the role configuration", "display_name": "Load Balancer Process Health Test", "name": "hue_load_balancer_scm_health_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Port to use to connect to the Hue through the Load Balancer.", "display_name": "Hue Load Balancer Port", "name": "listen", "value": "8889" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hue_load_balancer_file_descriptor", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "When computing the overall Load Balancer health, consider the host's health.", "display_name": "Load Balancer Host Health Test", "name": "hue_load_balancer_host_health_enabled", "value": "true" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hue_load_balancer_swap_memory_usage", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Load Balancer Environment Advanced Configuration Snippet (Safety Valve)", "name": "HUE_LOAD_BALANCER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf parameter.", "display_name": "Suppress Parameter Validation: Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf", "name": "role_config_suppression_hue_load_balancer_safety_valve", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hue_load_balancer_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hue_load_balancer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hue_load_balancer_host_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer Log Directory", "name": "role_config_suppression_hue_load_balancer_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-datanode.json0000666000175100017510000012704513245514472025107 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This behavior will always be disabled for workloads that read only short sections of a block (e.g HBase random-IO workloads). This property is supported in CDH3u3 or later deployments.", "display_name": "Enable purging cache after reads", "name": "dfs_datanode_drop_cache_behind_reads", "value": "false" }, { "desc": "Whether to suppress the results of the Data Directory Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Data Directory Status", "name": "role_health_suppression_data_node_volume_failures", "value": "false" }, { "desc": "The health test thresholds of free space in a DataNode. Specified as a percentage of the capacity on the DataNode.", "display_name": "DataNode Free Space Monitoring Thresholds", "name": "datanode_free_space_thresholds", "value": "{\"critical\":\"10.0\",\"warning\":\"20.0\"}" }, { "desc": "Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations.", "display_name": "Available Space Policy Balanced Preference", "name": "dfs_datanode_available_space_balanced_preference", "value": "0.75" }, { "desc": "The number of volumes that are allowed to fail before a DataNode stops offering service. By default, any volume failure will cause a DataNode to shutdown.", "display_name": "DataNode Failed Volumes Tolerated", "name": "dfs_datanode_failed_volumes_tolerated", "value": "0" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Maximum amount of bandwidth that each DataNode can use for balancing. Specified in bytes per second.", "display_name": "DataNode Balancing Bandwidth", "name": "dfs_balance_bandwidthPerSec", "value": "10485760" }, { "desc": "Whether to suppress the results of the DataNode Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: DataNode Data Directory Free Space", "name": "role_health_suppression_datanode_data_directories_free_space", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for DataNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for DataNode", "name": "role_config_suppression_datanode_java_opts", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_data_node_unexpected_exits", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The base port where the secure DataNode web UI listens. Combined with the DataNode's hostname to build its secure web UI address.", "display_name": "Secure DataNode Web UI Port (TLS/SSL)", "name": "dfs_datanode_https_port", "value": "50475" }, { "desc": "The maximum size, in megabytes, per log file for DataNode logs. Typically used by log4j or logback.", "display_name": "DataNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Port for the DataNode HTTP web UI. Combined with the DataNode's hostname to build its HTTP address.", "display_name": "DataNode HTTP Web UI Port", "name": "dfs_datanode_http_port", "value": "50075" }, { "desc": "When computing the overall DataNode health, consider the host's health.", "display_name": "DataNode Host Health Test", "name": "datanode_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Block Count heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Block Count", "name": "role_health_suppression_data_node_block_count", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds of failed volumes in a DataNode.", "display_name": "DataNode Volume Failures Thresholds", "name": "datanode_volume_failures_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Permissions for the directories on the local file system where the DataNode stores its blocks. The permissions must be octal. 755 and 700 are typical values.", "display_name": "DataNode Data Directory Permissions", "name": "dfs_datanode_data_dir_perm", "value": "700" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a DataNode Data Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "DataNode Data Directory Free Space Monitoring Percentage Thresholds", "name": "datanode_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_data_node_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "If this configuration is enabled, the DataNode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. This property is supported in CDH3u3 or later deployments.", "display_name": "Enable immediate enqueuing of data to disk after writes", "name": "dfs_datanode_sync_behind_writes", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_data_node_swap_memory_usage", "value": "false" }, { "desc": "If enabled, the DataNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind DataNode to Wildcard Address", "name": "dfs_datanode_bind_wildcard", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: DataNode Log Directory", "name": "role_config_suppression_datanode_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of DataNode in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of DataNode in Bytes", "name": "role_config_suppression_datanode_java_heapsize", "value": "false" }, { "desc": "In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This property is supported in CDH3u3 or later deployments.", "display_name": "Enable purging cache after writes", "name": "dfs_datanode_drop_cache_behind_writes", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_data_node_file_descriptor", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "DataNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "DATANODE_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "datanode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "While reading block files, the DataNode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader's position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the DataNode will attempt to read ahead. A value of 0 disables this feature. This property is supported in CDH3u3 or later deployments.", "display_name": "Number of read ahead bytes", "name": "dfs_datanode_readahead_bytes", "value": "4194304" }, { "desc": "The health test thresholds of the number of blocks on a DataNode", "display_name": "DataNode Block Count Thresholds", "name": "datanode_block_count_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"500000.0\"}" }, { "desc": "The maximum amount of memory a DataNode may use to cache data blocks in memory. Setting it to zero will disable caching.", "display_name": "Maximum Memory Used for Caching", "name": "dfs_datanode_max_locked_memory", "value": "4294967296" }, { "desc": "Comma-separated list of DataNode plug-ins to be activated. If one plug-in cannot be loaded, all the plug-ins are ignored.", "display_name": "DateNode Plugins", "name": "dfs_datanode_plugins_list", "value": "" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_data_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "datanode_web_metric_collection_enabled", "value": "true" }, { "desc": "The number of server threads for the DataNode.", "display_name": "Handler Count", "name": "dfs_datanode_handler_count", "value": "3" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Directory where DataNode will place its log files.", "display_name": "DataNode Log Directory", "name": "datanode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "DataNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the NameNode Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Connectivity", "name": "role_health_suppression_data_node_ha_connectivity", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory.", "display_name": "DataNode Data Directory Free Space Monitoring Absolute Thresholds", "name": "datanode_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_datanode_config_safety_valve", "value": "false" }, { "desc": "Enables the health test that the DataNode's process state is consistent with the role configuration", "display_name": "DataNode Process Health Test", "name": "datanode_scm_health_enabled", "value": "true" }, { "desc": "Timeout in seconds for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Timeout", "name": "dfs_thrift_timeout", "value": "60" }, { "desc": "DataNode Policy for picking which volume should get a new block. The Available Space policy is only available starting with CDH 4.3.", "display_name": "DataNode Volume Choosing Policy", "name": "dfs_datanode_volume_choosing_policy", "value": "org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy" }, { "desc": "Comma-delimited list of directories on the local file system where the DataNode stores HDFS block data. Typical values are /data/N/dfs/dn for N = 1, 2, 3.... In CDH 5.7 and higher, these directories can be optionally tagged with their storage types, e.g., [SSD]/data/1/dns/dn. HDFS supports the following storage types: [DISK], [SSD], [ARCHIVE], [RAM_DISK]. The default storage type of a directory will be [DISK] if it does not have a storage type tagged explicitly. These directories should be mounted using the noatime option, and the disks should be configured using JBOD. RAID is not recommended. Warning: Be very careful when modifying this property. Removing or changing entries can result in data loss. To hot swap drives in CDH 5.4 and higher, override the value of this property for the specific DataNode role instance that has the drive to be hot-swapped; do not modify the property value in the role group. See Configuring Hot Swap for DataNodes for more information.", "display_name": "DataNode Data Directory", "name": "dfs_data_dir_list", "value": null }, { "desc": "The minimum log level for DataNode logs", "display_name": "DataNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_data_node_host_health", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "datanode_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Port for the various DataNode Protocols. Combined with the DataNode's hostname to build its IPC port address.", "display_name": "DataNode Protocol Port", "name": "dfs_datanode_ipc_port", "value": "50020" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Reserved Space Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Reserved Space Validator", "name": "role_config_suppression_datanode_reserved_space_validator", "value": "false" }, { "desc": "The amount of time to wait for the DataNode to fully start up and connect to the NameNode before enforcing the connectivity check.", "display_name": "DataNode Connectivity Tolerance at Startup", "name": "datanode_connectivity_tolerance", "value": "180" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: DataNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_data_node_pause_duration", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory parameter.", "display_name": "Suppress Parameter Validation: DataNode Data Directory", "name": "role_config_suppression_dfs_data_dir_list", "value": "false" }, { "desc": "Minimum number of running threads for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Min Threadcount", "name": "dfs_thrift_threads_min", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Failed Volumes Tolerated Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Failed Volumes Tolerated Validator", "name": "role_config_suppression_datanode_failed_volumes_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Transceiver Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Transceiver Usage", "name": "role_health_suppression_data_node_transceivers_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory Permissions parameter.", "display_name": "Suppress Parameter Validation: DataNode Data Directory Permissions", "name": "role_config_suppression_dfs_datanode_data_dir_perm", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_data_node_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Reserved space in bytes per volume for non Distributed File System (DFS) use.", "display_name": "Reserved Space for Non DFS Use", "name": "dfs_datanode_du_reserved", "value": "10737418240" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Free Space", "name": "role_health_suppression_data_node_free_space_remaining", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Port for DataNode's XCeiver Protocol. Combined with the DataNode's hostname to build its address.", "display_name": "DataNode Transceiver Port", "name": "dfs_datanode_port", "value": "50010" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DateNode Plugins parameter.", "display_name": "Suppress Parameter Validation: DateNode Plugins", "name": "role_config_suppression_dfs_datanode_plugins_list", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "datanode_config_safety_valve", "value": null }, { "desc": "Maximum number of running threads for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Max Threadcount", "name": "dfs_thrift_threads_max", "value": "20" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_data_node_scm_health", "value": "false" }, { "desc": "The health test thresholds of transceivers usage in a DataNode. Specified as a percentage of the total configured number of transceivers.", "display_name": "DataNode Transceivers Usage Thresholds", "name": "datanode_transceivers_usage_thresholds", "value": "{\"critical\":\"95.0\",\"warning\":\"75.0\"}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "datanode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The maximum number of rolled log files to keep for DataNode logs. Typically used by log4j or logback.", "display_name": "DataNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether DataNodes should use DataNode hostnames when connecting to DataNodes for data transfer. This property is supported in CDH3u4 or later deployments.", "display_name": "Use DataNode Hostname", "name": "dfs_datanode_use_datanode_hostname", "value": "false" }, { "desc": "Enables the health test that verifies the DataNode is connected to the NameNode", "display_name": "DataNode Connectivity Health Test", "name": "datanode_connectivity_health_enabled", "value": "true" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of DataNode in Bytes", "name": "datanode_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls how much DataNode volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis.", "display_name": "Available Space Policy Balanced Threshold", "name": "dfs_datanode_available_space_balanced_threshold", "value": "10737418240" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 5, \"content\":\"Datanode registration failed\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Got a command from standby NN - ignoring command:.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Specifies the maximum number of threads to use for transferring data in and out of the DataNode.", "display_name": "Maximum Number of Transfer Threads", "name": "dfs_datanode_max_xcievers", "value": "4096" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for DataNode", "name": "datanode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: DataNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_datanode_role_env_safety_valve", "value": "false" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "datanode_pause_duration_window", "value": "5" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/impala-service.json0000666000175100017510000013400013245514472025274 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for the assignment locality health test. Specified as a percentage of total assignments.", "display_name": "Assignment Locality Ratio Thresholds", "name": "impala_assignment_locality_thresholds", "value": "{\"critical\":\"5.0\",\"warning\":\"80.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Local Path Access User Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Local Path Access User Validator", "name": "service_config_suppression_local_path_access_user_validator", "value": "false" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Enable HDFS Block Metadata API Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Enable HDFS Block Metadata API Configuration Validator", "name": "service_config_suppression_impala_hdfs_dfs_datanode_hdfs_blocks_metadata_enabled_set_validator", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "impala" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala System User (except Llama) parameter.", "display_name": "Suppress Parameter Validation: Impala System User (except Llama)", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Impala is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_private_key", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_impala_cmd_args_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_impalad_sentry_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Short-Circuit Read Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Short-Circuit Read Validator", "name": "service_config_suppression_impala_dssd_shortcircuit_read_enable_validator", "value": "false" }, { "desc": "Controls which queries admin users can see in the queries list view", "display_name": "Admin Users Query List Visibility Settings", "name": "admin_query_list_settings", "value": "ALL" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Allocations (Deployed)", "name": "impala_scheduled_allocations", "value": "{\"defaultFairSharePreemptionThreshold\":null,\"defaultFairSharePreemptionTimeout\":null,\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":null,\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":null,\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null}],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null}],\"userMaxAppsDefault\":null,\"users\":[]}" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Daemon Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Daemon Count Validator", "name": "service_config_suppression_impalad_count_validator", "value": "false" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Impala events:\n

\n\n
    \n
  • userName: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • operation: the Impala operation being performed.
  • \n
  • databaseName: the databaseName for the operation.
  • \n
  • tableName: the tableName for the operation.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Proxy User Configuration parameter.", "display_name": "Suppress Parameter Validation: Proxy User Configuration", "name": "service_config_suppression_impala_authorized_proxy_user_config", "value": "false" }, { "desc": "The time period over which to compute the assignment locality ratio. Specified in minutes.", "display_name": "Assignment Locality Monitoring Period", "name": "impala_assignment_locality_window", "value": "15" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Impala is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_server_certificate", "value": null }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. Typically edited using the Rules configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Configuration Rules (Staged)", "name": "impala_schedule_rules_draft", "value": null }, { "desc": "Name of the HDFS service that this Impala service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "The location on disk of the certificate, in PEM format, used to confirm the authenticity of SSL/TLS servers that the Impala daemons might connect to. Since the Impala daemons connect to each other, this should also include the CA certificate used to sign all the SSL/TLS Certificates. Without this parameter, SSL/TLS between Impala daemons will not be enabled.", "display_name": "Impala TLS/SSL CA Certificate", "name": "ssl_client_ca_certificate", "value": null }, { "desc": "Whether to suppress the results of the Impala Llama ApplicationMaster Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Llama ApplicationMaster Health", "name": "service_health_suppression_impala_llamas_healthy", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Impala Daemon command-line flags. Applies to all roles in this service. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "impala_cmd_args_safety_valve", "value": null }, { "desc": "The password for the private key in the Impala TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Impala TLS/SSL Private Key Password", "name": "ssl_private_key_password", "value": null }, { "desc": "Name of the ZooKeeper service to use for leader election and fencing when Llama is configured for high availability. This service dependency is required when more than one Llama role is present.", "display_name": "ZooKeeper Service for Llama HA", "name": "zookeeper_service", "value": null }, { "desc": "Use Dynamic Resource Pools to configure Impala admission control or RM for this Impala service. These features are only supported in Impala 1.3 or higher deployments.", "display_name": "Enable Dynamic Resource Pools", "name": "admission_control_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Query Aggregates parameter.", "display_name": "Suppress Parameter Validation: Impala Query Aggregates", "name": "service_config_suppression_impala_query_aggregates", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CA Certificate File Validator configuration validator.", "display_name": "Suppress Configuration Validator: CA Certificate File Validator", "name": "service_config_suppression_pem_ca_cert_recommended_for_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Private Key Password", "name": "service_config_suppression_ssl_private_key_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Server Private Key File (PEM Format)", "name": "service_config_suppression_ssl_private_key", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "impalad_sentry_safety_valve", "value": null }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. The value specified here is what gets deployed to all the hosts. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Configuration Rules (Deployed)", "name": "impala_schedule_rules", "value": "[]" }, { "desc": "Maximum number of HBase client retries for Impala. Used as a maximum for all operations such as fetching of the root region from the root RegionServer, getting a cell's value, and starting a row update. Overrides configuration in the HBase service.", "display_name": "Maximum HBase Client Retries", "name": "hbase_client_retries_number", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP Configuration Validator", "name": "service_config_suppression_impala_ldap_validator", "value": "false" }, { "desc": "Encrypt communication between clients and Impala using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Impala", "name": "client_services_ssl_enabled", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Impala events:\n

\n\n
    \n
  • operation: the Impala operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • databaseName: the database affected by the operation.
  • \n
  • tableName: the table affected by the operation.
  • \n
  • objectType: the type of object affected by the operation.
  • \n
  • privilege: the privilege associated with the operation.
  • \n
\n\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Specifies the set of authorized proxy users (users who can impersonate other users during authorization) and whom they are allowed to impersonate. Input is a semicolon-separated list of key=value pairs of authorized proxy users to the user(s) they can impersonate. These users are specified as a comma separated list of short usernames, or '*' to indicate all users. For example: joe=alice,bob;hue=*;admin=*. Only valid when Sentry is enabled.", "display_name": "Proxy User Configuration", "name": "impala_authorized_proxy_user_config", "value": "hue=*" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "impala_ldap_uri", "value": null }, { "desc": "Name of YARN service to use for resource management integration between Impala and YARN. This service dependency and the existence of a Llama role is required for using said integration.", "display_name": "YARN Service for Resource Management", "name": "yarn_service", "value": null }, { "desc": "Use Impala Admission Control to throttle Impala requests. Unless 'Enable Dynamic Resource Pools' is enabled, Impala uses a single, default pool that is configured using the Single Pool configurations below. These features are only supported in Impala 1.3 or higher deployments.", "display_name": "Enable Impala Admission Control", "name": "all_admission_control_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "When computing the overall IMPALA health, consider Impala StateStore's health", "display_name": "Impala StateStore Role Health Test", "name": "impala_statestore_health_enabled", "value": "true" }, { "desc": "Used to generate a core dump to get more information about an Impala crash. Unless otherwise configured systemwide using /proc/sys/kernel/core_pattern, the dump is generated in the 'current directory' of the Impala process (usually a subdirectory of the /var/run/cloudera-scm-agent/process directory). The core file can be very large.", "display_name": "Enable Core Dump", "name": "enable_core_dump", "value": "false" }, { "desc": "This parameter is useful when authenticating against a non Active Directory server such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server will look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "ldap_baseDN", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Pattern parameter.", "display_name": "Suppress Parameter Validation: LDAP Pattern", "name": "service_config_suppression_ldap_bind_pattern", "value": "false" }, { "desc": "When computing the overall IMPALA health, consider Impala Catalog Server's health", "display_name": "Impala Catalog Server Role Health Test", "name": "impala_catalogserver_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Llama Supported Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Llama Supported Validator", "name": "service_config_suppression_impala_llama_supported_validator", "value": "false" }, { "desc": "Configures the maximum number of queued queries for admission control when using a single pool. -1 or 0 disables queuing, i.e. incoming requests are rejected if they can not be executed immediately. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Max Queued Queries", "name": "admission_control_single_pool_max_queued", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Bypass Hive Metastore Validator configuration validator.", "display_name": "Suppress Configuration Validator: Bypass Hive Metastore Validator", "name": "service_config_suppression_impala_bypass_hms_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Llama ApplicationMaster Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Llama ApplicationMaster Count Validator", "name": "service_config_suppression_llama_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Sentry Validator", "name": "service_config_suppression_impala_sentry_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Daemon Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Health", "name": "service_health_suppression_impala_impalads_healthy", "value": "false" }, { "desc": "Maximum amount of time (in milliseconds) that a request waits to be admitted before timing out. Must be a positive integer.", "display_name": "Admission Control Queue Timeout", "name": "admission_control_queue_timeout", "value": "60000" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "The health test thresholds of the overall Impala Daemon health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Impala Daemons falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Impala Daemons falls below the critical threshold.", "display_name": "Healthy Impala Daemon Monitoring Thresholds", "name": "impala_impalads_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.lineage.client.properties.", "display_name": "Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "navigator_lineage_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Impala Short-Circuit Read Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Short-Circuit Read Validator", "name": "service_config_suppression_impala_dfs_client_read_shortcircuit_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Server Certificate File (PEM Format)", "name": "service_config_suppression_ssl_server_certificate", "value": "false" }, { "desc": "When set, this parameter allows arbitrary mapping from usernames into a Distinguished Name (DN). The string specified must have a placeholder named \"#UID\" inside it, and that #UID is replaced with the username. For example, you could mimic the behavior of LDAP BaseDN by specifying \"uid=#UID,ou=People,dc=cloudera,dc=com\". When the username of \"mike\" comes in, it replaces the #UID and the result is \"uid=mike,ou=People,dc=cloudera,dc=com\". This option should be used when more control over the DN is needed. This parameter is mutually exclusive with LDAP Domain and LDAP BaseDN.", "display_name": "LDAP Pattern", "name": "ldap_bind_pattern", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_impala_service_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL CA Certificate parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL CA Certificate", "name": "service_config_suppression_ssl_client_ca_certificate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala System Group (except Llama) parameter.", "display_name": "Suppress Parameter Validation: Impala System Group (except Llama)", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_ldap_basedn", "value": "false" }, { "desc": "Enable collection of lineage from the service's roles.", "display_name": "Enable Lineage Collection", "name": "navigator_lineage_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Impala Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "impala_service_env_safety_valve", "value": null }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using Sentry service instead, add Sentry service as a dependency to Impala service. Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry is supported only on Impala 1.1 or later deployments.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "sentry_enabled", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "enable_ldap_auth", "value": "false" }, { "desc": "Configures the max memory of running queries for admission control when using a single pool. -1 or 0 indicates no limit. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Mem Limit", "name": "admission_control_single_pool_mem_limit", "value": "-1" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Name of the Sentry service that this Impala service instance depends on. If selected, Impala uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties parameter.", "display_name": "Suppress Parameter Validation: Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "service_config_suppression_navigator_lineage_client_config_safety_valve", "value": "false" }, { "desc": "Configures the maximum number of concurrently running queries for admission control when using a single pool. -1 indicates no limit and 0 indicates all incoming requests will be rejected. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Max Running Queries", "name": "admission_control_single_pool_max_requests", "value": "200" }, { "desc": "Name of the HBase service that this Impala service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Use debug build of Impala binaries when starting roles. Useful when performing diagnostic activities to get more information in the stacktrace or core dump.", "display_name": "Use Debug Build", "name": "use_debug_build", "value": "false" }, { "desc": "Name of the Hive service that this Impala service instance depends on", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Impala Catalog Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Catalog Server Count Validator", "name": "service_config_suppression_catalogserver_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Permissions Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Permissions Validator", "name": "service_config_suppression_short_circuit_reads_data_directory_permissions_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_ldap_domain", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_impala_ldap_uri", "value": "false" }, { "desc": "Number of minutes between reestablishing our ticket with the Kerberos server.", "display_name": "Kerberos Re-init Interval", "name": "kerberos_reinit_interval", "value": "60" }, { "desc": "Whether to suppress configuration warnings produced by the Impala StateStore Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala StateStore Count Validator", "name": "service_config_suppression_statestore_count_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Assignment Locality heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Assignment Locality", "name": "service_health_suppression_impala_assignment_locality", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Catalog Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Catalog Server Health", "name": "service_health_suppression_impala_catalogserver_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress the results of the Impala StateStore Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala StateStore Health", "name": "service_health_suppression_impala_statestore_health", "value": "false" }, { "desc": "The minimum number of assignments that must occur during the test time period before the threshold values will be checked. Until this number of assignments have been observed in the test time period the health test will be disabled.", "display_name": "Assignment Locality Minimum Assignments", "name": "impala_assignment_locality_minimum", "value": "10" }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "enable_ldap_tls", "value": "false" }, { "desc": "Controls the aggregate metrics generated for Impala queries. The structure is a JSON list of the attributes to aggregate and the entities to aggregate to. For example, if the attributeName is 'hdfs_bytes_read' and the aggregationTargets is ['USER'] then the Service Monitor will create the metric 'impala_query_hdfs_bytes_read_rate' and, every ten minutes, will record the total hdfs bytes read for each user across all their Impala queries. By default it will also record the number of queries issues ('num_impala_queries_rate') for both users and pool. For a full list of the supported attributes see the Impala search page. Note that the valid aggregation targets are USER, YARN_POOL, and IMPALA (the service), and that these aggregate metrics can be viewed on both the reports and charts search pages.", "display_name": "Impala Query Aggregates", "name": "impala_query_aggregates", "value": "[\n {\n \"attributeName\": \"hdfs_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"thread_cpu_time\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"bytes_streamed\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cm_cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"query_duration\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"admission_wait\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"memory_accrual\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"thread_cpu_time\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"memory_spilled\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n }\n]\n" }, { "desc": "Time in seconds before Impala Daemon or Catalog Server times out with the StateStore.", "display_name": "StateStoreSubscriber Timeout", "name": "statestore_subscriber_timeout", "value": "30" }, { "desc": "The group that this Impala's processes should run as (except Llama, which has its own group).", "display_name": "Impala System Group (except Llama)", "name": "process_groupname", "value": "impala" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "ldap_domain", "value": null }, { "desc": "The user that this Impala's processes should run as (except Llama, which has its own user).", "display_name": "Impala System User (except Llama)", "name": "process_username", "value": "impala" }, { "desc": "Timeout in milliseconds for all HBase RPCs made by Impala. Overrides configuration in HBase service.", "display_name": "HBase RPC Timeout", "name": "hbase_rpc_timeout", "value": "3000" }, { "desc": "Controls which queries a non-admin user can see in the queries list view", "display_name": "Non-Admin Users Query List Visibility Settings", "name": "user_query_list_settings", "value": "ALL" }, { "desc": "The health test thresholds of the overall Impala Llama ApplicationMaster health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Impala Llama ApplicationMasters falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Impala Llama ApplicationMasters falls below the critical threshold.", "display_name": "Healthy Impala Llama ApplicationMaster Monitoring Thresholds", "name": "impala_llamas_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. Typically edited using the Pools configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Allocations (Staged)", "name": "impala_scheduled_allocations_draft", "value": null }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hive-hivemetastore.json0000666000175100017510000006022013245514472026205 0ustar zuulzuul00000000000000[ { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hive/metrics-hivemetastore/metrics.log. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Location", "name": "hive_metrics_sample_file_location", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hivemetastore_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_metastore_env_safety_valve", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Hive Metastore Server", "name": "hive_metastore_java_opts", "value": "-XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_metastore_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hivemetastore_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hivemetastore_scm_health", "value": "false" }, { "desc": "The minimum log level for Hive Metastore Server logs", "display_name": "Hive Metastore Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hivemetastore_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hive_metrics_sample_file_location", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hivemetastore_log_directory_free_space", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Logging Frequency", "name": "hive_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "The maximum number of rolled log files to keep for Hive Metastore Server logs. Typically used by log4j or logback.", "display_name": "Hive Metastore Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_metastore_config_safety_valve", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hivemetastore_heap_dump_directory_free_space", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Enables the health test that the Hive Metastore Server's process state is consistent with the role configuration", "display_name": "Hive Metastore Server Process Health Test", "name": "hivemetastore_scm_health_enabled", "value": "true" }, { "desc": "Controls whether the Hive metrics subsystem is enabled for the role.", "display_name": "Enable Metrics Subsystem", "name": "hive_metrics_enabled", "value": "true" }, { "desc": "Port on which Hive Metastore Server will listen for connections.", "display_name": "Hive Metastore Server Port", "name": "hive_metastore_port", "value": "9083" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Log Directory", "name": "role_config_suppression_hive_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Minimum number of worker threads in the Hive Metastore Server's thread pool", "display_name": "Min Hive Metastore Server Threads", "name": "hive_metastore_min_threads", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Hive Metastore Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Hive Metastore Server", "name": "role_config_suppression_hive_metastore_java_opts", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "hivemetastore_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "hivemetastore_pause_duration_window", "value": "5" }, { "desc": "Maximum number of worker threads in the Hive Metastore Server's thread pool", "display_name": "Max Hive Metastore Server Threads", "name": "hive_metastore_max_threads", "value": "100000" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hivemetastore_file_descriptor", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_hivemetastore_pause_duration", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "When computing the overall Hive Metastore Server health, consider the host's health.", "display_name": "Hive Metastore Server Host Health Test", "name": "hivemetastore_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hivemetastore_unexpected_exits", "value": "false" }, { "desc": "The delegation token store implementation class. Use DBTokenStore for Highly Available Metastore Configuration.", "display_name": "Hive Metastore Delegation Token Store", "name": "hive_metastore_delegation_token_store", "value": "org.apache.hadoop.hive.thrift.MemoryTokenStore" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress the results of the Hive Metastore Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Metastore Canary", "name": "role_health_suppression_hivemetastore_canary_health", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Hive Metastore Server logs. Typically used by log4j or logback.", "display_name": "Hive Metastore Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Directory where Hive Metastore Server will place its log files.", "display_name": "Hive Metastore Server Log Directory", "name": "hive_log_dir", "value": "/var/log/hive" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_metastore_env_safety_valve", "value": null }, { "desc": "Enables the health test that checks that basic Hive Metastore operations succeed", "display_name": "Hive Metastore Canary Health Test", "name": "metastore_canary_health_enabled", "value": "true" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Hive Metastore Server in Bytes", "name": "hive_metastore_java_heapsize", "value": "8589934592" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hbase-service.json0000666000175100017510000013451113245514472025122 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "The timeout before injecting a snapshot timeout error when waiting for a snapshot completion.", "display_name": "HBase Master Snapshot Waiting Timeout", "name": "hbase_snapshot_master_timeoutMillis", "value": "60000" }, { "desc": "The password for the HBase JKS keystore file.", "display_name": "HBase TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hbase/audit" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Thrift Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Thrift Server Count Validator", "name": "service_config_suppression_hbasethriftserver_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Region Health Canary Exclude Tables parameter.", "display_name": "Suppress Parameter Validation: HBase Region Health Canary Exclude Tables", "name": "service_config_suppression_hbase_region_health_canary_exclude_tables", "value": "false" }, { "desc": "Number of rows to fetch when calling next on a scanner if it is not served from memory. Higher caching values enable faster scanners but require more memory and some calls of next may take longer when the cache is empty.", "display_name": "HBase Client Scanner Caching", "name": "hbase_client_scanner_caching", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hbase" }, { "desc": "The health test thresholds of the overall RegionServer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" RegionServers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" RegionServers falls below the critical threshold.", "display_name": "Healthy RegionServer Monitoring Thresholds", "name": "hbase_regionservers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "Enable HBase row-level authorization.", "display_name": "HBase Row-Level Authorization", "name": "hbase_row_level_authorization", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode Rootserver parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode Rootserver", "name": "service_config_suppression_zookeeper_znode_rootserver", "value": "false" }, { "desc": "Timeout (in ms) for the distributed log splitting manager to receive response from a worker.", "display_name": "SplitLog Manager Timeout", "name": "hbase_service_splitlog_manager_timeout", "value": "120000" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Authentication And Authorization Validation configuration validator.", "display_name": "Suppress Configuration Validator: HBase Authentication And Authorization Validation", "name": "service_config_suppression_hbase_authentication_and_authorization_validator", "value": "false" }, { "desc": "Timeout for graceful shutdown of this HBase service. Once this timeout is reached, any remaining running roles are abruptly shutdown. A value of 0 means no timeout.", "display_name": "Graceful Shutdown Timeout", "name": "hbase_graceful_stop_timeout", "value": "180" }, { "desc": "Whether the step to reload regions back onto the original RegionServers should be skipped during rolling restart. This can be used to increase the speed of rolling restart or upgrade operations, but can result in regions being moved multiple times, decreasing performance for clients during rolling restart.", "display_name": "Skip Region Reload During Rolling Restart", "name": "hbase_skip_reload_during_rr", "value": "false" }, { "desc": "The number of times to retry connections to ZooKeeper. Used for reading and writing root region location. Used together with ${zookeeper.pause} in an exponential backoff fashion when making queries to ZooKeeper.", "display_name": "ZooKeeper Connection Retries", "name": "zookeeper_retries", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HBase Proxy User Hosts", "name": "service_config_suppression_hbase_proxy_user_hosts_list", "value": "false" }, { "desc": "Maximum number of hlog entries to replicate in one go. If this is large, and a consumer takes a while to process the events, the HBase RPC call will time out.", "display_name": "Replication Batch Size", "name": "hbase_replication_source_nb_capacity", "value": "1000" }, { "desc": "Tables to exclude in the HBase Region Health Canary which will scan a row from every region.", "display_name": "HBase Region Health Canary Exclude Tables", "name": "hbase_region_health_canary_exclude_tables", "value": "" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for HBase events:\n

\n
    \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • username: the user performing the action.
  • \n
  • tableName: the table affected by the operation.
  • \n
  • family: the column family affected by the operation.
  • \n
  • qualifier: the qualifier the operation.
  • \n
  • action: the action being performed.
  • \n
\n\n

\nThe default HBase audit event filter discards events that affect the internal\n-ROOT-, .META. and _acl_ tables.\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default HBase audit event filter discards events that affect the \",\n \"internal -ROOT-, .META. and _acl_ tables.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"tableName\", \"match\" : \"(?:-ROOT-|.META.|_acl_|hbase:meta|hbase:acl)\" }\n ]\n }\n ]\n}\n" }, { "desc": "If this is set to \"kerberos\", HBase REST Server will authenticate its clients. HBase Proxy User Hosts and Groups should be configured to allow specific users to access HBase through REST Server.", "display_name": "HBase REST Authentication", "name": "hbase_restserver_security_authentication", "value": "simple" }, { "desc": "Name of the HDFS service that this HBase service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the AWS S3 Secret Access Key for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: AWS S3 Secret Access Key for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_secret_access_key", "value": "false" }, { "desc": "Use this to enable Http server usage on thrift, which is mainly needed for \"doAs\" functionality.", "display_name": "Enable HBase Thrift Http Server", "name": "hbase_thriftserver_http", "value": "false" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase is acting as a TLS/SSL server.", "display_name": "HBase TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "true" }, { "desc": "Period of time, in milliseconds, to pause between connection retries to ZooKeeper. Used together with ${zookeeper.retries} in an exponential backoff fashion when making queries to ZooKeeper.", "display_name": "ZooKeeper Connection Retry Pause Duration", "name": "zookeeper_pause", "value": null }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_service_config_safety_valve", "value": null }, { "desc": "Name of the ZooKeeper service that this HBase service instance depends on.", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Enable snapshots. Disabling snapshots requires deletion of all snapshots before restarting the HBase master; the HBase master will not start if snapshots are disabled and snapshots exist.", "display_name": "Enable Snapshots", "name": "hbase_snapshot_enabled", "value": "true" }, { "desc": "ZooKeeper session timeout in milliseconds. HBase passes this to the ZooKeeper quorum as the suggested maximum time for a session. See http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions The client sends a requested timeout, the server responds with the timeout that it can give the client.", "display_name": "ZooKeeper Session Timeout", "name": "zookeeper_session_timeout", "value": "60000" }, { "desc": "The root znode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file paths are configured with a relative path, so they will all go under this directory unless changed.", "display_name": "ZooKeeper Znode Parent", "name": "zookeeper_znode_parent", "value": "/hbase" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Amazon S3 Access Key ID for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: Amazon S3 Access Key ID for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_access_key_id", "value": "false" }, { "desc": "Use this to allow proxy users on thrift gateway, which is mainly needed for \"doAs\" functionality.", "display_name": "Enable HBase Thrift Proxy Users", "name": "hbase_thriftserver_support_proxyuser", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Size of the threadpool used for hedged reads in hdfs clients. If a read from a block is slow, a parallel 'hedged' read will be started against a different block replica. The first one to return with a result is used while the other one is cancelled. This 'hedged' read feature helps rein in the outliers. A value of zero disables the feature.", "display_name": "HDFS Hedged Read Threadpool Size", "name": "hbase_server_dfs_client_hedged_read_threadpool_size", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the RegionServer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: RegionServer Count Validator", "name": "service_config_suppression_regionserver_count_validator", "value": "false" }, { "desc": "The tolerance window that will be used in HBase service tests that depend on detection of the active HBase Master.", "display_name": "HBase Active Master Detection Window", "name": "hbase_active_master_detecton_window", "value": "3" }, { "desc": "Maximum number of client retries. Used as a maximum for all operations such as fetching of the root region from the root RegionServer, getting a cell's value, and starting a row update.", "display_name": "Maximum HBase Client Retries", "name": "hbase_client_retries_number", "value": "35" }, { "desc": "Write buffer size in bytes. A larger buffer requires more memory on both the client and the server because the server instantiates the passed write buffer to process it but reduces the number of remote procedure calls (RPC). To estimate the amount of server memory used, multiply the value of 'hbase.client.write.buffer' by the value of 'hbase.regionserver.handler.count'.", "display_name": "HBase Client Write Buffer", "name": "hbase_client_write_buffer", "value": "2097152" }, { "desc": "AWS secret access key required to access S3 to store remote snapshots.", "display_name": "AWS S3 Secret Access Key for Remote Snapshots", "name": "hbase_snapshot_s3_secret_access_key", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Scheduler Pool for Remote Snapshots in AWS S3 parameter.", "display_name": "Suppress Parameter Validation: Scheduler Pool for Remote Snapshots in AWS S3", "name": "service_config_suppression_hbase_snapshot_s3_scheduler_pool", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • username: treats the field value as a user name, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can used to compare HBase events:\n

\n\n
    \n
  • operation: the HBase operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • tableName: the name of the table affected by the operation.
  • \n
  • family: the column family affected by the operation.
  • \n
  • qualifier: the qualifier of the operation.
  • \n
\n\n

\nThe default event tracker for HBase services defines equality by comparing the\nusername, operation, table name, family, and qualifier of the events.\n

\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": "{\n \"comment\" : [\n \"The default event tracker for HBase services defines equality by \",\n \"comparing the username, operation, table name, family, and qualifier of \",\n \"the events.\"\n ],\n \"timeToLive\" : 60000,\n \"fields\" : [\n { \"type\": \"value\", \"name\" : \"tableName\" },\n { \"type\": \"value\", \"name\" : \"family\" },\n { \"type\": \"value\", \"name\" : \"qualifier\" },\n { \"type\": \"value\", \"name\" : \"operation\" },\n { \"type\": \"username\", \"name\" : \"username\" }\n ]\n}\n" }, { "desc": "Enable TLS/SSL encryption for HBase web UIs.", "display_name": "Web UI TLS/SSL Encryption Enabled", "name": "hbase_hadoop_ssl_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode Parent parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode Parent", "name": "service_config_suppression_zookeeper_znode_parent", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Specifies the combined maximum allowed size of a KeyValue instance. This option configures an upper boundary for a single entry saved in a storage file. This option prevents a region from splitting if the data is too large. Set this option to a fraction of the maximum region size. To disable this check, use a value of zero or less.", "display_name": "Maximum Size of HBase Client KeyValue", "name": "hbase_client_keyvalue_maxsize", "value": "10485760" }, { "desc": "Allow indexing of tables in HBase by Lily HBase Indexer. Note: Replication must be enabled for indexing to work.", "display_name": "Enable Indexing", "name": "hbase_enable_indexing", "value": "false" }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "When computing the overall HBase cluster health, consider the health of the backup HBase Masters.", "display_name": "Backup Masters Health Test", "name": "hbase_backup_masters_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_hbase_core_site_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_hbase_ssl_server_safety_valve", "value": "false" }, { "desc": "Configure the type of encrypted communication to be used with RPC.", "display_name": "HBase Transport Security", "name": "hbase_rpc_protection", "value": "authentication" }, { "desc": "Period of time, in milliseconds, to pause between searches for work. Used as a sleep interval by service threads such as a META scanner and log roller.", "display_name": "HBase Server Thread Wake Frequency", "name": "hbase_server_thread_wakefrequency", "value": "10000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Root Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Root Directory", "name": "service_config_suppression_hdfs_rootdir", "value": "false" }, { "desc": "Set to true to cause the hosting server (Master or RegionServer) to abort if a coprocessor throws a Throwable object that is not IOException or a subclass of IOException. Setting it to true might be useful in development environments where one wants to terminate the server as soon as possible to simplify coprocessor failure analysis.", "display_name": "HBase Coprocessor Abort on Error", "name": "hbase_coprocessor_abort_on_error", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "An alert is published if the HBase region health canary detects at least this percentage of total regions are unhealthy. This threshold is used if the explicit count is not set via the hbase_canary_alert_unhealthy_region_count_threshold config.", "display_name": "HBase Canary Unhealthy Region Percentage Alert Threshold", "name": "hbase_canary_alert_unhealthy_region_percent_threshold", "value": "0.1" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the HBase user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'. Note: This property is used only if HBase REST/Thrift Server Authentication is enabled.", "display_name": "HBase Proxy User Groups", "name": "hbase_proxy_user_groups_list", "value": "*" }, { "desc": "The maximum amount of time the HBase master waits for a snapshot to complete.", "display_name": "HBase Master Snapshot Timeout", "name": "hbase_snapshot_master_timeout_millis", "value": "60000" }, { "desc": "The user the management services impersonate when connecting to HBase. If no value is specified, the HBase superuser is used.", "display_name": "HBase User to Impersonate", "name": "hbase_user_to_impersonate", "value": null }, { "desc": "An alert is published if the HBase region health canary detects at least this many unhealthy regions. This setting takes precedence over the hbase_canary_alert_unhealthy_region_percent_threshold config.", "display_name": "HBase Canary Unhealthy Region Count Alert Threshold", "name": "hbase_canary_alert_unhealthy_region_count_threshold", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase User to Impersonate parameter.", "display_name": "Suppress Parameter Validation: HBase User to Impersonate", "name": "service_config_suppression_hbase_user_to_impersonate", "value": "false" }, { "desc": "Path to ZooKeeper Node holding root region location. This is written by the HBase Master and read by clients and RegionServers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, the root location is stored at /hbase/root-region-server.", "display_name": "ZooKeeper Znode Rootserver", "name": "zookeeper_znode_rootserver", "value": "root-region-server" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "A general client pause time value. Used mostly as a time period to wait before retrying operations such as a failed get or region lookup.", "display_name": "HBase Client Pause", "name": "hbase_client_pause", "value": "100" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "An alert is published if the HBase region health canary runs slowly.", "display_name": "HBase Region Health Canary Slow Run Alert Enabled", "name": "hbase_region_health_canary_slow_run_alert_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Superusers parameter.", "display_name": "Suppress Parameter Validation: HBase Superusers", "name": "service_config_suppression_hbase_superuser", "value": "false" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "zookeeper.recovery.retry0zookeeper.recovery.retry.intervalmill3000hbase.zookeeper.recoverable.waittime1000zookeeper.session.timeout30000hbase.rpc.timeout10000hbase.client.retries.number1hbase.client.rpc.maxattempts1hbase.client.operation.timeout10000" }, { "desc": "Ratio of Lily HBase Indexers used by each HBase RegionServer while doing replication.", "display_name": "Replication Source Ratio", "name": "hbase_replication_source_ratio", "value": "1.0" }, { "desc": "Duration to wait before starting up a 'hedged' read.", "display_name": "HDFS Hedged Read Delay Threshold", "name": "hbase_server_dfs_client_hedged_read_threshold_millis", "value": "500" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hbase_service_env_safety_valve", "value": null }, { "desc": "Amazon S3 path where remote snapshots should be stored.", "display_name": "Amazon S3 Path for Remote Snapshots", "name": "hbase_snapshot_s3_path", "value": null }, { "desc": "When computing the overall HBase cluster health, consider the active HBase Master's health.", "display_name": "Active Master Health Test", "name": "hbase_master_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HBase Proxy User Groups", "name": "service_config_suppression_hbase_proxy_user_groups_list", "value": "false" }, { "desc": "Set to true to use HBase Secure RPC Engine for remote procedure calls (RPC). This is only effective in simple authentication mode. Does not provide authentication for RPC calls, but provides user information in the audit logs. Changing this setting requires a restart of this and all dependent services and redeployment of client configurations, along with a restart of the Service Monitor management role.", "display_name": "HBase Secure RPC Engine", "name": "hbase_secure_rpc_engine", "value": "false" }, { "desc": "Choose the authentication mechanism used by HBase.", "display_name": "HBase Secure Authentication", "name": "hbase_security_authentication", "value": "simple" }, { "desc": "Whether to suppress configuration warnings produced by the ZooKeeper Max Session Timeout Validator configuration validator.", "display_name": "Suppress Configuration Validator: ZooKeeper Max Session Timeout Validator", "name": "service_config_suppression_zookeeper_max_session_timeout_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The HDFS directory shared by HBase RegionServers.", "display_name": "HDFS Root Directory", "name": "hdfs_rootdir", "value": "/hbase" }, { "desc": "Enables the canary that checks HBase region availability by scanning a row from every region.", "display_name": "HBase Region Health Canary", "name": "hbase_region_health_canary_enabled", "value": "true" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Access key ID required to access Amazon S3 to store remote snapshots.", "display_name": "Amazon S3 Access Key ID for Remote Snapshots", "name": "hbase_snapshot_s3_access_key_id", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "service_config_suppression_hbase_service_config_safety_valve", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hbase" }, { "desc": "If this is set, HBase Thrift Server authenticates its clients. HBase Proxy User Hosts and Groups should be configured to allow specific users to access HBase through Thrift Server.", "display_name": "HBase Thrift Authentication", "name": "hbase_thriftserver_security_authentication", "value": "none" }, { "desc": "Whether to suppress the results of the RegionServer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: RegionServer Health", "name": "service_health_suppression_hbase_region_servers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Comma-delimited list of hosts where you want to allow the HBase user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'. Note: This property is used only if HBase REST/Thrift Server Authentication is enabled.", "display_name": "HBase Proxy User Hosts", "name": "hbase_proxy_user_hosts_list", "value": "*" }, { "desc": "Name of the scheduler pool to use for MR jobs created during export/import of remote snapshots in AWS S3.", "display_name": "Scheduler Pool for Remote Snapshots in AWS S3", "name": "hbase_snapshot_s3_scheduler_pool", "value": null }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "hbase_core_site_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hbase_service_env_safety_valve", "value": "false" }, { "desc": "The maximum amount of time the Hbase RegionServer waits for a snapshot to complete.", "display_name": "HBase RegionServer Snapshot Timeout", "name": "hbase_snapshot_region_timeout", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Amazon S3 Path for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: Amazon S3 Path for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_path", "value": "false" }, { "desc": "Start a process to periodically check that RegionServer is alive when RegionServer is started. Note: This canary is different from the Cloudera Service Monitoring canary and is provided by the HBase service itself.", "display_name": "Enable HBase Canary", "name": "hbase_regionserver_enable_canary", "value": "false" }, { "desc": "List of users or groups, who are allowed full privileges, regardless of stored ACLs, across the cluster. Only used when HBase security is enabled.", "display_name": "HBase Superusers", "name": "hbase_superuser", "value": "" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "hbase_ssl_server_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the HBase Master Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HBase Master Health", "name": "service_health_suppression_hbase_master_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Allow HBase tables to be replicated.", "display_name": "Enable Replication", "name": "hbase_enable_replication", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hbase" }, { "desc": "Enable HBase authorization.", "display_name": "HBase Secure Authorization", "name": "hbase_security_authorization", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HBase REST Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase REST Server Count Validator", "name": "service_config_suppression_hbaserestserver_count_validator", "value": "false" }, { "desc": "Timeout for all HBase RPCs in milliseconds.", "display_name": "RPC Timeout", "name": "hbase_rpc_timeout", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Master Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Master Count Validator", "name": "service_config_suppression_master_count_validator", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-gateway.json0000666000175100017510000001402413245514472024761 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "role_config_suppression_hdfs_client_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hbase_client_java_opts", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh", "display_name": "HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "hdfs_client_env_safety_valve", "value": null }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hadoop" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Trash Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Trash Enabled Validator", "name": "role_config_suppression_hdfs_trash_disabled_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for hdfs-site.xml.", "display_name": "HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "hdfs_client_config_safety_valve", "value": null }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hdfs_client_java_heapsize", "value": "268435456" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether HDFS clients will use the legacy block reader.", "display_name": "Use Legacy Blockreader", "name": "dfs_client_use_legacy_blockreader", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_hdfs_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hbase_client_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Move deleted files to the trash so that they can be recovered if necessary. This client side configuration takes effect only if the HDFS service-wide trash is disabled (NameNode Filesystem Trash Interval set to 0) and is ignored otherwise. The trash is not automatically emptied when enabled with this configuration.", "display_name": "Use Trash", "name": "dfs_client_use_trash", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/cdh_config.sh0000777000175100017510000000010113245514472024114 0ustar zuulzuul00000000000000#!/bin/bash tox -evenv -- python $(dirname $0)/cdh_config.py $* sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/zookeeper-server.json0000666000175100017510000011531513245514472025712 0ustar zuulzuul00000000000000[ { "desc": "Enables the quorum membership check for this ZooKeeper Server.", "display_name": "Enable the Quorum Membership Check", "name": "zookeeper_server_quorum_membership_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_server_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Enables the health test that the Server's process state is consistent with the role configuration", "display_name": "Server Process Health Test", "name": "zookeeper_server_scm_health_enabled", "value": "true" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The port to monitor for client connections. This is the port that clients attempt to connect to.", "display_name": "Client Port", "name": "clientPort", "value": "2181" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The port to monitor for inter-server communication", "display_name": "Quorum Port", "name": "quorumPort", "value": "3181" }, { "desc": "The health check thresholds of the weighted average size of the ZooKeeper Server outstanding requests queue over a recent period. See ZooKeeper Server Outstanding Requests Monitoring Period.", "display_name": "ZooKeeper Server Outstanding Requests Thresholds", "name": "zookeeper_server_outstanding_requests_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of ZooKeeper Server in Bytes", "name": "zookeeper_server_java_heapsize", "value": "1073741824" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The maximum size, in megabytes, per log file for Server logs. Typically used by log4j or logback.", "display_name": "Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The period to review when computing the moving average of the outstanding requests queue size. Specified in minutes.", "display_name": "ZooKeeper Server Outstanding Requests Monitoring Period", "name": "zookeeper_server_outstanding_requests_window", "value": "3" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "zookeeper_server_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The disk location that ZooKeeper will use to store its transaction logs.", "display_name": "Transaction Log Directory", "name": "dataLogDir", "value": "/var/lib/zookeeper" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_zookeeper_server_scm_health", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Quorum Membership heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Quorum Membership", "name": "role_health_suppression_zookeeper_server_quorum_membership", "value": "false" }, { "desc": "The port to monitor for leadership election", "display_name": "Election Port", "name": "electionPort", "value": "4181" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Transaction Log Directory parameter.", "display_name": "Suppress Parameter Validation: Transaction Log Directory", "name": "role_config_suppression_datalogdir", "value": "false" }, { "desc": "Unique identifier for each ZooKeeper server, typically starts at 1", "display_name": "ZooKeeper Server ID", "name": "serverId", "value": null }, { "desc": "Specifies the name of the user that has read-only privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Name of User with Read-Only access to the JMX Agent", "name": "jmx_passwd_file_readonly_user", "value": "monitorRole" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enables the JMX agent on the ZooKeeper server. Turning this off on any of the ZooKeeper servers that are part of a service will prevent Cloudera Manager from being able to monitor that server and may affect the monitoring provided on the entire service.", "display_name": "Enable JMX Agent", "name": "enable_jmx_agent", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "The minimum session timeout, in milliseconds, that the ZooKeeper Server will allow the client to negotiate", "display_name": "Minimum Session Timeout", "name": "minSessionTimeout", "value": "4000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_zookeeper_server_host_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Data Directory.", "display_name": "Data Directory Free Space Monitoring Absolute Thresholds", "name": "zookeeper_server_data_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_zookeeper_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "zookeeper_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health check thresholds of the weighted average size of the ZooKeeper Server connection count over a recent period. See ZooKeeper Server Connection Count Monitoring Period.", "display_name": "ZooKeeper Server Connection Count Thresholds", "name": "zookeeper_server_connection_count_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Maximum Request Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Maximum Request Latency", "name": "role_health_suppression_zookeeper_server_max_latency", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into zoo.cfg for this role only.", "display_name": "Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg", "name": "zookeeper_config_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SERVER_role_env_safety_valve", "value": null }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "zookeeper_server_gc_duration_window", "value": "5" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_zookeeper_server_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Transaction Log Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Transaction Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Transaction Log Directory Free Space Monitoring Percentage Thresholds", "name": "zookeeper_server_data_log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Log Directory parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Log Directory", "name": "role_config_suppression_zk_server_log_dir", "value": "false" }, { "desc": "Enables authentication when interacting with the JMX agent on the ZooKeeper server.", "display_name": "Enable Authenticated Communication with the JMX Agent", "name": "enable_jmx_authentication", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg parameter.", "display_name": "Suppress Parameter Validation: Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg", "name": "role_config_suppression_zookeeper_config_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_zookeeper_server_unexpected_exits", "value": "false" }, { "desc": "Specifies the password of the user that has read-write privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Password of user with read-write access to the JMX agent", "name": "jmx_passwd_file_readwrite_user_password", "value": "CONTROL" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Data Directory Free Space", "name": "role_health_suppression_zookeeper_server_data_directory_free_space", "value": "false" }, { "desc": "The address (IPv4, IPv6, or hostname) to monitor for client connections. This is the address that clients attempt to connect to. This setting is optional, because by default, ZooKeeper binds in such a way that any connection to the client port for any address/interface/NIC on the server will be accepted.", "display_name": "Client Port Address", "name": "clientPortAddress", "value": null }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_zookeeper_server_gc_duration", "value": "false" }, { "desc": "Whether to suppress the results of the Outstanding Requests heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Outstanding Requests", "name": "role_health_suppression_zookeeper_server_outstanding_requests", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Data Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Data Directory Free Space Monitoring Percentage Thresholds", "name": "zookeeper_server_data_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Transaction Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Transaction Log Directory Free Space", "name": "role_health_suppression_zookeeper_server_data_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Port Address parameter.", "display_name": "Suppress Parameter Validation: Client Port Address", "name": "role_config_suppression_clientportaddress", "value": "false" }, { "desc": "The minimum log level for Server logs", "display_name": "Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The maximum session timeout, in milliseconds, that the ZooKeeper Server will allow the client to negotiate", "display_name": "Maximum Session Timeout", "name": "maxSessionTimeout", "value": "40000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Name of User with Read-Write Access to the JMX Agent parameter.", "display_name": "Suppress Parameter Validation: Name of User with Read-Write Access to the JMX Agent", "name": "role_config_suppression_jmx_passwd_file_readwrite_user", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Zookeeper Server", "name": "zk_server_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The tolerance window that will be used in the detection of a ZooKeeper server's membership in a quorum. Specified in minutes.", "display_name": "Quorum Membership Detection Window", "name": "zookeeper_server_quorum_membership_detection_window", "value": "3" }, { "desc": "Whether to suppress the results of the Connection Count heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Connection Count", "name": "role_health_suppression_zookeeper_server_connection_count", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The period to review when computing the moving average of the connection count. Specified in minutes.", "display_name": "ZooKeeper Server Connection Count Monitoring Period", "name": "zookeeper_server_connection_count_window", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The disk location that ZooKeeper will use to store its database snapshots.", "display_name": "Data Directory", "name": "dataDir", "value": "/var/lib/zookeeper" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_zookeeper_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Data Directory parameter.", "display_name": "Suppress Parameter Validation: Data Directory", "name": "role_config_suppression_datadir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Password of user with read-write access to the JMX agent parameter.", "display_name": "Suppress Parameter Validation: Password of user with read-write access to the JMX agent", "name": "role_config_suppression_jmx_passwd_file_readwrite_user_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Name of User with Read-Only access to the JMX Agent parameter.", "display_name": "Suppress Parameter Validation: Name of User with Read-Only access to the JMX Agent", "name": "role_config_suppression_jmx_passwd_file_readonly_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Zookeeper Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Zookeeper Server", "name": "role_config_suppression_zk_server_java_opts", "value": "false" }, { "desc": "Directory where ZooKeeper will place its log files.", "display_name": "ZooKeeper Log Directory", "name": "zk_server_log_dir", "value": "/var/log/zookeeper" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The port used by the ZooKeeper Server's RMI server to handle JMX RMI requests. This is added as \"-Dcom.sun.management.jmxremote.rmi.port=\" to the ZooKeeper Server's JVM command line. This has an effect only in Oracle JDK 7u4 and higher. If the setting is left blank, the JMX Remote Port value is used. If set to 0 or -1, this setting is ignored. When this setting is not in effect, the JVM uses a random port for the RMI server.", "display_name": "JMX RMI Server Port", "name": "server_jmx_rmi_port", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Password of User with Read-Only Access to the JMX agent parameter.", "display_name": "Suppress Parameter Validation: Password of User with Read-Only Access to the JMX agent", "name": "role_config_suppression_jmx_passwd_file_readonly_user_password", "value": "false" }, { "desc": "Specifies the password of the user that has read-only privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Password of User with Read-Only Access to the JMX agent", "name": "jmx_passwd_file_readonly_user_password", "value": "MONITOR" }, { "desc": "The maximum number of rolled log files to keep for Server logs. Typically used by log4j or logback.", "display_name": "Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The percentage thresholds of the ratio of the maximum request latency to the maximum client-negotiable session timeout since the server was started.", "display_name": "Maximum Latency Monitoring Thresholds", "name": "zookeeper_server_max_latency_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"75.0\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_zookeeper_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The port used by the ZooKeeper Server's RMI registry. This is required to enable JMX access through RMI which is required for Cloudera Manager ZooKeeper monitoring. This is added as \"-Dcom.sun.management.jmxremote.port\" to the ZooKeeper Server's JVM command line.", "display_name": "JMX Remote Port", "name": "server_jmx_agent_port", "value": "9010" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Specifies the name of the user that has read-write privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Name of User with Read-Write Access to the JMX Agent", "name": "jmx_passwd_file_readwrite_user", "value": "controlRole" }, { "desc": "The maximum number of concurrent connections (at the socket level) that a single client, identified by the IP address, may make to a single member of the ZooKeeper ensemble. This setting is used to prevent certain classes of DoS attacks, including file descriptor exhaustion. To remove the limit on concurrent connections, set this value to 0.", "display_name": "Maximum Client Connections", "name": "maxClientCnxns", "value": "60" }, { "desc": "When computing the overall Server health, consider the host's health.", "display_name": "Server Host Health Test", "name": "zookeeper_server_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Transaction Log Directory.", "display_name": "Transaction Log Directory Free Space Monitoring Absolute Thresholds", "name": "zookeeper_server_data_log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/spark-gateway.json0000666000175100017510000002101513245514472025153 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-env.sh.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_client_config_safety_valve", "value": null }, { "desc": "Which deploy mode to use by default. Can be overridden by users when launching applications.", "display_name": "Default Application Deploy Mode", "name": "spark_deploy_mode", "value": "client" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "51" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The minimum log level for the Spark shell.", "display_name": "Shell Logging Threshold", "name": "spark_gateway_shell_logging_threshold", "value": "WARN" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "When dynamic allocation is enabled, time after which idle executors will be stopped.", "display_name": "Executor Idle Timeout", "name": "spark_dynamic_allocation_idle_timeout", "value": "60" }, { "desc": "When dynamic allocation is enabled, timeout before requesting new executors after the initial backlog timeout has already expired. By default this is the same value as the initial backlog timeout.", "display_name": "Sustained Scheduler Backlog Timeout", "name": "spark_dynamic_allocation_sustained_scheduler_backlog_timeout", "value": null }, { "desc": "Name of class implementing org.apache.spark.serializer.Serializer to use in Spark applications.", "display_name": "Spark Data Serializer", "name": "spark_data_serializer", "value": "org.apache.spark.serializer.KryoSerializer" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Enables the external shuffle service. The external shuffle service preserves shuffle files written by executors so that the executors can be deallocated without losing work. Must be enabled if Enable Dynamic Allocation is enabled. Recommended and enabled by default for CDH 5.5 and higher.", "display_name": "Enable Shuffle Service", "name": "spark_shuffle_service_enabled", "value": "true" }, { "desc": "When dynamic allocation is enabled, number of executors to allocate when the application starts. By default, this is the same value as the minimum number of executors.", "display_name": "Initial Executor Count", "name": "spark_dynamic_allocation_initial_executors", "value": null }, { "desc": "When dynamic allocation is enabled, time after which idle executors with cached RDDs blocks will be stopped. By default, they're never stopped. This configuration is only available starting in CDH 5.5.", "display_name": "Caching Executor Idle Timeout", "name": "spark_dynamic_allocation_cached_idle_timeout", "value": null }, { "desc": "When dynamic allocation is enabled, minimum number of executors to keep alive while the application is running.", "display_name": "Minimum Executor Count", "name": "spark_dynamic_allocation_min_executors", "value": "0" }, { "desc": "Python library paths to add to PySpark applications.", "display_name": "Extra Python Path", "name": "spark_python_path", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "role_config_suppression_spark-conf/spark-env.sh_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "When dynamic allocation is enabled, maximum number of executors to allocate. By default, Spark relies on YARN to control the maximum number of executors for the application.", "display_name": "Maximum Executor Count", "name": "spark_dynamic_allocation_max_executors", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Extra Python Path parameter.", "display_name": "Suppress Parameter Validation: Extra Python Path", "name": "role_config_suppression_spark_python_path", "value": "false" }, { "desc": "Write Spark application history logs to HDFS.", "display_name": "Enable History", "name": "spark_history_enabled", "value": "true" }, { "desc": "When dynamic allocation is enabled, timeout before requesting new executors when there are backlogged tasks.", "display_name": "Scheduler Backlog Timeout", "name": "spark_dynamic_allocation_scheduler_backlog_timeout", "value": "1" }, { "desc": "Enable dynamic allocation of executors in Spark applications.", "display_name": "Enable Dynamic Allocation", "name": "spark_dynamic_allocation_enabled", "value": "true" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/spark" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Data Serializer parameter.", "display_name": "Suppress Parameter Validation: Spark Data Serializer", "name": "role_config_suppression_spark_data_serializer", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", "name": "role_config_suppression_spark-conf/spark-defaults.conf_client_config_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-defaults.conf.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", "name": "spark-conf/spark-defaults.conf_client_config_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-service.json0000666000175100017510000026331413245514472024770 0ustar zuulzuul00000000000000[ { "desc": "Allows the flume user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Flume Proxy User Groups", "name": "flume_proxy_user_groups_list", "value": "*" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "Hadoop User Group Mapping LDAP URL", "name": "hadoop_group_mapping_ldap_url", "value": null }, { "desc": "Whether to suppress the results of the Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Free Space", "name": "service_health_suppression_hdfs_free_space_remaining", "value": "false" }, { "desc": "The maximal block replication.", "display_name": "Maximal Block Replication", "name": "dfs_replication_max", "value": "512" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Whether or not to use TLS/SSL when connecting to the LDAP server.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Enabled", "name": "hadoop_group_mapping_ldap_use_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Balancer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Balancer Count Validator", "name": "service_config_suppression_balancer_count_validator", "value": "false" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hadoop-hdfs/audit" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Groups", "name": "service_config_suppression_http_proxy_user_groups_list", "value": "false" }, { "desc": "File path to a jks-format truststore containing the TLS/SSL certificate used sign the LDAP server's certificate. Note that in previous releases this was erroneously referred to as a \"keystore\".", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Truststore", "name": "hadoop_group_mapping_ldap_keystore", "value": "" }, { "desc": "SSH connection timeout, in milliseconds, to use with the built-in sshfence fencer.", "display_name": "Timeout for SSH Fencing Strategy", "name": "dfs_ha_fencing_ssh_connect_timeout", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Groups", "name": "service_config_suppression_mapred_proxy_user_groups_list", "value": "false" }, { "desc": "Comma-separated list of groups authorized to perform admin operations on Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Admin Groups", "name": "hadoop_authorized_admin_groups", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hdfs" }, { "desc": "Password for the server keystore file used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Trusted Kerberos Realms parameter.", "display_name": "Suppress Parameter Validation: Trusted Kerberos Realms", "name": "service_config_suppression_trusted_realms", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS replication jobs.", "display_name": "HDFS Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_replication_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Health Canary Directory Permissions parameter.", "display_name": "Suppress Parameter Validation: HDFS Health Canary Directory Permissions", "name": "service_config_suppression_firehose_hdfs_canary_directory_permissions", "value": "false" }, { "desc": "The health check thresholds of free space in HDFS. Specified as a percentage of total HDFS capacity.", "display_name": "HDFS Free Space Monitoring Thresholds", "name": "hdfs_free_space_thresholds", "value": "{\"critical\":\"10.0\",\"warning\":\"20.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_shell_cmd_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Groups", "name": "service_config_suppression_hue_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Groups", "name": "service_config_suppression_hive_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Single User Mode Overrides Validator configuration validator.", "display_name": "Suppress Configuration Validator: Single User Mode Overrides Validator", "name": "service_config_suppression_single_user_mode_override_validator", "value": "false" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Comma-delimited list of hosts where you want to allow the Cloudera Service Monitor user to impersonate other users. The default '*' allows all hosts. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'.", "display_name": "Service Monitor Proxy User Hosts", "name": "smon_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "service_config_suppression_hdfs_service_config_safety_valve", "value": "false" }, { "desc": "Comma separated list of users allowed to do short circuit read. A short circuit read allows a client co-located with the data to read HDFS file blocks directly from HDFS. If empty, will default to the DataNode process' user.", "display_name": "DataNode Local Path Access Users", "name": "dfs_block_local_path_access_user", "value": null }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Comma-delimited list of groups that you want to allow the Hive user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hive Proxy User Groups", "name": "hive_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Whether to suppress the results of the NameNode Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Health", "name": "service_health_suppression_hdfs_ha_namenode_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Hosts", "name": "service_config_suppression_yarn_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Hosts", "name": "service_config_suppression_http_proxy_user_hosts_list", "value": "false" }, { "desc": "Enable encryption of data transfer between DataNodes and clients, and among DataNodes. For effective data transfer protection, enable Kerberos authentication and choose Privacy Quality of RPC Protection.", "display_name": "Enable Data Transfer Encryption", "name": "dfs_encrypt_data_transfer", "value": "false" }, { "desc": "The Key Management Server used by HDFS. This must be set to use encryption for data at rest.", "display_name": "KMS Service", "name": "kms_service", "value": null }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to all roles and client configurations in this HDFS service as well as all its dependent services. Any configs added here will be overridden by their default values in HDFS (which can be found in hdfs-default.xml).", "display_name": "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "core_site_safety_valve", "value": null }, { "desc": "The amount of time after NameNode(s) start that the lack of an active NameNode will be tolerated. This is intended to allow either the auto-failover daemon to make a NameNode active, or a specifically issued failover command to take effect. This is an advanced option that does not often need to be changed.", "display_name": "NameNode Activation Startup Tolerance", "name": "hdfs_namenode_activation_startup_tolerance", "value": "180" }, { "desc": "

Note: Do not edit this property in the classic layout. Switch to the new layout to use preconfigured redaction rules and test your rules inline.

Use this property to define a list of rules to be followed for redacting sensitive information from log files and query strings. Click + to add a new redaction rule. You can choose one of the preconfigured rules or add a custom rule. When specifying a custom rule, the Search field should contain a regular expression that will be matched against the data. If a match is found, it is replaced by the contents of the Replace field.

Trigger is an optional field. It can be used to specify a simple string to be searched in the data. If the string is found, the redactor attempts to find a match for the Search regex. If no trigger is specified, redaction occurs by matching the Search regular expression. Use the Trigger field to enhance performance: simple string matching is faster than regular expression matching.

Test your rules by entering sample text into the Test Redaction Rules text box and clicking Test Redaction. If no rules match, the text you entered is returned unchanged.

", "display_name": "Log and Query Redaction Policy", "name": "redaction_policy", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Groups", "name": "service_config_suppression_hdfs_proxy_user_groups_list", "value": "false" }, { "desc": "The minimal block replication.", "display_name": "Minimal Block Replication", "name": "dfs_replication_min", "value": "1" }, { "desc": "The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory.", "display_name": "System User's Home Directory", "name": "hdfs_user_home_dir", "value": "/var/lib/hadoop-hdfs" }, { "desc": "If false, permission checking is turned off for files in HDFS.", "display_name": "Check HDFS Permissions", "name": "dfs_permissions", "value": "true" }, { "desc": "The domain to use for the HTTP cookie that stores the authentication token. In order for authentiation to work correctly across all Hadoop nodes' web-consoles the domain must be correctly set. Important: when using IP addresses, browsers ignore cookies with domain settings. For this setting to work properly all nodes in the cluster must be configured to generate URLs with hostname.domain names on it.", "display_name": "Hadoop HTTP Authentication Cookie Domain", "name": "hadoop_http_auth_cookie_domain", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP User Search Filter parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP User Search Filter", "name": "service_config_suppression_hadoop_group_mapping_ldap_user_filter", "value": "false" }, { "desc": "The service monitor will use this directory to create files to test if the hdfs service is healthy. The directory and files are created with permissions specified by 'HDFS Health Canary Directory Permissions'", "display_name": "HDFS Health Canary Directory", "name": "firehose_hdfs_canary_directory", "value": "/tmp/.cloudera_health_monitoring_canary_files" }, { "desc": "Whether to suppress configuration warnings produced by the Redaction Policy Validator configuration validator.", "display_name": "Suppress Configuration Validator: Redaction Policy Validator", "name": "service_config_suppression_redaction_policy_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Corrupt Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Corrupt Blocks", "name": "service_health_suppression_hdfs_blocks_with_corrupt_replicas", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_replication_env_safety_valve", "value": "false" }, { "desc": "Name of the ZooKeeper service that this HDFS service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "The health check thresholds of the number of missing blocks. Specified as a percentage of the total number of blocks.", "display_name": "Missing Block Monitoring Thresholds", "name": "hdfs_missing_blocks_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health check thresholds of the number of blocks that have at least one corrupt replica. Specified as a percentage of the total number of blocks.", "display_name": "Blocks With Corrupt Replicas Monitoring Thresholds", "name": "hdfs_blocks_with_corrupt_replicas_thresholds", "value": "{\"critical\":\"1.0\",\"warning\":\"0.5\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Hosts", "name": "service_config_suppression_oozie_proxy_user_hosts_list", "value": "false" }, { "desc": "The SSH private key files to use with the built-in sshfence fencer. These are to be accessible to the hdfs user on the machines running the NameNodes.", "display_name": "Private Keys for SSH Fencing Strategy", "name": "dfs_ha_fencing_ssh_private_key_files", "value": null }, { "desc": "ACLs (Access Control Lists) enhance the existing HDFS permission model to support controlling file access for arbitrary combinations of users and groups instead of a single owner, single group, and all other users. When ACLs are disabled, the NameNode rejects all attempts to set an ACL.", "display_name": "Enable Access Control Lists", "name": "dfs_namenode_acls_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Path on the DataNode's local file system to a UNIX domain socket used for communication between the DataNode and local HDFS clients. This socket is used for Short Circuit Reads. Only the HDFS System User and \"root\" should have write access to the parent directory and all of its ancestors. This property is supported in CDH 4.2 or later deployments.", "display_name": "UNIX Domain Socket path", "name": "dfs_domain_socket_path", "value": "/var/run/hdfs-sockets/dn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Hosts", "name": "service_config_suppression_httpfs_proxy_user_hosts_list", "value": "false" }, { "desc": "Distinguished name of the user to bind as. This is used to connect to LDAP/AD for searching user and group information. This may be left blank if the LDAP server supports anonymous binds.", "display_name": "Hadoop User Group Mapping LDAP Bind User Distinguished Name", "name": "hadoop_group_mapping_ldap_bind_user", "value": "" }, { "desc": "Typically, HDFS clients and servers communicate by opening sockets via an IP address. In certain networking configurations, it is preferable to open sockets after doing a DNS lookup on the hostname. Enable this property to open sockets after doing a DNS lookup on the hostname. This property is supported in CDH3u4 or later deployments.", "display_name": "Use DataNode Hostname", "name": "dfs_client_use_datanode_hostname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Authentication And Authorization Validation configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Authentication And Authorization Validation", "name": "service_config_suppression_hdfs_authentication_and_authorization_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop HTTP Authentication Cookie Domain parameter.", "display_name": "Suppress Parameter Validation: Hadoop HTTP Authentication Cookie Domain", "name": "service_config_suppression_hadoop_http_auth_cookie_domain", "value": "false" }, { "desc": "Whether to suppress the results of the Under-Replicated Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Under-Replicated Blocks", "name": "service_health_suppression_hdfs_under_replicated_blocks", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Admin Groups parameter.", "display_name": "Suppress Parameter Validation: Authorized Admin Groups", "name": "service_config_suppression_hadoop_authorized_admin_groups", "value": "false" }, { "desc": "Quality of protection for secured RPC connections between NameNode and HDFS clients. For effective RPC protection, enable Kerberos authentication.", "display_name": "Hadoop RPC Protection", "name": "hadoop_rpc_protection", "value": "authentication" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Replication Factor", "name": "service_config_suppression_dfs_replication", "value": "false" }, { "desc": "A list of path prefixes that define the HDFS regions where authorization is enforced using Sentry information. Only relevant when Sentry Synchronization is enabled.", "display_name": "Sentry Synchronization Path Prefixes", "name": "hdfs_sentry_sync_path_prefixes", "value": "/user/hive/warehouse" }, { "desc": "For advanced use only. Key-value pairs (one on each line) to be inserted into the HDFS replication configuration for hadoop-env.sh.", "display_name": "HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "hdfs_replication_haoop_env_sh_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the FailoverProxyProvider Class parameter.", "display_name": "Suppress Parameter Validation: FailoverProxyProvider Class", "name": "service_config_suppression_dfs_ha_proxy_provider", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • username: treats the field value as a user name, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare HDFS events:\n

\n\n
    \n
  • operation: the HDFS operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • src: the source path for the operation.
  • \n
  • dest: the destination path for the operation.
  • \n
  • permissions: the permissions associated with the operation.
  • \n
\n\n

\nThe default event tracker for HDFS services defines equality by comparing the\nusername, operation, and source path of the events.\n

\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": "{\n \"comment\" : [\n \"The default event tracker for HDFS services defines equality by \",\n \"comparing the username, operation, and source path of the events.\"\n ],\n \"timeToLive\" : 60000,\n \"fields\" : [\n { \"type\": \"value\", \"name\" : \"src\" },\n { \"type\": \"value\", \"name\" : \"operation\" },\n { \"type\": \"username\", \"name\" : \"username\" }\n ]\n}\n" }, { "desc": "The user the management services impersonates when connecting to HDFS. If no value is specified, the HDFS superuser is used.", "display_name": "HDFS User to Impersonate", "name": "hdfs_user_to_impersonate", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Hosts", "name": "service_config_suppression_mapred_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Groups", "name": "service_config_suppression_flume_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "service_config_suppression_hadoop_policy_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Users parameter.", "display_name": "Suppress Parameter Validation: Authorized Users", "name": "service_config_suppression_hadoop_authorized_users", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS snapshot shell command.", "display_name": "HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_shell_cmd_env_safety_valve", "value": null }, { "desc": "Default block replication. The number of replications to make when the file is created. The default value is used if a replication number is not specified.", "display_name": "Replication Factor", "name": "dfs_replication", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Authorization Provider Group parameter.", "display_name": "Suppress Parameter Validation: Sentry Authorization Provider Group", "name": "service_config_suppression_sentry_authorization_provider_hdfs_group", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Comma-delimited list of groups to allow the HDFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "HDFS Proxy User Groups", "name": "hdfs_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS High Availability Fencing Methods parameter.", "display_name": "Suppress Parameter Validation: HDFS High Availability Fencing Methods", "name": "service_config_suppression_dfs_ha_fencing_methods", "value": "false" }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "The health test thresholds of the overall DataNode health. The check returns \"Concerning\" health if the percentage of \"Healthy\" DataNodes falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" DataNodes falls below the critical threshold.", "display_name": "Healthy DataNode Monitoring Thresholds", "name": "hdfs_datanodes_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. This is used by WebHCat.", "display_name": "HTTP Proxy User Hosts", "name": "HTTP_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Log and Query Redaction Policy parameter.", "display_name": "Suppress Parameter Validation: Log and Query Redaction Policy", "name": "service_config_suppression_redaction_policy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Auto Failover Validator configuration validator.", "display_name": "Suppress Configuration Validator: Auto Failover Validator", "name": "service_config_suppression_auto_failover_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the UNIX Domain Socket path parameter.", "display_name": "Suppress Parameter Validation: UNIX Domain Socket path", "name": "service_config_suppression_dfs_domain_socket_path", "value": "false" }, { "desc": "The default block size in bytes for new HDFS files. Note that this value is also used as the HBase Region Server HLog block size.", "display_name": "HDFS Block Size", "name": "dfs_block_size", "value": "134217728" }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Heap Size Validator", "name": "service_config_suppression_nameservice_namenodes_heap_size_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-Wide Default TLS/SSL Client Truststore Location parameter.", "display_name": "Suppress Parameter Validation: Cluster-Wide Default TLS/SSL Client Truststore Location", "name": "service_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Enables Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Note: This is effective only if Kerberos is enabled for the HDFS service.", "display_name": "Enable Kerberos Authentication for HTTP Web-Consoles", "name": "hadoop_secure_web_ui", "value": "false" }, { "desc": "Password for the TLS/SSL client truststore. Defines a cluster-wide default that can be overridden by individual services.", "display_name": "Cluster-Wide Default TLS/SSL Client Truststore Password", "name": "ssl_client_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Count Validator", "name": "service_config_suppression_namenode_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Proxy User Hosts", "name": "service_config_suppression_smon_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Membership Attribute parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Membership Attribute", "name": "service_config_suppression_hadoop_group_mapping_ldap_member_attr", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Health Canary Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Health Canary Directory", "name": "service_config_suppression_firehose_hdfs_canary_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Proxy User Groups", "name": "service_config_suppression_smon_proxy_user_groups_list", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the mapred user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Mapred Proxy User Hosts", "name": "mapred_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Synchronization Path Prefixes parameter.", "display_name": "Suppress Parameter Validation: Sentry Synchronization Path Prefixes", "name": "service_config_suppression_hdfs_sentry_sync_path_prefixes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Bind User Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Bind User Password", "name": "service_config_suppression_hadoop_group_mapping_ldap_bind_passwd", "value": "false" }, { "desc": "Enable WebHDFS interface", "display_name": "Enable WebHDFS", "name": "dfs_webhdfs_enabled", "value": "true" }, { "desc": "The password for the TLS/SSL truststore.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Truststore Password", "name": "hadoop_group_mapping_ldap_keystore_passwd", "value": "" }, { "desc": "Comma-delimited list of hosts where you allow the HttpFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HttpFS Proxy User Hosts", "name": "httpfs_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress the results of the Failover Controllers Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Failover Controllers Health", "name": "service_health_suppression_hdfs_failover_controllers_healthy", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_service_env_safety_valve", "value": null }, { "desc": "Comma-delimited list of groups to allow the HttpFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "HttpFS Proxy User Groups", "name": "httpfs_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Hosts", "name": "service_config_suppression_hive_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Groups", "name": "service_config_suppression_oozie_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Bind User Distinguished Name parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Bind User Distinguished Name", "name": "service_config_suppression_hadoop_group_mapping_ldap_bind_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Superuser Group parameter.", "display_name": "Suppress Parameter Validation: Superuser Group", "name": "service_config_suppression_dfs_permissions_supergroup", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Groups", "name": "service_config_suppression_yarn_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP TLS/SSL Truststore Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP TLS/SSL Truststore Password", "name": "service_config_suppression_hadoop_group_mapping_ldap_keystore_passwd", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hive Proxy User Hosts", "name": "hive_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "service_config_suppression_hdfs_ssl_client_safety_valve", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the YARN user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "YARN Proxy User Groups", "name": "yarn_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_core_site_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Count Validator", "name": "service_config_suppression_datanode_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP TLS/SSL Truststore parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP TLS/SSL Truststore", "name": "service_config_suppression_hadoop_group_mapping_ldap_keystore", "value": "false" }, { "desc": "Whether to suppress the results of the DataNode Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: DataNode Health", "name": "service_health_suppression_hdfs_data_nodes_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Rules to Map Kerberos Principals to Short Names parameter.", "display_name": "Suppress Parameter Validation: Additional Rules to Map Kerberos Principals to Short Names", "name": "service_config_suppression_extra_auth_to_local_rules", "value": "false" }, { "desc": "List of Kerberos realms that Hadoop services should trust. If empty, defaults to the default_realm property configured in the krb5.conf file. After changing this value and restarting the service, all services depending on this service must also be restarted. Adds mapping rules for each domain to the hadoop.security.auth_to_local property in core-site.xml.", "display_name": "Trusted Kerberos Realms", "name": "trusted_realms", "value": "" }, { "desc": "Enable authorization", "display_name": "Hadoop Secure Authorization", "name": "hadoop_security_authorization", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "hdfs_ssl_server_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Mountpoints Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Mountpoints Validator", "name": "service_config_suppression_nameservice_mountpoints_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Hosts", "name": "service_config_suppression_hdfs_proxy_user_hosts_list", "value": "false" }, { "desc": "The tolerance window that will be used in HDFS service tests that depend on detection of the active NameNode.", "display_name": "Active NameNode Detection Window", "name": "hdfs_active_namenode_detecton_window", "value": "3" }, { "desc": "Comma-delimited list of groups that you want to allow the mapred user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Mapred Proxy User Groups", "name": "mapred_proxy_user_groups_list", "value": "*" }, { "desc": "The health check thresholds of the number of under-replicated blocks. Specified as a percentage of the total number of blocks.", "display_name": "Under-replicated Block Monitoring Thresholds", "name": "hdfs_under_replicated_blocks_thresholds", "value": "{\"critical\":\"40.0\",\"warning\":\"10.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_hdfs_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Missing Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Missing Blocks", "name": "service_health_suppression_hdfs_missing_blocks", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-Wide Default TLS/SSL Client Truststore Password parameter.", "display_name": "Suppress Parameter Validation: Cluster-Wide Default TLS/SSL Client Truststore Password", "name": "service_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "dfs.socket.timeout3000dfs.datanode.socket.write.timeout3000ipc.client.connect.max.retries1fs.permissions.umask-mode000" }, { "desc": "Comma-separated list of users authorized to perform admin operations on Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Admin Users", "name": "hadoop_authorized_admin_users", "value": "*" }, { "desc": "Comma-delimited list of hosts where you want to allow the HDFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HDFS Proxy User Hosts", "name": "hdfs_proxy_user_hosts_list", "value": "*" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "hdfs_service_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Checkpoint Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Checkpoint Configuration Validator", "name": "service_config_suppression_nameservice_checkpoint_configuration_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_service_env_safety_valve", "value": "false" }, { "desc": "The short name of the Hue Kerberos principal. Normally, you do not need to specify this configuration. Cloudera Manager auto-configures this property so that Hue and Cloudera Manamgent Service work properly.", "display_name": "Hue's Kerberos Principal Short Name", "name": "hue_kerberos_principal_shortname", "value": null }, { "desc": "Maximum bandwidth used for image transfer in bytes per second. This can help keep normal NameNode operations responsive during checkpointing. A default value of 0 indicates that throttling is disabled.", "display_name": "FsImage Transfer Bandwidth", "name": "dfs_image_transfer_bandwidthPerSec", "value": "0" }, { "desc": "The name of the system group shared by all the core Hadoop services.", "display_name": "Shared Hadoop Group Name", "name": "hdfs_hadoop_group_name", "value": "hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Private Keys for SSH Fencing Strategy parameter.", "display_name": "Suppress Parameter Validation: Private Keys for SSH Fencing Strategy", "name": "service_config_suppression_dfs_ha_fencing_ssh_private_key_files", "value": "false" }, { "desc": "The name of the group of superusers.", "display_name": "Superuser Group", "name": "dfs_permissions_supergroup", "value": "supergroup" }, { "desc": "Enables the health check that verifies that the failover controllers associated with this service are healthy and running.", "display_name": "Failover Controllers Healthy", "name": "failover_controllers_healthy_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS User to Impersonate parameter.", "display_name": "Suppress Parameter Validation: HDFS User to Impersonate", "name": "service_config_suppression_hdfs_user_to_impersonate", "value": "false" }, { "desc": "Enable TLS/SSL encryption for HDFS, MapReduce, and YARN web UIs, as well as encrypted shuffle for MapReduce and YARN.", "display_name": "Hadoop TLS/SSL Enabled", "name": "hdfs_hadoop_ssl_enabled", "value": "false" }, { "desc": "Enables DataNode support for the experimental DistributedFileSystem.getFileVBlockStorageLocations API. Applicable to CDH 4.1 and onwards.", "display_name": "Enable HDFS Block Metadata API", "name": "dfs_datanode_hdfs_blocks_metadata_enabled", "value": "true" }, { "desc": "Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Oozie Proxy User Hosts", "name": "oozie_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Groups", "name": "service_config_suppression_httpfs_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codecs parameter.", "display_name": "Suppress Parameter Validation: Compression Codecs", "name": "service_config_suppression_io_compression_codecs", "value": "false" }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted shuffle and encrypted web UIs. Applies to all configurations of daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Timeout in milliseconds for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations(). This value is only emitted for Impala.", "display_name": "HDFS File Block Storage Location Timeout", "name": "dfs_client_file_block_storage_locations_timeout", "value": "10000" }, { "desc": "Comma-delimited list of hosts where you want to allow the flume user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Flume Proxy User Hosts", "name": "flume_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue's Kerberos Principal Short Name parameter.", "display_name": "Suppress Parameter Validation: Hue's Kerberos Principal Short Name", "name": "service_config_suppression_hue_kerberos_principal_shortname", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Admin Users parameter.", "display_name": "Suppress Parameter Validation: Authorized Admin Users", "name": "service_config_suppression_hadoop_authorized_admin_users", "value": "false" }, { "desc": "Allows the Cloudera Service Monitor user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "Service Monitor Proxy User Groups", "name": "smon_proxy_user_groups_list", "value": "*" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Enable automatic synchronization of HDFS ACLs with Sentry privileges. HDFS Access Control Lists and Check HDFS Permissions must be enabled when this feature is enabled. Use Sentry Synchronization Path Prefixes to define the HDFS regions where authorization is enforced using Sentry information. For more information, see Synchronizing HDFS ACLs and Sentry Authorization.", "display_name": "Enable Sentry Synchronization", "name": "hdfs_sentry_sync_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP URL parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP URL", "name": "service_config_suppression_hadoop_group_mapping_ldap_url", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping Search Base parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping Search Base", "name": "service_config_suppression_hadoop_group_mapping_ldap_base", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Path to the TLS/SSL client truststore file. Defines a cluster-wide default that can be overridden by individual services. This truststore must be in JKS format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. The contents of the truststore can be modified without restarting any roles. By default, changes to its contents are picked up within ten seconds. If not set, the default Java truststore is used to verify certificates.", "display_name": "Cluster-Wide Default TLS/SSL Client Truststore Location", "name": "ssl_client_truststore_location", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "service_config_suppression_hdfs_replication_haoop_env_sh_safety_valve", "value": "false" }, { "desc": "Algorithm to encrypt data transfer between DataNodes and clients, and among DataNodes. If 3des or rc4 are chosen, the entire communication is encrypted with that algorithm. In CDH 5.4 and higher, if AES/CTR/NoPadding is chosen, 3des is used for the initial key exchange, and then AES/CTR/NoPadding is used for the transfer of data. This is the most secure option, and is recommended for clusters running CDH 5.4 or higher. It also requires that the \"openssl-devel\" package be installed on all machines in the cluster. When this parameter is changed, a full, nonrolling restart of the cluster must be performed.", "display_name": "Data Transfer Encryption Algorithm", "name": "dfs_encrypt_data_transfer_algorithm", "value": "rc4" }, { "desc": "Comma-separated list of groups authorized to used Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Groups", "name": "hadoop_authorized_groups", "value": "" }, { "desc": "Enter a FailoverProxyProvider implementation to configure two URIs to connect to during fail-over. The first configured address is tried first, and on a fail-over event the other address is tried.", "display_name": "FailoverProxyProvider Class", "name": "dfs_ha_proxy_provider", "value": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" }, { "desc": "Default umask for file and directory creation, specified in an octal value (with a leading 0)", "display_name": "Default Umask", "name": "dfs_umaskmode", "value": "022" }, { "desc": "Comma-delimited list of hosts where you want to allow the Hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hue Proxy User Hosts", "name": "hue_proxy_user_hosts_list", "value": "*" }, { "desc": "For paths where authorization is enforced by Sentry Synchronization, file permissions will use this parameter as the group. This group should normally include the hive and impala users.", "display_name": "Sentry Authorization Provider Group", "name": "sentry_authorization_provider_hdfs_group", "value": "hive" }, { "desc": "Comma-separated list of compression codecs that can be used in job or map compression.", "display_name": "Compression Codecs", "name": "io_compression_codecs", "value": "org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec" }, { "desc": "Whether to suppress configuration warnings produced by the JournalNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: JournalNode Count Validator", "name": "service_config_suppression_journalnode_count_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The service monitor will use these permissions to create the directory and files to test if the hdfs service is healthy. Permissions are specified using the 10-character unix-symbolic format e.g. '-rwxr-xr-x'.", "display_name": "HDFS Health Canary Directory Permissions", "name": "firehose_hdfs_canary_directory_permissions", "value": "-rwxrwxrwx" }, { "desc": "Whether to suppress configuration warnings produced by the SecondaryNameNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: SecondaryNameNode Count Validator", "name": "service_config_suppression_secondarynamenode_count_validator", "value": "false" }, { "desc": "An additional filter to use when searching for LDAP users. The default will usually be appropriate for Active Directory installations. If connecting to a generic LDAP server, ''sAMAccountName'' will likely be replaced with ''uid''. {0} is a special string used to denote where the username fits into the filter.", "display_name": "Hadoop User Group Mapping LDAP User Search Filter", "name": "hadoop_group_mapping_ldap_user_filter", "value": "(&(objectClass=user)(sAMAccountName={0}))" }, { "desc": "The timeout, in milliseconds, to use with the Cloudera Manager agent-based fencer.", "display_name": "Timeout for Cloudera Manager Fencing Strategy", "name": "dfs_ha_fencing_cloudera_manager_timeout_millis", "value": "10000" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_datanode_read_shortcircuit", "value": "true" }, { "desc": "Comma-delimited list of hosts that you want to allow the YARN user to impersonate. The default '*' allows all hosts. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'.", "display_name": "YARN Proxy User Hosts", "name": "yarn_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Local Path Access Users parameter.", "display_name": "Suppress Parameter Validation: DataNode Local Path Access Users", "name": "service_config_suppression_dfs_block_local_path_access_user", "value": "false" }, { "desc": "Whether to suppress the results of the HDFS Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HDFS Canary", "name": "service_health_suppression_hdfs_canary_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User's Home Directory parameter.", "display_name": "Suppress Parameter Validation: System User's Home Directory", "name": "service_config_suppression_hdfs_user_home_dir", "value": "false" }, { "desc": "When computing the overall HDFS cluster health, consider the health of the standby NameNode.", "display_name": "Standby NameNode Health Check", "name": "hdfs_standby_namenodes_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Groups parameter.", "display_name": "Suppress Parameter Validation: Authorized Groups", "name": "service_config_suppression_hadoop_authorized_groups", "value": "false" }, { "desc": "The search base for the LDAP connection. This is a distinguished name, and will typically be the root of the LDAP directory.", "display_name": "Hadoop User Group Mapping Search Base", "name": "hadoop_group_mapping_ldap_base", "value": "" }, { "desc": "Comma-delimited list of groups that you want to allow the Hue user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hue Proxy User Groups", "name": "hue_proxy_user_groups_list", "value": "*" }, { "desc": "Comma-separated list of users authorized to used Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Users", "name": "hadoop_authorized_users", "value": "*" }, { "desc": "Enable/Disable the Log and Query Redaction Policy for this cluster.", "display_name": "Enable Log and Query Redaction", "name": "redaction_policy_enabled", "value": "false" }, { "desc": "The attribute of the group object that identifies the group name. The default will usually be appropriate for all LDAP systems.", "display_name": "Hadoop User Group Mapping LDAP Group Name Attribute", "name": "hadoop_group_mapping_ldap_group_name_attr", "value": "cn" }, { "desc": "Whether to suppress configuration warnings produced by the NFS High Availability Validator configuration validator.", "display_name": "Suppress Configuration Validator: NFS High Availability Validator", "name": "service_config_suppression_nfs_ha_validator", "value": "false" }, { "desc": "The amount of time to wait for HDFS filesystem image transfer from NameNode to complete.", "display_name": "FsImage Transfer Timeout", "name": "dfs_image_transfer_timeout", "value": "60000" }, { "desc": "List of fencing methods to use for service fencing. shell(./cloudera_manager_agent_fencer.py) is a fencing mechanism designed to use the Cloudera Manager agent. The sshfence method uses SSH. If using custom fencers (that may communicate with shared store, power units, or network switches), use the shell to invoke them.", "display_name": "HDFS High Availability Fencing Methods", "name": "dfs_ha_fencing_methods", "value": "shell(./cloudera_manager_agent_fencer.py)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Hadoop Group Name parameter.", "display_name": "Suppress Parameter Validation: Shared Hadoop Group Name", "name": "service_config_suppression_hdfs_hadoop_group_name", "value": "false" }, { "desc": "Additional mapping rules that will be inserted before rules generated from the list of trusted realms and before the default rule. After changing this value and restarting the service, any services depending on this one must be restarted as well. The hadoop.security.auth_to_local property is configured using this information.", "display_name": "Additional Rules to Map Kerberos Principals to Short Names", "name": "extra_auth_to_local_rules", "value": null }, { "desc": "The password of the bind user.", "display_name": "Hadoop User Group Mapping LDAP Bind User Password", "name": "hadoop_group_mapping_ldap_bind_passwd", "value": "" }, { "desc": "Choose the authentication mechanism used by Hadoop", "display_name": "Hadoop Secure Authentication", "name": "hadoop_security_authentication", "value": "simple" }, { "desc": "For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "hadoop_policy_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Failover Controller Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Failover Controller Count Validator", "name": "service_config_suppression_failovercontroller_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Hosts", "name": "service_config_suppression_flume_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HttpFS Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HttpFS Count Validator", "name": "service_config_suppression_httpfs_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Encryption Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Encryption Validator", "name": "service_config_suppression_hdfs_encryption_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Name Attribute parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Name Attribute", "name": "service_config_suppression_hadoop_group_mapping_ldap_group_name_attr", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the HTTP user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. This is used by WebHCat.", "display_name": "HTTP Proxy User Groups", "name": "HTTP_proxy_user_groups_list", "value": "*" }, { "desc": "Enables the health check that a client can create, read, write, and delete files", "display_name": "HDFS Canary Health Check", "name": "hdfs_canary_health_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into ssl-client.xml. Applies cluster-wide, but can be overridden by individual services.", "display_name": "HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "hdfs_ssl_client_safety_valve", "value": null }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "When computing the overall HDFS cluster health, consider the active NameNode's health", "display_name": "Active NameNode Role Health Check", "name": "hdfs_namenode_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "The group that this service's processes should run as (except the HttpFS server, which has its own group)", "display_name": "System Group", "name": "process_groupname", "value": "hdfs" }, { "desc": "The attribute of the group object that identifies the users that are members of the group. The default will usually be appropriate for any LDAP installation.", "display_name": "Hadoop User Group Mapping LDAP Group Membership Attribute", "name": "hadoop_group_mapping_ldap_member_attr", "value": "member" }, { "desc": "An additional filter to use when searching for groups.", "display_name": "Hadoop User Group Mapping LDAP Group Search Filter", "name": "hadoop_group_mapping_ldap_group_filter", "value": "(objectClass=group)" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for HDFS events:\n

\n\n
    \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • command: the HDFS operation being performed.
  • \n
  • src: the source path for the operation.
  • \n
  • dest: the destination path for the operation.
  • \n
  • permissions: the permissions associated with the operation.
  • \n
\n\n

\nThe default HDFS audit event filter discards events generated by the internal\nCloudera and Hadoop users (cloudera-scm, dr.who, hbase, hive, impala, mapred,\nsolr, and spark), 'ls' actions performed by the hdfs user, operations in the\nCloudera Hive canary directory, and events that affect files in the /tmp\ndirectory.\"\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default HDFS audit event filter discards events generated by the internal \",\n \"Cloudera and Hadoop users (cloudera-scm, dr.who, hbase, hive, impala, \",\n \"mapred, solr, and spark), 'ls' actions performed by the hdfs user, \",\n \"operations in the Cloudera Hive canary directory, and events that affect \",\n \"files in the /tmp directory.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n {\n \"name\" : \"username\",\n \"match\" : \"(?:cloudera-scm|dr.who|hbase|hive|impala|mapred|solr|spark)(?:/.+)?\"\n }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n {\n \"name\" : \"username\",\n \"match\" : \"(?:hdfs)(?:/.+)?\"\n },\n {\n \"name\" : \"operation\",\n \"match\" : \"(?:listStatus|listCachePools|listCacheDirectives|getfileinfo)\"\n }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/user/hue/\\\\.cloudera_manager_hive_metastore_canary(?:/.*)?\" }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/user/hue/\\\\.Trash/Current/user/hue/\\\\.cloudera_manager_hive_metastore_canary(?:/.*)?\" }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/tmp(?:/.*)?\" }\n ]\n }\n ]\n}\n" }, { "desc": "The user that this service's processes should run as (except the HttpFS server, which has its own user)", "display_name": "System User", "name": "process_username", "value": "hdfs" }, { "desc": "Class for user to group mapping (get groups for a given user).", "display_name": "Hadoop User Group Mapping Implementation", "name": "hadoop_security_group_mapping", "value": "org.apache.hadoop.security.ShellBasedUnixGroupsMapping" }, { "desc": "Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Oozie Proxy User Groups", "name": "oozie_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Search Filter parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Search Filter", "name": "service_config_suppression_hadoop_group_mapping_ldap_group_filter", "value": "false" }, { "desc": "If AES/CTR/NoPadding is chosen for the Data Transfer Encryption Algorithm, this specifies the length (in bits) of the AES key. When this parameter is changed, a full, non-rolling restart of the cluster must be performed.", "display_name": "Data Transfer Cipher Suite Key Strength", "name": "dfs_encrypt_data_transfer_cipher_keybits", "value": "256" }, { "desc": "SASL protection mode for secured connections to the DataNodes when reading or writing data.", "display_name": "DataNode Data Transfer Protection", "name": "dfs_data_transfer_protection", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Hosts", "name": "service_config_suppression_hue_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the NFS Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NFS Gateway Count Validator", "name": "service_config_suppression_nfsgateway_count_validator", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hbase-gateway.json0000666000175100017510000001504313245514472025121 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only, a string to be inserted into the client configuration for hbase-site.xml.", "display_name": "HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_client_config_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hbase-env.sh", "display_name": "HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh", "name": "hbase_client_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh parameter.", "display_name": "Suppress Parameter Validation: HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh", "name": "role_config_suppression_hbase_client_env_safety_valve", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hbase" }, { "desc": "Whether to enable interruption of RPC threads at the client. The default value of true enables primary RegionServers to access data from other regions' secondary replicas.", "display_name": "Enable Client RPC Threads Interruption", "name": "hbase_ipc_client_allowsInterrupt", "value": "true" }, { "desc": "If a multiget operation is performed with 'Consistency.TIMELINE', the read RPC is sent to the primary region server first. After this timeout, a parallel RPC for secondary region replicas is also sent if the primary does not respond. After this, the result is returned from whichever RPC is finished first. If the response returns from the primary region replica, that the data is the most recent. Result.isStale() API has been added to inspect the staleness. If the result is from a secondary region, Result.isStale() is set to true.", "display_name": "HBase Client Multiget Timeout For Secondary Region Replicas", "name": "hbase_client_primaryCallTimeout_multiget", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_client_config_safety_valve", "value": "false" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "If a get operation is performed with 'Consistency.TIMELINE', the read RPC is sent to the primary region server first. After this timeout, parallel RPC for secondary region replicas is also sent if the primary does not respond. After this, the result is returned from whichever RPC is finished first. If the response returns from the primary region replica, that data is the most recent. Result.isStale() API has been added to inspect the staleness. If the result is from a secondary region, Result.isStale() is set to true.", "display_name": "HBase Client Get Timeout For Secondary Region Replicas", "name": "hbase_client_primaryCallTimeout_get", "value": "10" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hbase_client_java_heapsize", "value": "268435456" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hbase_client_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hbase_client_java_opts", "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/sqoop-sqoop_server.json0000666000175100017510000005733713245514472026302 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_sqoop_server_unexpected_exits", "value": "false" }, { "desc": "Name of Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Name", "name": "sqoop_repository_database_name", "value": "sqoop" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SQOOP_SERVER_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_sqoop_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Maximum number of clients allowed to connect to the Sqoop 2 Server.", "display_name": "Maximum Client Connections", "name": "max_client_connections", "value": "10" }, { "desc": "User for Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database User", "name": "sqoop_repository_database_user", "value": "sa" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_sqoop_server_file_descriptor", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Sqoop 2 Server", "name": "sqoop_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Metastore Directory parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Metastore Directory", "name": "role_config_suppression_sqoop_metastore_data_dir", "value": "false" }, { "desc": "Directory where Sqoop 2 Server will place its log files.", "display_name": "Sqoop 2 Server Log Directory", "name": "sqoop_log_dir", "value": "/var/log/sqoop2" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Host parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Host", "name": "role_config_suppression_sqoop_repository_database_host", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database User parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database User", "name": "role_config_suppression_sqoop_repository_database_user", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Sqoop 2 Server logs. Typically used by log4j or logback.", "display_name": "Sqoop 2 Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Sqoop 2 Server in Bytes", "name": "sqoop_java_heapsize", "value": "1073741824" }, { "desc": "Host name of Sqoop repository database. If the port is non-default for your database type, use host:port notation. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Host", "name": "sqoop_repository_database_host", "value": "localhost" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The minimum log level for Sqoop 2 Server logs", "display_name": "Sqoop 2 Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_sqoop_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Password for Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Password", "name": "sqoop_repository_database_password", "value": "" }, { "desc": "Enables the health test that the Sqoop 2 Server's process state is consistent with the role configuration", "display_name": "Sqoop 2 Server Process Health Test", "name": "sqoop_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Log Directory", "name": "role_config_suppression_sqoop_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_sqoop_server_scm_health", "value": "false" }, { "desc": "When computing the overall Sqoop 2 Server health, consider the host's health.", "display_name": "Sqoop 2 Server Host Health Test", "name": "sqoop_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_sqoop_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Directory where the Sqoop 2 Server places its metastore data. This is used only when Sqoop Repository Database Type is Derby.", "display_name": "Sqoop 2 Server Metastore Directory", "name": "sqoop_metastore_data_dir", "value": "/var/lib/sqoop2" }, { "desc": "For advanced use only. A string to be inserted into sqoop.properties for this role only.", "display_name": "Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties", "name": "sqoop_config_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties", "name": "role_config_suppression_sqoop_config_safety_valve", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Sqoop 2 Server logs. Typically used by log4j or logback.", "display_name": "Sqoop 2 Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Sqoop 2 Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Sqoop 2 Server", "name": "role_config_suppression_sqoop_java_opts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_sqoop_server_role_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_sqoop_server_log_directory_free_space", "value": "false" }, { "desc": "Type of Sqoop repository database.", "display_name": "Sqoop Repository Database Type", "name": "sqoop_repository_database_type", "value": "derby" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Admin port of Sqoop 2 Server. (Note: This configuration only applies from CDH 4.3 onwards.)", "display_name": "Sqoop 2 Admin Port", "name": "sqoop_admin_port", "value": "8005" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "HTTP port of Sqoop 2 Server. (Note: This configuration only applies from CDH 4.3 onwards.)", "display_name": "Sqoop 2 HTTP Port", "name": "sqoop_http_port", "value": "12000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Password parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Password", "name": "role_config_suppression_sqoop_repository_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Name parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Name", "name": "role_config_suppression_sqoop_repository_database_name", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "sqoop_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/0000775000175100017510000000000013245515027022742 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/common/0000775000175100017510000000000013245515027024232 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/common/install_extjs0000666000175100017510000000115213245514476027050 0ustar zuulzuul00000000000000#!/bin/bash EXTJS_DESTINATION_DIR="/var/lib/oozie" EXTJS_DOWNLOAD_URL="http://tarballs.openstack.org/sahara/dist/common-artifacts/ext-2.2.zip" extjs_archive=/tmp/$(basename $EXTJS_DOWNLOAD_URL) if [ ! -n "$EXTJS_DESTINATION_DIR" ]; then if [ $test_only -eq 0 ]; then wget -O $extjs_archive $EXTJS_DOWNLOAD_URL mkdir -p $EXTJS_DESTINATION_DIR if [ -z "${EXTJS_NO_UNPACK:-}" ]; then unzip -d "$EXTJS_DESTINATION_DIR" $extjs_archive rm -f $extjs_archive else mv $extjs_archive $EXTJS_DESTINATION_DIR fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/common/add_jar0000666000175100017510000000132113245514476025547 0ustar zuulzuul00000000000000#!/usr/bin/env bash hadoop="2.6.0" HADOOP_SWIFT_JAR_NAME="hadoop-openstack-$hadoop-cdh$plugin_version.jar" HADOOP_SWIFT_BUILD_LOCATION="http://tarballs.openstack.org/sahara/dist/hadoop-openstack/master" SWIFT_LIB_URI="$HADOOP_SWIFT_BUILD_LOCATION/hadoop-openstack-${hadoop}.jar" HADOOP_SWIFT_JAR_NAME="hadoop-openstack.jar" if [ ! -f $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME ]; then if [ $test_only -eq 0 ]; then wget -O $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME $SWIFT_LIB_URI if [ $? -ne 0 ]; then echo -e "Could not download Swift Hadoop FS implementation.\nAborting" exit 1 fi chmod 0644 $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/common/install_cloudera0000666000175100017510000000130113245514472027501 0ustar zuulzuul00000000000000#!/bin/bash HADOOP_OPENSTACK_5_7_x_URL="https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-openstack/2.6.0-cdh$plugin_version/hadoop-openstack-2.6.0-cdh$plugin_version.jar" dest=/usr/lib/hadoop-mapreduce/hadoop-openstack.jar if [ ! -L "/usr/lib/oozie/oozie-sharelib-yarn.tar.gz" ]; then if [ $test_only -eq 0 ]; then wget -O $dest $HADOOP_OPENSTACK_5_7_x_URL ln -s /usr/lib/oozie/oozie-sharelib-yarn /usr/lib/oozie/oozie-sharelib-yarn.tar.gz ln -s /usr/lib/oozie/oozie-sharelib-mr1 /usr/lib/oozie/oozie-sharelib-mr1.tar.gz ln -s /usr/lib/oozie/oozie-sharelib-yarn.tar.gz /usr/lib/oozie/oozie-sharelib.tar.gz else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/common/unlimited_security_artifacts0000666000175100017510000000055513245514472032150 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -n /tmp/UnlimitedPolicy/ ]; then if [ $test_only -eq 0 ]; then mkdir /tmp/UnlimitedPolicy/ wget $unlimited_security_location/local_policy.jar -O /tmp/UnlimitedPolicy/local_policy.jar wget $unlimited_security_location/US_export_policy.jar -O /tmp/UnlimitedPolicy/US_export_policy.jar else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/image.yaml0000666000175100017510000000617513245514472024726 0ustar zuulzuul00000000000000arguments: java_distro: default: cloudera-jdk description: The distribution of Java to install. Defaults to openjdk. choices: - openjdk - oracle-java - cloudera-jdk plugin_version: default: 5.7.0 description: The distribution of CDH to install. Defaults to 5.7.0. hidden: True required: False choices: - 5.7.0 - 5.7.1 - 5.7.2 - 5.7.3 - 5.7.4 - 5.7.5 hdfs_lib_dir: default: /usr/lib/hadoop-mapreduce description: The path to HDFS_LIB_DIR. Default to /usr/lib/hadoop-mapreduce required: False unlimited_security_location: default: http://sahara-files.mirantis.com/kerberos-artifacts/ description: Place where UnlimitedSecurity polices are located required: False validators: - script: common/install_extjs - os_case: - centos: - package: wget - script: centos/wget_repo - ubuntu: - script: ubuntu/wget_repo: env_vars: [plugin_version] - argument_case: argument_name: java_distro cases: openjdk: - any: - all: - package: java-1.8.0-openjdk-devel - all: - package: java-1.7.0-openjdk-devel cloudera-jdk: - all: - package: oracle-j2sdk1.7 - package: ntp - package: - cloudera-manager-agent - cloudera-manager-daemons - cloudera-manager-server - cloudera-manager-server-db-2 - package: - hadoop-hdfs-namenode - hadoop-hdfs-datanode - hadoop-hdfs-secondarynamenode - hadoop-mapreduce - hadoop-mapreduce-historyserver - hadoop-yarn-nodemanager - hadoop-yarn-resourcemanager - package: - hbase - hbase-solr - package: - hive-hcatalog - hive-metastore - hive-server2 - hive-webhcat-server - hue - package: - oozie - spark-core - os_case: - centos: - package: spark-history-server - package: zookeeper - package: unzip - package: flume-ng - package: hadoop-kms - package: - impala - impala-server - impala-state-store - impala-catalog - impala-shell - package: keytrustee-keyprovider - package: - sentry - solr-server - solr-doc - search - sqoop2 - package: - kafka - kafka-server - script: common/install_cloudera: env_vars: [plugin_version] - os_case: - centos: - script: centos/turn_off_services - ubuntu: - script: ubuntu/turn_off_services - script: common/add_jar: env_vars: [plugin_version, hdfs_lib_dir] - script: common/unlimited_security_artifacts: env_vars: [unlimited_security_location] - os_case: - centos: - package: - krb5-server - krb5-libs - krb5-workstation - rng-tools - package: iptables-services - script: centos/selinux_permissive - script: centos/disable_firewall - package: nmap-ncat - ubuntu: - package: - krb5-admin-server - libpam-krb5 - ldap-utils - krb5-user - rng-tools sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/centos/0000775000175100017510000000000013245515027024235 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/centos/turn_off_services0000666000175100017510000000147013245514472027714 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files oozie.service | grep 'enabled' | wc -l) if [ "$check" != "on" ]; then if [ $test_only -eq 0 ]; then for i in cloudera-scm-agent \ cloudera-scm-server \ cloudera-scm-server-db \ hadoop-hdfs-datanode \ hadoop-hdfs-namenode \ hadoop-hdfs-secondarynamenode \ hadoop-mapreduce-historyserver \ hadoop-yarn-nodemanager \ hadoop-yarn-resourcemanager \ hive-metastore \ hive-server2 \ hive-webhcat-server \ hue \ oozie \ postgresql \ impala-catalog \ impala-server \ impala-state-store \ solr-server \ spark-history-server do chkconfig $i off done else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/centos/selinux_permissive0000666000175100017510000000046113245514472030123 0ustar zuulzuul00000000000000#!/bin/bash check=$(cat /etc/selinux/config | grep "SELINUX=permissive" | wc -l) echo $check if [ $check -eq 0 ]; then if [ $test_only -eq 0 ]; then echo "SELINUX=permissive" > /etc/selinux/config echo "SELINUXTYPE=targeted" >> /etc/selinux/config else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/centos/wget_repo0000666000175100017510000000423013245514472026157 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -f /etc/yum.repos.d/cloudera-cdh5.repo ]; then if [ $test_only -eq 0 ]; then echo '[cloudera-cdh5]' > /etc/yum.repos.d/cloudera-cdh5.repo echo "name=Cloudera's Distribution for Hadoop, Version 5" >> /etc/yum.repos.d/cloudera-cdh5.repo echo "baseurl=http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.7.0/" >> /etc/yum.repos.d/cloudera-cdh5.repo echo "gpgkey = http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-cdh5.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/cloudera-cdh5.repo echo '[cloudera-manager]' > /etc/yum.repos.d/cloudera-manager.repo echo 'name=Cloudera Manager' >> /etc/yum.repos.d/cloudera-manager.repo echo "baseurl=http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.7.0/" >> /etc/yum.repos.d/cloudera-manager.repo echo "gpgkey = http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-manager.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/cloudera-manager.repo echo '[navigator-keytrustee]' > /etc/yum.repos.d/kms.repo echo "name=Cloudera's Distribution for navigator-Keytrustee, Version 5" >> /etc/yum.repos.d/kms.repo echo "baseurl=http://archive.cloudera.com/navigator-keytrustee5/redhat/7/x86_64/navigator-keytrustee/5.7.0/" >> /etc/yum.repos.d/kms.repo echo "gpgkey = http://archive.cloudera.com/navigator-keytrustee5/redhat/7/x86_64/navigator-keytrustee/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/kms.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/kms.repo echo "[cloudera-kafka]" > /etc/yum.repos.d/cloudera-kafka.repo echo "name=Cloudera's Distribution for kafka, Version 2.0.2" >> /etc/yum.repos.d/cloudera-kafka.repo echo "baseurl=http://archive.cloudera.com/kafka/redhat/7/x86_64/kafka/2.0.2/" >> /etc/yum.repos.d/cloudera-kafka.repo echo "gpgkey = http://archive.cloudera.com/kafka/redhat/7/x86_64/kafka/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-kafka.repo echo "gpgcheck = 1" >> /etc/yum.repos.d/cloudera-kafka.repo yum clean all else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/centos/disable_firewall0000666000175100017510000000103513245514472027454 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files iptables.service | grep 'enabled' | wc -l) if [ $check -eq 1 ]; then if [ $test_only -eq 0 ]; then if type -p systemctl && [[ "$(systemctl --no-pager list-unit-files firewalld)" =~ 'enabled' ]]; then systemctl disable firewalld fi if type -p service; then service ip6tables save service iptables save chkconfig ip6tables off chkconfig iptables off fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/ubuntu/0000775000175100017510000000000013245515027024264 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/ubuntu/turn_off_services0000666000175100017510000000150013245514472027735 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files oozie.service | grep 'enabled' | wc -l) if [ "$check" != "on" ]; then if [ $test_only -eq 0 ]; then for i in cloudera-scm-agent \ cloudera-scm-server \ cloudera-scm-server-db \ hadoop-hdfs-datanode \ hadoop-hdfs-namenode \ hadoop-hdfs-secondarynamenode \ hadoop-mapreduce-historyserver \ hadoop-yarn-nodemanager \ hadoop-yarn-resourcemanager \ hive-metastore \ hive-server2 \ hive-webhcat-server \ hue \ oozie \ postgresql \ impala-catalog \ impala-server \ impala-state-store \ solr-server \ spark-history-server do update-rc.d -f $i remove done else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/images/ubuntu/wget_repo0000666000175100017510000000411013245514472026203 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -f /etc/apt/sources.list.d/cdh5.list ]; then if [ $test_only -eq 0 ]; then # Add repository with postgresql package (it's dependency of cloudera packages) # Base image doesn't contain this repo echo "deb http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty universe multiverse main" >> /etc/apt/sources.list # Cloudera repositories echo "deb [arch=amd64] http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh$plugin_version contrib" > /etc/apt/sources.list.d/cdh5.list echo "deb-src http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh$plugin_version contrib" >> /etc/apt/sources.list.d/cdh5.list wget -qO - http://archive-primary.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key | apt-key add - echo "deb [arch=amd64] http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm$plugin_version contrib" > /etc/apt/sources.list.d/cm5.list echo "deb-src http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm$plugin_version contrib" >> /etc/apt/sources.list.d/cm5.list wget -qO - http://archive-primary.cloudera.com/cm5/ubuntu/trusty/amd64/cm/archive.key | apt-key add - wget -O /etc/apt/sources.list.d/kms.list http://archive.cloudera.com/navigator-keytrustee5/ubuntu/trusty/amd64/navigator-keytrustee/cloudera.list wget -qO - http://archive.cloudera.com/navigator-keytrustee5/ubuntu/trusty/amd64/navigator-keytrustee/archive.key | apt-key add - # add Kafka repository echo 'deb http://archive.cloudera.com/kafka/ubuntu/trusty/amd64/kafka/ trusty-kafka2.0.2 contrib' >> /etc/apt/sources.list wget -qO - https://archive.cloudera.com/kafka/ubuntu/trusty/amd64/kafka/archive.key | apt-key add - #change repository priority echo 'Package: zookeeper' >> /etc/apt/preferences.d/cloudera-pin echo 'Pin: origin "archive.cloudera.com"' >> /etc/apt/preferences.d/cloudera-pin echo 'Pin-Priority: 1001' >> /etc/apt/preferences.d/cloudera-pin apt-get update else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/kms-kms.json0000666000175100017510000007672113245514472023774 0ustar zuulzuul00000000000000[ { "desc": "Comma-delimited list of hosts where you want to allow the hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hue Proxy User Hosts", "name": "hadoop_kms_proxyuser_hue_hosts", "value": "*" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kms_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "When computing the overall Key Management Server health, consider the host's health.", "display_name": "Key Management Server Host Health Test", "name": "kms_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hive_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hue_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the KMS Blacklist Users parameter.", "display_name": "Suppress Parameter Validation: KMS Blacklist Users", "name": "role_config_suppression_kms_blacklist_users", "value": "false" }, { "desc": "Allows the httpfs superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HttpFS Proxy User Groups", "name": "hadoop_kms_proxyuser_httpfs_groups", "value": "*" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_flume_groups", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kms_kms_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JavaKeyStoreProvider Directory parameter.", "display_name": "Suppress Parameter Validation: JavaKeyStoreProvider Directory", "name": "role_config_suppression_hadoop_security_key_provider_dir", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the mapred user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Mapred Proxy User Hosts", "name": "hadoop_kms_proxyuser_mapred_hosts", "value": "*" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kms_kms_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_yarn_groups", "value": "false" }, { "desc": "Allows the hdfs superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HDFS Proxy User Groups", "name": "hadoop_kms_proxyuser_hdfs_groups", "value": "*" }, { "desc": "Port used to access the KMS' embedded Tomcat admin console.", "display_name": "KMS Admin Port", "name": "kms_admin_port", "value": "16001" }, { "desc": "A comma-separated list of users (no spaces) for whom to disallow access to key material. These users can still fetch key metadata and create encrypted encryption keys, but are unable to do any other KMS operations. Typically, HDFS superusers will be specified here.", "display_name": "KMS Blacklist Users", "name": "kms_blacklist_users", "value": "" }, { "desc": "Comma-delimited list of hosts where you want to allow the yarn user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "YARN Proxy User Hosts", "name": "hadoop_kms_proxyuser_yarn_hosts", "value": "*" }, { "desc": "The maximum size, in megabytes, per log file for Key Management Server logs. Typically used by log4j or logback.", "display_name": "Key Management Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Comma-delimited list of hosts where you want to allow the hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hive Proxy User Hosts", "name": "hadoop_kms_proxyuser_hive_hosts", "value": "*" }, { "desc": "The minimum log level for Key Management Server logs", "display_name": "Key Management Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_http_hosts", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Key Management Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hive_groups", "value": "false" }, { "desc": "Allows the mapred superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Mapred Proxy User Groups", "name": "hadoop_kms_proxyuser_mapred_groups", "value": "*" }, { "desc": "Maximum number of threads used to handle KMS requests.", "display_name": "KMS Max Threads", "name": "kms_max_threads", "value": "250" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Key Management Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "KMS_role_env_safety_valve", "value": null }, { "desc": "Comma-delimited list of hosts where you want to allow the hdfs user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HDFS Proxy User Hosts", "name": "hadoop_kms_proxyuser_hdfs_hosts", "value": "*" }, { "desc": "The log directory for log files of the role Key Management Server.", "display_name": "Key Management Server Log Directory", "name": "log_dir", "value": "/var/log/hadoop-kms" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Key Management Server TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kms_kms_file_descriptor", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "For advanced use only. A string to be inserted into kms-acls.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml", "name": "kms-acls.xml_role_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hue_groups", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Oozie Proxy User Hosts", "name": "hadoop_kms_proxyuser_oozie_hosts", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hdfs_hosts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the httpfs user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HttpFS Proxy User Hosts", "name": "hadoop_kms_proxyuser_httpfs_hosts", "value": "*" }, { "desc": "Port used by clients to interact with the KMS.", "display_name": "KMS HTTP Port", "name": "kms_http_port", "value": "16000" }, { "desc": "Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Oozie Proxy User Groups", "name": "hadoop_kms_proxyuser_oozie_groups", "value": "*" }, { "desc": "Maximum heap size of the KMS.", "display_name": "KMS Heap Size", "name": "kms_heap_size", "value": "1073741824" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kms_kms_swap_memory_usage", "value": "false" }, { "desc": "Directory where configuration and binaries are staged before starting KMS. Does not normally need to be modified.", "display_name": "KMS Staging Directory", "name": "kms_staging_dir", "value": "/var/lib/hadoop-kms" }, { "desc": "Allows the hue superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hue Proxy User Groups", "name": "hadoop_kms_proxyuser_hue_groups", "value": "*" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_oozie_hosts", "value": "false" }, { "desc": "Allows the HTTP superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HTTP Proxy User Groups", "name": "hadoop_kms_proxyuser_HTTP_groups", "value": "*" }, { "desc": "For advanced use only. A string to be inserted into kms-site.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml", "name": "kms-site.xml_role_safety_valve", "value": null }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Key Management Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Key Management Server TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kms_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into core-site.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "core-site.xml_role_safety_valve", "value": null }, { "desc": "Allows the yarn superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "YARN Proxy User Groups", "name": "hadoop_kms_proxyuser_yarn_groups", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_http_groups", "value": "false" }, { "desc": "Allows the hive superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hive Proxy User Groups", "name": "hadoop_kms_proxyuser_hive_groups", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_yarn_hosts", "value": "false" }, { "desc": "Directory of the keystore file kms.keystore used by JavaKeyStoreProvider that backs the KMS.", "display_name": "JavaKeyStoreProvider Directory", "name": "hadoop_security_key_provider_dir", "value": "/var/lib/kms" }, { "desc": "Enables the health test that the Key Management Server's process state is consistent with the role configuration", "display_name": "Key Management Server Process Health Test", "name": "kms_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_flume_hosts", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Key Management Server logs. Typically used by log4j or logback.", "display_name": "Key Management Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "role_config_suppression_core-site.xml_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the KMS Staging Directory parameter.", "display_name": "Suppress Parameter Validation: KMS Staging Directory", "name": "role_config_suppression_kms_staging_dir", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HTTP Proxy User Hosts", "name": "hadoop_kms_proxyuser_HTTP_hosts", "value": "*" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_oozie_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hdfs_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_httpfs_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_mapred_groups", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kms_kms_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml", "name": "role_config_suppression_kms-acls.xml_role_safety_valve", "value": "false" }, { "desc": "Encrypt communication between clients and Key Management Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Key Management Server", "name": "ssl_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_httpfs_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Key Management Server TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Allows the flume superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Flume Proxy User Groups", "name": "hadoop_kms_proxyuser_flume_groups", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_mapred_hosts", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The password for the Key Management Server JKS keystore file.", "display_name": "Key Management Server TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml", "name": "role_config_suppression_kms-site.xml_role_safety_valve", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the flume user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Flume Proxy User Hosts", "name": "hadoop_kms_proxyuser_flume_hosts", "value": "*" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hbase-regionserver.json0000666000175100017510000016502613245514472026201 0ustar zuulzuul00000000000000[ { "desc": "If this number of HStoreFiles in any one HStore is exceeded, then a compaction is run to rewrite all HStoreFiles files as one HStoreFile. (One HStoreFile is written per flush of memstore.) You can delay compaction by specifying a larger number, but the compaction will take longer when it does run. During a compaction, updates cannot be flushed to disk. Long compactions require memory sufficient to carry the logging of all updates across the duration of the compaction. If too large, clients timeout during compaction.", "display_name": "HStore Compaction Threshold", "name": "hbase_hstore_compactionThreshold", "value": "3" }, { "desc": "Directory where RegionServer will place its log files.", "display_name": "RegionServer Log Directory", "name": "hbase_regionserver_log_dir", "value": "/var/log/hbase" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "regionserver_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_regionserver_config_safety_valve", "value": "false" }, { "desc": "The period of time that an HRegion will block updates after reaching the HStoreFile limit that is specified by 'hbase.hstore.blockingStoreFiles'. After this time has elapsed, the HRegion will stop blocking updates even if a compaction has not been completed.", "display_name": "HStore Blocking Wait Time", "name": "hbase_hstore_blockingWaitTime", "value": "90000" }, { "desc": "The period over which to compute the moving average of the compaction queue size.", "display_name": "HBase RegionServer Compaction Queue Monitoring Period", "name": "regionserver_compaction_queue_window", "value": "5" }, { "desc": "If the memstores in a region are this size or larger when closing, run a pre-flush process to clear out memstores before putting up the region closed flag and taking the region offline. On close, a flush process is run under the close flag up to empty memory. During this time, the region is offline and no writes are taken. If the memstore content is large, the flush process could take a long time to complete. The pre-flush process cleans out the bulk of the memstore before putting up the close flag and taking the region offline, so that the flush process that runs under the close flag has little to do.", "display_name": "HBase Memstore Pre-close Flush Size", "name": "hbase_hregion_preclose_flush_size", "value": "5242880" }, { "desc": "The health test thresholds of the average size of the HBase RegionServer flush queue over a recent period. See HBase RegionServer Flush Queue Monitoring Period.", "display_name": "HBase RegionServer Flush Queue Monitoring Thresholds", "name": "regionserver_flush_queue_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "Enables the health test that the RegionServer is connected to the Master", "display_name": "HBase RegionServer to Master Connectivity Test", "name": "regionserver_master_connectivity_enabled", "value": "true" }, { "desc": "When memstores are being forced to flush to make room in memory, keep flushing until this amount is reached. If this amount is equal to 'hbase.regionserver.global.memstore.upperLimit', then minimum possible flushing will occur when updates are blocked due to memstore limiting.", "display_name": "Low Watermark for Memstore Flush", "name": "hbase_regionserver_global_memstore_lowerLimit", "value": "0.38" }, { "desc": "The maximum number of rolled log files to keep for RegionServer logs. Typically used by log4j or logback.", "display_name": "RegionServer Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The address for the HBase RegionServer web UI", "display_name": "HBase RegionServer Web UI Address", "name": "hbase_regionserver_info_bindAddress", "value": null }, { "desc": "Whether to suppress the results of the Memstore Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Memstore Size", "name": "role_health_suppression_region_server_memstore_size", "value": "false" }, { "desc": "Number of handlers for processing priority requests in a RegionServer.", "display_name": "HBase RegionServer Meta-Handler Count", "name": "hbase_regionserver_metahandler_count", "value": "10" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": "32768" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HLog Reader Implementation parameter.", "display_name": "Suppress Parameter Validation: HLog Reader Implementation", "name": "role_config_suppression_hbase_regionserver_hlog_reader_impl", "value": "false" }, { "desc": "The port for the HBase RegionServer web UI. Set to -1 to disable RegionServer web UI.", "display_name": "HBase RegionServer Web UI port", "name": "hbase_regionserver_info_port", "value": "60030" }, { "desc": "The health test thresholds of the total size of RegionServer's memstores. Specified as a percentage of the configured upper limit. See Maximum Size of All Memstores in RegionServer.", "display_name": "HBase RegionServer Memstore Size Thresholds", "name": "regionserver_memstore_size_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"95.0\"}" }, { "desc": "Maximum size of all memstores in a RegionServer before new updates are blocked and flushes are forced.", "display_name": "Maximum Size of All Memstores in RegionServer", "name": "hbase_regionserver_global_memstore_upperLimit", "value": "0.4" }, { "desc": "List of coprocessors that are loaded by default on all tables. For any override coprocessor method, these classes will be called in order. After implementing your own coprocessor, just put it in HBase's classpath and add the fully qualified class name here. A coprocessor can also be loaded on demand by setting HTableDescriptor.", "display_name": "HBase Coprocessor Region Classes", "name": "hbase_coprocessor_region_classes", "value": "" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "RegionServer Environment Advanced Configuration Snippet (Safety Valve)", "name": "REGIONSERVER_role_env_safety_valve", "value": null }, { "desc": "Maximum number of HStoreFiles to compact per minor compaction.", "display_name": "Maximum Number of HStoreFiles Compaction", "name": "hbase_hstore_compaction_max", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase RegionServer", "name": "hbase_regionserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the RegionServer File Descriptor Limit Validator configuration validator.", "display_name": "Suppress Configuration Validator: RegionServer File Descriptor Limit Validator", "name": "role_config_suppression_regionserver_fd_limit_validator", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When using Multiple HDFS WAL as the WALProvider, sets how many write-ahead-logs each region server should run. Will result in this number of HDFS pipelines. Writes for a given Region only go to a single pipeline, spreading total RegionServer load.", "display_name": "Per-RegionServer Number of WAL Pipelines", "name": "hbase_regionserver_wal_pipelines", "value": "1" }, { "desc": "Timeout for Canary to perform its checks.", "display_name": "Canary Timeout", "name": "hbase_regionserver_canary_timeout", "value": "15" }, { "desc": "The maximum size, in megabytes, per log file for RegionServer logs. Typically used by log4j or logback.", "display_name": "RegionServer Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase RegionServer Web UI Address parameter.", "display_name": "Suppress Parameter Validation: HBase RegionServer Web UI Address", "name": "role_config_suppression_hbase_regionserver_info_bindaddress", "value": "false" }, { "desc": "Number of RPC Server instances spun up on RegionServers.", "display_name": "HBase RegionServer Handler Count", "name": "hbase_regionserver_handler_count", "value": "30" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period over which to compute the moving average of the HDFS read latency of the HBase RegionServer.", "display_name": "HBase RegionServer HDFS Read Latency Monitoring Period", "name": "regionserver_read_latency_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase RegionServer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase RegionServer", "name": "role_config_suppression_hbase_regionserver_java_opts", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_region_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Compaction Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Compaction Queue Size", "name": "role_health_suppression_region_server_compaction_queue", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Number of threads for completing small compactions.", "display_name": "RegionServer Small Compactions Thread Count", "name": "hbase_regionserver_thread_compaction_small", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Codecs parameter.", "display_name": "Suppress Parameter Validation: RegionServer Codecs", "name": "role_config_suppression_hbase_regionserver_codecs", "value": "false" }, { "desc": "Maximum number of Write-Ahead Log (WAL) files. This value multiplied by HDFS Block Size (dfs.blocksize) is the size of the WAL that will need to be replayed when a server crashes. This value is inversely proportional to the frequency of flushes to disk.", "display_name": "Maximum number of Write-Ahead Log (WAL) files", "name": "hbase_regionserver_maxlogs", "value": "32" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_region_server_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Coprocessor Region Classes parameter.", "display_name": "Suppress Parameter Validation: HBase Coprocessor Region Classes", "name": "role_config_suppression_hbase_coprocessor_region_classes", "value": "false" }, { "desc": "The size of the chunks allocated by MSLAB, in bytes.", "display_name": "Chunk Size Allocated by MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_chunksize", "value": "2097152" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HLog Writer Implementation parameter.", "display_name": "Suppress Parameter Validation: HLog Writer Implementation", "name": "role_config_suppression_hbase_regionserver_hlog_writer_impl", "value": "false" }, { "desc": "Limit for the number of regions after which no more region splitting should take place. This is not a hard limit for the number of regions but acts as a guideline for the RegionServer to stop splitting after a certain limit.", "display_name": "HBase Region Split Limit", "name": "hbase_regionserver_regionSplitLimit", "value": "2147483647" }, { "desc": "Enables the health test that the RegionServer's process state is consistent with the role configuration", "display_name": "RegionServer Process Health Test", "name": "regionserver_scm_health_enabled", "value": "true" }, { "desc": "When computing the overall RegionServer health, consider the host's health.", "display_name": "RegionServer Host Health Test", "name": "regionserver_host_health_enabled", "value": "true" }, { "desc": "Time period in seconds to reset long-running metrics (e.g. compactions). This is an HBase specific configuration.", "display_name": "Extended Period", "name": "hbase_metrics_extended_period", "value": "3600" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "regionserver_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Secure Bulk Load Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Secure Bulk Load Directory", "name": "role_config_suppression_hbase_bulkload_staging_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: RegionServer DNS Network Interface", "name": "role_config_suppression_hbase_regionserver_dns_interface", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "regionserver_gc_duration_window", "value": "5" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "regionserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The period over which to compute the moving average of the flush queue size.", "display_name": "HBase RegionServer Flush Queue Monitoring Period", "name": "regionserver_flush_queue_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "RegionServer Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Read threadpool size used by the RegionServer HBase IPC Server.", "display_name": "RegionServer IPC Read Threadpool Size", "name": "hbase_ipc_server_read_threadpool_size", "value": "10" }, { "desc": "Whether to suppress the results of the Flush Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Flush Queue Size", "name": "role_health_suppression_region_server_flush_queue", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: RegionServer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_regionserver_role_env_safety_valve", "value": "false" }, { "desc": "The number of reservation blocks that are used to prevent unstable RegionServers caused by an OOME.", "display_name": "RegionServer Reservation Blocks", "name": "hbase_regionserver_nbreservationblocks", "value": "4" }, { "desc": "The host name or IP address of the DNS name server which a RegionServer should use to determine the host name used by the HBase Master for communication and display purposes.", "display_name": "RegionServer DNS Nameserver", "name": "hbase_regionserver_dns_nameserver", "value": null }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_regionserver_config_safety_valve", "value": null }, { "desc": "The hashing algorithm for use in HashFunction. Two values are supported: 'murmur' (for MurmurHash) and 'jenkins' (for JenkinsHash).", "display_name": "HBase Hash Type", "name": "hbase_hash_type", "value": "murmur" }, { "desc": "The time, in milliseconds, between 'major' compactions of all HStoreFiles in a region. To disable automated major compactions, set this value to 0.", "display_name": "HBase HRegion Major Compaction", "name": "hbase_hregion_majorcompaction", "value": "604800000" }, { "desc": "Percentage of maximum heap (-Xmx setting) to allocate to block cache used by HFile/StoreFile. To disable, set this value to 0 .", "display_name": "HFile Block Cache Size", "name": "hfile_block_cache_size", "value": "0.4" }, { "desc": "The minimum log level for RegionServer logs", "display_name": "RegionServer Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The HLog file reader implementation.", "display_name": "HLog Reader Implementation", "name": "hbase_regionserver_hlog_reader_impl", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer DNS Nameserver parameter.", "display_name": "Suppress Parameter Validation: RegionServer DNS Nameserver", "name": "role_config_suppression_hbase_regionserver_dns_nameserver", "value": "false" }, { "desc": "The health test thresholds of the latency that the RegionServer sees for HDFS read operations", "display_name": "HBase RegionServer HDFS Read Latency Thresholds", "name": "regionserver_read_latency_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"50.0\"}" }, { "desc": "The port that an HBase RegionServer binds to.", "display_name": "HBase RegionServer Port", "name": "hbase_regionserver_port", "value": "60020" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "regionserver_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: RegionServer Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Cluster Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Cluster Connectivity", "name": "role_health_suppression_region_server_master_connectivity", "value": "false" }, { "desc": "The directory for HBase secure bulk Load.", "display_name": "HBase Secure Bulk Load Directory", "name": "hbase_bulkload_staging_dir", "value": "/tmp/hbase-staging" }, { "desc": "Duration between consecutive checks done by the Canary.", "display_name": "Canary Interval", "name": "hbase_regionserver_canary_interval", "value": "6" }, { "desc": "The maximum size byte array that should come from the MSLAB, in bytes.", "display_name": "Maximum Byte Array from MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_max_allocation", "value": "262144" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_region_server_swap_memory_usage", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase RegionServer in Bytes", "name": "hbase_regionserver_java_heapsize", "value": "4294967296" }, { "desc": "Whether asynchronous WAL replication to the secondary region replicas is enabled.\u2028 If enabled, a replication peer named 'region_replica_replication' is created\u2028 that tails the logs and replicates the mutations to region replicas for tables that\u2028 have region replication > 1. Disabling this replication also\u2028 requires disabling the replication peer using shell or the ReplicationAdmin Java class.\u2028 Replication to secondary region replicas works over standard intercluster replication. If disabled explicitly, enable replication by setting 'hbase.replication' to true for this feature to work.\u2028", "display_name": "Enable Replication To Secondary Region Replicas", "name": "hbase_region_replica_replication_enabled", "value": "false" }, { "desc": "The health test thresholds for the latency of HDFS write operations that the RegionServer detects", "display_name": "HBase RegionServer HDFS Sync Latency Thresholds", "name": "regionserver_sync_latency_thresholds", "value": "{\"critical\":\"5000.0\",\"warning\":\"500.0\"}" }, { "desc": "When true, HBase RegionServer will bind to 0.0.0.0. Only available in CDH 4.3 and later.", "display_name": "HBase RegionServer Bind to Wildcard Address", "name": "hbase_regionserver_bind_to_wildcard_address", "value": "true" }, { "desc": "If there are more than this number of HStoreFiles in any one HStore, then updates are blocked for this HRegion until a compaction is completed, or until the value specified for 'hbase.hstore.blockingWaitTime' has been exceeded.", "display_name": "HStore Blocking Store Files", "name": "hbase_hstore_blockingStoreFiles", "value": "10" }, { "desc": "Enable MemStore-Local Allocation Buffer (MSLAB) Allocation Scheme. Note: This feature is experimental in CDH3.", "display_name": "Enable MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase HRegion Major Compaction parameter.", "display_name": "Suppress Parameter Validation: HBase HRegion Major Compaction", "name": "role_config_suppression_hbase_hregion_majorcompaction", "value": "false" }, { "desc": "Sync the HLog to HDFS after this interval, in milliseconds, if it has not accumulated the number of HLog Entries specified to trigger a sync.", "display_name": "Synch Interval of HLog Entries", "name": "hbase_regionserver_optionallogflushinterval", "value": "1000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Store File Index Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Store File Index Size", "name": "role_health_suppression_region_server_store_file_idx_size", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_region_server_scm_health", "value": "false" }, { "desc": "Memstore will be flushed to disk if size of the memstore exceeds this value in number of bytes. This value is checked by a thread that runs the frequency specified by hbase.server.thread.wakefrequency.", "display_name": "HBase Memstore Flush Size", "name": "hbase_hregion_memstore_flush_size", "value": "134217728" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_region_server_file_descriptor", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Number of threads to use while loading and unloading regions to or from a RegionServer. Can be used to increase the speed of decommissioning or rolling restart operations.", "display_name": "Region Mover Threads", "name": "hbase_regionserver_regionmover_thread_count", "value": "1" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The lease period, in milliseconds, for the HBase RegionServer. Clients must report in within this period or else they are considered dead.", "display_name": "HBase RegionServer Lease Period", "name": "hbase_regionserver_lease_period", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of HBase RegionServer in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of HBase RegionServer in Bytes", "name": "role_config_suppression_hbase_regionserver_java_heapsize", "value": "false" }, { "desc": "Maximum HStoreFile size. If any one of a column families' HStoreFiles has grown to exceed this value, the hosting HRegion is split in two.", "display_name": "HBase Maximum File Size", "name": "hbase_hregion_max_filesize", "value": "10737418240" }, { "desc": "Comma-separated list of codecs that the RegionServer requires to start. Use this setting to make sure that all RegionServers joining a cluster are installed with a particular set of codecs.", "display_name": "RegionServer Codecs", "name": "hbase_regionserver_codecs", "value": "" }, { "desc": "The name of the DNS Network Interface from which a RegionServer should report its IP address.", "display_name": "RegionServer DNS Network Interface", "name": "hbase_regionserver_dns_interface", "value": null }, { "desc": "The period over which to compute the moving average of the HDFS sync latency of the HBase RegionServer.", "display_name": "HBase RegionServer HDFS Sync Latency Monitoring Period", "name": "regionserver_sync_latency_window", "value": "5" }, { "desc": "The health test thresholds of the weighted average size of the HBase RegionServer compaction queue over a recent period. See HBase RegionServer Compaction Queue Monitoring Period.", "display_name": "HBase RegionServer Compaction Queue Monitoring Thresholds", "name": "regionserver_compaction_queue_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_region_server_host_health", "value": "false" }, { "desc": "Period, in milliseconds, at which to roll the commit log.", "display_name": "HBase RegionServer Log Roll Period", "name": "hbase_regionserver_logroll_period", "value": "3600000" }, { "desc": "The HLog file writer implementation.", "display_name": "HLog Writer Implementation", "name": "hbase_regionserver_hlog_writer_impl", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Log Directory parameter.", "display_name": "Suppress Parameter Validation: RegionServer Log Directory", "name": "role_config_suppression_hbase_regionserver_log_dir", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \"IPC Server handler.*took.*appending an edit to hlog.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"ABORTING region server serverName.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"DFSOutputStream ResponseProcessor exception.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Error Recovery for block blk.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Failed init\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Problem renewing lease for DFSClient.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"remote error telling master we are up\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Session.*for server.*closing socket connection and attempting reconnect.\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting .*compaction on region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_REGION_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^completed compaction on region (.+) after (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_REGION_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting compaction on (.+) in region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"COLUMN_FAMILY\", \"group1\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^completed compaction: regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Completed compaction: Request = regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^aborted compaction: regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_ABORTED\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Finished memstore flush of .+ for region (.+) in (.+), sequenceid=(.+), compaction requested=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DURATION\", \"group2\": \"SEQUENCE_ID\", \"group3\": \"COMPACTION_REQUESTED\" },\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Flush of region (.+) due to global heap pressure$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_DUE_TO_HEAP_PRESSURE\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"WARN\", \"content\": \"^Region (.+) has too many store files; delaying flush up to 90000ms$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_DELAYED_TOO_MANY_STORE_FILES\", \"attribute:SEVERITY\": \"CRITICAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting split of region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Running rollback/cleanup of failed split of (.+);.+$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_ABORTED\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Region split, hbase:meta updated, and report to master. Parent=(.+), new regions: (.+,.*,.+), (.+,.*,.+). Split took (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DAUGHTER_REGIONS\", \"group2\": \"DAUGHTER_REGIONS\", \"group3\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Region split, META updated, and report to master. Parent=(.+), new regions: (.+,.*,.+), (.+,.*,.+). Split took (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DAUGHTER_REGIONS\", \"group2\": \"DAUGHTER_REGIONS\", \"group3\": \"DURATION\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "The amount of time to wait for the HBase Region Server to fully start up and connect to the HBase Master before enforcing the connectivity check.", "display_name": "HBase Region Server Connectivity Tolerance at Startup", "name": "regionserver_connectivity_tolerance", "value": "180" }, { "desc": "The health test thresholds of the size used by the HStoreFile index. Specified as a percentage of the total heap size.", "display_name": "Percentage of Heap Used by HStoreFile Index", "name": "regionserver_store_file_idx_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Interval, in milliseconds, between messages from the RegionServer to the HBase Master. Use a high value such as 3000 for clusters that have more than 10 hosts.", "display_name": "HBase RegionServer Message Interval", "name": "hbase_regionserver_msginterval", "value": "3000" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_region_server_gc_duration", "value": "false" }, { "desc": "Jitter outer bound for major compactions.\u2028 On each RegionServer, the hbase.region.majorcompaction\u2028 interval is multiplied by a random fraction that is inside the bounds of this\u2028 maximum. This + or - product is added to when the next\u2028 major compaction is to run. Major compaction\u2028 should not occur on every RegionServer at the same time. The\u2028 smaller this number, the closer together the compactions.", "display_name": "HBase HRegion Major Compaction Jitter", "name": "hbase_hregion_majorcompaction_jitter", "value": "0.5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase RegionServer Interface Class parameter.", "display_name": "Suppress Parameter Validation: HBase RegionServer Interface Class", "name": "role_config_suppression_hbase_regionserver_class", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_region_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_region_server_web_metric_collection", "value": "false" }, { "desc": "The implementation that should be used by the RegionServer for the write-ahead-log.", "display_name": "WAL Provider", "name": "hbase_regionserver_wal_provider", "value": "multiwal" }, { "desc": "Whether to suppress configuration warnings produced by the HBase RegionServer Multiple HDFS WAL with Replication Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase RegionServer Multiple HDFS WAL with Replication Validator", "name": "role_config_suppression_hbase_regionserver_multiwal_replication_validator", "value": "false" }, { "desc": "An interface that is assignable to HRegionInterface. Used in HBase Client for opening a proxy to a remote HBase RegionServer.", "display_name": "HBase RegionServer Interface Class", "name": "hbase_regionserver_class", "value": null }, { "desc": "Blocks writes if the size of the memstore increases to the value of 'hbase.hregion.block.memstore' multiplied by the value of 'hbase.hregion.flush.size' bytes. This setting is useful for preventing runaway memstore during spikes in update traffic. Without an upper-bound, memstore fills such that when it flushes, the resultant process of flushing files take a long time to compact or split, or worse, an \"out of memory\" error occurs.", "display_name": "HBase Memstore Block Multiplier", "name": "hbase_hregion_memstore_block_multiplier", "value": "2" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/impala-llama.json0000666000175100017510000007306613245514472024740 0ustar zuulzuul00000000000000[ { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client ACLs parameter.", "display_name": "Suppress Parameter Validation: Client ACLs", "name": "role_config_suppression_llama_am_server_thrift_client_acl", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_llama_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Enables the health test that the Impala Llama ApplicationMaster's process state is consistent with the role configuration", "display_name": "Impala Llama ApplicationMaster Process Health Test", "name": "llama_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Administrative Interface ACLs parameter.", "display_name": "Suppress Parameter Validation: Administrative Interface ACLs", "name": "role_config_suppression_llama_am_server_thrift_admin_acl", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Llama ApplicationMaster heartbeat interval, in milliseconds. On each heartbeat the ApplicationMaster submits new reservations to YARN ResourceManager and gets updates from it.", "display_name": "AM Heartbeat Interval", "name": "llama_am_server_thrift_client_notifier_heartbeat_ms", "value": "5000" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_llama_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Minimum amount of time the backed off reservations will be in 'backed off' state. The actual amount time is a random value between the minimum and the maximum.", "display_name": "Anti-Deadlock Minimum Delay", "name": "llama_am_gang_anti_deadlock_min_delay_ms", "value": "10000" }, { "desc": "ACL for Impala ApplicationMaster admins. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. \"alice,bob users,wheel\". A special value of \"*\" means all users are allowed. These take effect only if security is enabled.", "display_name": "Administrative Interface ACLs", "name": "llama_am_server_thrift_admin_acl", "value": "*" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Percentage of resources that will be backed off by the Impala ApplicationMaster anti-deadlock logic. Random reservations will be backed off until the percentage of backed off resources reaches this percentage.", "display_name": "Anti-Deadlock Backoff Percentage", "name": "llama_am_gang_anti_deadlock_backoff_percent", "value": "30" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The user that the Llama process should run as.", "display_name": "Llama System User", "name": "process_username", "value": "llama" }, { "desc": "Maximum number of threads used by the Llama ApplicationMaster auxiliary service uses for serving client requests.", "display_name": "Thrift Server Maximum Threads", "name": "llama_am_server_thrift_server_max_threads", "value": "50" }, { "desc": "Maximum number of retries for a client notification. After the maximum number of client notification retries has been reached without success the client is considered lost and all its reservations are released. A successful client notification resets the retries count.", "display_name": "Maximum Client Notification Retries", "name": "llama_am_server_thrift_client_notifier_max_retries", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Log Directory parameter.", "display_name": "Suppress Parameter Validation: Llama Log Directory", "name": "role_config_suppression_llama_log_dir", "value": "false" }, { "desc": "Kerberos principal used by the Impala Llama ApplicationMaster roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "llama" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "llama_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Interval of time without any new allocation that will trigger the Impala ApplicationMaster anti-deadlock logic.", "display_name": "Anti-Deadlock No Allocation Limit Interval", "name": "llama_am_gang_anti_deadlock_no_allocation_limit_ms", "value": "30000" }, { "desc": "Time in milliseconds after which Llama will discard its AM for a queue that has been empty of reservations. Does not apply to queues specified with the Core Queues property.", "display_name": "Queue Expiration Age", "name": "llama_am_queue_expire_ms", "value": "300000" }, { "desc": "The maximum number of rolled log files to keep for Impala Llama ApplicationMaster logs. Typically used by log4j or logback.", "display_name": "Impala Llama ApplicationMaster Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Timeout policy for resources being cached.", "display_name": "Resource Caching Idle Timeout", "name": "llama_am_cache_eviction_timeout_policy_idle_timeout_ms", "value": "30000" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_llama_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_llama_role_env_safety_valve", "value": "false" }, { "desc": "The minimum log level for Impala Llama ApplicationMaster logs", "display_name": "Impala Llama ApplicationMaster Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_llama_log_directory_free_space", "value": "false" }, { "desc": "When computing the overall Impala Llama ApplicationMaster health, consider the host's health.", "display_name": "Impala Llama ApplicationMaster Host Health Test", "name": "llama_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_llama_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into llama-site.xml for this role only.", "display_name": "Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml", "name": "llama_config_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Minimum number of threads used by the Llama ApplicationMaster auxiliary service uses for serving client requests.", "display_name": "Thrift Server Minimum Threads", "name": "llama_am_server_thrift_server_min_threads", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_llama_fair_scheduler_safety_valve", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Llama Server", "name": "llama_java_opts", "value": "" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Queues Llama ApplicationMaster should connect to at start up.", "display_name": "Core Queues", "name": "llama_am_core_queues", "value": "" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_llama_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Port on which the Llama ApplicationMaster serves its Thrift interface.", "display_name": "Llama Thrift Port", "name": "llama_port", "value": "15000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Llama Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Llama Server", "name": "role_config_suppression_llama_java_opts", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve)", "name": "LLAMA_role_env_safety_valve", "value": null }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Core Queues parameter.", "display_name": "Suppress Parameter Validation: Core Queues", "name": "role_config_suppression_llama_am_core_queues", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama System User parameter.", "display_name": "Suppress Parameter Validation: Llama System User", "name": "role_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "If enabled, the Impala Llama ApplicationMaster binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind Impala Llama ApplicationMaster to Wildcard Address", "name": "llama_bind_wildcard", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Maximum amount of time the backed off reservations will be in 'backed off' state. The actual amount time is a random value between the minimum and the maximum.", "display_name": "Anti-Deadlock Maximum Delay", "name": "llama_am_gang_anti_deadlock_max_delay_ms", "value": "30000" }, { "desc": "Directory where Llama will place its log files.", "display_name": "Llama Log Directory", "name": "llama_log_dir", "value": "/var/log/impala-llama" }, { "desc": "The group that the Llama processes should run as.", "display_name": "Llama System Group", "name": "process_groupname", "value": "llama" }, { "desc": "The maximum size, in megabytes, per log file for Impala Llama ApplicationMaster logs. Typically used by log4j or logback.", "display_name": "Impala Llama ApplicationMaster Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Port on which the Llama ApplicationMaster listens to HTTP requests.", "display_name": "Llama HTTP Port", "name": "llama_http_port", "value": "15001" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Client notification retry interval, in milliseconds.", "display_name": "Client Notification Retry Interval", "name": "llama_am_server_thrift_client_notifier_retry_interval_ms", "value": "5000" }, { "desc": "ACL for Impala ApplicationMaster clients. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. \"alice,bob users,wheel\". A special value of \"*\" means all users are allowed. These take effect only if security is enabled.", "display_name": "Client ACLs", "name": "llama_am_server_thrift_client_acl", "value": "*" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Impala Llama ApplicationMaster in Bytes", "name": "llama_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_llama_scm_health", "value": "false" }, { "desc": "Whether Llama should cache allocated resources on release.", "display_name": "Enable Resource Caching", "name": "llama_am_cache_enabled", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Enter an XML string that will be inserted verbatim into the Fair Scheduler allocations file. Overrides the configuration set using the Pools configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "llama_fair_scheduler_safety_valve", "value": null }, { "desc": "Port on which the Llama ApplicationMaster listens to administrative requests on its administrative Thrift interface.", "display_name": "Llama Thrift Admin Port", "name": "llama_am_server_thrift_admin_address", "value": "15002" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml", "name": "role_config_suppression_llama_config_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to break resource requests into smaller requests of standard size before the resource cache. The sizes are taken from Yarn settings Container Memory Increment and Container Virtual CPU Cores Increment.", "display_name": "Enable Resource Cache Normalization", "name": "llama_am_resource_normalizing_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Socket time, in milliseconds, used Llama ApplicationMaster auxiliary service for all its server and client Thrift connections.", "display_name": "Thrift Transport Timeout", "name": "llama_am_server_thrift_transport_timeout_ms", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama System Group parameter.", "display_name": "Suppress Parameter Validation: Llama System Group", "name": "role_config_suppression_process_groupname", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/yarn-gateway.json0000666000175100017510000006553713245514472025025 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "role_config_suppression_mapreduce_client_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Java Opts Base", "name": "role_config_suppression_yarn_app_mapreduce_am_command_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Reduce Task Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: Reduce Task Java Opts Base", "name": "role_config_suppression_mapreduce_reduce_java_opts", "value": "false" }, { "desc": "Maximum sleep time between failover attempts. Used only if RM HA is enabled.", "display_name": "Client Failover Sleep Max Time", "name": "client_failover_sleep_max", "value": "2000" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "92" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the I/O Sort Factor parameter.", "display_name": "Suppress Parameter Validation: I/O Sort Factor", "name": "role_config_suppression_io_sort_factor", "value": "false" }, { "desc": "The default number of reduce tasks per job. Will be part of generated client configuration.", "display_name": "Default Number of Reduce Tasks per Job", "name": "mapred_reduce_tasks", "value": "1" }, { "desc": "The amount of physical memory, in MiB, allocated for each reduce task of a job. For versions before CDH 5.5, if not specified, by default it is set to 1024. For CDH 5.5 and higher, a value less than 128 is not supported but if it is specified as 0, the amount of physical memory to request is inferred from Reduce Task Maximum Heap Size and Heap to Container Size Ratio. If Reduce Task Maximum Heap Size is not specified, by default the amount of physical memory to request is set to 1024. This parameter has no effect prior to CDH 4.4.", "display_name": "Reduce Task Memory", "name": "mapreduce_reduce_memory_mb", "value": "0" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The application framework to run jobs with. If not set, jobs will be run with the local job runner.", "display_name": "Application Framework", "name": "mapreduce_framework_name", "value": "yarn" }, { "desc": "Classpaths to include for MapReduce applications. During evaluation, the string '{version}' in the value of this parameter will be replaced by the actual MapReduce version.", "display_name": "MR Application Classpath", "name": "mapreduce_application_classpath", "value": "$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH" }, { "desc": "Whether map tasks should attempt to use the optimized native implementation of the map-side output collector. This can improve performance of many jobs that are shuffle-intensive. Experimental in CDH 5.2.", "display_name": "Enable Optimized Map-side Output Collector", "name": "mapreduce_enable_native_map_output_collector", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codec of MapReduce Map Output parameter.", "display_name": "Suppress Parameter Validation: Compression Codec of MapReduce Map Output", "name": "role_config_suppression_mapred_map_output_compression_codec", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Map Task Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: Map Task Java Opts Base", "name": "role_config_suppression_mapreduce_map_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_mapreduce_client_config_safety_valve", "value": "false" }, { "desc": "The physical memory requirement, in MiB, for the ApplicationMaster.", "display_name": "ApplicationMaster Memory", "name": "yarn_app_mapreduce_am_resource_mb", "value": "1024" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "mapreduce_client_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For MapReduce map outputs that are compressed, specify the compression codec to use. Will be part of generated client configuration.", "display_name": "Compression Codec of MapReduce Map Output", "name": "mapred_map_output_compression_codec", "value": "org.apache.hadoop.io.compress.SnappyCodec" }, { "desc": "Compress the output of MapReduce jobs. Will be part of generated client configuration.", "display_name": "Compress MapReduce Job Output", "name": "mapred_output_compress", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for mapred-site.xml.", "display_name": "MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "mapreduce_client_config_safety_valve", "value": null }, { "desc": "For MapReduce job outputs that are compressed, specify the compression codec to use. Will be part of generated client configuration.", "display_name": "Compression Codec of MapReduce Job Output", "name": "mapred_output_compression_codec", "value": "org.apache.hadoop.io.compress.DefaultCodec" }, { "desc": "The total amount of memory buffer, in megabytes, to use while sorting files. Note that this memory comes out of the user JVM heap size (meaning total user JVM heap - this amount of memory = total user usable heap space. Note that Cloudera's default differs from Hadoop's default; Cloudera uses a bigger buffer by default because modern machines often have more RAM. The smallest value across all TaskTrackers will be part of generated client configuration.", "display_name": "I/O Sort Memory Buffer (MiB)", "name": "io_sort_mb", "value": "256" }, { "desc": "For MapReduce job outputs that are compressed as SequenceFiles, you can select one of these compression type options: NONE, RECORD or BLOCK. Cloudera recommends BLOCK. Will be part of generated client configuration.", "display_name": "Compression Type of MapReduce Job Output", "name": "mapred_output_compression_type", "value": "BLOCK" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codec of MapReduce Job Output parameter.", "display_name": "Suppress Parameter Validation: Compression Codec of MapReduce Job Output", "name": "role_config_suppression_mapred_output_compression_codec", "value": "false" }, { "desc": "The number of streams to merge at the same time while sorting files. That is, the number of sort heads to use during the merge sort on the reducer side. This determines the number of open file handles. Merging more files in parallel reduces merge sort iterations and improves run time by eliminating disk I/O. Note that merging more files in parallel uses more memory. If 'io.sort.factor' is set too high or the maximum JVM heap is set too low, excessive garbage collection will occur. The Hadoop default is 10, but Cloudera recommends a higher value. Will be part of generated client configuration.", "display_name": "I/O Sort Factor", "name": "io_sort_factor", "value": "64" }, { "desc": "The maximum number of application attempts for MapReduce jobs. The value of this parameter overrides ApplicationMaster Maximum Attempts for MapReduce jobs.", "display_name": "Maximum Number of Attempts for MapReduce Jobs", "name": "mapreduce_am_max_attempts", "value": "2" }, { "desc": "Threshold for number of maps, beyond which a job is considered too big for ubertask optimization.", "display_name": "Ubertask Maximum Maps", "name": "mapreduce_job_ubertask_maxmaps", "value": "9" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Running Job History Location parameter.", "display_name": "Suppress Parameter Validation: Running Job History Location", "name": "role_config_suppression_hadoop_job_history_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Job Submit Replication Validator configuration validator.", "display_name": "Suppress Configuration Validator: Job Submit Replication Validator", "name": "role_config_suppression_mapreduce_replication_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Environment parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Environment", "name": "role_config_suppression_yarn_app_mapreduce_am_admin_user_env", "value": "false" }, { "desc": "Base sleep time between failover attempts. Used only if RM HA is enabled.", "display_name": "Client Failover Sleep Base Time", "name": "client_failover_sleep_base", "value": "100" }, { "desc": "If enabled, multiple instances of some map tasks may be executed in parallel.", "display_name": "Map Tasks Speculative Execution", "name": "mapred_map_tasks_speculative_execution", "value": "false" }, { "desc": "If enabled, multiple instances of some reduce tasks may be executed in parallel.", "display_name": "Reduce Tasks Speculative Execution", "name": "mapred_reduce_tasks_speculative_execution", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for yarn-site.xml.", "display_name": "YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_client_config_safety_valve", "value": null }, { "desc": "Threshold for number of input bytes, beyond which a job is considered too big for ubertask optimization. If no value is specified, dfs.block.size is used as a default.", "display_name": "Ubertask Maximum Job Size", "name": "mapreduce_job_ubertask_maxbytes", "value": null }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "mapreduce_client_java_heapsize", "value": "825955249" }, { "desc": "Additional execution environment entries for map and reduce task processes.", "display_name": "MR Application Environment", "name": "mapreduce_admin_user_env", "value": "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH" }, { "desc": "The number of virtual CPU cores for each reduce task of a job.", "display_name": "Reduce Task CPU Virtual Cores", "name": "mapreduce_reduce_cpu_vcores", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_mapreduce_client_java_opts", "value": "false" }, { "desc": "If enabled, uses compression on the map outputs before they are sent across the network. Will be part of generated client configuration.", "display_name": "Use Compression on Map Outputs", "name": "mapred_compress_map_output", "value": "true" }, { "desc": "Limit on the number of counters allowed per job.", "display_name": "Job Counters Limit", "name": "mapreduce_job_counters_limit", "value": "120" }, { "desc": "Whether to suppress configuration warnings produced by the ApplicationMaster Java Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: ApplicationMaster Java Maximum Heap Size Validator", "name": "role_config_suppression_yarn_app_mapreduce_am_max_heap_yarn_app_mapreduce_am_resource_mb_validator", "value": "false" }, { "desc": "Compression level for the codec used to compress MapReduce outputs. Default compression is a balance between speed and compression ratio.", "display_name": "Compression Level of Codecs", "name": "zlib_compress_level", "value": "DEFAULT_COMPRESSION" }, { "desc": "Fraction of the number of map tasks in the job which should be completed before reduce tasks are scheduled for the job.", "display_name": "Number of Map Tasks to Complete Before Reduce Tasks", "name": "mapred_reduce_slowstart_completed_maps", "value": "0.8" }, { "desc": "The maximum Java heap size, in bytes, of the map processes. This number will be formatted and concatenated with 'Map Task Java Opts Base' to pass to Hadoop.", "display_name": "Map Task Maximum Heap Size", "name": "mapreduce_map_java_opts_max_heap", "value": "0" }, { "desc": "The amount of physical memory, in MiB, allocated for each map task of a job. For versions before CDH 5.5, if not specified, by default it is set to 1024. For CDH 5.5 and higher, a value less than 128 is not supported but if it is specified as 0, the amount of physical memory to request is inferred from Map Task Maximum Heap Size and Heap to Container Size Ratio. If Map Task Maximum Heap Size is not specified, by default the amount of physical memory to request is set to 1024.", "display_name": "Map Task Memory", "name": "mapreduce_map_memory_mb", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The number of virtual CPU cores allocated for each map task of a job. This parameter has no effect prior to CDH 4.4.", "display_name": "Map Task CPU Virtual Cores", "name": "mapreduce_map_cpu_vcores", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Classpath parameter.", "display_name": "Suppress Parameter Validation: MR Application Classpath", "name": "role_config_suppression_mapreduce_application_classpath", "value": "false" }, { "desc": "Threshold for number of reduces, beyond which a job is considered too big for ubertask optimization. Note: As of CDH 5, MR2 does not support more than one reduce in an ubertask. (Zero is valid.)", "display_name": "Ubertask Maximum Reduces", "name": "mapreduce_job_ubertask_maxreduces", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the Reduce Task Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Reduce Task Maximum Heap Size Validator", "name": "role_config_suppression_mapreduce_reduce_java_opts_max_heap_mapreduce_reduce_memory_mb_validator", "value": "false" }, { "desc": "Maximum allowed connections for the shuffle. Set to 0 (zero) to indicate no limit on the number of connections.", "display_name": "Max Shuffle Connections", "name": "mapreduce_shuffle_max_connections", "value": "80" }, { "desc": "Environment variables for the MapReduce ApplicationMaster. These settings can be overridden in the ApplicationMaster User Environment (yarn.app.mapreduce.am.env).", "display_name": "ApplicationMaster Environment", "name": "yarn_app_mapreduce_am_admin_user_env", "value": "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH" }, { "desc": "Whether to suppress configuration warnings produced by the Map Task Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Map Task Maximum Heap Size Validator", "name": "role_config_suppression_mapreduce_map_java_opts_max_heap_mapreduce_map_memory_mb_validator", "value": "false" }, { "desc": "Java opts for the map processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp pass a value of: \"-verbose:gc -Xloggc:/tmp/@taskid@.gc\". The configuration variable 'Map Task Memory' can be used to control the maximum memory of the map processes.", "display_name": "Map Task Java Opts Base", "name": "mapreduce_map_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Location to store the job history files of running jobs. This is a path on the host where the JobTracker is running.", "display_name": "Running Job History Location", "name": "hadoop_job_history_dir", "value": "/var/log/hadoop-mapreduce/history" }, { "desc": "Java command line arguments passed to the MapReduce ApplicationMaster.", "display_name": "ApplicationMaster Java Opts Base", "name": "yarn_app_mapreduce_am_command_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "The ratio of heap size to container size for both map and reduce tasks. The heap should be smaller than the container size to allow for some overhead of the JVM.", "display_name": "Heap to Container Size Ratio", "name": "mapreduce_job_heap_memory_mb_ratio", "value": "0.8" }, { "desc": "The maximum Java heap size, in bytes, of the reduce processes. This number will be formatted and concatenated with 'Reduce Task Java Opts Base' to pass to Hadoop.", "display_name": "Reduce Task Maximum Heap Size", "name": "mapreduce_reduce_java_opts_max_heap", "value": "0" }, { "desc": "Whether to enable ubertask optimization, which runs \"sufficiently small\" jobs sequentially within a single JVM. \"Small\" is defined by the mapreduce.job.ubertask.maxmaps, mapreduce.job.ubertask.maxreduces, and mapreduce.job.ubertask.maxbytes settings.", "display_name": "Enable Ubertask Optimization", "name": "mapreduce_job_ubertask_enabled", "value": "false" }, { "desc": "The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string.", "display_name": "Mapreduce Task Timeout", "name": "mapred_task_timeout", "value": "600000" }, { "desc": "Size of buffer for read and write operations of SequenceFiles.", "display_name": "SequenceFile I/O Buffer Size", "name": "io_file_buffer_size", "value": "65536" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hadoop" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The soft limit in either the buffer or record collection buffers. When this limit is reached, a thread will begin to spill the contents to disk in the background. Note that this does not imply any chunking of data to the spill. A value less than 0.5 is not recommended. The syntax is in decimal units; the default is 80% and is formatted 0.8. Will be part of generated client configuration.", "display_name": "I/O Sort Spill Percent", "name": "io_sort_spill_percent", "value": "0.8" }, { "desc": "The replication level for submitted job files.", "display_name": "Mapreduce Submit Replication", "name": "mapred_submit_replication", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Maximum Number of Attempts for MapReduce Jobs parameter.", "display_name": "Suppress Parameter Validation: Maximum Number of Attempts for MapReduce Jobs", "name": "role_config_suppression_mapreduce_am_max_attempts", "value": "false" }, { "desc": "The virtual CPU cores requirement, for the ApplicationMaster. This parameter has no effect prior to CDH 4.4.", "display_name": "ApplicationMaster Virtual CPU Cores", "name": "yarn_app_mapreduce_am_resource_cpu_vcores", "value": "1" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Java opts for the reduce processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp pass a value of: \"-verbose:gc -Xloggc:/tmp/@taskid@.gc\". The configuration variable 'Reduce Task Memory' can be used to control the maximum memory of the reduce processes.", "display_name": "Reduce Task Java Opts Base", "name": "mapreduce_reduce_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_yarn_client_config_safety_valve", "value": "false" }, { "desc": "The maximum permissible size of the split metainfo file. The JobTracker won't attempt to read split metainfo files bigger than the configured value. No limits if set to -1.", "display_name": "JobTracker MetaInfo Maxsize", "name": "mapreduce_jobtracker_split_metainfo_maxsize", "value": "10000000" }, { "desc": "The maximum heap size, in bytes, of the Java MapReduce ApplicationMaster. This number will be formatted and concatenated with 'ApplicationMaster Java Opts Base' to pass to Hadoop.", "display_name": "ApplicationMaster Java Maximum Heap Size", "name": "yarn_app_mapreduce_am_max_heap", "value": "825955249" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Environment parameter.", "display_name": "Suppress Parameter Validation: MR Application Environment", "name": "role_config_suppression_mapreduce_admin_user_env", "value": "false" }, { "desc": "The default number of parallel transfers run by reduce during the copy (shuffle) phase. This number should be between sqrt(nodes*number_of_map_slots_per_node) and nodes*number_of_map_slots_per_node/2. Will be part of generated client configuration.", "display_name": "Default Number of Parallel Transfers During Shuffle", "name": "mapred_reduce_parallel_copies", "value": "10" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh", "display_name": "Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "mapreduce_client_env_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hue-hue_server.json0000666000175100017510000005364613245514472025341 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hue_metrics_sample_file_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini parameter.", "display_name": "Suppress Parameter Validation: Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini", "name": "role_config_suppression_hue_server_hue_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Hue Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HUE_SERVER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_ssl_certificate", "value": "false" }, { "desc": "Timeout in seconds for Thrift calls to HiveServer2 and Impala.", "display_name": "HiveServer2 and Impala Thrift Connection Timeout", "name": "hs2_conn_timeout", "value": "120" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hue_server_host_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hue_server_unexpected_exits", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_hue_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Jobsub Examples and Templates Directory parameter.", "display_name": "Suppress Parameter Validation: Jobsub Examples and Templates Directory", "name": "role_config_suppression_hue_server_remote_data_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Top Banner Custom HTML parameter.", "display_name": "Suppress Parameter Validation: Top Banner Custom HTML", "name": "role_config_suppression_banner_html", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "hue_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hue/metrics-hue_server/metrics.log.", "display_name": "Metrics Sample File Location", "name": "hue_metrics_sample_file_location", "value": null }, { "desc": "Location on HDFS where the jobsub examples and templates are stored.", "display_name": "Jobsub Examples and Templates Directory", "name": "hue_server_remote_data_dir", "value": "/user/hue/jobsub" }, { "desc": "The frequency at which the metrics are logged to the sample file.", "display_name": "Metrics Sample File Logging Frequency", "name": "hue_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_certificate", "value": null }, { "desc": "Encrypt communication between clients and Hue using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Hue", "name": "ssl_enable", "value": "false" }, { "desc": "An optional, custom one-line HTML code to display as a banner on top of all Hue Server web pages. Useful in displaying cluster identity of the Hue Server.", "display_name": "Top Banner Custom HTML", "name": "banner_html", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hue_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_private_key", "value": null }, { "desc": "When computing the overall Hue Server health, consider the host's health.", "display_name": "Hue Server Host Health Test", "name": "hue_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hue_server_scm_health", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "hue_server_web_metric_collection_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hue Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hue_server_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hue_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "If enabled, the Hue server binds to the wildcard address (\"0.0.0.0\") for its ports.", "display_name": "Bind Hue Server to Wildcard Address", "name": "hue_server_bind_wildcard", "value": "false" }, { "desc": "Enables the health test that the Hue Server's process state is consistent with the role configuration", "display_name": "Hue Server Process Health Test", "name": "hue_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Server Log Directory", "name": "role_config_suppression_hue_server_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hue_server_log_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the certificate of the certificate authority (CA) and any intermediate certificates used to sign the server certificate. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format, and is usually created by concatenating together all of the appropriate root and intermediate certificates.", "display_name": "Hue TLS/SSL Server CA Certificate (PEM Format)", "name": "ssl_cacerts", "value": null }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hue_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server CA Certificate (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server CA Certificate (PEM Format)", "name": "role_config_suppression_ssl_cacerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Secret Key parameter.", "display_name": "Suppress Parameter Validation: Secret Key", "name": "role_config_suppression_secret_key", "value": "false" }, { "desc": "Port to use to connect to the Hue server.", "display_name": "Hue HTTP Port", "name": "hue_http_port", "value": "8888" }, { "desc": "Whether to suppress configuration warnings produced by the Hue TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hue TLS/SSL Validator", "name": "role_config_suppression_hue_ssl_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Directory where Hue Server will place its log files.", "display_name": "Hue Server Log Directory", "name": "hue_server_log_dir", "value": "/var/log/hue" }, { "desc": "For advanced use only. A string to be inserted into hue_safety_valve_server.ini for this role only.", "display_name": "Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini", "name": "hue_server_hue_safety_valve", "value": null }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Random string used for secure hashing in the session store.", "display_name": "Secret Key", "name": "secret_key", "value": null }, { "desc": "The password for the private key in the Hue TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Hue TLS/SSL Private Key Password", "name": "ssl_private_key_password", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hue_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Private Key Password", "name": "role_config_suppression_ssl_private_key_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_ssl_private_key", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hive-metastore-sentry-safety.xml0000666000175100017510000000135213245514472027774 0ustar zuulzuul00000000000000 hive.metastore.client.impl org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient Sets custom Hive metastore client which Sentry uses to filter out metadata. hive.metastore.pre.event.listeners org.apache.sentry.binding.metastore.MetastoreAuthzBinding list of comma separated listeners for metastore events. hive.metastore.event.listeners org.apache.sentry.binding.metastore.SentryMetastorePostEventListener list of comma separated listeners for metastore, post events. sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hive-webhcat.json0000666000175100017510000005134213245514472024750 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_webhcat_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Directory where WebHCat Server will place its log files.", "display_name": "WebHCat Server Log Directory", "name": "hcatalog_log_dir", "value": "/var/log/hcatalog" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_webhcat_env_safety_valve", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "WebHCat Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_webhcat_env_safety_valve", "value": null }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_webhcat_hive_config_safety_valve", "value": null }, { "desc": "The minimum log level for WebHCat Server logs", "display_name": "WebHCat Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_webhcat_swap_memory_usage", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of WebHCat Server in Bytes", "name": "hive_webhcat_java_heapsize", "value": "268435456" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When computing the overall WebHCat Server health, consider the host's health.", "display_name": "WebHCat Server Host Health Test", "name": "webhcat_host_health_enabled", "value": "true" }, { "desc": "The maximum number of rolled log files to keep for WebHCat Server logs. Typically used by log4j or logback.", "display_name": "WebHCat Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_webhcat_log_directory_free_space", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_webhcat_heap_dump_directory_free_space", "value": "false" }, { "desc": "Enables the health test that the WebHCat Server's process state is consistent with the role configuration", "display_name": "WebHCat Server Process Health Test", "name": "webhcat_scm_health_enabled", "value": "true" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_webhcat_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml", "name": "role_config_suppression_hive_webhcat_config_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into webhcat-site.xml for this role only.", "display_name": "WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml", "name": "hive_webhcat_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_webhcat_host_health", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for WebHCat Server", "name": "hive_webhcat_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for WebHCat Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for WebHCat Server", "name": "role_config_suppression_hive_webhcat_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_webhcat_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Log Directory", "name": "role_config_suppression_hcatalog_log_dir", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for WebHCat Server logs. Typically used by log4j or logback.", "display_name": "WebHCat Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Port on which WebHCat Server will listen for connections.", "display_name": "WebHCat Server Port", "name": "hive_webhcat_address_port", "value": "50111" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "WebHCat Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "webhcat_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_webhcat_hive_config_safety_valve", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/mapred-site.xml0000666000175100017510000001050113245514472024433 0ustar zuulzuul00000000000000 mapreduce.job.split.metainfo.maxsize 10000000 mapreduce.job.counters.max 120 mapreduce.output.fileoutputformat.compress false mapreduce.output.fileoutputformat.compress.type BLOCK mapreduce.output.fileoutputformat.compress.codec org.apache.hadoop.io.compress.DefaultCodec mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.compress true zlib.compress.level DEFAULT_COMPRESSION mapreduce.task.io.sort.factor 64 mapreduce.map.sort.spill.percent 0.8 mapreduce.reduce.shuffle.parallelcopies 10 mapreduce.task.timeout 600000 mapreduce.client.submit.file.replication 10 mapreduce.job.reduces 1 mapreduce.task.io.sort.mb 256 mapreduce.map.speculative false mapreduce.reduce.speculative false mapreduce.job.reduce.slowstart.completedmaps 0.8 mapreduce.jobhistory.address test-master-001.novalocal:10020 mapreduce.jobhistory.webapp.address test-master-001.novalocal:19888 mapreduce.jobhistory.webapp.https.address test-master-001.novalocal:19890 mapreduce.jobhistory.admin.address test-master-001.novalocal:10033 mapreduce.framework.name yarn yarn.app.mapreduce.am.staging-dir /user yarn.app.mapreduce.am.resource.mb 1024 yarn.app.mapreduce.am.resource.cpu-vcores 1 mapreduce.job.ubertask.enable false yarn.app.mapreduce.am.command-opts -Djava.net.preferIPv4Stack=true -Xmx825955249 mapreduce.map.java.opts -Djava.net.preferIPv4Stack=true -Xmx825955249 mapreduce.reduce.java.opts -Djava.net.preferIPv4Stack=true -Xmx825955249 yarn.app.mapreduce.am.admin.user.env LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH mapreduce.map.memory.mb 1024 mapreduce.map.cpu.vcores 1 mapreduce.reduce.memory.mb 1024 mapreduce.reduce.cpu.vcores 1 mapreduce.application.classpath $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH mapreduce.admin.user.env LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH mapreduce.shuffle.max.connections 80 sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/sqoop-service.json0000666000175100017510000001606713245514472025206 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "MapReduce jobs are run against this service.", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "sqoop" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "sqoop2" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "sqoop2" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "sqoop_env_safety_valve", "value": null }, { "desc": "When computing the overall SQOOP health, consider Sqoop 2 Server's health", "display_name": "Sqoop 2 Server Role Health Test", "name": "sqoop_sqoop_server_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_sqoop_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Sqoop 2 Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Sqoop 2 Server Count Validator", "name": "service_config_suppression_sqoop_server_count_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Sqoop 2 Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sqoop 2 Server Health", "name": "service_health_suppression_sqoop_sqoop_server_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/solr-gateway.json0000666000175100017510000000261613245514472025020 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/solr" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/impala-impalad.json0000666000175100017510000012370613245514472025256 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Local UDF Library Dir parameter.", "display_name": "Suppress Parameter Validation: Local UDF Library Dir", "name": "role_config_suppression_local_library_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_impalad_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_default_query_options", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_impalad_web_metric_collection", "value": "false" }, { "desc": "The directory in which Impala daemon lineage log files are written. If \"Impala Lineage Generation\" property is enabled, Impala generates its lineage logs in this directory.", "display_name": "Impala Daemon Lineage Log Directory", "name": "lineage_event_log_dir", "value": "/var/log/impalad/lineage" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml for this role only.", "display_name": "Impala Daemon HBase Advanced Configuration Snippet (Safety Valve)", "name": "impalad_hbase_conf_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "The directory in which Impala daemon audit event log files are written. If \"Impala Audit Event Generation\" property is enabled, Impala will generate its audit logs in this directory.", "display_name": "Impala Daemon Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/impalad/audit" }, { "desc": "Password for Impala Daemon webserver authentication.", "display_name": "Impala Daemon Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Encrypt and verify the integrity of all data spilled to disk as part of a query. This feature is only supported for Impala 2.0 and higher and CDH 5.2 and higher (which includes Impala 2.0).", "display_name": "Disk Spill Encryption", "name": "disk_spill_encryption", "value": "false" }, { "desc": "Port on which Beeswax client requests are served by Impala Daemons.", "display_name": "Impala Daemon Beeswax Port", "name": "beeswax_port", "value": "21000" }, { "desc": "An XML string to use verbatim for the contents of fair-scheduler.xml for Impala Daemons. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve)", "name": "impalad_fair_scheduler_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Impala Lineage Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Lineage Enabled Validator", "name": "role_config_suppression_impala_lineage_enabled_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Daemon Scratch Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Scratch Directories Free Space", "name": "role_health_suppression_impalad_scratch_directories_free_space", "value": "false" }, { "desc": "Port on which ImpalaBackendService is exported.", "display_name": "Impala Daemon Backend Port", "name": "be_port", "value": "22000" }, { "desc": "Timeout for requests to the Hive Metastore Server from Impala. Consider increasing this if you have tables with a lot of metadata and see timeout errors.", "display_name": "Impala Daemon Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "3600" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Enables the health test that verifies the Impala Daemon is connected to the StateStore.", "display_name": "Impala Daemon Connectivity Health Test", "name": "impalad_connectivity_health_enabled", "value": "true" }, { "desc": "Port on which HiveServer2 client requests are served by Impala Daemons.", "display_name": "Impala Daemon HiveServer2 Port", "name": "hs2_port", "value": "21050" }, { "desc": "The maximum size (in entries) of the Impala daemon lineage log file before a new one is created.", "display_name": "Impala Daemon Maximum Lineage Log File Size", "name": "max_lineage_log_file_size", "value": "5000" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The period to review when computing query monitoring failures.", "display_name": "Monitoring Period For Query Monitoring Failures", "name": "impala_query_monitoring_failure_window", "value": "5" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_impalad_memory_rss_health", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "impalad_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The location on disk of the certificate, in .pem format, used to confirm the authenticity of the LDAP server certificate. This is the Certificate Authority (CA) certificate, and it was used to sign the LDAP server certificate. If not set, all certificates are trusted, which means that an attacker could potentially intercept otherwise encrypted usernames and passwords.", "display_name": "LDAP Server CA Certificate", "name": "impalad_ldap_ca_certificate", "value": null }, { "desc": "Port where Llama notification callback should be started", "display_name": "Llama Callback Port", "name": "llama_callback_port", "value": "28000" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_impalad_scm_health", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml for this role only.", "display_name": "Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve)", "name": "impala_hdfs_site_conf_safety_valve", "value": null }, { "desc": "The maximum size, in megabytes, per log file for Impala Daemon logs. Typically used by log4j or logback.", "display_name": "Impala Daemon Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Username for Impala Daemon webserver authentication.", "display_name": "Impala Daemon Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Enables audit event generation by Impala daemons. The audit log file will be placed in the directory specified by 'Impala Daemon Audit Log Directory' parameter.", "display_name": "Enable Impala Audit Event Generation", "name": "enable_audit_event_log", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_impalad_host_health", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When computing the overall Impala Daemon health, consider the host's health.", "display_name": "Impala Daemon Host Health Test", "name": "impalad_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Enable or disable the Impala Daemon web server. This web server contains useful information about Impala Daemon.", "display_name": "Enable Impala Daemon Web Server", "name": "impalad_enable_webserver", "value": "true" }, { "desc": "Number of seconds to wait between attempts during Llama registration.", "display_name": "Llama Registration Wait Seconds", "name": "llama_registration_wait_secs", "value": "3" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Maximum number of query results a client may request to be cached on a per-query basis to support restarting fetches. This option guards against unreasonably large result caches requested by clients. Requests exceeding this maximum will be rejected.", "display_name": "Result Cache Maximum Size", "name": "impalad_result_cache_max_size", "value": "100000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Audit Log Directory", "name": "role_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "The amount of time at Impala Daemon startup allowed for the Impala Daemon to start accepting new queries for processing.", "display_name": "Impala Daemon Ready Status Startup Tolerance", "name": "impalad_ready_status_check_startup_tolerance", "value": "180" }, { "desc": "Enables the health test that the Impala Daemon's process state is consistent with the role configuration", "display_name": "Impala Daemon Process Health Test", "name": "impalad_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Hive Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Hive Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_hive_conf_safety_valve", "value": "false" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "Impala Daemon Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "The health test thresholds for failures encountered when monitoring queries within a recent period specified by the impala_query_monitoring_failure_window configuration for the role.", "display_name": "Query Monitoring Failures Thresholds", "name": "impala_query_monitoring_failure_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "An XML snippet to append to llama-site.xml for Impala Daemons. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve)", "name": "impala_llama_site_conf_safety_valve", "value": null }, { "desc": "Port where StateStoreSubscriberService is running.", "display_name": "StateStoreSubscriber Service Port", "name": "state_store_subscriber_port", "value": "23000" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "Impala Maximum Log Files", "name": "max_log_files", "value": "10" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_impalad_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_cmd_args_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Scratch Directories parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Scratch Directories", "name": "role_config_suppression_scratch_dirs", "value": "false" }, { "desc": "Maximum number of seconds that Impala attempts to register or re-register with Llama. If registration is unsuccessful, Impala cancels the action with an error, which could result in an impalad startup failure or a cancelled query. A setting of -1 seconds means try indefinitely.", "display_name": "Llama Registration Timeout Seconds", "name": "llama_registration_timeout_secs", "value": "30" }, { "desc": "The amount of time to wait for the Impala Daemon to fully start up and connect to the StateStore before enforcing the connectivity check.", "display_name": "Impala Daemon Connectivity Tolerance at Startup", "name": "impalad_connectivity_tolerance", "value": "180" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Impala Daemon Scratch Directories.", "display_name": "Impala Daemon Scratch Directories Free Space Monitoring Absolute Thresholds", "name": "impalad_scratch_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_fair_scheduler_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml for this role only.", "display_name": "Impala Daemon Hive Advanced Configuration Snippet (Safety Valve)", "name": "impala_hive_conf_safety_valve", "value": null }, { "desc": "Directories where Impala Daemon will write data such as spilling information to disk to free up memory. This can potentially be large amounts of data.", "display_name": "Impala Daemon Scratch Directories", "name": "scratch_dirs", "value": null }, { "desc": "A list of key-value pairs of additional query options to pass to the Impala Daemon command line, separated by ','.", "display_name": "Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve)", "name": "default_query_options", "value": "" }, { "desc": "The maximum size (in queries) of the Impala Daemon audit event log file before a new one is created.", "display_name": "Impala Daemon Maximum Audit Log File Size", "name": "max_audit_event_log_file_size", "value": "5000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Directory where an Impala Daemon core dump is placed.", "display_name": "Impala Daemon Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/impalad" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Lineage Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Lineage Log Directory", "name": "role_config_suppression_lineage_event_log_dir", "value": "false" }, { "desc": "Abort Impala startup if there are improper configs or running on unsupported hardware.", "display_name": "Abort on Config Error", "name": "abort_on_config_error", "value": "true" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Impala Daemon Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Server CA Certificate parameter.", "display_name": "Suppress Parameter Validation: LDAP Server CA Certificate", "name": "role_config_suppression_impalad_ldap_ca_certificate", "value": "false" }, { "desc": "The minimum log level for Impala Daemon logs", "display_name": "Impala Daemon Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Enables the health check that determines if the Impala daemon is ready to process queries.", "display_name": "Impala Daemon Ready Status Health Check", "name": "impalad_ready_status_check_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemons Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Impala Daemons Load Balancer", "name": "role_config_suppression_impalad_load_balancer", "value": "false" }, { "desc": "Directory where Impala Daemon will place its log files.", "display_name": "Impala Daemon Log Directory", "name": "log_dir", "value": "/var/log/impalad" }, { "desc": "Enables the health check that determines if query collection for the Impala Daemon is successful.", "display_name": "Impala Daemon Query Collection Status Health Check", "name": "impala_query_monitoring_status_check_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon HBase Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon HBase Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_hbase_conf_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_hdfs_site_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The timeout used by the Cloudera Manager Agent's query monitor when communicating with the Impala Daemon web server, specified in seconds.", "display_name": "Query Monitoring Timeout", "name": "executing_queries_timeout_seconds", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Audit Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Audit Enabled Validator", "name": "role_config_suppression_impala_audit_enabled_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_role_env_safety_valve", "value": "false" }, { "desc": "User-defined function (UDF) libraries are copied from HDFS into this local directory.", "display_name": "Local UDF Library Dir", "name": "local_library_dir", "value": "/var/lib/impala/udfs" }, { "desc": "The password for the private key in the Impala Daemon Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Impala Daemon Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "Whether to suppress the results of the Impala Daemon Ready Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Ready Check", "name": "role_health_suppression_impalad_ready_status", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Impala Daemon Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "Whether to suppress the results of the Query Monitoring Status Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Query Monitoring Status Check", "name": "role_health_suppression_impalad_query_monitoring_status", "value": "false" }, { "desc": "Port where Impala debug web server runs.", "display_name": "Impala Daemon HTTP Server Port", "name": "impalad_webserver_port", "value": "25000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Impala Daemon Scratch Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Impala Daemon Scratch Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Impala Daemon Scratch Directories Free Space Monitoring Percentage Thresholds", "name": "impalad_scratch_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Maximum number of times a request to reserve, expand, or release resources is attempted until the request is cancelled.", "display_name": "Llama Maximum Request Attempts", "name": "llama_max_request_attempts", "value": "5" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "impalad_web_metric_collection_enabled", "value": "true" }, { "desc": "Address of the load balancer used for Impala daemons. Should be specified in host:port format. If this is specified and Kerberos is enabled, Cloudera Manager adds a principal for 'impala/<load_balancer_host>@<realm>' to the keytab for all Impala daemons.", "display_name": "Impala Daemons Load Balancer", "name": "impalad_load_balancer", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_impalad_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)", "name": "IMPALAD_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the StateStore Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: StateStore Connectivity", "name": "role_health_suppression_impalad_connectivity", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "Enables lineage generation by Impala daemons. The lineage log file is placed in the directory specified by the 'Impala Daemon Lineage Log Directory' parameter.", "display_name": "Enable Impala Lineage Generation", "name": "enable_lineage_log", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_llama_site_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_impalad_file_descriptor", "value": "false" }, { "desc": "Memory limit in bytes for Impala Daemon, enforced by the daemon itself. If reached, queries running on the Impala Daemon may be killed. Leave it blank to let Impala pick its own limit. Use a value of -1 B to specify no limit.", "display_name": "Impala Daemon Memory Limit", "name": "impalad_memory_limit", "value": null }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above. ", "display_name": "Impala Daemon Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The polling period of the Impala query monitor in the Cloudera Manager Agent, specified in seconds. If set to zero, query monitoring is disabled.", "display_name": "Query Monitoring Period", "name": "query_monitoring_period_seconds", "value": "1.0" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_impalad_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "impalad_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Impala Daemon command-line flags. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "impalad_cmd_args_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-namenode.json0000666000175100017510000015076013245514472025116 0ustar zuulzuul00000000000000[ { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "namenode_pause_duration_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "namenode_startup_tolerance", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: NameNode Log Directory", "name": "role_config_suppression_namenode_log_dir", "value": "false" }, { "desc": "Mount points that are mapped to this NameNode's nameservice.", "display_name": "Mount Points", "name": "nameservice_mountpoints", "value": "/" }, { "desc": "The number of server threads for the NameNode.", "display_name": "NameNode Handler Count", "name": "dfs_namenode_handler_count", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Data Directories parameter.", "display_name": "Suppress Parameter Validation: NameNode Data Directories", "name": "role_config_suppression_dfs_name_dir_list", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "namenode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The health test thresholds of the age of the HDFS namespace checkpoint. Specified as a percentage of the configured checkpoint interval.", "display_name": "Filesystem Checkpoint Age Monitoring Thresholds", "name": "namenode_checkpoint_age_thresholds", "value": "{\"critical\":\"400.0\",\"warning\":\"200.0\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Checkpoint Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Checkpoint Status", "name": "role_health_suppression_name_node_ha_checkpoint_age", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Rolling Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Rolling Upgrade Status", "name": "role_health_suppression_name_node_rolling_upgrade_status", "value": "false" }, { "desc": "Full path to a custom topology script on the host file system. The topology script is used to determine the rack location of nodes. If left blank, a topology script will be provided that uses your hosts' rack information, visible in the \"Hosts\" page.", "display_name": "Topology Script File Name", "name": "topology_script_file_name", "value": null }, { "desc": "Comma-separated list of NameNode plug-ins to be activated. If one plug-in cannot be loaded, all the plug-ins are ignored.", "display_name": "NameNode Plugins", "name": "dfs_namenode_plugins_list", "value": "" }, { "desc": "This determines the total amount of block transfers to begin in parallel at a DataNode for replication, when such a command list is being sent over a DataNode heartbeat by the NameNode. The actual number is obtained by multiplying this value by the total number of live nodes in the cluster. The result number is the number of blocks to transfer immediately, per DataNode heartbeat.", "display_name": "Replication Work Multiplier Per Iteration", "name": "dfs_namenode_replication_work_multiplier_per_iteration", "value": "10" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NameNode Data Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NameNode Data Directories Free Space Monitoring Percentage Thresholds", "name": "namenode_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The absolute maximum number of outgoing replication threads a given node can have at one time. The regular limit (dfs.namenode.replication.max-streams) is waived for highest-priority block replications. Highest replication priority is for blocks that are at a very high risk of loss if the disk or server on which they remain fails. These are usually blocks with only one copy, or blocks with zero live copies but a copy in a node being decommissioned. dfs.namenode.replication.max-streams-hard-limit provides a limit on the total number of outgoing replication threads, including threads of all priorities.", "display_name": "Hard Limit on the Number of Replication Threads on a Datanode", "name": "dfs_namenode_replication_max_streams_hard_limit", "value": "40" }, { "desc": "If set to false and if one of the replicas of the NameNode storage fails, such as temporarily failure of NFS, this directory is not used until the NameNode restarts. If enabled, failed storage is re-checked on every checkpoint and, if it becomes valid, the NameNode will try to restore the edits and fsimage.", "display_name": "Restore NameNode Directories at Checkpoint Time", "name": "dfs_name_dir_restore", "value": "false" }, { "desc": "Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.replication.min. Enter a value less than or equal to 0 to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safemode permanent.", "display_name": "Safemode Threshold Percentage", "name": "dfs_safemode_threshold_pct", "value": "0.999" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topology Script File Name parameter.", "display_name": "Suppress Parameter Validation: Topology Script File Name", "name": "role_config_suppression_topology_script_file_name", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for NameNode logs. Typically used by log4j or logback.", "display_name": "NameNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The base port where the DFS NameNode web UI listens. If the port number is 0, then the server starts on a free port. Combined with the NameNode's hostname to build its HTTP address.", "display_name": "NameNode Web UI Port", "name": "dfs_http_port", "value": "50070" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Edits Directories parameter.", "display_name": "Suppress Parameter Validation: NameNode Edits Directories", "name": "role_config_suppression_dfs_namenode_edits_dir", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NameNode in Bytes", "name": "namenode_java_heapsize", "value": "4294967296" }, { "desc": "Timeout when writing edits to a JournalNode. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Write Transactions Timeout", "name": "dfs_qjournal_write_txns_timeout_ms", "value": "20000" }, { "desc": "The maximum number of outgoing replication threads a node can have at one time. This limit is waived for the highest priority replications. Configure dfs.namenode.replication.max-streams-hard-limit to set the absolute limit, including the highest-priority replications.", "display_name": "Maximum Number of Replication Threads on a DataNode", "name": "dfs_namenode_replication_max_streams", "value": "20" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt", "name": "role_config_suppression_namenode_hosts_allow_safety_valve", "value": "false" }, { "desc": "Directories on the local file system to store the NameNode edits. If not set, the edits are stored in the NameNode's Data Directories. The value of this configuration is automatically generated to be the Quorum-based Storage URI if there are JournalNodes and this NameNode is not Highly Available.", "display_name": "NameNode Edits Directories", "name": "dfs_namenode_edits_dir", "value": null }, { "desc": "Timeout in seconds for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Timeout", "name": "dfs_thrift_timeout", "value": "60" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_name_node_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NameNode", "name": "namenode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The health check thresholds of the NameNode's RPC latency.", "display_name": "NameNode RPC Latency Thresholds", "name": "namenode_rpc_latency_thresholds", "value": "{\"critical\":\"5000.0\",\"warning\":\"1000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where NameNode will place its log files.", "display_name": "NameNode Log Directory", "name": "namenode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "namenode_config_safety_valve", "value": null }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Determines extension of safemode in milliseconds after the threshold level is reached.", "display_name": "Safemode Extension", "name": "dfs_safemode_extension", "value": "30000" }, { "desc": "Whether to suppress the results of the JournalNode Sync Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JournalNode Sync Status", "name": "role_health_suppression_name_node_journal_node_sync_status", "value": "false" }, { "desc": "The health check thresholds for the number of out-of-sync JournalNodes for this NameNode.", "display_name": "NameNode Out-Of-Sync JournalNodes Thresholds", "name": "namenode_out_of_sync_journal_nodes_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Number of minutes between trash checkpoints. Also controls the number of minutes after which a trash checkpoint directory is deleted. To disable the trash feature, enter 0.", "display_name": "Filesystem Trash Interval", "name": "fs_trash_interval", "value": "1440" }, { "desc": "Optional port for the service-rpc address which can be used by HDFS daemons instead of sharing the RPC address used by the clients.", "display_name": "NameNode Service RPC Port", "name": "dfs_namenode_servicerpc_address", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt", "name": "role_config_suppression_namenode_hosts_exclude_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the NameNode Data Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Data Directories Free Space", "name": "role_health_suppression_name_node_data_directories_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_name_node_log_directory_free_space", "value": "false" }, { "desc": "The period to review when computing the moving average of the NameNode's RPC latency.", "display_name": "NameNode RPC Latency Monitoring Window", "name": "namenode_rpc_latency_window", "value": "5" }, { "desc": "Enable Automatic Failover to maintain High Availability. Requires a ZooKeeper service and a High Availability NameNode partner.", "display_name": "Enable Automatic Failover", "name": "autofailover_enabled", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the Filesystem Trash Interval On Validator configuration validator.", "display_name": "Suppress Configuration Validator: Filesystem Trash Interval On Validator", "name": "role_config_suppression_fs_trash_interval_minimum_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Determines where on the local file system the NameNode should store the name table (fsimage). For redundancy, enter a comma-delimited list of directories to replicate the name table in all of the directories. Typical values are /data/N/dfs/nn where N=1..3.", "display_name": "NameNode Data Directories", "name": "dfs_name_dir_list", "value": null }, { "desc": "The base port where the secure NameNode web UI listens.", "display_name": "Secure NameNode Web UI Port (TLS/SSL)", "name": "dfs_https_port", "value": "50470" }, { "desc": "Enables the health test of the rolling metadata upgrade status of the NameNode. This covers rolling metadata upgrades. Nonrolling metadata upgrades are covered in a separate health test.", "display_name": "HDFS Rolling Metadata Upgrade Status Health Test", "name": "namenode_rolling_upgrade_status_enabled", "value": "true" }, { "desc": "When computing the overall NameNode health, consider the host's health.", "display_name": "NameNode Host Health Test", "name": "namenode_host_health_enabled", "value": "true" }, { "desc": "Enables the health test of the metadata upgrade status of the NameNode. This covers nonrolling metadata upgrades. Rolling metadata upgrades are covered in a separate health test.", "display_name": "HDFS Metadata Upgrade Status Health Test", "name": "namenode_upgrade_status_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_name_node_host_health", "value": "false" }, { "desc": "If enabled, the NameNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind NameNode to Wildcard Address", "name": "namenode_bind_wildcard", "value": "false" }, { "desc": "Indicate whether or not to avoid reading from stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Stale DataNodes are moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.", "display_name": "Avoid Reading Stale DataNode", "name": "dfs_namenode_avoid_read_stale_datanode", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Service Handler Count Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Service Handler Count Minimum Validator", "name": "role_config_suppression_dfs_namenode_service_handler_count_minimum_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Quorum-based Storage Journal name parameter.", "display_name": "Suppress Parameter Validation: Quorum-based Storage Journal name", "name": "role_config_suppression_dfs_namenode_quorum_journal_name", "value": "false" }, { "desc": "This determines the percentage amount of block invalidations (deletes) to do over a single DataNode heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DataNode.", "display_name": "Invalidate Work Percentage Per Iteration", "name": "dfs_namenode_invalidate_work_pct_per_iteration", "value": "0.32" }, { "desc": "Whether to suppress configuration warnings produced by the Java Heap Size of NameNode in Bytes Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: Java Heap Size of NameNode in Bytes Minimum Validator", "name": "role_config_suppression_namenode_java_heapsize_minimum_validator", "value": "false" }, { "desc": "The minimum log level for NameNode logs", "display_name": "NameNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_name_node_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Name Directory Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Name Directory Status", "name": "role_health_suppression_name_node_directory_failures", "value": "false" }, { "desc": "Enables the health test that the NameNode's process state is consistent with the role configuration", "display_name": "NameNode Process Health Test", "name": "namenode_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mount Points parameter.", "display_name": "Suppress Parameter Validation: Mount Points", "name": "role_config_suppression_nameservice_mountpoints", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into dfs_hosts_exclude.txt for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt", "name": "namenode_hosts_exclude_safety_valve", "value": null }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "namenode_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_name_node_scm_health", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_name_node_pause_duration", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Minimum number of running threads for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Min Threadcount", "name": "dfs_thrift_threads_min", "value": "10" }, { "desc": "The access time for HDFS file is precise upto this value. Setting the value of 0 disables access times for HDFS. When using the NFS Gateway role, make sure this property is enabled.", "display_name": "Access Time Precision", "name": "dfs_access_time_precision", "value": "3600000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories.", "display_name": "NameNode Data Directories Free Space Monitoring Absolute Thresholds", "name": "namenode_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The minimum log level for NameNode block state change log messages. Setting this to WARN or higher greatly reduces the amount of log output related to block state changes.", "display_name": "NameNode Block State Change Logging Threshold", "name": "namenode_blockstatechange_log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Upgrade Status", "name": "role_health_suppression_name_node_upgrade_status", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the RPC Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: RPC Latency", "name": "role_health_suppression_name_node_rpc_latency", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "namenode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Nameservice of this NameNode. The Nameservice represents the interface to this NameNode and its High Availability partner. The Nameservice also represents the namespace associated with a federated NameNode.", "display_name": "NameNode Nameservice", "name": "dfs_federation_namenode_nameservice", "value": null }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "namenode_web_metric_collection_enabled", "value": "true" }, { "desc": "The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired.", "display_name": "Filesystem Checkpoint Transaction Threshold", "name": "fs_checkpoint_txns", "value": "1000000" }, { "desc": "When the ratio of number stale DataNodes to total DataNodes marked is greater than this ratio, permit writing to stale nodes to prevent causing hotspots.", "display_name": "Write Stale DataNode Ratio", "name": "dfs_namenode_write_stale_datanode_ratio", "value": "0.5" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds of failed status directories in a NameNode.", "display_name": "NameNode Directory Failures Thresholds", "name": "namenode_directory_failures_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Plugins parameter.", "display_name": "Suppress Parameter Validation: NameNode Plugins", "name": "role_config_suppression_dfs_namenode_plugins_list", "value": "false" }, { "desc": "Directory on a shared storage device, such as a Quorum-based Storage URI or a local directory that is an NFS mount from a NAS, to store the NameNode edits. The value of this configuration is automatically generated to be the Quourm Journal URI if there are JournalNodes and this NameNode is Highly Available.", "display_name": "Shared Edits Directory", "name": "dfs_namenode_shared_edits_dir", "value": null }, { "desc": "Enables the health test that the NameNode is not in safemode", "display_name": "NameNode Safemode Health Test", "name": "namenode_safe_mode_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Handler Count Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Handler Count Minimum Validator", "name": "role_config_suppression_dfs_namenode_handler_count_minimum_validator", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into dfs_hosts_allow.txt for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt", "name": "namenode_hosts_allow_safety_valve", "value": null }, { "desc": "Maximum number of running threads for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Max Threadcount", "name": "dfs_thrift_threads_max", "value": "20" }, { "desc": "Default time interval for marking a DataNode as \"stale\". If the NameNode has not received heartbeat messages from a DataNode for more than this time interval, the DataNode is marked and treated as \"stale\" by default.", "display_name": "Stale DataNode Time Interval", "name": "dfs_namenode_stale_datanode_interval", "value": "30000" }, { "desc": "The maximum number of rolled log files to keep for NameNode logs. Typically used by log4j or logback.", "display_name": "NameNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NameNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NameNode", "name": "role_config_suppression_namenode_java_opts", "value": "false" }, { "desc": "The health test thresholds of the number of transactions since the last HDFS namespace checkpoint. Specified as a percentage of the configured checkpointing transaction limit.", "display_name": "Filesystem Checkpoint Transactions Monitoring Thresholds", "name": "namenode_checkpoint_transactions_thresholds", "value": "{\"critical\":\"400.0\",\"warning\":\"200.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_namenode_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "NAMENODE_role_env_safety_valve", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The number of server threads for the NameNode used for service calls. Only used when NameNode Service RPC Port is configured.", "display_name": "NameNode Service Handler Count", "name": "dfs_namenode_service_handler_count", "value": "30" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"},\n {\"alert\": false, \"rate\": 1, \"threshold\":\"INFO\", \"content\":\"Triggering checkpoint.*\"}\n ]\n}\n" }, { "desc": "Name of the journal located on each JournalNode filesystem.", "display_name": "Quorum-based Storage Journal name", "name": "dfs_namenode_quorum_journal_name", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_name_node_file_descriptor", "value": "false" }, { "desc": "The port where the NameNode runs the HDFS protocol. Combined with the NameNode's hostname to build its address.", "display_name": "NameNode Port", "name": "namenode_port", "value": "8020" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Edits Directory parameter.", "display_name": "Suppress Parameter Validation: Shared Edits Directory", "name": "role_config_suppression_dfs_namenode_shared_edits_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_namenode_config_safety_valve", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_name_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_name_node_swap_memory_usage", "value": "false" }, { "desc": "The time between two periodic file system checkpoints.", "display_name": "Filesystem Checkpoint Period", "name": "fs_checkpoint_period", "value": "3600" }, { "desc": "Specifies the number of DataNodes that must be live before the name node exits safemode. Enter a value less than or equal to 0 to take the number of live DataNodes into account when deciding whether to remain in safemode during startup. Values greater than the number of DataNodes in the cluster will make safemode permanent.", "display_name": "Safemode Minimum DataNodes", "name": "dfs_safemode_min_datanodes", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Nameservice parameter.", "display_name": "Suppress Parameter Validation: NameNode Nameservice", "name": "role_config_suppression_dfs_federation_namenode_nameservice", "value": "false" }, { "desc": "Whether to suppress the results of the Safe Mode Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Safe Mode Status", "name": "role_health_suppression_name_node_safe_mode", "value": "false" }, { "desc": "Indicate whether or not to avoid writing to stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Writes avoid using stale DataNodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of DataNodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads.", "display_name": "Avoid Writing Stale DataNode", "name": "dfs_namenode_avoid_write_stale_datanode", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/spark-service.json0000666000175100017510000002251113245514472025154 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Spark Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_spark_on_yarn_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "The location of the Spark JAR in HDFS. If left blank, Cloudera Manager will use the Spark JAR installed on the cluster nodes.", "display_name": "Spark JAR Location (HDFS)", "name": "spark_jar_hdfs_path", "value": "" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "spark" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Spark Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "SPARK_ON_YARN_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark History Location (HDFS) parameter.", "display_name": "Suppress Parameter Validation: Spark History Location (HDFS)", "name": "service_config_suppression_spark_history_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the History Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: History Server Count Validator", "name": "service_config_suppression_spark_yarn_history_server_count_validator", "value": "false" }, { "desc": "Name of the YARN (MR2 Included) service that this Spark service instance depends on", "display_name": "YARN (MR2 Included) Service", "name": "yarn_service", "value": null }, { "desc": "The location of Spark application history logs in HDFS. Changing this value will not move existing logs to the new location.", "display_name": "Spark History Location (HDFS)", "name": "spark_history_log_dir", "value": "/user/spark/applicationHistory" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "The port the Spark Shuffle Service listens for fetch requests.", "display_name": "Spark Shuffle Service Port", "name": "spark_shuffle_service_port", "value": "7337" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark JAR Location (HDFS) parameter.", "display_name": "Suppress Parameter Validation: Spark JAR Location (HDFS)", "name": "service_config_suppression_spark_jar_hdfs_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "service_config_suppression_spark-conf/spark-env.sh_service_safety_valve", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "spark" }, { "desc": "Enable whether the Spark communication protocols do authentication using a shared secret.", "display_name": "Spark Authentication", "name": "spark_authenticate", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into spark-conf/spark-env.sh. Applies to configurations of all roles in this service except client configuration.", "display_name": "Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_service_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-secondarynamenode.json0000666000175100017510000007566213245514472027035 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "secondarynamenode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired.", "display_name": "Filesystem Checkpoint Transaction Threshold", "name": "fs_checkpoint_txns", "value": "1000000" }, { "desc": "Directory where SecondaryNameNode will place its log files.", "display_name": "SecondaryNameNode Log Directory", "name": "secondarynamenode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_secondary_name_node_web_metric_collection", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_secondarynamenode_config_safety_valve", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for SecondaryNameNode logs. Typically used by log4j or logback.", "display_name": "SecondaryNameNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When computing the overall SecondaryNameNode health, consider the host's health.", "display_name": "SecondaryNameNode Host Health Test", "name": "secondarynamenode_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_secondary_name_node_host_health", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the HDFS Checkpoint Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HDFS Checkpoint Directories Free Space", "name": "role_health_suppression_secondary_name_node_checkpoint_directories_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Determines where on the local file system the HDFS SecondaryNameNode should store the temporary images to merge. For redundancy, enter a comma-delimited list of directories to replicate the image in all of the directories. Typical values are /data/N/dfs/snn for N = 1, 2, 3...", "display_name": "HDFS Checkpoint Directories", "name": "fs_checkpoint_dir_list", "value": null }, { "desc": "The minimum log level for SecondaryNameNode logs", "display_name": "SecondaryNameNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_secondary_name_node_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "secondarynamenode_config_safety_valve", "value": null }, { "desc": "The time between two periodic file system checkpoints.", "display_name": "Filesystem Checkpoint Period", "name": "fs_checkpoint_period", "value": "3600" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories.", "display_name": "HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds", "name": "secondarynamenode_checkpoint_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "HDFS Checkpoint Directories Free Space Monitoring Percentage Thresholds", "name": "secondarynamenode_checkpoint_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "secondarynamenode_gc_duration_window", "value": "5" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "SECONDARYNAMENODE_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Secondary NameNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Secondary NameNode", "name": "role_config_suppression_secondarynamenode_java_opts", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_secondary_name_node_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Enables the health test that the SecondaryNameNode's process state is consistent with the role configuration", "display_name": "SecondaryNameNode Process Health Test", "name": "secondarynamenode_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "secondarynamenode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "secondarynamenode_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The base port where the secure SecondaryNameNode web UI listens.", "display_name": "Secure SecondaryNameNode Web UI Port (TLS/SSL)", "name": "dfs_secondary_https_port", "value": "50495" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"},\n {\"alert\": false, \"rate\": 1, \"threshold\":\"INFO\", \"content\":\"Triggering checkpoint.*\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_secondary_name_node_file_descriptor", "value": "false" }, { "desc": "Nameservice of this SecondaryNameNode", "display_name": "SecondaryNameNode Nameservice", "name": "dfs_secondarynamenode_nameservice", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "If enabled, the SecondaryNameNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind SecondaryNameNode to Wildcard Address", "name": "secondary_namenode_bind_wildcard", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Log Directory", "name": "role_config_suppression_secondarynamenode_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_secondary_name_node_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The maximum size, in megabytes, per log file for SecondaryNameNode logs. Typically used by log4j or logback.", "display_name": "SecondaryNameNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_secondary_name_node_unexpected_exits", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Secondary NameNode", "name": "secondarynamenode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Checkpoint Directories parameter.", "display_name": "Suppress Parameter Validation: HDFS Checkpoint Directories", "name": "role_config_suppression_fs_checkpoint_dir_list", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_secondarynamenode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_secondary_name_node_gc_duration", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_secondary_name_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Secondary NameNode in Bytes", "name": "secondary_namenode_java_heapsize", "value": "4294967296" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Nameservice parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Nameservice", "name": "role_config_suppression_dfs_secondarynamenode_nameservice", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "secondarynamenode_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The SecondaryNameNode HTTP port. If the port is 0, then the server starts on a free port. Combined with the SecondaryNameNode's hostname to build its HTTP address.", "display_name": "SecondaryNameNode Web UI Port", "name": "dfs_secondary_http_port", "value": "50090" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/yarn-jobhistory.json0000666000175100017510000007532313245514472025552 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_jobhistory_swap_memory_usage", "value": "false" }, { "desc": "The user that the JobHistory Server process should run as.", "display_name": "System User", "name": "history_process_username", "value": "mapred" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The HTTP port of the MapReduce JobHistory Server web application. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Web Application HTTP Port", "name": "mapreduce_jobhistory_webapp_address", "value": "19888" }, { "desc": "Kerberos principal used by the JobHistory Server roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "mapred" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "jobhistory_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_jobhistory_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "jobhistory_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_jobhistory_host_health", "value": "false" }, { "desc": "The port of the MapReduce JobHistory Server. Together with the hostname of the JobHistory role, forms the address.", "display_name": "MapReduce JobHistory Server Port", "name": "mapreduce_jobhistory_address", "value": "10020" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Log Directory", "name": "role_config_suppression_mr2_jobhistory_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_jobhistory_scm_health", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "jobhistory_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The port of the MapReduce JobHistory Server administrative interface. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Server Admin Interface Port", "name": "mapreduce_jobhistory_admin_address", "value": "10033" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "jobhistory_config_safety_valve", "value": null }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "jobhistory_web_metric_collection_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for JobHistory Server logs. Typically used by log4j or logback.", "display_name": "JobHistory Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_jobhistory_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_jobhistory_file_descriptor", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_history_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_jobhistory_gc_duration", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_jobhistory_heap_dump_directory_free_space", "value": "false" }, { "desc": "The HTTPS port of the MapReduce JobHistory Server web application. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Web Application HTTPS Port (TLS/SSL)", "name": "mapreduce_jobhistory_webapp_https_address", "value": "19890" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The minimum log level for JobHistory Server logs", "display_name": "JobHistory Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_jobhistory_log_directory_free_space", "value": "false" }, { "desc": "The group that the JobHistory Server process should run as.", "display_name": "System Group", "name": "history_process_groupname", "value": "hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "jobhistory_mapred_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_history_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_jobhistory_mapred_safety_valve", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for JobHistory Server", "name": "mr2_jobhistory_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "jobhistory_gc_duration_window", "value": "5" }, { "desc": "Time interval for history cleaner to check for files to delete. Files are only deleted if they are older than mapreduce.jobhistory.max-age-ms.", "display_name": "Job History Files Cleaner Interval", "name": "mapreduce_jobhistory_cleaner_interval", "value": "86400000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_jobhistory_unexpected_exits", "value": "false" }, { "desc": "Enables the health test that the JobHistory Server's process state is consistent with the role configuration", "display_name": "JobHistory Server Process Health Test", "name": "jobhistory_scm_health_enabled", "value": "true" }, { "desc": "Job history files older than this time duration will deleted when the history cleaner runs.", "display_name": "Job History Files Maximum Age", "name": "mapreduce_jobhistory_max_age_ms", "value": "604800000" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "JobHistory Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "JOBHISTORY_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MapReduce ApplicationMaster Staging Root Directory parameter.", "display_name": "Suppress Parameter Validation: MapReduce ApplicationMaster Staging Root Directory", "name": "role_config_suppression_yarn_app_mapreduce_am_staging_dir", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_jobhistory_web_metric_collection", "value": "false" }, { "desc": "The root HDFS directory of the staging area for users' MR2 jobs; for example /user. The staging directories are always named after the user.", "display_name": "MapReduce ApplicationMaster Staging Root Directory", "name": "yarn_app_mapreduce_am_staging_dir", "value": "/user" }, { "desc": "If enabled, the JobHistory Server binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind JobHistory Server to Wildcard Address", "name": "yarn_jobhistory_bind_wildcard", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for JobHistory Server logs. Typically used by log4j or logback.", "display_name": "JobHistory Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "JobHistory Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When computing the overall JobHistory Server health, consider the host's health.", "display_name": "JobHistory Server Host Health Test", "name": "jobhistory_host_health_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of JobHistory Server in Bytes", "name": "mr2_jobhistory_java_heapsize", "value": "1073741824" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Directory where JobHistory Server will place its log files.", "display_name": "JobHistory Server Log Directory", "name": "mr2_jobhistory_log_dir", "value": "/var/log/hadoop-mapreduce" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JobHistory Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for JobHistory Server", "name": "role_config_suppression_mr2_jobhistory_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/flume-service.json0000666000175100017510000002445313245514472025153 0ustar zuulzuul00000000000000[ { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Sets the maximum number of Flume components that will be returned under Flume Metric Details. Increasing this value will negatively impact the interactive performance of the Flume Metrics Details page.", "display_name": "Maximum displayed Flume metrics components", "name": "flume_context_groups_request_limit", "value": "1000" }, { "desc": "Whether to suppress configuration warnings produced by the Agent Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Agent Count Validator", "name": "service_config_suppression_agent_count_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The health test thresholds of the overall Agent health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Agents falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Agents falls below the critical threshold.", "display_name": "Healthy Agent Monitoring Thresholds", "name": "flume_agents_healthy_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"95.0\"}" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Flume Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "flume_env_safety_valve", "value": null }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "flume" }, { "desc": "The password for the Flume TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Flume TLS/SSL Certificate Trust Store Password", "name": "flume_truststore_password", "value": null }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Flume might connect to. This is used when Flume is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Flume TLS/SSL Certificate Trust Store File", "name": "flume_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress the results of the Agent Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Agent Health", "name": "service_health_suppression_flume_agents_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Flume Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_flume_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Flume TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_flume_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Name of the HDFS service that this Flume service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "flume" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "flume" }, { "desc": "Name of the Solr service that this Flume service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Flume TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_flume_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the Hbase service that this Flume service instance depends on", "display_name": "Hbase Service", "name": "hbase_service", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/kafka-service.json0000666000175100017510000005267413245514472025126 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Controlled Shutdown Maximum Attempts parameter.", "display_name": "Suppress Parameter Validation: Controlled Shutdown Maximum Attempts", "name": "service_config_suppression_controlled.shutdown.max.retries", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_kafka_service_env_safety_valve", "value": "false" }, { "desc": "The default number of partitions for automatically created topics.", "display_name": "Default Number of Partitions", "name": "num.partitions", "value": "1" }, { "desc": "The amount of time to retain delete messages for log compacted topics. Once a consumer has seen an original message you need to ensure it also sees the delete message. If you removed the delete message too quickly, this might not happen. As a result there is a configurable delete retention time.", "display_name": "Log Compaction Delete Record Retention Time", "name": "log.cleaner.delete.retention.ms", "value": "604800000" }, { "desc": "Enables auto creation of topics on the server. If this is set to true, then attempts to produce, consume, or fetch metadata for a non-existent topic automatically create the topic with the default replication factor and number of partitions.", "display_name": "Topic Auto Creation", "name": "auto.create.topics.enable", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Number of threads used to replicate messages from leaders. Increasing this value increases the degree of I/O parallelism in the follower broker.", "display_name": "Number of Replica Fetchers", "name": "num.replica.fetchers", "value": "1" }, { "desc": "Enables Kafka monitoring.", "display_name": "Enable Kafka Monitoring (Note: Requires Kafka-1.3.0 parcel or higher)", "name": "monitoring.enabled", "value": "true" }, { "desc": "If automatic leader rebalancing is enabled, the controller tries to balance leadership for partitions among the brokers by periodically returning leadership for each partition to the preferred replica, if it is available.", "display_name": "Enable Automatic Leader Rebalancing", "name": "auto.leader.rebalance.enable", "value": "true" }, { "desc": "Number of unsuccessful controlled shutdown attempts before executing an unclean shutdown. For example, the default value of 3 means that the system will attempt a controlled shutdown 3 times before executing an unclean shutdown.", "display_name": "Controlled Shutdown Maximum Attempts", "name": "controlled.shutdown.max.retries", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "The number of partitions for the offset commit topic. Since changing this after deployment is currently unsupported, we recommend using a higher setting for production (for example, 100-200).", "display_name": "Offset Commit Topic Number of Partitions", "name": "offsets.topic.num.partitions", "value": "50" }, { "desc": "If a follower has not sent any fetch requests, nor has it consumed up to the leader's log end offset during this time, the leader removes the follower from the ISR set.", "display_name": "Allowed Replica Time Lag", "name": "replica.lag.time.max.ms", "value": "10000" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Replica Maximum Fetch Size parameter.", "display_name": "Suppress Parameter Validation: Replica Maximum Fetch Size", "name": "service_config_suppression_replica.fetch.max.bytes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The replication factor for the offset commit topic. A higher setting is recommended in order to ensure higher availability (for example, 3 or 4) . If the offsets topic is created when there are fewer brokers than the replication factor, then the offsets topic is created with fewer replicas.", "display_name": "Offset Commit Topic Replication Factor", "name": "offsets.topic.replication.factor", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the Kafka Broker Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kafka Broker Count Validator", "name": "service_config_suppression_kafka_broker_count_validator", "value": "false" }, { "desc": "Controls how frequently the log cleaner will attempt to clean the log. This ratio bounds the maximum space wasted in the log by duplicates. For example, at 0.5 at most 50% of the log could be duplicates. A higher ratio will mean fewer, more efficient cleanings but will mean more wasted space in the log.", "display_name": "Log Cleaner Clean Ratio", "name": "log.cleaner.min.cleanable.ratio", "value": "0.5" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "kafka" }, { "desc": "Enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so might result in data loss.", "display_name": "Enable Unclean Leader Election", "name": "unclean.leader.election.enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Default Replication Factor", "name": "service_config_suppression_default.replication.factor", "value": "false" }, { "desc": "Enables controlled shutdown of the broker. If enabled, the broker moves all leaders on it to other brokers before shutting itself down. This reduces the unavailability window during shutdown.", "display_name": "Enable Controlled Shutdown", "name": "controlled.shutdown.enable", "value": "true" }, { "desc": "The frequency with which to check for leader imbalance.", "display_name": "Leader Imbalance Check Interval", "name": "leader.imbalance.check.interval.seconds", "value": "300" }, { "desc": "The maximum number of bytes to fetch for each partition in fetch requests replicas send to the leader. This value should be larger than message.max.bytes.", "display_name": "Replica Maximum Fetch Size", "name": "replica.fetch.max.bytes", "value": "1048576" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Session Timeout parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Session Timeout", "name": "service_config_suppression_zookeeper.session.timeout.ms", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of Replica Fetchers parameter.", "display_name": "Suppress Parameter Validation: Number of Replica Fetchers", "name": "service_config_suppression_num.replica.fetchers", "value": "false" }, { "desc": "Any producer distinguished by clientId will get throttled if it produces more bytes than this value per-second. Only respected by Kafka 2.0 or later.", "display_name": "Default Producer Quota", "name": "quota.producer.default", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Kafka Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_service_env_safety_valve", "value": null }, { "desc": "The maximum size of a message that the server can receive. It is important that this property be in sync with the maximum fetch size the consumers use, or else an unruly producer could publish messages too large for consumers to consume.", "display_name": "Maximum Message Size", "name": "message.max.bytes", "value": "1000000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Leader Imbalance Allowed Per Broker parameter.", "display_name": "Suppress Parameter Validation: Leader Imbalance Allowed Per Broker", "name": "service_config_suppression_leader.imbalance.per.broker.percentage", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Enables the log cleaner to compact topics with cleanup.policy=compact on this cluster.", "display_name": "Enable Log Compaction", "name": "log.cleaner.enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Minimum Number of Replicas in ISR parameter.", "display_name": "Suppress Parameter Validation: Minimum Number of Replicas in ISR", "name": "service_config_suppression_min.insync.replicas", "value": "false" }, { "desc": "ZNode in ZooKeeper that should be used as a root for this Kafka cluster.", "display_name": "ZooKeeper Root", "name": "zookeeper.chroot", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the Kafka MirrorMaker Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kafka MirrorMaker Count Validator", "name": "service_config_suppression_kafka_mirror_maker_count_validator", "value": "false" }, { "desc": "The number of messages written to a log partition before triggering an fsync on the log. Setting this lower syncs data to disk more often, but has a major impact on performance. We recommend use of replication for durability rather than depending on single-server fsync; however, this setting can be used to be extra certain. If used in conjunction with log.flush.interval.ms, the log is flushed when either criteria is met.", "display_name": "Log Flush Message Interval", "name": "log.flush.interval.messages", "value": null }, { "desc": "The number of background threads to use for log cleaning.", "display_name": "Number of Log Cleaner Threads", "name": "log.cleaner.threads", "value": "1" }, { "desc": "Enable Kerberos authentication for this KAFKA service.", "display_name": "Enable Kerberos Authentication", "name": "kerberos.auth.enable", "value": "false" }, { "desc": "List of metric reporter class names. HTTP reporter is included by default.", "display_name": "List of Metric Reporters", "name": "kafka.metrics.reporters", "value": "nl.techop.kafka.KafkaHttpMetricsReporter" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Root parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Root", "name": "service_config_suppression_zookeeper.chroot", "value": "false" }, { "desc": "The frequency, in ms, with which the log flusher checks whether any log is eligible to be flushed to disk.", "display_name": "Log Flush Scheduler Interval", "name": "log.flush.scheduler.interval.ms", "value": null }, { "desc": "The minimum number of replicas in the in-sync replica needed to satisfy a produce request where required.acks=-1 (that is, all).", "display_name": "Minimum Number of Replicas in ISR", "name": "min.insync.replicas", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Offset Commit Topic Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Offset Commit Topic Replication Factor", "name": "service_config_suppression_offsets.topic.replication.factor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Leader Imbalance Check Interval parameter.", "display_name": "Suppress Parameter Validation: Leader Imbalance Check Interval", "name": "service_config_suppression_leader.imbalance.check.interval.seconds", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "If the server fails to send a heartbeat to ZooKeeper within this period of time, it is considered dead. If set too low, ZooKeeper might falsely consider a server dead; if set too high, ZooKeeper might take too long to recognize a dead server.", "display_name": "ZooKeeper Session Timeout", "name": "zookeeper.session.timeout.ms", "value": "6000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Maximum Message Size parameter.", "display_name": "Suppress Parameter Validation: Maximum Message Size", "name": "service_config_suppression_message.max.bytes", "value": "false" }, { "desc": "Enables topic deletion using admin tools. When delete topic is disabled, deleting topics through the admin tools has no effect.", "display_name": "Enable Delete Topic", "name": "delete.topic.enable", "value": "true" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "If a replica falls more than this number of messages behind the leader, the leader removes the follower from the ISR and treats it as dead. This property is deprecated in Kafka 1.4.0; higher versions use only replica.lag.time.max.ms.", "display_name": "Allowed Replica Message Lag", "name": "replica.lag.max.messages", "value": "4000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Number of Partitions parameter.", "display_name": "Suppress Parameter Validation: Default Number of Partitions", "name": "service_config_suppression_num.partitions", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Kafka service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The total memory used for log deduplication across all cleaner threads. This memory is statically allocated and will not cause GC problems.", "display_name": "Log Cleaner Deduplication Buffer Size", "name": "log.cleaner.dedupe.buffer.size", "value": "134217728" }, { "desc": "Any consumer distinguished by clientId/consumer group will get throttled if it fetches more bytes than this value per-second. Only respected by Kafka 2.0 or later.", "display_name": "Default Consumer Quota", "name": "quota.consumer.default", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Offset Commit Topic Number of Partitions parameter.", "display_name": "Suppress Parameter Validation: Offset Commit Topic Number of Partitions", "name": "service_config_suppression_offsets.topic.num.partitions", "value": "false" }, { "desc": "The default replication factor for automatically created topics.", "display_name": "Default Replication Factor", "name": "default.replication.factor", "value": "1" }, { "desc": "The maximum time between fsync calls on the log. If used in conjuction with log.flush.interval.messages, the log is flushed when either criteria is met.", "display_name": "Log Flush Time Interval", "name": "log.flush.interval.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the List of Metric Reporters parameter.", "display_name": "Suppress Parameter Validation: List of Metric Reporters", "name": "service_config_suppression_kafka.metrics.reporters", "value": "false" }, { "desc": "The percentage of leader imbalance allowed per broker. The controller rebalances leadership if this ratio goes above the configured value per broker.", "display_name": "Leader Imbalance Allowed Per Broker", "name": "leader.imbalance.per.broker.percentage", "value": "10" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/hdfs-httpfs.json0000666000175100017510000006400413245514472024633 0ustar zuulzuul00000000000000[ { "desc": "The user that the HttpFS server process should run as.", "display_name": "System User", "name": "httpfs_process_username", "value": "httpfs" }, { "desc": "The password for the HttpFS JKS keystore file.", "display_name": "HttpFS TLS/SSL Server JKS Keystore File Password", "name": "httpfs_https_keystore_password", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_httpfs_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_httpfs_https_keystore_file", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HttpFS Environment Advanced Configuration Snippet (Safety Valve)", "name": "HTTPFS_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "httpfs_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_httpfs_log_directory_free_space", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Enables the health test that the HttpFS's process state is consistent with the role configuration", "display_name": "HttpFS Process Health Test", "name": "httpfs_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Load Balancer parameter.", "display_name": "Suppress Parameter Validation: HttpFS Load Balancer", "name": "role_config_suppression_httpfs_load_balancer", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml", "name": "role_config_suppression_httpfs_config_safety_valve", "value": "false" }, { "desc": "Directory where HttpFS will place its log files.", "display_name": "HttpFS Log Directory", "name": "httpfs_log_dir", "value": "/var/log/hadoop-httpfs" }, { "desc": "The minimum log level for HttpFS logs", "display_name": "HttpFS Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HttpFS in Bytes", "name": "httpfs_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_httpfs_https_keystore_password", "value": "false" }, { "desc": "The password for the HttpFS TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HttpFS TLS/SSL Certificate Trust Store Password", "name": "httpfs_https_truststore_password", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_httpfs_swap_memory_usage", "value": "false" }, { "desc": "The port where the REST interface to HDFS is available. The REST interface is served over HTTPS if TLS/SSL is enabled for HttpFS, or over HTTP otherwise.", "display_name": "REST Port", "name": "hdfs_httpfs_http_port", "value": "14000" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HttpFS logs. Typically used by log4j or logback.", "display_name": "HttpFS Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "When computing the overall HttpFS health, consider the host's health.", "display_name": "HttpFS Host Health Test", "name": "httpfs_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_httpfs_https_truststore_password", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The secret to use for signing client authentication tokens.", "display_name": "Signature Secret", "name": "hdfs_httpfs_signature_secret", "value": "hadoop httpfs secret" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_httpfs_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HttpFS might connect to. This is used when HttpFS is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HttpFS TLS/SSL Certificate Trust Store File", "name": "httpfs_https_truststore_file", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_httpfs_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HttpFS Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_httpfs_role_env_safety_valve", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HttpFS", "name": "httpfs_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HttpFS Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Encrypt communication between clients and HttpFS using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HttpFS", "name": "httpfs_use_ssl", "value": "false" }, { "desc": "The group that the HttpFS server process should run as.", "display_name": "System Group", "name": "httpfs_process_groupname", "value": "httpfs" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_httpfs_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Signature Secret parameter.", "display_name": "Suppress Parameter Validation: Signature Secret", "name": "role_config_suppression_hdfs_httpfs_signature_secret", "value": "false" }, { "desc": "The port for the administration interface.", "display_name": "Administration Port", "name": "hdfs_httpfs_admin_port", "value": "14001" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_httpfs_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Log Directory parameter.", "display_name": "Suppress Parameter Validation: HttpFS Log Directory", "name": "role_config_suppression_httpfs_log_dir", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_httpfs_https_truststore_file", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for HttpFS logs. Typically used by log4j or logback.", "display_name": "HttpFS Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HttpFS is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HttpFS TLS/SSL Server JKS Keystore File Location", "name": "httpfs_https_keystore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_httpfs_process_username", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only. A string to be inserted into httpfs-site.xml for this role only.", "display_name": "HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml", "name": "httpfs_config_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HttpFS Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Kerberos principal used by the HttpFS roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "httpfs" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_httpfs_unexpected_exits", "value": "false" }, { "desc": "Address of the load balancer used for HttpFS roles. Should be specified in host:port format. Note: Changing this property will regenerate Kerberos keytabs for all HttpFS roles.", "display_name": "HttpFS Load Balancer", "name": "httpfs_load_balancer", "value": null }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HttpFS parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HttpFS", "name": "role_config_suppression_httpfs_java_opts", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/oozie-service.json0000666000175100017510000003204613245514472025165 0ustar zuulzuul00000000000000[ { "desc": "Service to run MapReduce jobs against", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null }, { "desc": "Encrypt communication between clients and Oozie using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Oozie", "name": "oozie_use_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_oozie_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to configure Oozie properties needed for JMS integration", "display_name": "Enable JMS Integration", "name": "oozie_use_jms", "value": "false" }, { "desc": "Whether to configure Oozie properties needed for SLA integration", "display_name": "Enable SLA Integration", "name": "oozie_use_sla", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Coordinator Job Lookup trigger command is scheduled at this interval (in seconds).", "display_name": "Coordinator Job Lookup Interval", "name": "oozie_service_coord_lookup_interval", "value": "300" }, { "desc": "The health test thresholds of the overall Oozie Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Oozie Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Oozie Servers falls below the critical threshold.", "display_name": "Healthy Oozie Server Monitoring Thresholds", "name": "oozie_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "oozie" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Oozie Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "oozie_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Oozie Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Oozie Server Health", "name": "service_health_suppression_oozie_oozie_servers_healthy", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "oozie" }, { "desc": "Name of the Hive service that this Oozie service instance depends on. This is used to configure Oozie HCat integration.", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Credential Classes parameter.", "display_name": "Suppress Parameter Validation: Oozie Credential Classes", "name": "service_config_suppression_oozie_credential_classes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Oozie Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Oozie Server Count Validator", "name": "service_config_suppression_oozie_server_count_validator", "value": "false" }, { "desc": "Name of the Spark on Yarn service that this Oozie service instance depends on", "display_name": "Spark on Yarn Service", "name": "spark_on_yarn_service", "value": null }, { "desc": "Namespace used by this Oozie service in ZooKeeper when High Availability is enabled.", "display_name": "ZooKeeper Namespace", "name": "oozie_zookeeper_namespace", "value": "oozie" }, { "desc": "A list of credential class mappings for CredentialsProvider.", "display_name": "Oozie Credential Classes", "name": "oozie_credential_classes", "value": "hcat=org.apache.oozie.action.hadoop.HCatCredentials,hbase=org.apache.oozie.action.hadoop.HbaseCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Event Listeners parameter.", "display_name": "Suppress Parameter Validation: Oozie Event Listeners", "name": "service_config_suppression_oozie_event_listeners", "value": "false" }, { "desc": "Address of the load balancer used if Oozie HA is enabled. Should be specified in host:port format.", "display_name": "Oozie Load Balancer", "name": "oozie_load_balancer", "value": null }, { "desc": "List of event listeners used by the Oozie service. Listeners needed for JMS or SLA integration are automatically emitted if they are enabled.", "display_name": "Oozie Event Listeners", "name": "oozie_event_listeners", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JMS Broker parameter.", "display_name": "Suppress Parameter Validation: JMS Broker", "name": "service_config_suppression_oozie_jms_broker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Namespace parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Namespace", "name": "service_config_suppression_oozie_zookeeper_namespace", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Oozie service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Use ACLs on Znode while a secure ZooKeeper is used for Oozie High Availability. Note: This config is not emitted if ZooKeeper is not secure.", "display_name": "Use ACLs on Znode", "name": "oozie_zk_secure", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Root of the directory in HDFS where the Oozie ShareLibs are stored. The libraries are stored in the share/lib subdirectory under the specified root directory.", "display_name": "ShareLib Root Directory", "name": "oozie_sharelib_rootdir", "value": "/user/oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Oozie Load Balancer", "name": "service_config_suppression_oozie_load_balancer", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The timeout in seconds used for the Oozie Upload ShareLib command. When the value is zero, there is no timeout for the command.", "display_name": "Oozie Upload ShareLib Command Timeout", "name": "oozie_upload_sharelib_cmd_timeout", "value": "270" }, { "desc": "URL of the JMS Broker used by the Oozie service in JMS integration is enabled.", "display_name": "JMS Broker", "name": "oozie_jms_broker", "value": "tcp://localhost:61616" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ShareLib Root Directory parameter.", "display_name": "Suppress Parameter Validation: ShareLib Root Directory", "name": "service_config_suppression_oozie_sharelib_rootdir", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_7_0/resources/kafka-kafka_broker.json0000666000175100017510000006744113245514472026105 0ustar zuulzuul00000000000000[ { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The port to give out to producers, consumers, and other brokers to use in establishing connections. This only needs to be set if this port is different from the port the server should bind to.", "display_name": "Advertised Port", "name": "advertised.port", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Broker Java Options parameter.", "display_name": "Suppress Parameter Validation: Additional Broker Java Options", "name": "role_config_suppression_broker_java_opts", "value": "false" }, { "desc": "If set, this is the hostname given out to producers, consumers, and other brokers to use in establishing connections. Never set this property at the group level; it should always be overriden on instance level.", "display_name": "Advertised Host", "name": "advertised.host.name", "value": null }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Metric Report Host parameter.", "display_name": "Suppress Parameter Validation: HTTP Metric Report Host", "name": "role_config_suppression_kafka.http.metrics.host", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kafka_kafka_broker_scm_health", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka Broker might connect to. This is used when Kafka Broker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Kafka Broker TLS/SSL Certificate Trust Store File", "name": "ssl_client_truststore_location", "value": "" }, { "desc": "Port the HTTP metric reporter listens on.", "display_name": "HTTP Metric Report Port", "name": "kafka.http.metrics.port", "value": "24042" }, { "desc": "Kafka broker port.", "display_name": "TCP Port", "name": "port", "value": "9092" }, { "desc": "The log for a topic partition is stored as a directory of segment files. This setting controls the size to which a segment file can grow before a new segment is rolled over in the log. This value should be larger than message.max.bytes.", "display_name": "Segment File Size", "name": "log.segment.bytes", "value": "1073741824" }, { "desc": "For advanced use only. A string to be inserted into kafka-monitoring.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties", "name": "kafka-monitoring.properties_role_safety_valve", "value": null }, { "desc": "The frequency, in milliseconds, that the log cleaner checks whether any log segment is eligible for deletion, per retention policies.", "display_name": "Data Retention Check Interval", "name": "log.retention.check.interval.ms", "value": "300000" }, { "desc": "The maximum number of rolled log files to keep for Kafka Broker logs. Typically used by log4j or logback.", "display_name": "Kafka Broker Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Advertised Host parameter.", "display_name": "Suppress Parameter Validation: Advertised Host", "name": "role_config_suppression_advertised.host.name", "value": "false" }, { "desc": "The minimum log level for Kafka Broker logs", "display_name": "Kafka Broker Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "When computing the overall Kafka Broker health, consider the host's health.", "display_name": "Kafka Broker Host Health Test", "name": "kafka_broker_host_health_enabled", "value": "true" }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in megabytes. Kafka does not generally require setting large heap sizes. It is better to let the file system cache utilize the available memory.", "display_name": "Java Heap Size of Broker", "name": "broker_max_heap_size", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kafka_kafka_broker_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "The number of I/O threads that the server uses for executing requests. You should have at least as many threads as you have disks.", "display_name": "Number of I/O Threads", "name": "num.io.threads", "value": "8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Protocol to be used for inter-broker communication.", "display_name": "Inter Broker Protocol", "name": "security.inter.broker.protocol", "value": "PLAINTEXT" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties", "name": "role_config_suppression_ssl.properties_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "For advanced use only. A string to be inserted into kafka.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties", "name": "kafka.properties_role_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out (in hours). Secondary to the log.retention.ms property. The special value of -1 is interpreted as unlimited. This property is deprecated in Kafka 1.4.0. Use log.retention.ms.", "display_name": "Data Retention Hours", "name": "log.retention.hours", "value": "168" }, { "desc": "Kafka broker secure port.", "display_name": "TLS/SSL Port", "name": "ssl_port", "value": "9093" }, { "desc": "The amount of data to retain in the log for each topic-partition. This is the limit per partition: multiply by the number of partitions to get the total data retained for the topic. The special value of -1 is interpreted as unlimited. If both log.retention.ms and log.retention.bytes are set, a segment is deleted when either limit is exceeded.", "display_name": "Data Retention Size", "name": "log.retention.bytes", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Host the HTTP metric reporter binds to.", "display_name": "HTTP Metric Report Host", "name": "kafka.http.metrics.host", "value": "0.0.0.0" }, { "desc": "The maximum size, in megabytes, per log file for Kafka Broker logs. Typically used by log4j or logback.", "display_name": "Kafka Broker Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties", "name": "role_config_suppression_kafka-monitoring.properties_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kafka_broker_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_BROKER_role_env_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Port for JMX.", "display_name": "JMX Port", "name": "jmx_port", "value": "9393" }, { "desc": "The log directory for log files of the role Kafka Broker.", "display_name": "Kafka Broker Log Directory", "name": "log_dir", "value": "/var/log/kafka" }, { "desc": "The maximum time before a new log segment is rolled out. This property is used in Cloudera Kafka 1.4.0 and later in place of log.roll.hours.", "display_name": "Data Log Roll Time", "name": "log.roll.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties", "name": "role_config_suppression_kafka.properties_role_safety_valve", "value": "false" }, { "desc": "The password that protects the private key contained in the JKS keystore used when Kafka Broker is acting as a TLS/SSL server.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Kafka Broker Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kafka_kafka_broker_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Data Directories parameter.", "display_name": "Suppress Parameter Validation: Data Directories", "name": "role_config_suppression_log.dirs", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out. If both log.retention.ms and log.retention.bytes are set, a segment is deleted when either limit is exceeded. The special value of -1 is interpreted as unlimited. This property is used in Kafka 1.4.0 and later in place of log.retention.hours.", "display_name": "Data Retention Time", "name": "log.retention.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out (in hours). This property is deprecated in Cloudera Kafka 1.4.0; use log.roll.ms.", "display_name": "Data Log Roll Hours", "name": "log.roll.hours", "value": "168" }, { "desc": "These arguments are passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags are passed here.", "display_name": "Additional Broker Java Options", "name": "broker_java_opts", "value": "-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka Broker is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kafka_kafka_broker_host_health", "value": "false" }, { "desc": "A list of one or more directories in which Kafka data is stored. Each new partition created is placed in the directory that currently has the fewest partitions. Each directory should be on its own separate drive.", "display_name": "Data Directories", "name": "log.dirs", "value": "/var/local/kafka/data" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Segment File Size parameter.", "display_name": "Suppress Parameter Validation: Segment File Size", "name": "role_config_suppression_log.segment.bytes", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Broker ID parameter.", "display_name": "Suppress Parameter Validation: Broker ID", "name": "role_config_suppression_broker.id", "value": "false" }, { "desc": "The password for the Kafka Broker JKS keystore file.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kafka_kafka_broker_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kafka_broker_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into ssl.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties", "name": "ssl.properties_role_safety_valve", "value": null }, { "desc": "Encrypt communication between clients and Kafka Broker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Kafka Broker", "name": "ssl_enabled", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "ID uniquely identifying each broker. Never set this property at the group level; it should always be overridden on instance level.", "display_name": "Broker ID", "name": "broker.id", "value": null }, { "desc": "Maximum number of connections allowed from each IP address.", "display_name": "Maximum Connections per IP Address", "name": "max.connections.per.ip", "value": null }, { "desc": "Enables the health test that the Kafka Broker's process state is consistent with the role configuration", "display_name": "Kafka Broker Process Health Test", "name": "kafka_broker_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of I/O Threads parameter.", "display_name": "Suppress Parameter Validation: Number of I/O Threads", "name": "role_config_suppression_num.io.threads", "value": "false" }, { "desc": "The password for the Kafka Broker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Kafka Broker TLS/SSL Certificate Trust Store Password", "name": "ssl_client_truststore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of Broker parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of Broker", "name": "role_config_suppression_broker_max_heap_size", "value": "false" }, { "desc": "Client authentication mode for SSL connections. Default is none, could be set to \"required\", i.e., client authentication is required or to \"requested\", i.e., client authentication is requested and client without certificates can still connect.", "display_name": "SSL Client Authentication", "name": "ssl.client.auth", "value": "none" }, { "desc": "Authenticate a SASL connection with zookeeper, if Kerberos authentication is enabled. It also allows a broker to set SASL ACL on zookeeper nodes which locks these nodes down so that only kafka broker can modify.", "display_name": "Authenticate Zookeeper Connection", "name": "authenticate.zookeeper.connection", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/exceptions.py0000666000175100017510000000445313245514472021232 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import exceptions as e from sahara.i18n import _ class CMApiVersionError(e.SaharaException): """Exception indicating that CM API Version does not meet requirement. A message indicating the reason for failure must be provided. """ base_message = _("CM API version not meet requirement: %s") def __init__(self, message): self.code = "CM_API_VERSION_ERROR" self.message = self.base_message % message super(CMApiVersionError, self).__init__() class CMApiException(e.SaharaException): """Exception Type from CM API Errors. Any error result from the CM API is converted into this exception type. This handles errors from the HTTP level as well as the API level. """ base_message = _("CM API error: %s") def __init__(self, message): self.code = "CM_API_EXCEPTION" self.message = self.base_message % message super(CMApiException, self).__init__() class CMApiAttributeError(e.SaharaException): """Exception indicating a CM API attribute error. A message indicating the reason for failure must be provided. """ base_message = _("CM API attribute error: %s") def __init__(self, message): self.code = "CM_API_ATTRIBUTE_ERROR" self.message = self.base_message % message super(CMApiAttributeError, self).__init__() class CMApiValueError(e.SaharaException): """Exception indicating a CM API value error. A message indicating the reason for failure must be provided. """ base_message = _("CM API value error: %s") def __init__(self, message): self.code = "CM_API_VALUE_ERROR" self.message = self.base_message % message super(CMApiValueError, self).__init__() sahara-8.0.0/sahara/plugins/cdh/v5_9_0/0000775000175100017510000000000013245515027017465 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_9_0/deploy.py0000666000175100017510000001260213245514472021341 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ from sahara.plugins.cdh import commands as cmd from sahara.plugins.cdh import deploy as common_deploy from sahara.plugins.cdh.v5_9_0 import cloudera_utils as cu from sahara.plugins import utils as gu from sahara.service.edp import hdfs_helper as h from sahara.utils import cluster_progress_ops as cpo CU = cu.ClouderaUtilsV590() PACKAGES = common_deploy.PACKAGES def configure_cluster(cluster): instances = gu.get_instances(cluster) if not cmd.is_pre_installed_cdh(CU.pu.get_manager(cluster).remote()): CU.pu.configure_os(instances) CU.pu.install_packages(instances, PACKAGES) CU.pu.start_cloudera_agents(instances) CU.pu.start_cloudera_manager(cluster) CU.update_cloudera_password(cluster) CU.configure_rack_awareness(cluster) CU.await_agents(cluster, instances) CU.create_mgmt_service(cluster) CU.create_services(cluster) CU.configure_services(cluster) CU.configure_instances(instances, cluster) CU.deploy_configs(cluster) @cpo.event_wrapper( True, step=_("Start roles: NODEMANAGER, DATANODE"), param=('cluster', 0)) def _start_roles(cluster, instances): for instance in instances: if 'HDFS_DATANODE' in instance.node_group.node_processes: hdfs = CU.get_service_by_role('DATANODE', instance=instance) CU.start_roles(hdfs, CU.pu.get_role_name(instance, 'DATANODE')) if 'YARN_NODEMANAGER' in instance.node_group.node_processes: yarn = CU.get_service_by_role('NODEMANAGER', instance=instance) CU.start_roles(yarn, CU.pu.get_role_name(instance, 'NODEMANAGER')) def scale_cluster(cluster, instances): if not instances: return if not cmd.is_pre_installed_cdh(instances[0].remote()): CU.pu.configure_os(instances) CU.pu.install_packages(instances, PACKAGES) CU.pu.start_cloudera_agents(instances) CU.await_agents(cluster, instances) CU.configure_rack_awareness(cluster) CU.configure_instances(instances, cluster) CU.update_configs(instances) common_deploy.prepare_scaling_kerberized_cluster( cluster, CU, instances) CU.pu.configure_swift(cluster, instances) _start_roles(cluster, instances) CU.refresh_datanodes(cluster) CU.refresh_yarn_nodes(cluster) CU.restart_stale_services(cluster) def decommission_cluster(cluster, instances): dns = [] dns_to_delete = [] nms = [] nms_to_delete = [] for i in instances: if 'HDFS_DATANODE' in i.node_group.node_processes: dns.append(CU.pu.get_role_name(i, 'DATANODE')) dns_to_delete.append( CU.pu.get_role_name(i, 'HDFS_GATEWAY')) if 'YARN_NODEMANAGER' in i.node_group.node_processes: nms.append(CU.pu.get_role_name(i, 'NODEMANAGER')) nms_to_delete.append( CU.pu.get_role_name(i, 'YARN_GATEWAY')) if dns: CU.decommission_nodes( cluster, 'DATANODE', dns, dns_to_delete) if nms: CU.decommission_nodes( cluster, 'NODEMANAGER', nms, nms_to_delete) CU.delete_instances(cluster, instances) CU.refresh_datanodes(cluster) CU.refresh_yarn_nodes(cluster) CU.restart_stale_services(cluster) @cpo.event_wrapper(True, step=_("Prepare cluster"), param=('cluster', 0)) def _prepare_cluster(cluster): if CU.pu.get_oozie(cluster): CU.pu.install_extjs(cluster) if CU.pu.get_hive_metastore(cluster): CU.pu.configure_hive(cluster) if CU.pu.get_sentry(cluster): CU.pu.configure_sentry(cluster) @cpo.event_wrapper( True, step=_("Finish cluster starting"), param=('cluster', 0)) def _finish_cluster_starting(cluster): if CU.pu.get_hive_metastore(cluster): CU.pu.put_hive_hdfs_xml(cluster) server = CU.pu.get_hbase_master(cluster) if CU.pu.c_helper.is_hbase_common_lib_enabled(cluster) and server: with server.remote() as r: h.create_hbase_common_lib(r) if CU.pu.get_flumes(cluster): flume = CU.get_service_by_role('AGENT', cluster) CU.start_service(flume) def start_cluster(cluster): _prepare_cluster(cluster) CU.first_run(cluster) CU.pu.configure_swift(cluster) if len(CU.pu.get_jns(cluster)) > 0: CU.enable_namenode_ha(cluster) # updating configs for NameNode role on needed nodes CU.update_role_config(CU.pu.get_secondarynamenode(cluster), 'HDFS_NAMENODE') if CU.pu.get_stdb_rm(cluster): CU.enable_resourcemanager_ha(cluster) # updating configs for ResourceManager on needed nodes CU.update_role_config(CU.pu.get_stdb_rm(cluster), 'YARN_STANDBYRM') _finish_cluster_starting(cluster) common_deploy.setup_kerberos_for_cluster(cluster, CU) def get_open_ports(node_group): ports = common_deploy.get_open_ports(node_group) return ports sahara-8.0.0/sahara/plugins/cdh/v5_9_0/validation.py0000666000175100017510000000136513245514472022203 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_9_0 import plugin_utils as pu from sahara.plugins.cdh import validation class ValidatorV590(validation.Validator): PU = pu.PluginUtilsV590() sahara-8.0.0/sahara/plugins/cdh/v5_9_0/versionhandler.py0000666000175100017510000000270213245514472023070 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import abstractversionhandler as avm from sahara.plugins.cdh.v5_9_0 import cloudera_utils from sahara.plugins.cdh.v5_9_0 import config_helper from sahara.plugins.cdh.v5_9_0 import deploy from sahara.plugins.cdh.v5_9_0 import edp_engine from sahara.plugins.cdh.v5_9_0 import images from sahara.plugins.cdh.v5_9_0 import plugin_utils from sahara.plugins.cdh.v5_9_0 import validation class VersionHandler(avm.BaseVersionHandler): def __init__(self): super(VersionHandler, self).__init__() self.config_helper = config_helper.ConfigHelperV590() self.cloudera_utils = cloudera_utils.ClouderaUtilsV590() self.plugin_utils = plugin_utils.PluginUtilsV590() self.deploy = deploy self.edp_engine = edp_engine self.images = images self.validation = validation.ValidatorV590() sahara-8.0.0/sahara/plugins/cdh/v5_9_0/config_helper.py0000666000175100017510000000737013245514472022657 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import config_helper as c_h from sahara.plugins import provisioning as p from sahara.utils import files as f class ConfigHelperV590(c_h.ConfigHelper): path_to_config = 'plugins/cdh/v5_9_0/resources/' CDH5_UBUNTU_REPO = ( 'deb [arch=amd64] http://archive.cloudera.com/cdh5' '/ubuntu/trusty/amd64/cdh trusty-cdh5.9.0 contrib' '\ndeb-src http://archive.cloudera.com/cdh5/ubuntu' '/trusty/amd64/cdh trusty-cdh5.9.0 contrib') DEFAULT_CDH5_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/cdh5/ubuntu' '/trusty/amd64/cdh/archive.key') CM5_UBUNTU_REPO = ( 'deb [arch=amd64] http://archive.cloudera.com/cm5' '/ubuntu/trusty/amd64/cm trusty-cm5.9.0 contrib' '\ndeb-src http://archive.cloudera.com/cm5/ubuntu' '/trusty/amd64/cm trusty-cm5.9.0 contrib') DEFAULT_CM5_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/cm5/ubuntu' '/trusty/amd64/cm/archive.key') CDH5_CENTOS_REPO = ( '[cloudera-cdh5]' '\nname=Cloudera\'s Distribution for Hadoop, Version 5' '\nbaseurl=http://archive.cloudera.com/cdh5/redhat/6' '/x86_64/cdh/5.9.0/' '\ngpgkey = http://archive.cloudera.com/cdh5/redhat/6' '/x86_64/cdh/RPM-GPG-KEY-cloudera' '\ngpgcheck = 1') CM5_CENTOS_REPO = ( '[cloudera-manager]' '\nname=Cloudera Manager' '\nbaseurl=http://archive.cloudera.com/cm5/redhat/6' '/x86_64/cm/5.9.0/' '\ngpgkey = http://archive.cloudera.com/cm5/redhat/6' '/x86_64/cm/RPM-GPG-KEY-cloudera' '\ngpgcheck = 1') KEY_TRUSTEE_UBUNTU_REPO_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/ubuntu/trusty/amd64/navigator-' 'keytrustee/cloudera.list') DEFAULT_KEY_TRUSTEE_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/ubuntu/trusty/amd64/navigator-' 'keytrustee/archive.key') KEY_TRUSTEE_CENTOS_REPO_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/redhat/6/x86_64/navigator-' 'keytrustee/navigator-keytrustee5.repo') DEFAULT_SWIFT_LIB_URL = ( 'https://repository.cloudera.com/artifactory/repo/org' '/apache/hadoop/hadoop-openstack/2.6.0-cdh5.9.0' '/hadoop-openstack-2.6.0-cdh5.9.0.jar') SWIFT_LIB_URL = p.Config( 'Hadoop OpenStack library URL', 'general', 'cluster', priority=1, default_value=DEFAULT_SWIFT_LIB_URL, description=("Library that adds Swift support to CDH. The file" " will be downloaded by VMs.")) HIVE_SERVER2_SENTRY_SAFETY_VALVE = f.get_file_text( path_to_config + 'hive-server2-sentry-safety.xml') HIVE_METASTORE_SENTRY_SAFETY_VALVE = f.get_file_text( path_to_config + 'hive-metastore-sentry-safety.xml') SENTRY_IMPALA_CLIENT_SAFETY_VALVE = f.get_file_text( path_to_config + 'sentry-impala-client-safety.xml') def __init__(self): super(ConfigHelperV590, self).__init__() self.priority_one_confs = self._load_json( self.path_to_config + 'priority-one-confs.json') self._init_all_ng_plugin_configs() sahara-8.0.0/sahara/plugins/cdh/v5_9_0/images.py0000666000175100017510000000277313245514472021322 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import images from sahara.plugins import utils as plugin_utils _validator = images.SaharaImageValidator.from_yaml( 'plugins/cdh/v5_9_0/resources/images/image.yaml', resource_roots=['plugins/cdh/v5_9_0/resources/images']) def get_image_arguments(): return _validator.get_argument_list() def pack_image(remote, test_only=False, image_arguments=None): _validator.validate(remote, test_only=test_only, image_arguments=image_arguments) def validate_images(cluster, test_only=False, image_arguments=None): image_arguments = get_image_arguments() if not test_only: instances = plugin_utils.get_instances(cluster) else: instances = plugin_utils.get_instances(cluster)[0] for instance in instances: with instance.remote() as r: _validator.validate(r, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/cdh/v5_9_0/cloudera_utils.py0000666000175100017510000000203113245514472023056 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import cloudera_utils as cu from sahara.plugins.cdh.v5_9_0 import config_helper from sahara.plugins.cdh.v5_9_0 import plugin_utils as pu from sahara.plugins.cdh.v5_9_0 import validation class ClouderaUtilsV590(cu.ClouderaUtils): def __init__(self): cu.ClouderaUtils.__init__(self) self.pu = pu.PluginUtilsV590() self.validator = validation.ValidatorV590 self.c_helper = config_helper.ConfigHelperV590() sahara-8.0.0/sahara/plugins/cdh/v5_9_0/__init__.py0000666000175100017510000000000013245514472021571 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_9_0/plugin_utils.py0000666000175100017510000000146013245514472022563 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import plugin_utils as pu from sahara.plugins.cdh.v5_9_0 import config_helper class PluginUtilsV590(pu.AbstractPluginUtils): def __init__(self): self.c_helper = config_helper.ConfigHelperV590() sahara-8.0.0/sahara/plugins/cdh/v5_9_0/edp_engine.py0000666000175100017510000000366413245514472022152 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import confighints_helper as ch_helper from sahara.plugins.cdh import edp_engine from sahara.plugins.cdh.v5_9_0 import cloudera_utils as cu from sahara.service.edp.oozie import engine as oozie_engine from sahara.utils import edp class EdpOozieEngine(edp_engine.EdpOozieEngine): def __init__(self, cluster): super(EdpOozieEngine, self).__init__(cluster) self.cloudera_utils = cu.ClouderaUtilsV590() @staticmethod def get_possible_job_config(job_type): if edp.compare_job_type(job_type, edp.JOB_TYPE_HIVE): return {'job_config': ch_helper.get_possible_hive_config_from( 'plugins/cdh/v5_9_0/resources/hive-site.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_MAPREDUCE_STREAMING): return {'job_config': ch_helper.get_possible_mapreduce_config_from( 'plugins/cdh/v5_9_0/resources/mapred-site.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_PIG): return {'job_config': ch_helper.get_possible_pig_config_from( 'plugins/cdh/v5_9_0/resources/mapred-site.xml')} return oozie_engine.OozieJobEngine.get_possible_job_config(job_type) class EdpSparkEngine(edp_engine.EdpSparkEngine): edp_base_version = "5.9.0" sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/0000775000175100017510000000000013245515027021477 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/sentry-sentry_server.json0000666000175100017510000004606113245514472026642 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When computing the overall Sentry Server health, consider the host's health.", "display_name": "Sentry Server Host Health Test", "name": "sentry_server_host_health_enabled", "value": "true" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_sentry_server_unexpected_exits", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Sentry Server in Bytes", "name": "sentry_server_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Sentry Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Sentry Server", "name": "role_config_suppression_sentry_server_java_opts", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Sentry Server logs. Typically used by log4j or logback.", "display_name": "Sentry Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Log Directory", "name": "role_config_suppression_sentry_server_log_dir", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Sentry Server", "name": "sentry_server_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_sentry_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Directory where Sentry Server will place its log files.", "display_name": "Sentry Server Log Directory", "name": "sentry_server_log_dir", "value": "/var/log/sentry" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Sentry Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_sentry_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_sentry_server_host_health", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_sentry_server_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "RPC port number of Sentry Server.", "display_name": "Sentry Server RPC Port", "name": "sentry_service_server_rpc_port", "value": "8038" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_sentry_server_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Sentry Server logs. Typically used by log4j or logback.", "display_name": "Sentry Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Enables the health test that the Sentry Server's process state is consistent with the role configuration", "display_name": "Sentry Server Process Health Test", "name": "sentry_server_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "sentry_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Sentry Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "sentry_env_safety_valve", "value": null }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for Sentry Server logs", "display_name": "Sentry Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_sentry_server_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_sentry_server_swap_memory_usage", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/solr-service.json0000666000175100017510000007053113245514472025022 0ustar zuulzuul00000000000000[ { "desc": "If Solr does not respond on its web URL within this time interval, the Catalina process is killed.", "display_name": "Solrd Watchdog Timeout", "name": "solrd_watchdog_timeout", "value": "70" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_solr_sentry_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Name of the Sentry service that this Solr service instance depends on. If selected, Solr uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Solr Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Solr Server Count Validator", "name": "service_config_suppression_solr_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_solr_core_site_safety_valve", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Solr events:\n

\n\n
    \n
  • operation: the Solr operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "solr_sentry_safety_valve", "value": null }, { "desc": "ZooKeeper znode used to store information about this Solr service.", "display_name": "ZooKeeper Znode", "name": "zookeeper_znode", "value": "/solr" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "solr" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Solr events:\n

\n\n
    \n
  • operation: the Solr operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_solr_ldap_uri", "value": "false" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "ldap_domain", "value": null }, { "desc": "The password for the Solr TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Solr TLS/SSL Certificate Trust Store Password", "name": "solr_https_truststore_password", "value": null }, { "desc": "The password for the Solr JKS keystore file.", "display_name": "Solr TLS/SSL Server JKS Keystore File Password", "name": "solr_https_keystore_password", "value": null }, { "desc": "Name of the HDFS service that this Search service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "solr_ldap_enable_starttls", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "solr_core_site_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_solr_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Data Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Data Directory", "name": "service_config_suppression_hdfs_data_dir", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Search service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Solr might connect to. This is used when Solr is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Solr TLS/SSL Certificate Trust Store File", "name": "solr_https_truststore_file", "value": null }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "solr_env_safety_valve", "value": null }, { "desc": "HDFS directory used for storage by this Solr service.", "display_name": "HDFS Data Directory", "name": "hdfs_data_dir", "value": "/solr" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP TLS Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP TLS Validator", "name": "service_config_suppression_solr_ldap_tls_validator", "value": "false" }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "sentry_solr_provider_resource", "value": "/user/solr/sentry/sentry-provider.ini" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/solr/audit" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "solr" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "solr_hdfs_site_safety_valve", "value": null }, { "desc": "The health test thresholds of the overall Solr Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Solr Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Solr Servers falls below the critical threshold.", "display_name": "Healthy Solr Server Monitoring Thresholds", "name": "solr_solr_servers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_sentry_solr_provider_resource", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solrd Watchdog Timeout parameter.", "display_name": "Suppress Parameter Validation: Solrd Watchdog Timeout", "name": "service_config_suppression_solrd_watchdog_timeout", "value": "false" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "solr_enable_ldap_auth", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "solr" }, { "desc": "Choose the authentication mechanism used by Solr.", "display_name": "Solr Secure Authentication", "name": "solr_security_authentication", "value": "simple" }, { "desc": "Encrypt communication between clients and Solr using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)). Additional manual steps must be performed; see Enabling TLS/SSL for Solr.", "display_name": "Enable TLS/SSL for Solr", "name": "solr_use_ssl", "value": "false" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access.", "display_name": "Sentry User to Group Mapping Class", "name": "solr_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP Secure URI and Start TLS Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP Secure URI and Start TLS Validator", "name": "service_config_suppression_solr_ldaps_or_tls_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_solr_https_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using Sentry service instead (supported in CDH 5.8 and later), add Sentry service as a dependency to Solr service. Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry with policy files is supported only on CDH 5 or later deployments. Sentry requires authentication to be turned on for Solr.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "solr_sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_solr_https_keystore_password", "value": "false" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Enable the background watchdog thread that can kill Catalina process if Solr is not responsive.", "display_name": "Enable Solrd Watchdog", "name": "solrd_enable_watchdog", "value": "true" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "service_config_suppression_solr_hdfs_site_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "solr_ldap_uri", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode", "name": "service_config_suppression_zookeeper_znode", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Solr is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Solr TLS/SSL Server JKS Keystore File Location", "name": "solr_https_keystore_file", "value": null }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_solr_ldap_basedn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_solr_https_keystore_file", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_solr_https_truststore_password", "value": "false" }, { "desc": "Whether to suppress the results of the Solr Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Solr Server Health", "name": "service_health_suppression_solr_solr_servers_healthy", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "This parameter is useful when authenticating against a non Active Directory server such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,<this parameter>. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server will look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "solr_ldap_basedn", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_ldap_domain", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/flume-agent.json0000666000175100017510000012255213245514472024612 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Used to select an agent configuration to use from flume.conf. Multiple agents may share the same agent name, in which case they will be assigned the same agent configuration.", "display_name": "Agent Name", "name": "agent_name", "value": "tier1" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Text that goes into morphlines.conf file used by the Flume-NG Solr sink. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service.", "display_name": "Morphlines File", "name": "agent_morphlines_conf_file", "value": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Application configuration file in HOCON format (Human-Optimized Config Object Notation). \n# HOCON syntax is defined at http://github.com/typesafehub/config/blob/master/HOCON.md\n# and also used by Akka (http://www.akka.io) and Play (http://www.playframework.org/).\n# For more examples see http://doc.akka.io/docs/akka/2.1.2/general/configuration.html\n\n# morphline.conf example file\n# this is a comment\n\n# Specify server locations in a SOLR_LOCATOR variable; used later in variable substitutions:\nSOLR_LOCATOR : {\n # Name of solr collection\n collection : collection1\n \n # ZooKeeper ensemble\n zkHost : \"$ZK_HOST\"\n \n # Relative or absolute path to a directory containing conf/solrconfig.xml and conf/schema.xml\n # If this path is uncommented it takes precedence over the configuration stored in ZooKeeper. \n # solrHomeDir : \"example/solr/collection1\"\n \n # The maximum number of documents to send to Solr per network batch (throughput knob)\n # batchSize : 100\n}\n\n# Specify an array of one or more morphlines, each of which defines an ETL \n# transformation chain. A morphline consists of one or more (potentially \n# nested) commands. A morphline is a way to consume records (e.g. Flume events, \n# HDFS files or blocks), turn them into a stream of records, and pipe the stream \n# of records through a set of easily configurable transformations on it's way to \n# Solr (or a MapReduceIndexerTool RecordWriter that feeds via a Reducer into Solr).\nmorphlines : [\n {\n # Name used to identify a morphline. E.g. used if there are multiple morphlines in a \n # morphline config file\n id : morphline1 \n \n # Import all morphline commands in these java packages and their subpackages.\n # Other commands that may be present on the classpath are not visible to this morphline.\n importCommands : [\"org.kitesdk.**\", \"org.apache.solr.**\"]\n \n commands : [ \n { \n # Parse Avro container file and emit a record for each avro object\n readAvroContainer {\n # Optionally, require the input record to match one of these MIME types:\n # supportedMimeTypes : [avro/binary]\n \n # Optionally, use a custom Avro schema in JSON format inline:\n # schemaString : \"\"\"\"\"\"\n \n # Optionally, use a custom Avro schema file in JSON format:\n # schemaFile : /path/to/syslog.avsc\n }\n } \n \n { \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # extractAvroPaths is a command that uses zero or more avro path expressions to extract \n # values from an Avro object. Each expression consists of a record output field name (on \n # the left side of the colon ':') as well as zero or more path steps (on the right hand \n # side), each path step separated by a '/' slash. Avro arrays are traversed with the '[]'\n # notation.\n #\n # The result of a path expression is a list of objects, each of which is added to the \n # given record output field.\n # \n # The path language supports all Avro concepts, including nested structures, records, \n # arrays, maps, unions, etc, as well as a flatten option that collects the primitives in \n # a subtree into a flat list.\n extractAvroPaths {\n flatten : false\n paths : { \n id : /id \n text : /text \n user_friends_count : /user_friends_count\n user_location : /user_location\n user_description : /user_description\n user_statuses_count : /user_statuses_count\n user_followers_count : /user_followers_count\n user_name : /user_name\n user_screen_name : /user_screen_name\n created_at : /created_at\n retweet_count : /retweet_count\n retweeted : /retweeted\n in_reply_to_user_id : /in_reply_to_user_id\n source : /source\n in_reply_to_status_id : /in_reply_to_status_id\n media_url_https : /media_url_https\n expanded_url : /expanded_url\n }\n }\n }\n \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # convert timestamp field to native Solr timestamp format\n # e.g. 2012-09-06T07:14:34Z to 2012-09-06T07:14:34.000Z\n {\n convertTimestamp {\n field : created_at\n inputFormats : [\"yyyy-MM-dd'T'HH:mm:ss'Z'\", \"yyyy-MM-dd\"]\n inputTimezone : America/Los_Angeles\n# outputFormat : \"yyyy-MM-dd'T'HH:mm:ss.SSSZ\" \n outputTimezone : UTC\n }\n }\n \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # Command that sanitizes record fields that are unknown to Solr schema.xml by either \n # deleting them (renameToPrefix is absent or a zero length string), or by moving them to a\n # field prefixed with the given renameToPrefix (e.g. renameToPrefix = \"ignored_\" to use \n # typical dynamic Solr fields).\n #\n # Recall that Solr throws an exception on any attempt to load a document that contains a \n # field that isn't specified in schema.xml.\n {\n sanitizeUnknownSolrFields {\n # Location from which to fetch Solr schema\n solrLocator : ${SOLR_LOCATOR}\n \n # renameToPrefix : \"ignored_\"\n }\n } \n \n # log the record at DEBUG level to SLF4J\n { logDebug { format : \"output record: {}\", args : [\"@{}\"] } } \n \n # load the record into a SolrServer or MapReduce SolrOutputFormat.\n { \n loadSolr {\n solrLocator : ${SOLR_LOCATOR}\n }\n }\n ]\n }\n]\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Plugin directories parameter.", "display_name": "Suppress Parameter Validation: Plugin directories", "name": "role_config_suppression_agent_plugin_dirs", "value": "false" }, { "desc": "When computing the overall Agent health, consider the host's health.", "display_name": "Agent Host Health Test", "name": "flume_agent_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "agent_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The maximum size, in megabytes, per log file for Agent logs. Typically used by log4j or logback.", "display_name": "Agent Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_flume_agent_swap_memory_usage", "value": "false" }, { "desc": "Text that goes verbatim into grok-dictionary.conf file used by the Flume-NG Solr sink.", "display_name": "Grok Dictionary File", "name": "agent_grok_dictionary_conf_file", "value": "USERNAME [a-zA-Z0-9._-]+\nUSER %{USERNAME}\nINT (?:[+-]?(?:[0-9]+))\nBASE10NUM (?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))\nNUMBER (?:%{BASE10NUM})\nBASE16NUM (?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))\nUUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}\n\n# Networking\nMAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})\nCISCOMAC (?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})\nWINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})\nCOMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})\nIP (?/(?>[\\w_%!$@:.,-]+|\\\\.)*)+\n#UNIXPATH (?/dev/pts/%{NONNEGINT})\nBSDTTY (?>/dev/tty[pq][a-z0-9])\nTTY (?:%{BSDTTY}|%{LINUXTTY})\nWINPATH (?>[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+\nURIPROTO [A-Za-z]+(\\+[A-Za-z+]+)?\nURIHOST %{IPORHOST}(?::%{POSINT:port})?\n# uripath comes loosely from RFC1738, but mostly from what Firefox\n# doesn't turn into %XX\nURIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=#%_\\-]*)+\n#URIPARAM \\?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?\nURIPARAM \\?[A-Za-z0-9$.+!*'|(){},~#%&/=:;_?\\-\\[\\]]*\nURIPATHPARAM %{URIPATH}(?:%{URIPARAM})?\nURI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?\n\n# Months: January, Feb, 3, 03, 12, December\nMONTH \\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b\nMONTHNUM (?:0?[1-9]|1[0-2])\nMONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\n\n# Days: Monday, Tue, Thu, etc...\nDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)\n\n# Years?\nYEAR (?>\\d\\d){1,2}\n# Time: HH:MM:SS\n#TIME \\d{2}:\\d{2}(?::\\d{2}(?:\\.\\d+)?)?\n# I'm still on the fence about using grok to perform the time match,\n# since it's probably slower.\n# TIME %{POSINT<24}:%{POSINT<60}(?::%{POSINT<60}(?:\\.%{POSINT})?)?\nHOUR (?:2[0123]|[01]?[0-9])\nMINUTE (?:[0-5][0-9])\n# '60' is a leap second in most time standards and thus is valid.\nSECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\nTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])\n# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)\nDATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}\nDATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}\nISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))\nISO8601_SECOND (?:%{SECOND}|60)\nTIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?\nDATE %{DATE_US}|%{DATE_EU}\nDATESTAMP %{DATE}[- ]%{TIME}\nTZ (?:[PMCE][SD]T)\nDATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}\nDATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}\n\n# Syslog Dates: Month Day HH:MM:SS\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}\nPROG (?:[\\w._/%-]+)\nSYSLOGPROG %{PROG:program}(?:\\[%{POSINT:pid}\\])?\nSYSLOGHOST %{IPORHOST}\nSYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>\nHTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}\n\n# Shortcuts\nQS %{QUOTEDSTRING}\n\n# Log formats\nSYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:\nCOMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}\n\n# Log Levels\nLOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Name parameter.", "display_name": "Suppress Parameter Validation: Agent Name", "name": "role_config_suppression_agent_name", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_flume_agent_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Morphlines File parameter.", "display_name": "Suppress Parameter Validation: Morphlines File", "name": "role_config_suppression_agent_morphlines_conf_file", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "agent_web_metric_collection_enabled", "value": "true" }, { "desc": "

Verbatim contents of flume.conf. Multiple agents may be configured from the same configuration file; the Agent Name setting can be overridden to select which agent configuration to use for each agent.

To integrate with a secured cluster, you can use the substitution strings \"$KERBEROS_PRINCIPAL\" and \"$KERBEROS_KEYTAB\", which will be replaced by the principal name and the keytab path respectively.

", "display_name": "Configuration File", "name": "agent_config_file", "value": "# Please paste flume.conf here. Example:\n\n# Sources, channels, and sinks are defined per\n# agent name, in this case 'tier1'.\ntier1.sources = source1\ntier1.channels = channel1\ntier1.sinks = sink1\n\n# For each source, channel, and sink, set\n# standard properties.\ntier1.sources.source1.type = netcat\ntier1.sources.source1.bind = 127.0.0.1\ntier1.sources.source1.port = 9999\ntier1.sources.source1.channels = channel1\ntier1.channels.channel1.type = memory\ntier1.sinks.sink1.type = logger\ntier1.sinks.sink1.channel = channel1\n\n# Other properties are specific to each type of\n# source, channel, or sink. In this case, we\n# specify the capacity of the memory channel.\ntier1.channels.channel1.capacity = 100\n" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_flume_agent_file_descriptor", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Agent logs. Typically used by log4j or logback.", "display_name": "Agent Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Agent Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Custom Mime-types File parameter.", "display_name": "Suppress Parameter Validation: Custom Mime-types File", "name": "role_config_suppression_agent_custom_mimetypes_file", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Agent Log Directory parameter.", "display_name": "Suppress Parameter Validation: Flume Agent Log Directory", "name": "role_config_suppression_flume_agent_log_dir", "value": "false" }, { "desc": "Enables the health test that the Agent's process state is consistent with the role configuration", "display_name": "Agent Process Health Test", "name": "flume_agent_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_flume_agent_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_flume_agent_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "flume_agent_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The port on which the Flume web server listens for requests.", "display_name": "HTTP Port", "name": "agent_http_port", "value": "41414" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Agent Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Agent in Bytes", "name": "agent_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Configuration File parameter.", "display_name": "Suppress Parameter Validation: Configuration File", "name": "role_config_suppression_agent_config_file", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Disables import of ZooKeeper configuration from the HBase classpath. This prevents zoo.cfg from overriding hbase-site.xml for Zookeeper quorum information. This option is only supported on CDH 4.4 or later deployments.", "display_name": "HBase sink prefer hbase-site.xml over Zookeeper config", "name": "agent_disable_zoo_cfg", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Agent Environment Advanced Configuration Snippet (Safety Valve)", "name": "AGENT_role_env_safety_valve", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Directory where Flume Agent will place its log files.", "display_name": "Flume Agent Log Directory", "name": "flume_agent_log_dir", "value": "/var/log/flume-ng" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Flume Agent parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Flume Agent", "name": "role_config_suppression_flume_agent_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Grok Dictionary File parameter.", "display_name": "Suppress Parameter Validation: Grok Dictionary File", "name": "role_config_suppression_agent_grok_dictionary_conf_file", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_flume_agent_log_directory_free_space", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Text that goes verbatim into custom-mimetypes.xml file used by the Flume-NG Solr sink.", "display_name": "Custom Mime-types File", "name": "agent_custom_mimetypes_file", "value": "\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Home Directory parameter.", "display_name": "Suppress Parameter Validation: Flume Home Directory", "name": "role_config_suppression_agent_home_dir", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here. Note that Flume agent only uses options that start with -D and -X (including -XX).", "display_name": "Java Configuration Options for Flume Agent", "name": "flume_agent_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_flume_agent_scm_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_flume_agent_host_health", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Agent Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_agent_role_env_safety_valve", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "List of Flume plugin directories. This overrides the default Flume plugin directory.", "display_name": "Plugin directories", "name": "agent_plugin_dirs", "value": "/usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d" }, { "desc": "Home directory for Flume user. The File Channel uses paths for checkpoint and data directories that are within the user home.", "display_name": "Flume Home Directory", "name": "agent_home_dir", "value": "/var/lib/flume-ng" }, { "desc": "The minimum log level for Agent logs", "display_name": "Agent Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/sentry-impala-client-safety.xml0000666000175100017510000000063613245514472027565 0ustar zuulzuul00000000000000 sentry.service.client.server.rpc-port 3893 sentry.service.client.server.rpc-address hostname sentry.service.client.server.rpc-connection-timeout 200000 sentry.service.security.mode none sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-failovercontroller.json0000666000175100017510000005615113245514472027244 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where Failover Controller will place its log files.", "display_name": "Failover Controller Log Directory", "name": "failover_controller_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hdfs_failovercontroller_swap_memory_usage", "value": "false" }, { "desc": "The RPC timeout for the HA health monitor.", "display_name": "HA Health Monitor RPC Timeout", "name": "ha_health_monitor_rpc_timeout_ms", "value": "45000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "failovercontroller_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hdfs_failovercontroller_unexpected_exits", "value": "false" }, { "desc": "When computing the overall Failover Controller health, consider the host's health.", "display_name": "Failover Controller Host Health Test", "name": "failovercontroller_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_failovercontroller_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hdfs_failovercontroller_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Log Directory parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Log Directory", "name": "role_config_suppression_failover_controller_log_dir", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Failover Controller logs. Typically used by log4j or logback.", "display_name": "Failover Controller Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_fc_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Failover Controller in Bytes", "name": "failover_controller_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hdfs_failovercontroller_scm_health", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hdfs_failovercontroller_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Failover Controller Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Failover Controller Environment Advanced Configuration Snippet (Safety Valve)", "name": "FAILOVERCONTROLLER_role_env_safety_valve", "value": null }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hdfs_failovercontroller_file_descriptor", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Failover Controller logs. Typically used by log4j or logback.", "display_name": "Failover Controller Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "fc_config_safety_valve", "value": null }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Enables the health test that the Failover Controller's process state is consistent with the role configuration", "display_name": "Failover Controller Process Health Test", "name": "failovercontroller_scm_health_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Failover Controller", "name": "failover_controller_java_opts", "value": "" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Failover Controller parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Failover Controller", "name": "role_config_suppression_failover_controller_java_opts", "value": "false" }, { "desc": "The minimum log level for Failover Controller logs", "display_name": "Failover Controller Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hdfs_failovercontroller_log_directory_free_space", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/oozie-oozie_server.json0000666000175100017510000011373113245514472026243 0ustar zuulzuul00000000000000[ { "desc": "The period over which to compute the moving average of the callable queue size.", "display_name": "Oozie Server Callable Queue Monitoring Period", "name": "oozie_server_callable_queue_window", "value": "5" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "oozie_server_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Comma-separated list of ActionService executor extension classes. Only action types with associated executors can be used in workflows. For CDH 5.4 and higher, this parameter is used only to specify additional classes for workflows. All executor extension classes included in that release will be added automatically and do not need to be specified.", "display_name": "Oozie ActionService Executor Extension Classes", "name": "oozie_executor_extension_classes", "value": "" }, { "desc": "Comma-separated list of Oozie plug-ins to be activated. If one plugin cannot be loaded, all the plugins are ignored.", "display_name": "Oozie Server Plugins", "name": "oozie_plugins_list", "value": "" }, { "desc": "Enables the health test that the Oozie Server's process state is consistent with the role configuration", "display_name": "Oozie Server Process Health Test", "name": "oozie_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Password parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Password", "name": "role_config_suppression_oozie_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Plugins parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Plugins", "name": "role_config_suppression_oozie_plugins_list", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Oozie Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_oozie_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_oozie_https_keystore_password", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "oozie_server_web_metric_collection_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Oozie Server", "name": "oozie_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Host parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Host", "name": "role_config_suppression_oozie_email_smtp_host", "value": "false" }, { "desc": "Completed workflow jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Workflow Jobs", "name": "purgeservice_older_than", "value": "30" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_oozie_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie SchemaService Workflow Extension Schemas parameter.", "display_name": "Suppress Parameter Validation: Oozie SchemaService Workflow Extension Schemas", "name": "role_config_suppression_oozie_workflow_extension_schemas", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The SMTP server port to use for Oozie email action", "display_name": "Oozie Email Action SMTP Port", "name": "oozie_email_smtp_prt", "value": "25" }, { "desc": "Whether to suppress the results of the Callable Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Callable Queue Size", "name": "role_health_suppression_oozie_server_callablequeue_size_health", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Data Directory parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Data Directory", "name": "role_config_suppression_oozie_data_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_oozie_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_oozie_server_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action From Address parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action From Address", "name": "role_config_suppression_oozie_email_from_address", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Log Directory", "name": "role_config_suppression_oozie_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Enable SMTP authentication for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Enabled", "name": "oozie_email_smtp_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml", "name": "role_config_suppression_oozie_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "SMTP password for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Password", "name": "oozie_email_smtp_password", "value": null }, { "desc": "Maximum concurrency for a given callable type. Each command is a callable type: submit, start, run, etc. Each action type is a callable type: MapReduce, SSH, sub-workflow, etc. All commands that use action executors (action-start, action-end. etc.) use the action type as the callable type.", "display_name": "Maximum concurrency for a given callable type", "name": "oozie_service_callablequeueservice_callable_concurrency", "value": "10" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "If true, enables the Oozie Server web console. ExtJS 2.2 zip archive must be extracted to /var/lib/oozie on the same host as the Oozie Server.", "display_name": "Enable Oozie Server Web Console", "name": "oozie_web_console", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "oozie_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Username for connecting to the database used by Oozie Server. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database User", "name": "oozie_database_user", "value": "sa" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_oozie_server_pause_duration", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_oozie_server_file_descriptor", "value": "false" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "oozie_server_pause_duration_window", "value": "5" }, { "desc": "The from address to be used for mailing all emails for Oozie email action", "display_name": "Oozie Email Action From Address", "name": "oozie_email_from_address", "value": "oozie@localhost" }, { "desc": "Port of Oozie Server", "display_name": "Oozie HTTP Port", "name": "oozie_http_port", "value": "11000" }, { "desc": "Whether to suppress the results of the Oozie Server Shared Library Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Oozie Server Shared Library Check", "name": "role_health_suppression_oozie_server_shared_lib_version_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where Oozie Server will place its log files.", "display_name": "Oozie Server Log Directory", "name": "oozie_log_dir", "value": "/var/log/oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Oozie Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Oozie Server", "name": "role_config_suppression_oozie_java_opts", "value": "false" }, { "desc": "Completed bundle jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Bundle Jobs", "name": "purgeservice_bundle_older_than", "value": "7" }, { "desc": "Maximum callable queue size", "display_name": "Maximum Callable Queue Size", "name": "oozie_service_callablequeueservice_queue_size", "value": "10000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The password for the Oozie JKS keystore file.", "display_name": "Oozie TLS/SSL Server JKS Keystore File Password", "name": "oozie_https_keystore_password", "value": null }, { "desc": "Completed coordinator jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Coordinator Jobs", "name": "purgeservice_coord_older_than", "value": "7" }, { "desc": "Workflow Status metrics collection interval.", "display_name": "Workflow Status Metrics Collection Interval", "name": "oozie_job_metric_collection_interval", "value": "1" }, { "desc": "The minimum log level for Oozie Server logs", "display_name": "Oozie Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_oozie_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_oozie_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Name parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Name", "name": "role_config_suppression_oozie_database_name", "value": "false" }, { "desc": "When computing the overall Oozie Server health, consider the host's health.", "display_name": "Oozie Server Host Health Test", "name": "oozie_server_host_health_enabled", "value": "true" }, { "desc": "Password for connecting to the database used by Oozie Server. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database Password", "name": "oozie_database_password", "value": "" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie ActionService Executor Extension Classes parameter.", "display_name": "Suppress Parameter Validation: Oozie ActionService Executor Extension Classes", "name": "role_config_suppression_oozie_executor_extension_classes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds of the weighted average size of the Oozie Server callable queue over a recent period. See also Oozie Server Callable Queue Monitoring Period.", "display_name": "Oozie Server Callable Queue Monitoring Threshold", "name": "oozie_server_callable_queue_threshold", "value": "{\"critical\":\"95.0\",\"warning\":\"80.0\"}" }, { "desc": "Directory where the Oozie Server places its data. Only applicable when using Derby as the database type.", "display_name": "Oozie Server Data Directory", "name": "oozie_data_dir", "value": "/var/lib/oozie/data" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_oozie_server_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Oozie is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Oozie TLS/SSL Server JKS Keystore File Location", "name": "oozie_https_keystore_file", "value": "/var/lib/oozie/.keystore" }, { "desc": "Comma-separated list of SchemaService workflow extension schemas for additional action types. From CDH 5.4 and higher, this parameter is used only to specify additional schemas for workflows. All schemas included in that release will be added automatically and do not need to be specified.", "display_name": "Oozie SchemaService Workflow Extension Schemas", "name": "oozie_workflow_extension_schemas", "value": "" }, { "desc": "Port of the Oozie Server when using TLS/SSL.", "display_name": "Oozie HTTPS Port", "name": "oozie_https_port", "value": "11443" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to use the Codehale-based metrics for instrumentation. Enabling this disables the 'instrumentation' REST endpoint and enables the 'metrics' REST endpoint (<hostname:port>/v2/admin/metrics).", "display_name": "Enable The Metrics Instrumentation Service", "name": "oozie_use_metric_instrumentation", "value": "true" }, { "desc": "Type of the database used by Oozie Server.", "display_name": "Oozie Server Database Type", "name": "oozie_database_type", "value": "derby" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Host parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Host", "name": "role_config_suppression_oozie_database_host", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_oozie_https_keystore_file", "value": "false" }, { "desc": "The SMTP server host to use for Oozie email action", "display_name": "Oozie Email Action SMTP Host", "name": "oozie_email_smtp_host", "value": "localhost" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_oozie_server_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_oozie_https_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Authentication Password parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Authentication Password", "name": "role_config_suppression_oozie_email_smtp_password", "value": "false" }, { "desc": "SMTP username for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Username", "name": "oozie_email_smtp_username", "value": null }, { "desc": "The password for the Oozie TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Oozie TLS/SSL Certificate Trust Store Password", "name": "oozie_https_truststore_password", "value": null }, { "desc": "Number of threads used for executing callables", "display_name": "Number Threads For Executing Callables", "name": "oozie_service_callablequeueservice_threads", "value": "50" }, { "desc": "The maximum number of rolled log files to keep for Oozie Server logs. Typically used by log4j or logback.", "display_name": "Oozie Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "720" }, { "desc": "The admin port Oozie server runs.", "display_name": "Oozie Admin Port", "name": "oozie_admin_port", "value": "11001" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database User parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database User", "name": "role_config_suppression_oozie_database_user", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Oozie might connect to. This is used when Oozie is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Oozie TLS/SSL Certificate Trust Store File", "name": "oozie_https_truststore_file", "value": null }, { "desc": "If true, enables version check for Oozie Server and installed shared libraries.", "display_name": "Enable Oozie Server Shared Libraries Version Check", "name": "oozie_server_shared_lib_version_check_enabled", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "oozie_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Name of the database used by Oozie Server.", "display_name": "Oozie Server Database Name", "name": "oozie_database_name", "value": "oozie" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to purge completed workflows and their corresponding coordinator actions of long-running coordinator jobs if the completed workflow jobs are older than the value specified in oozie.service.PurgeService.older.than.", "display_name": "Enable Purge for Long-Running Coordinator Jobs", "name": "purgeservice_purge_old_coord_action", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_oozie_https_truststore_password", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "For advanced use only. A string to be inserted into oozie-site.xml for this role only.", "display_name": "Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml", "name": "oozie_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Authentication Username parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Authentication Username", "name": "role_config_suppression_oozie_email_smtp_username", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Oozie Server in Bytes", "name": "oozie_java_heapsize", "value": "1073741824" }, { "desc": "Hostname of the database used by Oozie Server. If the port is non-default for your database type, use host:port notation. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database Host", "name": "oozie_database_host", "value": "localhost" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Oozie Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "OOZIE_SERVER_role_env_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/cdh_config.py0000666000175100017510000000757113245514472024153 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json import six from sahara.plugins.cdh.client import api_client # -- cm config -- cm_address = 'localhost' cm_port = 7180 cm_username = 'admin' cm_password = 'admin' hdfs_service_name = 'hdfs01' yarn_service_name = 'yarn01' oozie_service_name = 'oozie01' hive_service_name = 'hive01' hue_service_name = 'hue01' spark_service_name = 'spark_on_yarn01' zookeeper_service_name = 'zookeeper01' hbase_service_name = 'hbase01' flume_service_name = 'flume01' sqoop_service_name = 'sqoop01' solr_service_name = 'solr01' ks_indexer_service_name = 'ks_indexer01' impala_service_name = 'impala01' sentry_service_name = 'sentry01' kms_service_name = 'kms01' def get_cm_api(): return api_client.ApiResource(cm_address, server_port=cm_port, username=cm_username, password=cm_password) def get_cluster(api): return api.get_all_clusters()[0] def process_service(service, service_name): for role_cfgs in service.get_all_role_config_groups(): role_cm_cfg = role_cfgs.get_config(view='full') role_cfg = parse_config(role_cm_cfg) role_name = role_cfgs.roleType.lower() write_cfg(role_cfg, '%s-%s.json' % (service_name, role_name)) service_cm_cfg = service.get_config(view='full')[0] service_cfg = parse_config(service_cm_cfg) write_cfg(service_cfg, '%s-service.json' % service_name) def parse_config(config): cfg = [] for name, value in six.iteritems(config): p = { 'name': value.name, 'value': value.default, 'display_name': value.displayName, 'desc': value.description } cfg.append(p) return cfg def write_cfg(cfg, file_name): to_write = json.dumps(cfg, sort_keys=True, indent=4, separators=(',', ': ')) with open(file_name, 'w') as f: f.write(to_write) def main(): client = get_cm_api() cluster = get_cluster(client) hdfs = cluster.get_service(hdfs_service_name) process_service(hdfs, 'hdfs') yarn = cluster.get_service(yarn_service_name) process_service(yarn, 'yarn') oozie = cluster.get_service(oozie_service_name) process_service(oozie, 'oozie') hive = cluster.get_service(hive_service_name) process_service(hive, 'hive') hue = cluster.get_service(hue_service_name) process_service(hue, 'hue') spark = cluster.get_service(spark_service_name) process_service(spark, 'spark') zookeeper = cluster.get_service(zookeeper_service_name) process_service(zookeeper, 'zookeeper') hbase = cluster.get_service(hbase_service_name) process_service(hbase, 'hbase') flume = cluster.get_service(flume_service_name) process_service(flume, 'flume') sqoop = cluster.get_service(sqoop_service_name) process_service(sqoop, 'sqoop') solr = cluster.get_service(solr_service_name) process_service(solr, 'solr') ks_indexer = cluster.get_service(ks_indexer_service_name) process_service(ks_indexer, 'ks_indexer') impala = cluster.get_service(impala_service_name) process_service(impala, 'impala') sentry = cluster.get_service(sentry_service_name) process_service(sentry, 'sentry') kms = cluster.get_service(kms_service_name) process_service(kms, 'kms') if __name__ == '__main__': main() sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/impala-statestore.json0000666000175100017510000005662613245514472026054 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "statestore_web_metric_collection_enabled", "value": "true" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "statestore_startup_tolerance", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_statestore_cmd_args_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "statestore_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_statestore_web_metric_collection", "value": "false" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "StateStore Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Port where StateStoreService is exported.", "display_name": "StateStore Service Port", "name": "state_store_port", "value": "24000" }, { "desc": "When computing the overall Impala StateStore health, consider the host's health.", "display_name": "Impala StateStore Host Health Test", "name": "statestore_host_health_enabled", "value": "true" }, { "desc": "Directory where a StateStore core dump is placed.", "display_name": "StateStore Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/statestore" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_statestore_scm_health", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Breakpad Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Statestore Breakpad Dump Directory", "name": "role_config_suppression_minidump_path", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Username for Statestore webserver authentication.", "display_name": "Statestore Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when StateStore Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "StateStore Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "Maximum number of tasks allowed to be pending at the thread manager underlying the StateStore Thrift server (0 allows infinitely many pending tasks)", "display_name": "Maximum StateStore Pending Tasks", "name": "state_store_pending_task_count_max", "value": "0" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_statestore_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Directory where StateStore will place its log files.", "display_name": "StateStore Log Directory", "name": "log_dir", "value": "/var/log/statestore" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Enables the health test that the Impala StateStore's process state is consistent with the role configuration", "display_name": "Impala StateStore Process Health Test", "name": "statestore_scm_health_enabled", "value": "true" }, { "desc": "Port where StateStore debug web server runs.", "display_name": "StateStore HTTP Server Port", "name": "statestore_webserver_port", "value": "25010" }, { "desc": "Maximum number of Breakpad dump files stored by Statestore Role.", "display_name": "Statestore Max Breakpad Dump Files", "name": "max_minidumps", "value": "9" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Statestore Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when StateStore Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "StateStore Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_statestore_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_statestore_host_health", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to StateStore command line flags.", "display_name": "Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "statestore_cmd_args_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Number of worker threads for the thread manager underlying the StateStore Thrift server.", "display_name": "StateStore Worker Threads", "name": "state_store_num_server_worker_threads", "value": "4" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Log Directory parameter.", "display_name": "Suppress Parameter Validation: StateStore Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_statestore_memory_rss_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala StateStore Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala StateStore Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_statestore_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Statestore Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "The password for the private key in the StateStore Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "StateStore Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_statestore_file_descriptor", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above. ", "display_name": "StateStore Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The maximum size, in megabytes, per log file for Impala StateStore logs. Typically used by log4j or logback.", "display_name": "Impala StateStore Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_statestore_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "statestore_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: StateStore Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "Enable/Disable StateStore web server. This web server contains useful information about StateStore daemon.", "display_name": "Enable StateStore Web Server", "name": "statestore_enable_webserver", "value": "true" }, { "desc": "Directory for storing Statestore Breakpad dumps.", "display_name": "Statestore Breakpad Dump Directory", "name": "minidump_path", "value": "/var/log/impala-minidumps" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala StateStore Environment Advanced Configuration Snippet (Safety Valve)", "name": "STATESTORE_role_env_safety_valve", "value": null }, { "desc": "Password for Statestore webserver authentication.", "display_name": "Statestore Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "The minimum log level for Impala StateStore logs", "display_name": "Impala StateStore Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "StateStore Maximum Log Files", "name": "max_log_files", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: StateStore Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_statestore_swap_memory_usage", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/kafka-kafka_mirror_maker.json0000666000175100017510000007023613245514472027310 0ustar zuulzuul00000000000000[ { "desc": "The maximum size, in megabytes, per log file for Kafka MirrorMaker logs. Typically used by log4j or logback.", "display_name": "Kafka MirrorMaker Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Stop the entire mirror maker when a send failure occurs.", "display_name": "Abort on Send Failure", "name": "abort.on.send.failure", "value": "true" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Maximum number of bytes that can be buffered between producer and consumer. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Queue Size", "name": "queue.byte.size", "value": "100000000" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kafka_kafka_mirror_maker_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kafka_mirror_maker_role_env_safety_valve", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Only required if source Kafka cluster requires client authentication.", "display_name": "Source Kafka Cluster's Client Auth", "name": "source.ssl.client.auth", "value": "false" }, { "desc": "When computing the overall Kafka MirrorMaker health, consider the host's health.", "display_name": "Kafka MirrorMaker Host Health Test", "name": "kafka_mirror_maker_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties", "name": "role_config_suppression_ssl_client.properties_role_safety_valve", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Kafka MirrorMaker logs. Typically used by log4j or logback.", "display_name": "Kafka MirrorMaker Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topic Blacklist parameter.", "display_name": "Suppress Parameter Validation: Topic Blacklist", "name": "role_config_suppression_blacklist", "value": "false" }, { "desc": "The minimum log level for Kafka MirrorMaker logs", "display_name": "Kafka MirrorMaker Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Offset commit interval in milliseconds.", "display_name": "Offset Commit Interval", "name": "offset.commit.interval.ms", "value": "60000" }, { "desc": "For advanced use only. A string to be inserted into mirror_maker_producers.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties", "name": "mirror_maker_producers.properties_role_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kafka_kafka_mirror_maker_host_health", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kafka_mirror_maker_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties", "name": "role_config_suppression_ssl_server.properties_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The password for the Kafka MirrorMaker JKS keystore file.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Destination Broker List parameter.", "display_name": "Suppress Parameter Validation: Destination Broker List", "name": "role_config_suppression_bootstrap.servers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Number of producer instances. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Number of Producers", "name": "num.producers", "value": "1" }, { "desc": "For advanced use only. A string to be inserted into ssl_client.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties", "name": "ssl_client.properties_role_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_MIRROR_MAKER_role_env_safety_valve", "value": null }, { "desc": "Name of the consumer group used by MirrorMaker. When multiple role instances are configured with the same topics and same group ID, the role instances load-balance replication for the topics. When multiple role instances are configured with the same topics but different group ID, each role instance replicates all the events for those topics - this can be used to replicate the source cluster into multiple destination clusters.", "display_name": "Consumer Group ID", "name": "group.id", "value": "cloudera_mirrormaker" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kafka_kafka_mirror_maker_swap_memory_usage", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Consumer Group ID parameter.", "display_name": "Suppress Parameter Validation: Consumer Group ID", "name": "role_config_suppression_group.id", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Consumer Rebalance Listener Arguments parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Consumer Rebalance Listener Arguments", "name": "role_config_suppression_consumer.rebalance.listener.args", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka MirrorMaker might connect to. This is used when Kafka MirrorMaker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Kafka MirrorMaker TLS/SSL Certificate Trust Store File", "name": "ssl_client_truststore_location", "value": "" }, { "desc": "Enables the health test that the Kafka MirrorMaker's process state is consistent with the role configuration", "display_name": "Kafka MirrorMaker Process Health Test", "name": "kafka_mirror_maker_scm_health_enabled", "value": "true" }, { "desc": "Number of messages that are buffered between producer and consumer. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Message Queue Size", "name": "queue.size", "value": "10000" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "A consumer rebalance listener of type ConsumerRebalanceListener to be invoked when MirrorMaker's consumer rebalances.", "display_name": "MirrorMaker Consumer Rebalance Listener", "name": "consumer.rebalance.listener", "value": "" }, { "desc": "Run with MirrorMaker settings that eliminate potential loss of data. This impacts performance, but is highly recommended. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Avoid Data Loss", "name": "no.data.loss", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties", "name": "role_config_suppression_mirror_maker_producers.properties_role_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into mirror_maker_consumers.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties", "name": "mirror_maker_consumers.properties_role_safety_valve", "value": null }, { "desc": "Protocol to be used for communication with source kafka cluster.", "display_name": "Source Kafka Cluster's Security Protocol", "name": "source.security.protocol", "value": "PLAINTEXT" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Consumer Rebalance Listener parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Consumer Rebalance Listener", "name": "role_config_suppression_consumer.rebalance.listener", "value": "false" }, { "desc": "Protocol to be used for communication with destination kafka cluster.", "display_name": "Destination Kafka Cluster's Security Protocol", "name": "destination.security.protocol", "value": "PLAINTEXT" }, { "desc": "Regular expression that represents a set of topics to mirror. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used.", "display_name": "Topic Whitelist", "name": "whitelist", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kafka_kafka_mirror_maker_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topic Whitelist parameter.", "display_name": "Suppress Parameter Validation: Topic Whitelist", "name": "role_config_suppression_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Port for JMX.", "display_name": "JMX Port", "name": "jmx_port", "value": "9394" }, { "desc": "Arguments used by MirrorMaker message handler.", "display_name": "MirrorMaker Message Handler Arguments", "name": "message.handler.args", "value": "" }, { "desc": "List of brokers on destination cluster. This should be more than one, for high availability, but there's no need to list all brokers.", "display_name": "Destination Broker List", "name": "bootstrap.servers", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Source Broker List parameter.", "display_name": "Suppress Parameter Validation: Source Broker List", "name": "role_config_suppression_source.bootstrap.servers", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kafka_kafka_mirror_maker_file_descriptor", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka MirrorMaker is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Message Handler parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Message Handler", "name": "role_config_suppression_message.handler", "value": "false" }, { "desc": "Only required if destination Kafka cluster requires client authentication.", "display_name": "Destination Kafka Cluster's Client Auth", "name": "destination.ssl.client.auth", "value": "false" }, { "desc": "A MirrorMaker message handler of type MirrorMakerMessageHandler that will process every record in-between producer and consumer.", "display_name": "MirrorMaker Message Handler", "name": "message.handler", "value": "" }, { "desc": "List of brokers on source cluster. This should be more than one, for high availability, but there's no need to list all brokers.", "display_name": "Source Broker List", "name": "source.bootstrap.servers", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties", "name": "role_config_suppression_mirror_maker_consumers.properties_role_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "For advanced use only. A string to be inserted into ssl_server.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties", "name": "ssl_server.properties_role_safety_valve", "value": null }, { "desc": "Number of consumer threads.", "display_name": "Number of Consumer Threads", "name": "num.streams", "value": "1" }, { "desc": "The password that protects the private key contained in the JKS keystore used when Kafka MirrorMaker is acting as a TLS/SSL server.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": "" }, { "desc": "Encrypt communication between clients and Kafka MirrorMaker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Kafka MirrorMaker", "name": "ssl_enabled", "value": "false" }, { "desc": "Regular expression that represents a set of topics to avoid mirroring. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Topic Blacklist", "name": "blacklist", "value": "" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Arguments used by MirrorMaker consumer rebalance listener.", "display_name": "MirrorMaker Consumer Rebalance Listener Arguments", "name": "consumer.rebalance.listener.args", "value": "" }, { "desc": "The log directory for log files of the role Kafka MirrorMaker.", "display_name": "Kafka MirrorMaker Log Directory", "name": "log_dir", "value": "/var/log/kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Message Handler Arguments parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Message Handler Arguments", "name": "role_config_suppression_message.handler.args", "value": "false" }, { "desc": "The password for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Kafka MirrorMaker TLS/SSL Certificate Trust Store Password", "name": "ssl_client_truststore_password", "value": "" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hue-kt_renewer.json0000666000175100017510000003313313245514472025326 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_kt_renewer_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kt_renewer_swap_memory_usage", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Ticket Renewer Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kerberos Ticket Renewer Log Directory", "name": "role_config_suppression_kt_renewer_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kt_renewer_unexpected_exits", "value": "false" }, { "desc": "Interval in seconds with which Hue's Kerberos ticket will get renewed.", "display_name": "Hue Keytab Renewal Interval", "name": "keytab_reinit_frequency", "value": "3600" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Enables the health test that the Kerberos Ticket Renewer's process state is consistent with the role configuration", "display_name": "Kerberos Ticket Renewer Process Health Test", "name": "kt_renewer_scm_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve)", "name": "KT_RENEWER_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When computing the overall Kerberos Ticket Renewer health, consider the host's health.", "display_name": "Kerberos Ticket Renewer Host Health Test", "name": "kt_renewer_host_health_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kt_renewer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kt_renewer_host_health", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Directory where Kerberos Ticket Renewer will place its log files.", "display_name": "Kerberos Ticket Renewer Log Directory", "name": "kt_renewer_log_dir", "value": "/var/log/hue" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kt_renewer_role_env_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kt_renewer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_kt_renewer_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kt_renewer_scm_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/solr-solr_server.json0000666000175100017510000006440113245514472025726 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Solr Load Balancer", "name": "role_config_suppression_solr_load_balancer", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Number of blocks per cache slab. The size of the cache is 8 KB (the block size) times the number of blocks per slab times the number of slabs.", "display_name": "HDFS Block Cache Blocks per Slab", "name": "solr_hdfs_blockcache_blocksperbank", "value": "16384" }, { "desc": "When computing the overall Solr Server health, consider the host's health.", "display_name": "Solr Server Host Health Test", "name": "solr_server_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "solr_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Solr Server in Bytes", "name": "solr_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Solr Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Solr Server", "name": "role_config_suppression_solr_java_opts", "value": "false" }, { "desc": "Enables the health test that the Solr Server's process state is consistent with the role configuration", "display_name": "Solr Server Process Health Test", "name": "solr_server_scm_health_enabled", "value": "true" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Solr Server logs. Typically used by log4j or logback.", "display_name": "Solr Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Solr Server API Liveness heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Solr Server API Liveness", "name": "role_health_suppression_solr_core_status_collection_health", "value": "false" }, { "desc": "The maximum number of request processing threads to be created by Solr server, which determines the maximum number of simultaneous requests that can be handled.", "display_name": "Solr Max Connector Threads", "name": "solr_max_connector_thread", "value": "10000" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_solr_server_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds on the duration of the Solr Server API request.", "display_name": "Solr Server API Liveness Request Duration", "name": "solr_core_status_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_solr_server_host_health", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_solr_server_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Solr Server Log Directory", "name": "role_config_suppression_solr_log_dir", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of Solr Server in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of Solr Server in Bytes", "name": "role_config_suppression_solr_java_heapsize", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_solr_server_file_descriptor", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Solr Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The time in milliseconds a client is allowed to not talk to ZooKeeper before its session expires.", "display_name": "ZooKeeper Client Timeout", "name": "zookeeper_client_timeout", "value": "15000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "solr_server_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Use off-heap memory when caching HDFS blocks in Solr.", "display_name": "HDFS Block Cache Off-Heap Memory", "name": "solr_hdfs_blockcache_direct_memory_allocation", "value": "true" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather status of Solr Cores from the Solr Server with a simple API request.", "display_name": "Solr Server API Liveness", "name": "solr_core_status_collection_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Solr Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SOLR_SERVER_role_env_safety_valve", "value": null }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "HTTPS port of Solr Server.", "display_name": "Solr HTTPS port", "name": "solr_https_port", "value": "8985" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Timeout (in seconds) for graceful shutdown of this Solr server. Once this timeout is reached, Solr server is abruptly shutdown. A value of 0 means no timeout.", "display_name": "Graceful Shutdown Timeout", "name": "solr_graceful_stop_timeout", "value": "180" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_solr_server_gc_duration", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Admin port of the Solr Server.", "display_name": "Solr Admin Port", "name": "solr_admin_port", "value": "8984" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Data Directory parameter.", "display_name": "Suppress Parameter Validation: Solr Data Directory", "name": "role_config_suppression_solr_data_dir", "value": "false" }, { "desc": "Maximum amount of off-heap memory in bytes that may be allocated by the Java process. Passed to Java -XX:MaxDirectMemorySize. If unset, defaults to the size of the heap.", "display_name": "Java Direct Memory Size of Solr Server in Bytes", "name": "solr_java_direct_memory_size", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_solr_server_role_env_safety_valve", "value": "false" }, { "desc": "Enable caching of HDFS blocks in Solr. There is one block cache per Solr collection. configured to use off-heap memory, Maximum Off-Heap Memory must be set high enough to account for all block caches.", "display_name": "HDFS Block Cache", "name": "solr_hdfs_blockcache_enabled", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_solr_server_unexpected_exits", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Solr Server logs. Typically used by log4j or logback.", "display_name": "Solr Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Address of the load balancer, specified in host:port format.", "display_name": "Solr Load Balancer", "name": "solr_load_balancer", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "solr_server_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_solr_server_log_directory_free_space", "value": "false" }, { "desc": "Directory on local file system where Solr Server keeps the configurations for collections.", "display_name": "Solr Data Directory", "name": "solr_data_dir", "value": "/var/lib/solr" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "solr_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Number of slabs per block cache. The size of the cache is 8 KB (the block size) times the number of blocks per slab times the number of slabs.", "display_name": "HDFS Block Cache Number of Slabs", "name": "solr_hdfs_blockcache_slab_count", "value": "1" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_solr_server_web_metric_collection", "value": "false" }, { "desc": "Directory where Solr Server will place its log files.", "display_name": "Solr Server Log Directory", "name": "solr_log_dir", "value": "/var/log/solr" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "solr_server_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Solr Server", "name": "solr_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The minimum log level for Solr Server logs", "display_name": "Solr Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_solr_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "HTTP port of Solr Server.", "display_name": "Solr HTTP Port", "name": "solr_http_port", "value": "8983" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hbase-hbasethriftserver.json0000666000175100017510000007310413245514472027216 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbasethriftserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase Thrift Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase Thrift Server", "name": "role_config_suppression_hbase_thriftserver_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_thrift_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server DNS Network Interface", "name": "role_config_suppression_hbase_thriftserver_dns_interface", "value": "false" }, { "desc": "The \"core size\" of the thread pool. New threads are created on every connection until this many threads are created.", "display_name": "HBase Thrift Server Min Worker Threads", "name": "hbase_thriftserver_min_worker_threads", "value": "200" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_thrift_server_host_health", "value": "false" }, { "desc": "The port that HBase Thrift Server binds to.", "display_name": "HBase Thrift Server Port", "name": "hbase_thriftserver_port", "value": "9090" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Bind Address parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Bind Address", "name": "role_config_suppression_hbase_thriftserver_bindaddress", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_thrift_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server DNS Name Server", "name": "role_config_suppression_hbase_thriftserver_dns_nameserver", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Log Directory", "name": "role_config_suppression_hbase_thriftserver_log_dir", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HBase Thrift Server logs. Typically used by log4j or logback.", "display_name": "HBase Thrift Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_thrift_server_log_directory_free_space", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase Thrift Server over HTTP is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location", "name": "hbase_thriftserver_http_keystore_file", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase Thrift Server in Bytes", "name": "hbase_thriftserver_java_heapsize", "value": "1073741824" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The password for the HBase Thrift Server over HTTP JKS keystore file.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password", "name": "hbase_thriftserver_http_keystore_password", "value": null }, { "desc": "If true, HBase Thrift Server Web UI will bind to a wildcard address (0.0.0.0). Otherwise it will bind to a host name. Only available in CDH 4.3 and later.", "display_name": "HBase Thrift Server Web UI Bind to Wildcard Address", "name": "hbase_thriftserver_info_bind_to_wildcard", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_hbase_thriftserver_http_keystore_file", "value": "false" }, { "desc": "Address to bind the HBase Thrift Server to. When using the THsHaServer or the TNonblockingServer, always binds to 0.0.0.0 irrespective of this configuration value.", "display_name": "HBase Thrift Server Bind Address", "name": "hbase_thriftserver_bindaddress", "value": "0.0.0.0" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Type of HBase Thrift Server.", "display_name": "HBase Thrift Server Type", "name": "hbase_thriftserver_type", "value": "threadpool" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_hbase_thriftserver_http_keystore_password", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASETHRIFTSERVER_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When computing the overall HBase Thrift Server health, consider the host's health.", "display_name": "HBase Thrift Server Host Health Test", "name": "hbasethriftserver_host_health_enabled", "value": "true" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase Thrift Server over HTTP is acting as a TLS/SSL server.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password", "name": "hbase_thriftserver_http_keystore_keypassword", "value": null }, { "desc": "Enables the health test that the HBase Thrift Server's process state is consistent with the role configuration", "display_name": "HBase Thrift Server Process Health Test", "name": "hbasethriftserver_scm_health_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase Thrift Server", "name": "hbase_thriftserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Use framed transport. When using the THsHaServer or TNonblockingServer, framed transport is always used irrespective of this configuration value.", "display_name": "Enable HBase Thrift Server Framed Transport", "name": "hbase_thriftserver_framed", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for HBase Thrift Server logs. Typically used by log4j or logback.", "display_name": "HBase Thrift Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The port that HBase Thrift Server Web UI binds to.", "display_name": "HBase Thrift Server Web UI Port", "name": "hbase_thriftserver_info_port", "value": "9095" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_hbase_thriftserver_http_keystore_keypassword", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_thrift_server_swap_memory_usage", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The host name or IP address of the DNS name server which an HBase Thrift Server should use to determine the host name used for communication and display purposes.", "display_name": "HBase Thrift Server DNS Name Server", "name": "hbase_thriftserver_dns_nameserver", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_thrift_server_scm_health", "value": "false" }, { "desc": "Use the TCompactProtocol instead of the default TBinaryProtocol. TCompactProtocol is a binary protocol that is more compact than the default and typically more efficient.", "display_name": "Enable HBase Thrift Server Compact Protocol", "name": "hbase_thriftserver_compact", "value": "false" }, { "desc": "The name of the DNS network interface from which an HBase Thrift Server should report its IP address.", "display_name": "HBase Thrift Server DNS Network Interface", "name": "hbase_thriftserver_dns_interface", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Directory where HBase Thrift Server will place its log files.", "display_name": "HBase Thrift Server Log Directory", "name": "hbase_thriftserver_log_dir", "value": "/var/log/hbase" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_thriftserver_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_thrift_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbasethriftserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_thriftserver_config_safety_valve", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for HBase Thrift Server logs", "display_name": "HBase Thrift Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Encrypt communication between clients and HBase Thrift Server over HTTP using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HBase Thrift Server over HTTP", "name": "hbase_thriftserver_http_use_ssl", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-journalnode.json0000666000175100017510000007250613245514472025653 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "journalnode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_journal_node_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Fsync Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Fsync Latency", "name": "role_health_suppression_journal_node_fsync_latency", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: JournalNode Log Directory", "name": "role_config_suppression_journalnode_log_dir", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of JournalNode in Bytes", "name": "journalNode_java_heapsize", "value": "536870912" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "journalnode_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_journal_node_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_journal_node_host_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Enables the health test that the JournalNode's process state is consistent with the role configuration", "display_name": "JournalNode Process Health Test", "name": "journalnode_scm_health_enabled", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The maximum number of rolled log files to keep for JournalNode logs. Typically used by log4j or logback.", "display_name": "JournalNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JournalNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a JournalNode Edits Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "JournalNode Edits Directory Free Space Monitoring Percentage Thresholds", "name": "journalnode_edits_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_jn_config_safety_valve", "value": "false" }, { "desc": "Directory on the local file system where NameNode edits are written.", "display_name": "JournalNode Edits Directory", "name": "dfs_journalnode_edits_dir", "value": null }, { "desc": "Whether to suppress the results of the Sync Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sync Status", "name": "role_health_suppression_journal_node_sync_status", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_journal_node_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The base port where the secure JournalNode web UI listens. Combined with the JournalNode's hostname to build its secure web UI address.", "display_name": "Secure JournalNode Web UI Port (TLS/SSL)", "name": "dfs_journalnode_https_port", "value": "8481" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_journal_node_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "jn_config_safety_valve", "value": null }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "When computing the overall JournalNode health, consider the host's health.", "display_name": "JournalNode Host Health Test", "name": "journalnode_host_health_enabled", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JournalNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for JournalNode", "name": "role_config_suppression_journalnode_java_opts", "value": "false" }, { "desc": "The amount of time at JournalNode startup allowed for the active NameNode to get in sync with the JournalNode.", "display_name": "Active NameNode Sync Status Startup Tolerance", "name": "journalnode_sync_status_startup_tolerance", "value": "180" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The health test thresholds for JournalNode fsync latency.", "display_name": "JournalNode Fsync Latency Thresholds", "name": "journalnode_fsync_latency_thresholds", "value": "{\"critical\":\"3000.0\",\"warning\":\"1000.0\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "journalnode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_journal_node_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JournalNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_journalnode_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "JournalNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enables the health check that verifies the active NameNode's sync status to the JournalNode", "display_name": "Active NameNode Sync Status Health Check", "name": "journalnode_sync_status_enabled", "value": "true" }, { "desc": "Port for the JournalNode's RPC. Combined with the JournalNode's hostname to build its RPC address.", "display_name": "JournalNode RPC Port", "name": "dfs_journalnode_rpc_port", "value": "8485" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "journalnode_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Port for the JournalNode HTTP web UI. Combined with the JournalNode hostname to build its HTTP address.", "display_name": "JournalNode HTTP Port", "name": "dfs_journalnode_http_port", "value": "8480" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Edits Directory parameter.", "display_name": "Suppress Parameter Validation: JournalNode Edits Directory", "name": "role_config_suppression_dfs_journalnode_edits_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "If enabled, the JournalNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind JournalNode to Wildcard Address", "name": "journalnode_bind_wildcard", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_journal_node_web_metric_collection", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "journalnode_gc_duration_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory.", "display_name": "JournalNode Edits Directory Free Space Monitoring Absolute Thresholds", "name": "journalnode_edits_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The maximum size, in megabytes, per log file for JournalNode logs. Typically used by log4j or logback.", "display_name": "JournalNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_journal_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "Directory where JournalNode will place its log files.", "display_name": "JournalNode Log Directory", "name": "journalnode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "JournalNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "JOURNALNODE_role_env_safety_valve", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for JournalNode", "name": "journalNode_java_opts", "value": "" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the JournalNode Edits Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JournalNode Edits Directory Free Space", "name": "role_health_suppression_journal_node_edits_directory_free_space", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for JournalNode logs", "display_name": "JournalNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_journal_node_gc_duration", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/yarn-service.json0000666000175100017510000012735513245514472025023 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds of the overall NodeManager health. The check returns \"Concerning\" health if the percentage of \"Healthy\" NodeManagers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" NodeManagers falls below the critical threshold.", "display_name": "Healthy NodeManager Monitoring Thresholds", "name": "yarn_nodemanagers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Entries to add to the classpaths of YARN applications.", "display_name": "YARN Application Classpath", "name": "yarn_application_classpath", "value": "$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "service_config_suppression_yarn_service_replication_config_safety_valve", "value": "false" }, { "desc": "The user that this service's processes should run as. (Except the Job History Server, which has its own user)", "display_name": "System User", "name": "process_username", "value": "yarn" }, { "desc": "Name of the HDFS service that this YARN service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Whether to suppress the results of the ResourceManager Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ResourceManager Health", "name": "service_health_suppression_yarn_resourcemanagers_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Container Usage Output Directory parameter.", "display_name": "Suppress Parameter Validation: Container Usage Output Directory", "name": "service_config_suppression_cm_yarn_container_usage_output_dir", "value": "false" }, { "desc": "The tolerance window used in YARN service tests that depend on detection of the active ResourceManager.", "display_name": "Active ResourceManager Detection Window", "name": "yarn_active_resourcemanager_detecton_window", "value": "3" }, { "desc": "The amount of time after ResourceManager(s) start that the lack of an active ResourceManager will be tolerated. This is an advanced option that does not often need to be changed.", "display_name": "ResourceManager Activation Startup Tolerance", "name": "yarn_resourcemanager_activation_startup_tolerance", "value": "180" }, { "desc": "When computing the overall YARN service health, whether to consider the health of the standby ResourceManager.", "display_name": "Standby ResourceManager Health Check", "name": "yarn_standby_resourcemanager_health_enabled", "value": "true" }, { "desc": "Whether users and groups specified in Admin ACL should be checked for authorization to perform admin operations.", "display_name": "Enable ResourceManager ACLs", "name": "yarn_acl_enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "YARN pool which is used to submit the job to aggregate container usage metrics.", "display_name": "Container Usage MapReduce Job Pool", "name": "cm_yarn_container_usage_job_pool", "value": null }, { "desc": "For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "yarn_hadoop_policy_config_safety_valve", "value": null }, { "desc": "DFS directory where the container usage metrics from Cloudera Manager sink are stored by YARN NodeManagers. Cloudera Service Monitor will read the container usage metrics and aggregate them for generating usage reports. Note: If you change this, you will need to re-run the Create YARN Container Usage Metrics Dir command.", "display_name": "Cloudera Manager Container Usage Metrics Directory", "name": "cm_yarn_container_usage_input_dir", "value": "/tmp/cmYarnContainerMetrics" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Application Classpath parameter.", "display_name": "Suppress Parameter Validation: YARN Application Classpath", "name": "service_config_suppression_yarn_application_classpath", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "Password for the TLS/SSL client truststore. Overrides the cluster-wide default truststore password set in HDFS.", "display_name": "TLS/SSL Client Truststore File Password", "name": "ssl_client_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "yarn_core_site_safety_valve", "value": null }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the NodeManager Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NodeManager Count Validator", "name": "service_config_suppression_nodemanager_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "service_config_suppression_mapreduce_service_replication_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_yarn_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Container Usage MapReduce Job Pool parameter.", "display_name": "Suppress Parameter Validation: Container Usage MapReduce Job Pool", "name": "service_config_suppression_cm_yarn_container_usage_job_pool", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules.", "display_name": "Fair Scheduler Allocations (Deployed)", "name": "yarn_fs_scheduled_allocations", "value": "{\"defaultFairSharePreemptionThreshold\":null,\"defaultFairSharePreemptionTimeout\":null,\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":null,\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":[{\"create\":true,\"name\":\"specified\",\"queue\":null,\"rules\":null},{\"create\":null,\"name\":\"nestedUserQueue\",\"queue\":null,\"rules\":[{\"create\":null,\"name\":\"default\",\"queue\":\"users\",\"rules\":null}]},{\"create\":true,\"name\":\"default\",\"queue\":null,\"rules\":null}],\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null,\"type\":null},{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"users\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null,\"type\":\"parent\"}],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null,\"type\":null}],\"userMaxAppsDefault\":null,\"users\":[]}" }, { "desc": "Controls which applications non-admin users can see in the applications list view", "display_name": "Non-Admin Users Applications List Visibility Settings", "name": "user_application_list_settings", "value": "ALL" }, { "desc": "When computing the overall YARN health, consider JobHistory Server's health", "display_name": "JobHistory Server Role Health Test", "name": "yarn_jobhistoryserver_health_enabled", "value": "true" }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted shuffle and encrypted web UIs. Applies to all configurations of daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "service_config_suppression_yarn_hadoop_policy_config_safety_valve", "value": "false" }, { "desc": "Interval at which YARN container usage metrics are sampled. Increasing this configuration can reduce the accuracy of container usage metrics, whereas setting it too low will increase the resources used to compute container usage.", "display_name": "Container Metrics Sampling Interval", "name": "yarn_container_seconds_per_sample", "value": "3000" }, { "desc": "Password for the server keystore file used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Whether to suppress the results of the JobHistory Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JobHistory Server Health", "name": "service_health_suppression_yarn_jobhistory_health", "value": "false" }, { "desc": "Enables Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Note: This is effective only if Kerberos is enabled for the HDFS service.", "display_name": "Enable Kerberos Authentication for HTTP Web-Consoles", "name": "hadoop_secure_web_ui", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Secure Web UI Validator configuration validator.", "display_name": "Suppress Configuration Validator: Secure Web UI Validator", "name": "service_config_suppression_hadoop_secure_web_ui", "value": "false" }, { "desc": "Path to the truststore file used when roles of this service act as TLS/SSL clients. Overrides the cluster-wide default truststore location set in HDFS. This truststore must be in JKS format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. The contents of the truststore can be modified without restarting any roles. By default, changes to its contents are picked up within ten seconds. If not set, the default Java truststore is used to verify certificates.", "display_name": "TLS/SSL Client Truststore File Location", "name": "ssl_client_truststore_location", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Container Usage MapReduce Job User parameter.", "display_name": "Suppress Parameter Validation: Container Usage MapReduce Job User", "name": "service_config_suppression_cm_yarn_container_usage_job_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "io.compression.codecsorg.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress the results of the YARN Container Usage Aggregation heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: YARN Container Usage Aggregation", "name": "service_health_suppression_yarn_usage_aggregation_health", "value": "false" }, { "desc": "User which Cloudera Service Monitor uses to run the MapReduce job to aggregate container usage metrics. Note: If you change this user, you will need to change the owner of the existing Container Usage Metrics and Output Directories.", "display_name": "Container Usage MapReduce Job User", "name": "cm_yarn_container_usage_job_user", "value": null }, { "desc": "For advanced use only, a string to be inserted into yarn-site.xml. Applies to all HDFS Replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_service_replication_config_safety_valve", "value": null }, { "desc": "Number of reduce tasks to use for the MapReduce job to aggregate container usage metrics.", "display_name": "Reduce Tasks for Container Usage MapReduce Job", "name": "cm_yarn_container_usage_job_num_reduce_tasks", "value": "1" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "yarn_service_env_safety_valve", "value": null }, { "desc": "Whether to enable log aggregation", "display_name": "Enable Log Aggregation", "name": "yarn_log_aggregation_enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cloudera Manager Container Usage Metrics Directory parameter.", "display_name": "Suppress Parameter Validation: Cloudera Manager Container Usage Metrics Directory", "name": "service_config_suppression_cm_yarn_container_usage_input_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "service_config_suppression_yarn_ssl_client_safety_valve", "value": "false" }, { "desc": "ACL that determines which users and groups can submit and kill applications in any pool, and can issue commands on ResourceManager roles.", "display_name": "Admin ACL", "name": "yarn_admin_acl", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Client Truststore File Password parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Client Truststore File Password", "name": "service_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "Cluster ID used when ResourceManager is Highly Available.", "display_name": "RM-HA Cluster ID", "name": "yarn_rm_ha_cluster_id", "value": "yarnRM" }, { "desc": "For advanced use only, a string to be inserted into yarn-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_service_config_safety_valve", "value": null }, { "desc": "Enables storing YARN container usage metrics in HDFS and periodically running a MapReduce job from Cloudera Service Monitor to aggregate them into per application metrics. This is needed for YARN usage reporting feature to work.", "display_name": "Enable Container Usage Metrics Collection", "name": "cm_yarn_enable_container_usage_aggregation", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "yarn_ssl_server_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Llama Proxy User Hosts", "name": "service_config_suppression_llama_proxy_user_hosts_list", "value": "false" }, { "desc": "When set to true, pools specified in applications but not explicitly configured, are created at runtime with default settings. When set to false, applications specifying pools not explicitly configured run in a pool named default. This setting applies when an application explicitly specifies a pool and when the application runs in a pool named with the username associated with the application.", "display_name": "Allow Undeclared Pools", "name": "yarn_scheduler_fair_allow_undeclared_pools", "value": "true" }, { "desc": "How long to keep aggregation logs before deleting them.", "display_name": "Log Aggregation Retention Period", "name": "yarn_log_aggregation_retain_seconds", "value": "604800" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service MapReduce Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_yarn_service_mapred_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_yarn_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Llama Proxy User Groups", "name": "service_config_suppression_llama_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether YARN uses the Linux Container Executor both in secure (Kerberos) and insecure (not Kerberos) environments. Cgroups enforcement only works when the Linux Container Executor is used.", "display_name": "Always Use Linux Container Executor", "name": "yarn_service_lce_always", "value": "false" }, { "desc": "Whether to serve logs over HTTP from Yarn web servers. This includes listing the logs directory at the /logs endpoint, which may be a security concern.", "display_name": "Serve logs over HTTP", "name": "http_logs_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User's Home Directory parameter.", "display_name": "Suppress Parameter Validation: System User's Home Directory", "name": "service_config_suppression_hdfs_user_home_dir", "value": "false" }, { "desc": "UNIX user that containers run as when Linux-container-executor is used in nonsecure mode.", "display_name": "UNIX User for Nonsecure Mode with Linux Container Executor", "name": "yarn_nodemanager_linux_container_executor_nonsecure_mode_local_user", "value": "nobody" }, { "desc": "For advanced use only, a string to be inserted into mapred-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)", "name": "yarn_service_mapred_safety_valve", "value": null }, { "desc": "Comma-delimited list of hosts where you want to allow the Llama (AM for Impala) user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Llama Proxy User Hosts", "name": "llama_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the ResourceManager Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: ResourceManager Count Validator", "name": "service_config_suppression_resourcemanager_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Application Aggregates parameter.", "display_name": "Suppress Parameter Validation: YARN Application Aggregates", "name": "service_config_suppression_yarn_application_aggregates", "value": "false" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use.", "display_name": "Fair Scheduler Configuration Rules (Deployed)", "name": "yarn_fs_schedule_rules", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the JobHistory Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: JobHistory Server Count Validator", "name": "service_config_suppression_jobhistory_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_yarn_core_site_safety_valve", "value": "false" }, { "desc": "The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory.", "display_name": "System User's Home Directory", "name": "hdfs_user_home_dir", "value": "/var/lib/hadoop-yarn" }, { "desc": "Whether YARN creates a cgroup per container, thereby isolating the CPU usage of containers. When set, yarn.nodemanager.linux-container-executor.resources-handler.class is configured to org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler. The host (in Cloudera Manager) must have cgroups enabled. The number of shares allocated to all YARN containers is configured by adjusting the CPU shares value of the Node Manager in the Resource Management configuration group.", "display_name": "Use CGroups for Resource Management", "name": "yarn_service_cgroups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the UNIX User for Nonsecure Mode with Linux Container Executor parameter.", "display_name": "Suppress Parameter Validation: UNIX User for Nonsecure Mode with Linux Container Executor", "name": "service_config_suppression_yarn_nodemanager_linux_container_executor_nonsecure_mode_local_user", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. Typically edited using the Pools configuration UI.", "display_name": "Fair Scheduler Allocations (Staged)", "name": "yarn_fs_scheduled_allocations_draft", "value": null }, { "desc": "Name of the ZooKeeper service that this YARN service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "For advanced use only, a string to be inserted into ssl-client.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "yarn_ssl_client_safety_valve", "value": null }, { "desc": "When computing the overall YARN service health, whether to consider the active ResourceManager's health.", "display_name": "Active ResourceManager Role Health Check", "name": "yarn_resourcemanagers_health_enabled", "value": "true" }, { "desc": "Maximum number of hours in the past for which container usage aggregation is performed by Cloudera Service Monitor.", "display_name": "Maximum Hours to Aggregate Usage Metrics", "name": "cm_yarn_container_usage_job_go_back_window_hours", "value": "6" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "yarn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RM-HA Cluster ID parameter.", "display_name": "Suppress Parameter Validation: RM-HA Cluster ID", "name": "service_config_suppression_yarn_rm_ha_cluster_id", "value": "false" }, { "desc": "The group that this service's processes should run as. (Except the Job History Server, which has its own group.)", "display_name": "System Group", "name": "process_groupname", "value": "hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin ACL parameter.", "display_name": "Suppress Parameter Validation: Admin ACL", "name": "service_config_suppression_yarn_admin_acl", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Controls which applications an admin user can see in the applications list view", "display_name": "Admin Users Applications List Visibility Settings", "name": "admin_application_list_settings", "value": "ALL" }, { "desc": "DFS directory where the aggregated container usage metrics are stored by Cloudera Service Monitor. This directory is created by Cloudera Service Monitor before running the usage aggregation MapReduce job for the first time.", "display_name": "Container Usage Output Directory", "name": "cm_yarn_container_usage_output_dir", "value": "/tmp/cmYarnContainerMetricsAggregate" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Comma-delimited list of groups that you want to allow the Llama (AM for Impala) user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Llama Proxy User Groups", "name": "llama_proxy_user_groups_list", "value": "*" }, { "desc": "For advanced use only, a string to be inserted into mapred-site.xml. Applies to all HDFS Replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "mapreduce_service_replication_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the NodeManager Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Health", "name": "service_health_suppression_yarn_node_managers_healthy", "value": "false" }, { "desc": "Controls the aggregate metrics generated for YARN applications. The structure is a JSON list of the attributes to aggregate and the entities to aggregate to. For example, if the attributeName is 'maps_completed' and the aggregationTargets is ['USER'] then the Service Monitor will create the metric 'yarn_application_maps_completed_rate' and, every ten minutes, will record the total maps completed for each user across all their YARN applications. By default it will also record the number of applications submitted ('apps_submitted_rate') for both users and pool. For a full list of the supported attributes see the YARN search page. Note that the valid aggregation targets are USER, YARN_POOL, and YARN (the service), and that these aggregate metrics can be viewed on both the reports and charts search pages.", "display_name": "YARN Application Aggregates", "name": "yarn_application_aggregates", "value": "[\n {\n \"attributeName\": \"maps_total\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"reduces_total\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"mb_millis_maps\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"mb_millis_reduces\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"vcores_millis_maps\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"vcores_millis_reduces\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"file_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"file_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cm_cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"application_duration\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL_USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n }\n]\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Client Truststore File Location parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Client Truststore File Location", "name": "service_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "service_config_suppression_yarn_service_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "This is the maximum time to wait for running containers and applications to complete before transition a DECOMMISSIONING node into DECOMMISSIONED. (-1 indicates infinite timeout, 0 indicates non-graceful)", "display_name": "Node Manager Graceful Decommission Timeout", "name": "yarn_resourcemanager_nodemanager_graceful_decommission_timeout_secs", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. Typically edited using the Rules configuration UI.", "display_name": "Fair Scheduler Configuration Rules (Staged)", "name": "yarn_fs_schedule_rules_draft", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hive-hiveserver2.json0000666000175100017510000013537713245514472025614 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Downloaded Resources Directory.", "display_name": "Hive Downloaded Resources Directory Free Space Monitoring Absolute Thresholds", "name": "hiveserver2_downloaded_resources_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Local Directory where Hive stores jars and data when performing a MapJoin optimization. If not specified, Hive uses a default location.", "display_name": "Hive Local Scratch Directory", "name": "hiveserver2_exec_local_scratchdir", "value": null }, { "desc": "Local directory where Hive stores jars downloaded for remote file systems (HDFS). If not specified, Hive uses a default location.", "display_name": "Hive Downloaded Resources Directory", "name": "hiveserver2_downloaded_resources_dir", "value": null }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "hiveserver2_pause_duration_window", "value": "5" }, { "desc": "Minimum number of worker threads in HiveServer2's thread pool", "display_name": "Min HiveServer2 Threads", "name": "hiveserver2_min_threads", "value": "5" }, { "desc": "When enabled, Spark will add and remove executors dynamically to Hive jobs. This is done based on the workload.", "display_name": "Enable Dynamic Executor Allocation", "name": "hiveserver2_spark_dynamic_allocation_enabled", "value": "true" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The check interval for session/operation timeout, in milliseconds, which can be disabled by setting to zero or a negative value. ", "display_name": "Session Check Interval", "name": "hiveserver2_session_check_interval", "value": "900000" }, { "desc": "The maximum percentage of heap to be used for hash in ReduceSink operator for Top-K selection. 0 means the optimization is disabled. Accepted values are between 0 and 1.", "display_name": "Maximum ReduceSink Top-K Memory Usage", "name": "hiveserver2_limit_pushdown_memory_usage", "value": "0.1" }, { "desc": "Whether to try sorted merge bucket (SMB) join.", "display_name": "Hive Optimize Sorted Merge Bucket Join", "name": "hiveserver2_optimize_bucketmapjoin_sortedmerge", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Number of Spark executors assigned to each application. This should not be set when Dynamic Executor Allocation is enabled.", "display_name": "Spark Executors Per Application", "name": "hiveserver2_spark_executor_instances", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_hs2_config_safety_valve", "value": "false" }, { "desc": "Some select queries can be converted to a single FETCH task instead of a MapReduce task, minimizing latency. A value of none disables all conversion, minimal converts simple queries such as SELECT * and filter on partition columns, and more converts SELECT queries including FILTERS.", "display_name": "Fetch Task Query Conversion", "name": "hiveserver2_fetch_task_conversion", "value": "minimal" }, { "desc": "Portion of total memory used in map-side partial aggregation. When exceeded, the partially aggregated results will be flushed from the map task to the reducers.", "display_name": "Ratio of Memory Usage for Map-Side Aggregation", "name": "hiveserver2_map_aggr_hash_memory_ratio", "value": "0.5" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hiveserver2_host_health", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When enabled, HiveServer2 will temporarily save logs associated with ongoing operations. This enables clients like beeline and Hue to request and display logs for a particular ongoing operation. Logs are removed upon completion of operation.", "display_name": "Enable HiveServer2 Operations Logging", "name": "hive_server2_logging_operation_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Upper bound on the number of executors used by the application at any given time. This is used by dynamic executor allocation.", "display_name": "Upper Bound on Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_max_executors", "value": "2147483647" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HiveServer2 WebUI is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HiveServer2 WebUI TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "Whether column stats for a table are fetched during explain.", "display_name": "Load Column Statistics", "name": "hiveserver2_stats_fetch_column_stats", "value": "true" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HiveServer2 in Bytes", "name": "hiveserver2_java_heapsize", "value": "4294967296" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When the number of ReduceSink operators after merging is less than this number, the ReduceDeDuplication optimization will be disabled.", "display_name": "Mininum Reducers for ReduceDeDuplication Optimization", "name": "hiveserver2_optimize_reducededuplication_min_reducer", "value": "4" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hiveserver2_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The desired file size after merging. This should be larger than hive.merge.smallfiles.avgsize.", "display_name": "Desired File Size After Merging", "name": "hiveserver2_merge_size_per_task", "value": "268435456" }, { "desc": "Whether to use the indexing optimization for all queries.", "display_name": "Enable Automatic Use of Indexes", "name": "hiveserver2_optimize_index_filter", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hiveserver2_log_directory_free_space", "value": "false" }, { "desc": "HiveServer2 will impersonate the beeline client user when talking to other services such as MapReduce and HDFS.", "display_name": "HiveServer2 Enable Impersonation", "name": "hiveserver2_enable_impersonation", "value": "true" }, { "desc": "Enable optimization that checks if a query can be answered using statistics. If so, answers the query using only statistics stored in metastore.", "display_name": "Enable Stats Optimization", "name": "hiveserver2_compute_query_using_stats", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hive_metrics_sample_file_location", "value": "false" }, { "desc": "Enable optimization that converts common join into MapJoin based on input file size.", "display_name": "Enable MapJoin Optimization", "name": "hiveserver2_enable_mapjoin", "value": "true" }, { "desc": "An XML string that will be inserted verbatim into the Fair Scheduler allocations file. This configuration only has effect in CDH 5.8 or later.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "hiveserver2_fair_scheduler_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hiveserver2_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Local Scratch Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Hive Local Scratch Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Hive Local Scratch Directory Free Space Monitoring Percentage Thresholds", "name": "hiveserver2_exec_local_scratch_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Local Scratch Directory.", "display_name": "Hive Local Scratch Directory Free Space Monitoring Absolute Thresholds", "name": "hiveserver2_exec_local_scratch_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_hs2_config_safety_valve", "value": null }, { "desc": "Ratio between 0.0 and 1.0 of entries in the vectorized group-by aggregation hash that is flushed when the memory threshold is exceeded.", "display_name": "Vectorized GroupBy Flush Ratio", "name": "hiveserver2_vectorized_groupby_flush_ratio", "value": "0.1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 WebUI TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 WebUI TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HiveServer2 parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HiveServer2", "name": "role_config_suppression_hiveserver2_java_opts", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hiveserver2_heap_dump_directory_free_space", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "When computing the overall HiveServer2 health, consider the host's health.", "display_name": "HiveServer2 Host Health Test", "name": "hiveserver2_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hiveserver2_scm_health", "value": "false" }, { "desc": "In vectorized group-by, the number of row entries added to the hash table before re-checking average variable size for memory usage estimation.", "display_name": "Vectorized GroupBy Check Interval", "name": "hiveserver2_vectorized_groupby_checkinterval", "value": "4096" }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hive/metrics-hivemetastore/metrics.log. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Location", "name": "hive_metrics_sample_file_location", "value": null }, { "desc": "Initial number of executors used by the application at any given time. This is required if the dynamic executor allocation feature is enabled.", "display_name": "Initial Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_initial_executors", "value": "1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Merge small files at the end of a map-only job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Map-Only Job", "name": "hiveserver2_merge_mapfiles", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 WebUI TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 WebUI TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Lower bound on the number of executors used by the application at any given time. This is used by dynamic executor allocation.", "display_name": "Lower Bound on Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_min_executors", "value": "1" }, { "desc": "The number of rows with the same key value to be cached in memory per SMB-joined table.", "display_name": "Hive SMB Join Cache Rows", "name": "hiveserver2_smbjoin_cache_rows", "value": "10000" }, { "desc": "Maximum number of worker threads in HiveServer2's thread pool", "display_name": "Max HiveServer2 Threads", "name": "hiveserver2_max_threads", "value": "100" }, { "desc": "The minimum log level for HiveServer2 logs", "display_name": "HiveServer2 Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "This is the amount of extra off-heap memory that can be requested from YARN, per executor process. This, together with spark.executor.memory, is the total memory that YARN can use to create JVM for an executor process.", "display_name": "Spark Executor Memory Overhead", "name": "hiveserver2_spark_yarn_executor_memory_overhead", "value": "26" }, { "desc": "Enable optimization that vectorizes query execution by streamlining operations by processing a block of 1024 rows at a time.", "display_name": "Enable Vectorization Optimization", "name": "hiveserver2_vectorized_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_hs2_env_safety_valve", "value": "false" }, { "desc": "Number of cores per Spark executor.", "display_name": "Spark Executor Cores", "name": "hiveserver2_spark_executor_cores", "value": "1" }, { "desc": "The default execution engine for running hive queries. Can be set to mr for MapReduce or spark for Spark.", "display_name": "Default Execution Engine", "name": "hs2_execution_engine", "value": "mr" }, { "desc": "Address of the load balancer used for HiveServer2 roles, specified in host:port format. If port is not specified, the port used by HiveServer2 is used. Note: Changing this property regenerates Kerberos keytabs for all HiveServer2 roles.", "display_name": "HiveServer2 Load Balancer", "name": "hiverserver2_load_balancer", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Port on which HiveServer2 will listen for connections.", "display_name": "HiveServer2 Port", "name": "hs2_thrift_address_port", "value": "10000" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HiveServer2", "name": "hiveserver2_java_opts", "value": "-XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The server address of HiverServer2 host to be used for communication between Hive client and remote Spark driver. Default is empty, which means the address will be determined in the same way as for hive.server2.thrift.bind.host. This is only necessary if the host has mutiple network addresses and if a different network address other than hive.server2.thrift.bind.host is to be used.", "display_name": "Spark RPC Server Address", "name": "hive_spark_client_rpc_server_address", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HiveServer2 Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_hs2_env_safety_valve", "value": null }, { "desc": "Maximum size of each Spark executor's Java heap memory when Hive is running on Spark.", "display_name": "Spark Executor Maximum Java Heap Size", "name": "hiveserver2_spark_executor_memory", "value": "268435456" }, { "desc": "Maximum size of each Spark driver's Java heap memory when Hive is running on Spark.", "display_name": "Spark Driver Maximum Java Heap Size", "name": "hiveserver2_spark_driver_memory", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Encrypt communication between clients and HiveServer2 WebUI using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HiveServer2 WebUI", "name": "ssl_enabled", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Remove extra map-reduce jobs if the data is already clustered by the same key, eliminating the need to repartition the dataset again.", "display_name": "Enable ReduceDeDuplication Optimization", "name": "hiveserver2_optimize_reducededuplication", "value": "true" }, { "desc": "Merge small files at the end of a Spark job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Spark Job", "name": "hiveserver2_merge_sparkfiles", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Operations Log Directory parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Operations Log Directory", "name": "role_config_suppression_hive_server2_logging_operation_log_location", "value": "false" }, { "desc": "Whether to suppress the results of the Hive Local Scratch Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Local Scratch Directory Free Space", "name": "role_health_suppression_hiveserver2_exec_local_scratch_directory_free_space", "value": "false" }, { "desc": "Whether to vectorize the reduce side of query execution.", "display_name": "Enable Reduce-Side Vectorization", "name": "hiveserver2_vectorized_reduce_enabled", "value": "false" }, { "desc": "Merge small files at the end of a map-reduce job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Map-Reduce Job", "name": "hiveserver2_merge_mapredfiles", "value": "false" }, { "desc": "Session will be considered to be idle only if there is no activity, and there is no pending operation. This setting takes effect only if session idle timeout (hive.server2.idle.session.timeout) and checking (hive.server2.session.check.interval) are enabled.", "display_name": "Exclude Live Operations From Session Idle Time", "name": "hiveserver2_idle_session_timeout_check_operation", "value": "true" }, { "desc": "The frequency at which the metrics are logged to the sample file. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Logging Frequency", "name": "hive_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "The port the HiveServer2 WebUI will listen on. This can be set to 0 to disable the WebUI.", "display_name": "HiveServer2 WebUI Port", "name": "hiveserver2_webui_port", "value": "10002" }, { "desc": "Enables the health test that the HiveServer2's process state is consistent with the role configuration", "display_name": "HiveServer2 Process Health Test", "name": "hiveserver2_scm_health_enabled", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Directory where HiveServer2 will place its log files.", "display_name": "HiveServer2 Log Directory", "name": "hive_log_dir", "value": "/var/log/hive" }, { "desc": "If enabled, the HiveServer2 binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind HiveServer2 to Wildcard Address", "name": "hiveserver2_webui_bind_wildcard", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hiveserver2_fair_scheduler_safety_valve", "value": "false" }, { "desc": "Operation will be closed when not accessed for this duration of time, in milliseconds; disable by setting to zero. For a positive value, checked for operations in terminal state only (FINISHED, CANCELED, CLOSED, ERROR). For a negative value, checked for all of the operations regardless of state.", "display_name": "Idle Operation Timeout", "name": "hiveserver2_idle_operation_timeout", "value": "21600000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Downloaded Resources Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Downloaded Resources Directory", "name": "role_config_suppression_hiveserver2_downloaded_resources_dir", "value": "false" }, { "desc": "When dynamic partition is enabled, reducers keep only one record writer at all times, which lowers the memory pressure on reducers.", "display_name": "Enable Sorted Dynamic Partition Optimizer", "name": "hiveserver2_optimize_sort_dynamic_partition", "value": "false" }, { "desc": "Controls whether the Hive metrics subsystem is enabled for the role.", "display_name": "Enable Metrics Subsystem", "name": "hive_metrics_enabled", "value": "true" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "hiveserver2_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hiveserver2_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark RPC Server Address parameter.", "display_name": "Suppress Parameter Validation: Spark RPC Server Address", "name": "role_config_suppression_hive_spark_client_rpc_server_address", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Executor Cores parameter.", "display_name": "Suppress Parameter Validation: Spark Executor Cores", "name": "role_config_suppression_hiveserver2_spark_executor_cores", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HiveServer2 logs. Typically used by log4j or logback.", "display_name": "HiveServer2 Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "If Hive auto convert join is on, and the sum of the size for n-1 of the tables/partitions for a n-way join is smaller than the specified size, the join is directly converted to a MapJoin (there is no conditional task).", "display_name": "Hive Auto Convert Join Noconditional Size", "name": "hiveserver2_auto_convert_join_noconditionaltask_size", "value": "20971520" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_hiveserver2_pause_duration", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Log Directory parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Log Directory", "name": "role_config_suppression_hive_log_dir", "value": "false" }, { "desc": "When enabled, HiveServer2 logs EXPLAIN EXTENDED output for every query at INFO log4j level.", "display_name": "Enable Explain Logging", "name": "hiveserver2_enable_explain_output", "value": "false" }, { "desc": "Whether to suppress the results of the Hive Downloaded Resources Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Downloaded Resources Directory Free Space", "name": "role_health_suppression_hiveserver2_downloaded_resources_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Directory in HDFS where Hive writes intermediate data between MapReduce jobs. If not specified, Hive uses a default location.", "display_name": "Hive HDFS Scratch Directory", "name": "hiveserver2_exec_scratchdir", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Load Balancer parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Load Balancer", "name": "role_config_suppression_hiverserver2_load_balancer", "value": "false" }, { "desc": "Session will be closed when not accessed for this duration of time, in milliseconds; disable by setting to zero or a negative value.", "display_name": "Idle Session Timeout", "name": "hiveserver2_idle_session_timeout", "value": "43200000" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The maximum size, in megabytes, per log file for HiveServer2 logs. Typically used by log4j or logback.", "display_name": "HiveServer2 Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Enabled the Calcite-based Cost-Based Optimizer for HiveServer2.", "display_name": "Enable Cost-Based Optimizer for Hive", "name": "hiveserver2_enable_cbo", "value": "false" }, { "desc": "The password for the HiveServer2 WebUI JKS keystore file.", "display_name": "HiveServer2 WebUI TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Enable map-side partial aggregation, which cause the mapper to generate fewer rows. This reduces the data to be sorted and distributed to reducers.", "display_name": "Enable Map-Side Aggregation", "name": "hiveserver2_map_aggr", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The max threads for the HiveServer2 WebUI.", "display_name": "HiveServer2 WebUI Max Threads", "name": "hiveserver2_webui_max_threads", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Local Scratch Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Local Scratch Directory", "name": "role_config_suppression_hiveserver2_exec_local_scratchdir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive HDFS Scratch Directory parameter.", "display_name": "Suppress Parameter Validation: Hive HDFS Scratch Directory", "name": "role_config_suppression_hiveserver2_exec_scratchdir", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Downloaded Resources Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Hive Downloaded Resources Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Hive Downloaded Resources Directory Free Space Monitoring Percentage Thresholds", "name": "hiveserver2_downloaded_resources_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When the average output file size of a job is less than the value of this property, Hive will start an additional map-only job to merge the output files into bigger files. This is only done for map-only jobs if hive.merge.mapfiles is true, for map-reduce jobs if hive.merge.mapredfiles is true, and for Spark jobs if hive.merge.sparkfiles is true.", "display_name": "Small File Average Size Merge Threshold", "name": "hiveserver2_merge_smallfiles_avgsize", "value": "16777216" }, { "desc": "Above this size, queries are converted to fetch tasks.", "display_name": "Fetch Task Query Conversion Threshold", "name": "hiveserver2_fetch_task_conversion_threshold", "value": "268435456" }, { "desc": "This is the amount of extra off-heap memory that can be requested from YARN, per driver. This, together with spark.driver.memory, is the total memory that YARN can use to create JVM for a driver process.", "display_name": "Spark Driver Memory Overhead", "name": "hiveserver2_spark_yarn_driver_memory_overhead", "value": "26" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hiveserver2_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Top level directory where operation logs are temporarily stored if Enable HiveServer2 Operations Logging is true. Logs are stored in session and operation level sub-directories under this location and are removed upon completion of operation.", "display_name": "HiveServer2 Operations Log Directory", "name": "hive_server2_logging_operation_log_location", "value": "/var/log/hive/operation_logs" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hue-service.json0000666000175100017510000011317213245514472024623 0ustar zuulzuul00000000000000[ { "desc": "Create users in Hue when they try to login with their LDAP credentials. For use when using LdapBackend for Hue authentication.", "display_name": "Create LDAP users on login", "name": "create_users_on_login", "value": "true" }, { "desc": "Number of threads used by the Hue web server.", "display_name": "Hue Web Server Threads", "name": "cherrypy_server_threads", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Comma-separated list of regular expressions, which match 'host:port' of requested proxy target.", "display_name": "Whitelist", "name": "whitelist", "value": "(localhost|127\\.0\\.0\\.1):(50030|50070|50060|50075)" }, { "desc": "The health test thresholds of the overall Load Balancer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Load Balancers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Load Balancers falls below the critical threshold.", "display_name": "Healthy Load Balancer Monitoring Thresholds", "name": "hue_load_balancer_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Thrift Usage Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Thrift Usage Validator", "name": "service_config_suppression_hue_hbase_thrift_server_validator", "value": "false" }, { "desc": "The health test thresholds of the overall Kerberos Ticket Renewer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Kerberos Ticket Renewers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Kerberos Ticket Renewers falls below the critical threshold.", "display_name": "Healthy Kerberos Ticket Renewer Monitoring Thresholds", "name": "hue_kt_renewers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hue" }, { "desc": "The distinguished name to use as a search base for finding users and groups. This should be similar to 'dc=hadoop,dc=mycompany,dc=com'.", "display_name": "LDAP Search Base", "name": "base_dn", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Database Dump File parameter.", "display_name": "Suppress Parameter Validation: Database Dump File", "name": "service_config_suppression_database_dump_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Temporary Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Temporary Directory", "name": "service_config_suppression_hdfs_tmp_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Password parameter.", "display_name": "Suppress Parameter Validation: Hue Database Password", "name": "service_config_suppression_database_password", "value": "false" }, { "desc": "Whether to suppress the results of the Hue Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hue Server Health", "name": "service_health_suppression_hue_hue_servers_healthy", "value": "false" }, { "desc": "The name of a default group that users will be added to at creation time.", "display_name": "Default User Group", "name": "default_user_group", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Kerberos Ticket Renewer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kerberos Ticket Renewer Count Validator", "name": "service_config_suppression_kt_renewer_count_validator", "value": "false" }, { "desc": "Distinguished name of the user to bind as. This is used to connect to LDAP/AD for searching user and group information. This may be left blank if the LDAP server supports anonymous binds.", "display_name": "LDAP Bind User Distinguished Name", "name": "bind_dn", "value": null }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "use_start_tls", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Username Pattern parameter.", "display_name": "Suppress Parameter Validation: LDAP Username Pattern", "name": "service_config_suppression_ldap_username_pattern", "value": "false" }, { "desc": "The password of the bind user.", "display_name": "LDAP Bind Password", "name": "bind_password", "value": null }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hue/audit" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default User Group parameter.", "display_name": "Suppress Parameter Validation: Default User Group", "name": "service_config_suppression_default_user_group", "value": "false" }, { "desc": "Type of database used for Hue", "display_name": "Hue Database Type", "name": "database_type", "value": "sqlite3" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the PAM Backend Service Name parameter.", "display_name": "Suppress Parameter Validation: PAM Backend Service Name", "name": "service_config_suppression_pam_auth_service", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Web Interface Role parameter.", "display_name": "Suppress Parameter Validation: HDFS Web Interface Role", "name": "service_config_suppression_hue_webhdfs", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Bind User Distinguished Name parameter.", "display_name": "Suppress Parameter Validation: LDAP Bind User Distinguished Name", "name": "service_config_suppression_bind_dn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Username parameter.", "display_name": "Suppress Parameter Validation: Hue Database Username", "name": "service_config_suppression_database_user", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Name of host where the Hue database is running. Not necessary for SQLite3.", "display_name": "Hue Database Hostname", "name": "database_host", "value": "localhost" }, { "desc": "Name of the Sentry service that this Hue service instance depends on", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_ldap_url", "value": "false" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Name Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Name Attribute", "name": "service_config_suppression_group_name_attr", "value": "false" }, { "desc": "The location on disk of the certificate, in .pem format, used to confirm the authenticity of the LDAP server certificate. This is the Certificate Authority (CA) certificate, and it was used to sign the LDAP server certificate. If not set, all certificates are trusted, which means that an attacker could potentially intercept otherwise encrypted usernames and passwords.", "display_name": "LDAP Server CA Certificate", "name": "ldap_cert", "value": null }, { "desc": "HTTPFS role or Namenode (if webhdfs is enabled) that hue can use to communicate with HDFS.", "display_name": "HDFS Web Interface Role", "name": "hue_webhdfs", "value": null }, { "desc": "Whether to suppress the results of the Load Balancer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Load Balancer Health", "name": "service_health_suppression_hue_load_balancer_healthy", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Name of Hue database.", "display_name": "Hue Database Name", "name": "database_name", "value": "hue" }, { "desc": "The attribute of the group object that identifies the members of the group. For Active Directory, this is typically 'member'.", "display_name": "LDAP Group Membership Attribute", "name": "group_member_attr", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Blacklist parameter.", "display_name": "Suppress Parameter Validation: Blacklist", "name": "service_config_suppression_blacklist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Whitelist parameter.", "display_name": "Suppress Parameter Validation: Whitelist", "name": "service_config_suppression_whitelist", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "The base filter for searching for users. For Active Directory, this is typically '(objectClass=user)'.", "display_name": "LDAP User Filter", "name": "user_filter", "value": null }, { "desc": "Name of the Impala service that this Hue service instance depends on", "display_name": "Impala Service", "name": "impala_service", "value": null }, { "desc": "Name of the Oozie service that this Hue service instance depends on", "display_name": "Oozie Service", "name": "oozie_service", "value": null }, { "desc": "Thrift server to use for HBase app.", "display_name": "HBase Thrift Server", "name": "hue_hbase_thrift", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Membership Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Membership Attribute", "name": "service_config_suppression_group_member_attr", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Hostname parameter.", "display_name": "Suppress Parameter Validation: Hue Database Hostname", "name": "service_config_suppression_database_host", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini parameter.", "display_name": "Suppress Parameter Validation: Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", "name": "service_config_suppression_hue_service_safety_valve", "value": "false" }, { "desc": "Name of the Solr service that this Hue service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "Name of the Sqoop service that this Hue service instance depends on", "display_name": "Sqoop Service", "name": "sqoop_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Server CA Certificate parameter.", "display_name": "Suppress Parameter Validation: LDAP Server CA Certificate", "name": "service_config_suppression_ldap_cert", "value": "false" }, { "desc": "The username to use to log into the Hue database. Not necessary for SQLite3.", "display_name": "Hue Database Username", "name": "database_user", "value": "hue" }, { "desc": "In debug mode, Django displays a detailed traceback when an exception occurs. Debugging information may contain sensitive data. Django remembers every SQL query it executes in debug mode, which will rapidly consume memory.", "display_name": "Enable Django Debug Mode", "name": "django_debug_enable", "value": "false" }, { "desc": "Comma-separated list of regular expressions, which match any prefix of 'host:port/path' of requested proxy target. This does not support matching GET parameters.", "display_name": "Blacklist", "name": "blacklist", "value": "()" }, { "desc": "Search Bind Authentication connects to the LDAP server using credentials provided in the 'bind_dn' and 'bind_password' configurations. If these configurations are not set, then an anonymous search is performed.", "display_name": "Use Search Bind Authentication", "name": "search_bind_authentication", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Username Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Username Attribute", "name": "service_config_suppression_user_name_attr", "value": "false" }, { "desc": "Mode of authenticating login credentials. Select desktop.auth.backend.LdapBackend to use LDAP to authenticate login credentials. LDAP requires you to also set the LDAP URL, Active Directory Domain, and optionally LDAP certificate if you are using secure LDAP. Select desktop.auth.backend.PamBackend to use PAM to authenticate login credentials.", "display_name": "Authentication Backend", "name": "auth_backend", "value": "desktop.auth.backend.AllowFirstUserDjangoBackend" }, { "desc": "Whether to suppress configuration warnings produced by the Load Balancer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Load Balancer Count Validator", "name": "service_config_suppression_hue_load_balancer_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Filter parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Filter", "name": "service_config_suppression_group_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_nt_domain", "value": "false" }, { "desc": "Enable debug output in HTTP Internal Server Error (status 500) responses. Debugging information may contain sensitive data. If Enable Django Debug Mode is set, this is automatically enabled.", "display_name": "Enable Debugging of Internal Server Error Responses", "name": "http_500_debug_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Database Directory", "name": "service_config_suppression_database_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "hue_sentry_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hue_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the User Augmentor parameter.", "display_name": "Suppress Parameter Validation: User Augmentor", "name": "service_config_suppression_user_augmentor", "value": "false" }, { "desc": "The username attribute in the LDAP schema. For Active Directory, this is typically 'sAMAccountName'.", "display_name": "LDAP Username Attribute", "name": "user_name_attr", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Search Base parameter.", "display_name": "Suppress Parameter Validation: LDAP Search Base", "name": "service_config_suppression_base_dn", "value": "false" }, { "desc": "Password for Hue database. Not necessary for SQLite3.", "display_name": "Hue Database Password", "name": "database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS HTTPFS Usage Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS HTTPFS Usage Validator", "name": "service_config_suppression_hdfs_httpfs_present_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_hue_sentry_safety_valve", "value": "false" }, { "desc": "Name of the HBase service that this Hue service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Base filter for searching for groups. For Active Directory, this is typically '(objectClass=group)'.", "display_name": "LDAP Group Filter", "name": "group_filter", "value": null }, { "desc": "Time zone name.", "display_name": "Time Zone", "name": "time_zone", "value": "America/Los_Angeles" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Hue events:\n

\n\n
    \n
  • operation: the Hue operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Whether to suppress the results of the Kerberos Ticket Renewer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Kerberos Ticket Renewer Health", "name": "service_health_suppression_hue_kt_renewers_healthy", "value": "false" }, { "desc": "Name of the Hive service that this Hue service instance depends on", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress the results of the Beeswax Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Beeswax Server Health", "name": "service_health_suppression_hue_beeswax_server_health", "value": "false" }, { "desc": "The health test thresholds of the overall Hue Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Hue Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Hue Servers falls below the critical threshold.", "display_name": "Healthy Hue Server Monitoring Thresholds", "name": "hue_hue_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The group name attribute in the LDAP schema. For Active Directory, this is typically 'cn'.", "display_name": "LDAP Group Name Attribute", "name": "group_name_attr", "value": null }, { "desc": "HDFS directory used for storing temporary files.", "display_name": "HDFS Temporary Directory", "name": "hdfs_tmp_dir", "value": "/tmp" }, { "desc": "Default encoding for site data.", "display_name": "Default Site Encoding", "name": "default_site_encoding", "value": "utf" }, { "desc": "If the database is SQLite3, this is the filename of the database to use, and the directory of this file must be writable by the 'hue' user.", "display_name": "Hue Database Directory", "name": "database_dir", "value": "/var/lib/hue/desktop.db" }, { "desc": "When you enable anonymous usage data collection Hue tracks anonymised pages and application versions in order to gather information about each application's usage levels. The data collected does not include any hostnames or IDs. Data collection option is available on CDH 4.4 and later deployments.", "display_name": "Enable Usage Data Collection", "name": "usage_data_collection_enable", "value": "true" }, { "desc": "Name of the ZooKeeper service that this Hue service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "ldap_url", "value": null }, { "desc": "The PAM service name to use when authenticating over desktop.auth.backend.PamBackend. This is typically the name of a file under /etc/pam.d/ on the Hue host.", "display_name": "PAM Backend Service Name", "name": "pam_auth_service", "value": "login" }, { "desc": "Whether to suppress configuration warnings produced by the Hue Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hue Server Count Validator", "name": "service_config_suppression_hue_server_count_validator", "value": "false" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "nt_domain", "value": null }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hue" }, { "desc": "Class that defines extra accessor methods for user objects.", "display_name": "User Augmentor", "name": "user_augmentor", "value": "desktop.auth.backend.DefaultUserAugmentor" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hue Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hue_service_env_safety_valve", "value": "false" }, { "desc": "File where the database gets dumped to or loaded from.", "display_name": "Database Dump File", "name": "database_dump_file", "value": "/tmp/hue_database_dump.json" }, { "desc": "LDAP Username Pattern for use with non-Active Directory LDAP implementations. Must contain the special '<username>' string for replacement during authentication.", "display_name": "LDAP Username Pattern", "name": "ldap_username_pattern", "value": null }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hue" }, { "desc": "When computing the overall HUE health, consider Beeswax Server's health", "display_name": "Beeswax Server Role Health Test", "name": "hue_beeswax_server_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP User Filter parameter.", "display_name": "Suppress Parameter Validation: LDAP User Filter", "name": "service_config_suppression_user_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Bind Password parameter.", "display_name": "Suppress Parameter Validation: LDAP Bind Password", "name": "service_config_suppression_bind_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Time Zone parameter.", "display_name": "Suppress Parameter Validation: Time Zone", "name": "service_config_suppression_time_zone", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Name parameter.", "display_name": "Suppress Parameter Validation: Hue Database Name", "name": "service_config_suppression_database_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Site Encoding parameter.", "display_name": "Suppress Parameter Validation: Default Site Encoding", "name": "service_config_suppression_default_site_encoding", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hue_safety_valve.ini. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", "name": "hue_service_safety_valve", "value": null }, { "desc": "Port on host where the Hue database is running. Not necessary for SQLite3.", "display_name": "Hue Database Port", "name": "database_port", "value": "3306" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/spark-spark_yarn_history_server.json0000666000175100017510000004524613245514472031050 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "History Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SPARK_YARN_HISTORY_SERVER_role_env_safety_valve", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The port of the history server WebUI", "display_name": "History Server WebUI Port", "name": "history_server_web_port", "value": "18088" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "How often to poll HDFS for new applications.", "display_name": "HDFS Polling Interval", "name": "history_server_fs_poll_interval", "value": "10" }, { "desc": "The maximum size, in megabytes, per log file for History Server logs. Typically used by log4j or logback.", "display_name": "History Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: History Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "How often the History Server will clean up event log files.", "display_name": "Event Log Cleaner Interval", "name": "event_log_cleaner_interval", "value": "86400" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "For advanced use only. A string to be inserted into spark-conf/spark-history-server.conf for this role only.", "display_name": "History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-history-server.conf", "name": "spark-conf/spark-history-server.conf_role_safety_valve", "value": null }, { "desc": "Max number of application UIs to keep in the History Server's memory. All applications will still be available, but may take longer to load if they're not in memory.", "display_name": "Retained App Count", "name": "history_server_retained_apps", "value": "50" }, { "desc": "When computing the overall History Server health, consider the host's health.", "display_name": "History Server Host Health Test", "name": "spark_yarn_history_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The log directory for log files of the role History Server.", "display_name": "History Server Log Directory", "name": "log_dir", "value": "/var/log/spark" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-history-server.conf parameter.", "display_name": "Suppress Parameter Validation: History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-history-server.conf", "name": "role_config_suppression_spark-conf/spark-history-server.conf_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "spark_yarn_history_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "History Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes.", "display_name": "Java Heap Size of History Server in Bytes", "name": "history_server_max_heapsize", "value": "536870912" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_host_health", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_unexpected_exits", "value": "false" }, { "desc": "Specifies the maximum age of the event logs.", "display_name": "Maximum Event Log Age", "name": "event_log_cleaner_max_age", "value": "604800" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: History Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "role_config_suppression_spark-conf/spark-env.sh_role_safety_valve", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The maximum number of rolled log files to keep for History Server logs. Typically used by log4j or logback.", "display_name": "History Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Enables the health test that the History Server's process state is consistent with the role configuration", "display_name": "History Server Process Health Test", "name": "spark_yarn_history_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_swap_memory_usage", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: History Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_spark_yarn_history_server_role_env_safety_valve", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "For advanced use only. A string to be inserted into spark-conf/spark-env.sh for this role only.", "display_name": "History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_role_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for History Server logs", "display_name": "History Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Specifies whether the History Server should periodically clean up event logs from storage.", "display_name": "Enable Event Log Cleaner", "name": "event_log_cleaner_enabled", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/yarn-nodemanager.json0000666000175100017510000013666413245514472025646 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Containers Environment Variable parameter.", "display_name": "Suppress Parameter Validation: Containers Environment Variable", "name": "role_config_suppression_yarn_nodemanager_admin_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NodeManager parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NodeManager", "name": "role_config_suppression_node_manager_java_opts", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed System Users parameter.", "display_name": "Suppress Parameter Validation: Allowed System Users", "name": "role_config_suppression_container_executor_allowed_system_users", "value": "false" }, { "desc": "When computing the overall NodeManager health, consider the host's health.", "display_name": "NodeManager Host Health Test", "name": "nodemanager_host_health_enabled", "value": "true" }, { "desc": "Enables the health test that the NodeManager's process state is consistent with the role configuration", "display_name": "NodeManager Process Health Test", "name": "nodemanager_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the NodeManager Health Checker heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Health Checker", "name": "role_health_suppression_node_manager_health_checker", "value": "false" }, { "desc": "The minimum Linux user ID allowed. Used to prevent other super users.", "display_name": "Minimum User ID", "name": "container_executor_min_user_id", "value": "1000" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_node_manager_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Local Directories parameter.", "display_name": "Suppress Parameter Validation: NodeManager Local Directories", "name": "role_config_suppression_yarn_nodemanager_local_dirs", "value": "false" }, { "desc": "Whether to suppress the results of the ResourceManager Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ResourceManager Connectivity", "name": "role_health_suppression_node_manager_connectivity", "value": "false" }, { "desc": "The minimum space that must be available on a disk for it to be used. This applies to local dirs and log dirs.", "display_name": "Disk Health Checker Minimum Health Disk Space", "name": "yarn_nodemanager_disk_health_checker_min_free_space_per_disk_mb", "value": "0" }, { "desc": "Number of threads to use for localization fetching.", "display_name": "Localizer Fetch Thread Count", "name": "yarn_nodemanager_localizer_fetch_thread_count", "value": "4" }, { "desc": "Frequency, in milliseconds, of running disk health checker.", "display_name": "Disk Health Checker Frequency", "name": "yarn_nodemanager_disk_health_checker_interval_ms", "value": "120000" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "nodemanager_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Recovery Directory.", "display_name": "NodeManager Recovery Directory Free Space Monitoring Absolute Thresholds", "name": "nodemanager_recovery_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_node_manager_log_directory_free_space", "value": "false" }, { "desc": "Number of threads container manager uses.", "display_name": "Container Manager Thread Count", "name": "yarn_nodemanager_container_manager_thread_count", "value": "20" }, { "desc": "Number of virtual CPU cores that can be allocated for containers. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores", "name": "yarn_nodemanager_resource_cpu_vcores", "value": "8" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Address where the localizer IPC is.", "display_name": "Localizer Cache Cleanup Interval", "name": "yarn_nodemanager_localizer_cache_cleanup_interval_ms", "value": "600000" }, { "desc": "List of users banned from running containers.", "display_name": "Banned System Users", "name": "container_executor_banned_users", "value": "hdfs,yarn,mapred,bin" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Container Log Directories parameter.", "display_name": "Suppress Parameter Validation: NodeManager Container Log Directories", "name": "role_config_suppression_yarn_nodemanager_log_dirs", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "nodemanager_mapred_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Local Directories.", "display_name": "NodeManager Local Directories Free Space Monitoring Absolute Thresholds", "name": "nodemanager_local_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Banned System Users parameter.", "display_name": "Suppress Parameter Validation: Banned System Users", "name": "role_config_suppression_container_executor_banned_users", "value": "false" }, { "desc": "Amount of physical memory, in MiB, that can be allocated for containers.", "display_name": "Container Memory", "name": "yarn_nodemanager_resource_memory_mb", "value": "8192" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NodeManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_nodemanager_role_env_safety_valve", "value": "false" }, { "desc": "Environment variables that containers may override rather than use NodeManager's default.", "display_name": "Containers Environment Variables Whitelist ", "name": "yarn_nodemanager_env_whitelist", "value": "JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME" }, { "desc": "Comma-separated list of arguments which are to be passed to node health script when it is being launched.", "display_name": "Healthchecker Script Arguments", "name": "mapred_healthchecker_script_args", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_node_manager_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_node_manager_unexpected_exits", "value": "false" }, { "desc": "If enabled, adds 'org.apache.hadoop.mapred.ShuffleHandler' to the NodeManager auxiliary services. This is required for MapReduce applications.", "display_name": "Enable Shuffle Auxiliary Service", "name": "mapreduce_aux_service", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Container Log Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Container Log Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Container Log Directories Free Space Monitoring Percentage Thresholds", "name": "nodemanager_log_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Log Directory parameter.", "display_name": "Suppress Parameter Validation: NodeManager Log Directory", "name": "role_config_suppression_node_manager_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The maximum size, in megabytes, per log file for NodeManager logs. Typically used by log4j or logback.", "display_name": "NodeManager Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "List of users explicitly whitelisted to be allowed to run containers. Users with IDs lower than the \"Minimum User Id\" setting may be whitelisted by using this setting.", "display_name": "Allowed System Users", "name": "container_executor_allowed_system_users", "value": "nobody,impala,hive,llama,hbase" }, { "desc": "Directory where NodeManager will place its log files.", "display_name": "NodeManager Log Directory", "name": "node_manager_log_dir", "value": "/var/log/hadoop-yarn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Recovery Directory parameter.", "display_name": "Suppress Parameter Validation: NodeManager Recovery Directory", "name": "role_config_suppression_yarn_nodemanager_recovery_dir", "value": "false" }, { "desc": "Enables the health check that verifies the NodeManager is seen as healthy by the ResourceManager.", "display_name": "NodeManager Health Checker Health Check", "name": "nodemanager_health_checker_health_enabled", "value": "true" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "nodemanager_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The address of the NodeManager IPC.", "display_name": "NodeManager IPC Address", "name": "yarn_nodemanager_address", "value": "8041" }, { "desc": "Environment variables that should be forwarded from the NodeManager's environment to the container's.", "display_name": "Containers Environment Variable", "name": "yarn_nodemanager_admin_env", "value": "MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the CGroups Hierarchy parameter.", "display_name": "Suppress Parameter Validation: CGroups Hierarchy", "name": "role_config_suppression_linux_container_executor_cgroups_hierarchy", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_node_manager_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Remote App Log Directory Suffix parameter.", "display_name": "Suppress Parameter Validation: Remote App Log Directory Suffix", "name": "role_config_suppression_yarn_nodemanager_remote_app_log_dir_suffix", "value": "false" }, { "desc": "Number of threads used in cleanup.", "display_name": "Cleanup Thread Count", "name": "yarn_nodemanager_delete_thread_count", "value": "4" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NodeManager", "name": "node_manager_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Target size of localizer cache in MB, per local directory.", "display_name": "Localizer Cache Target Size", "name": "yarn_nodemanager_localizer_cache_target_size_mb", "value": "10240" }, { "desc": "Absolute path to the script which is periodically run by the node health monitoring service to determine if the node is healthy or not. If the value of this key is empty or the file does not exist in the location configured here, the node health monitoring service is not started.", "display_name": "Healthchecker Script Path", "name": "mapred_healthchecker_script_path", "value": "" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Recovery Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Recovery Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Recovery Directory Free Space Monitoring Percentage Thresholds", "name": "nodemanager_recovery_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_nodemanager_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "nodemanager_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Healthchecker Script Path parameter.", "display_name": "Suppress Parameter Validation: Healthchecker Script Path", "name": "role_config_suppression_mapred_healthchecker_script_path", "value": "false" }, { "desc": "The HTTPS port of the NodeManager web application.", "display_name": "NodeManager Web Application HTTPS Port (TLS/SSL)", "name": "nodemanager_webserver_https_port", "value": "8044" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "nodemanager_web_metric_collection_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "nodemanager_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_node_manager_heap_dump_directory_free_space", "value": "false" }, { "desc": "The local filesystem directory in which the NodeManager stores state when recovery is enabled. Recovery is enabled by default.", "display_name": "NodeManager Recovery Directory", "name": "yarn_nodemanager_recovery_dir", "value": "/var/lib/hadoop-yarn/yarn-nm-recovery" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NodeManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_node_manager_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the NodeManager Local Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Local Directories Free Space", "name": "role_health_suppression_nodemanager_local_data_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "HDFS directory where application logs are stored when an application completes.", "display_name": "Remote App Log Directory", "name": "yarn_nodemanager_remote_app_log_dir", "value": "/tmp/logs" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_node_manager_gc_duration", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "nodemanager_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Healthchecker Script Arguments parameter.", "display_name": "Suppress Parameter Validation: Healthchecker Script Arguments", "name": "role_config_suppression_mapred_healthchecker_script_args", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Path (rooted in the cgroups hierarchy on the machine) where to place YARN-managed cgroups.", "display_name": "CGroups Hierarchy", "name": "linux_container_executor_cgroups_hierarchy", "value": "/hadoop-yarn" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Number of threads to handle localization requests.", "display_name": "Localizer Client Thread Count", "name": "yarn_nodemanager_localizer_client_thread_count", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Local Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Local Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Local Directories Free Space Monitoring Percentage Thresholds", "name": "nodemanager_local_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Single User Mode Overrides Validator configuration validator.", "display_name": "Suppress Configuration Validator: Single User Mode Overrides Validator", "name": "role_config_suppression_single_user_mode_override_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_nodemanager_mapred_safety_valve", "value": "false" }, { "desc": "The HTTP Port of the NodeManager web application.", "display_name": "NodeManager Web Application HTTP Port", "name": "nodemanager_webserver_port", "value": "8042" }, { "desc": "Heartbeat interval to ResourceManager", "display_name": "Heartbeat Interval", "name": "yarn_nodemanager_heartbeat_interval_ms", "value": "1000" }, { "desc": "The amount of time to wait for the NodeManager to fully start up and connect to the ResourceManager before enforcing the connectivity check.", "display_name": "NodeManager Connectivity Tolerance at Startup", "name": "nodemanager_connectivity_tolerance_seconds", "value": "180" }, { "desc": "List of directories on the local filesystem where a NodeManager stores intermediate data files.", "display_name": "NodeManager Local Directories", "name": "yarn_nodemanager_local_dirs", "value": null }, { "desc": "Enables the health check that verifies the NodeManager is connected to the ResourceManager.", "display_name": "NodeManager Connectivity Health Check", "name": "nodemanager_connectivity_health_enabled", "value": "true" }, { "desc": "Number of seconds after an application finishes before the NodeManager's DeletionService will delete the application's localized file and log directory. To diagnose YARN application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories.", "display_name": "Localized Dir Deletion Delay", "name": "yarn_nodemanager_delete_debug_delay_sec", "value": "0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NodeManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The maximum number of rolled log files to keep for NodeManager logs. Typically used by log4j or logback.", "display_name": "NodeManager Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the NodeManager Container Log Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Container Log Directories Free Space", "name": "role_health_suppression_nodemanager_log_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Remote App Log Directory parameter.", "display_name": "Suppress Parameter Validation: Remote App Log Directory", "name": "role_config_suppression_yarn_nodemanager_remote_app_log_dir", "value": "false" }, { "desc": "List of directories on the local filesystem where a NodeManager stores container log files.", "display_name": "NodeManager Container Log Directories", "name": "yarn_nodemanager_log_dirs", "value": "/var/log/hadoop-yarn/container" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NodeManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "NODEMANAGER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the NodeManager Recovery Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Recovery Directory Free Space", "name": "role_health_suppression_nodemanager_recovery_directory_free_space", "value": "false" }, { "desc": "Time in seconds to retain user logs. Only applicable if log aggregation is disabled.", "display_name": "Log Retain Duration", "name": "yarn_nodemanager_log_retain_seconds", "value": "10800" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The minimum fraction of number of disks to be healthy for the NodeManager to launch new containers. This correspond to both local dirs and log dirs, i.e., if there are fewer number of healthy local dirs (or log dirs) available, then new containers will not be launched on this node.", "display_name": "Disk Health Checker Minimum Health Disks Fraction", "name": "yarn_nodemanager_disk_health_checker_min_healthy_disks", "value": "0.25" }, { "desc": "Address where the localizer IPC is.", "display_name": "Localizer Port", "name": "yarn_nodemanager_localizer_address", "value": "8040" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Container Executor Group parameter.", "display_name": "Suppress Parameter Validation: Container Executor Group", "name": "role_config_suppression_container_executor_group", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Containers Environment Variables Whitelist parameter.", "display_name": "Suppress Parameter Validation: Containers Environment Variables Whitelist ", "name": "role_config_suppression_yarn_nodemanager_env_whitelist", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The minimum log level for NodeManager logs", "display_name": "NodeManager Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}", "display_name": "Remote App Log Directory Suffix", "name": "yarn_nodemanager_remote_app_log_dir_suffix", "value": "logs" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum allowed connections for the shuffle. Set to 0 (zero) to indicate no limit on the number of connections.", "display_name": "Max Shuffle Connections", "name": "mapreduce_shuffle_max_connections", "value": "0" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the NodeManager will check for full disk. This applies to local dirs and log dirs.", "display_name": "Dish Health Checker Max Disk Utilization Percent", "name": "yarn_nodemanager_disk_health_checker_max_disk_utilization_per_disk_percentage", "value": "90.0" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_node_manager_host_health", "value": "false" }, { "desc": "The system group that owns the container-executor binary. This does not need to be changed unless the ownership of the binary is explicitly changed.", "display_name": "Container Executor Group", "name": "container_executor_group", "value": "yarn" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Container Log Directories.", "display_name": "NodeManager Container Log Directories Free Space Monitoring Absolute Thresholds", "name": "nodemanager_log_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum allowed threads for serving shuffle connections. Set to zero to indicate the default of 2 times the number of available processors.", "display_name": "Max Shuffle Threads", "name": "mapreduce_shuffle_max_threads", "value": "80" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NodeManager in Bytes", "name": "node_manager_java_heapsize", "value": "1073741824" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/sentry-gateway.json0000666000175100017510000000602213245514472025362 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/sentry" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Client Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Sentry Client Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "role_config_suppression_sentry_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for sentry-site.xml.", "display_name": "Sentry Client Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "sentry_client_config_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-balancer.json0000666000175100017510000001607213245514472025076 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Balancer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Balancer", "name": "role_config_suppression_balancer_java_opts", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "balancer_config_safety_valve", "value": null }, { "desc": "The policy that should be used to rebalance HDFS storage. The default DataNode policy balances the storage at the DataNode level. This is similar to the balancing policy from prior releases. The BlockPool policy balances the storage at the block pool level as well as at the DataNode level. The BlockPool policy is relevant only to a Federated HDFS service.", "display_name": "Rebalancing Policy", "name": "rebalancing_policy", "value": "DataNode" }, { "desc": "The percentage deviation from average utilization, after which a node will be rebalanced. (for example, '10.0' for 10%)", "display_name": "Rebalancing Threshold", "name": "rebalancer_threshold", "value": "10.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_balancer_config_safety_valve", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Balancer", "name": "balancer_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Balancer in Bytes", "name": "balancer_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/sentry-service.json0000666000175100017510000004526413245514472025374 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the Sentry Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Sentry Server Count Validator", "name": "service_config_suppression_sentry_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Password parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Password", "name": "service_config_suppression_sentry_server_database_password", "value": "false" }, { "desc": "Whether to suppress the results of the Sentry Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sentry Server Health", "name": "service_health_suppression_sentry_sentry_servers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Password for Sentry Server database.", "display_name": "Sentry Server Database Password", "name": "sentry_server_database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Sentry events:\n

\n\n
    \n
  • operation: the Sentry operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "sentry" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Sentry events:\n

\n\n
    \n
  • operation: the Sentry operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "User for Sentry Server database.", "display_name": "Sentry Server Database User", "name": "sentry_server_database_user", "value": "sentry" }, { "desc": "Type of Sentry Server database.", "display_name": "Sentry Server Database Type", "name": "sentry_server_database_type", "value": "mysql" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Host parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Host", "name": "service_config_suppression_sentry_server_database_host", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Name parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Name", "name": "service_config_suppression_sentry_server_database_name", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sentry Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "SENTRY_service_env_safety_valve", "value": null }, { "desc": "Name of the ZooKeeper service that this Sentry service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Connecting Users parameter.", "display_name": "Suppress Parameter Validation: Allowed Connecting Users", "name": "service_config_suppression_sentry_service_allow_connect", "value": "false" }, { "desc": "Host name of Sentry Server database.", "display_name": "Sentry Server Database Host", "name": "sentry_server_database_host", "value": "localhost" }, { "desc": "The health test thresholds of the overall Sentry Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Sentry Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Sentry Servers falls below the critical threshold.", "display_name": "Healthy Sentry Server Monitoring Thresholds", "name": "sentry_sentry_server_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Name of Sentry Server database.", "display_name": "Sentry Server Database Name", "name": "sentry_server_database_name", "value": "sentry" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "sentry" }, { "desc": "List of users allowed to connect to the Sentry Server. These are usually service users such as hive and impala, and the list does not usually need to include end users.", "display_name": "Allowed Connecting Users", "name": "sentry_service_allow_connect", "value": "hive,impala,hue,hdfs,solr,kafka" }, { "desc": "Enables the Sentry Web UI. This must be enabled when using Sentry in High Availability configuration.", "display_name": "Enable Web UI", "name": "sentry_service_web_enable", "value": "true" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "sentry" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "sentry_server_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_sentry_server_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/sentry/audit" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin Groups parameter.", "display_name": "Suppress Parameter Validation: Admin Groups", "name": "service_config_suppression_sentry_service_admin_group", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Port number of Sentry Server database.", "display_name": "Sentry Server Database Port", "name": "sentry_server_database_port", "value": "3306" }, { "desc": "If an end user is in one of these admin groups, that user has administrative privileges on the Sentry Server.", "display_name": "Admin Groups", "name": "sentry_service_admin_group", "value": "hive,impala,hue,solr,kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Name of the HDFS service that this Sentry service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_sentry_service_env_safety_valve", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database User parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database User", "name": "service_config_suppression_sentry_server_database_user", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/impala-catalogserver.json0000666000175100017510000007706213245514472026515 0ustar zuulzuul00000000000000[ { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Catalog Server in Bytes", "name": "catalogd_embedded_jvm_heapsize", "value": "4294967296" }, { "desc": "Directory where the Catalog Server core dump is placed.", "display_name": "Catalog Server Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/catalogd" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Catalog Server Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Catalog Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Catalog Server", "name": "role_config_suppression_catalogd_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The password for the private key in the Catalog Server Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Catalog Server Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "Number of retry attempts the Catalog Server will make when connecting to the Hive Metastore Server. By default, the Catalog Server waits one second between retries.", "display_name": "Catalog Server Hive Metastore Connection Retries", "name": "hive_metastore_connect_retries", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "Port where Catalog Server debug web server runs.", "display_name": "Catalog Server HTTP Server Port", "name": "catalogserver_webserver_port", "value": "25020" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "catalogserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Port where Catalog Server is exported.", "display_name": "Catalog Server Service Port", "name": "catalog_service_port", "value": "26000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server HBase Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server HBase Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hbase_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Hive Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Hive Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hive_conf_safety_valve", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Impala Catalog Server logs. Typically used by log4j or logback.", "display_name": "Impala Catalog Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Username for Catalog Server web server authentication.", "display_name": "Catalog Server Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Password for Catalog Server web server authentication.", "display_name": "Catalog Server Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Directory where Catalog Server will place its log files.", "display_name": "Catalog Server Log Directory", "name": "log_dir", "value": "/var/log/catalogd" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml for this role only.", "display_name": "Catalog Server HDFS Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hdfs_site_conf_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_catalogserver_unexpected_exits", "value": "false" }, { "desc": "Enable/Disable Catalog Server web server. This web server contains useful information about Catalog Server daemon.", "display_name": "Enable Catalog Server Web Server", "name": "catalogd_enable_webserver", "value": "true" }, { "desc": "When computing the overall Impala Catalog Server health, consider the host's health.", "display_name": "Impala Catalog Server Host Health Test", "name": "catalogserver_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Directory for storing Catalog Server Breakpad dumps.", "display_name": "Catalog Server Breakpad Dump Directory", "name": "minidump_path", "value": "/var/log/impala-minidumps" }, { "desc": "The minimum log level for Impala Catalog Server logs", "display_name": "Impala Catalog Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_catalogserver_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_catalogserver_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_catalogserver_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_catalogserver_host_health", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "catalogserver_web_metric_collection_enabled", "value": "true" }, { "desc": "The amount of time to wait for the Catalog Server to fully start up and connect to the StateStore before enforcing the connectivity check.", "display_name": "Catalog Server Connectivity Tolerance at Startup", "name": "catalogserver_connectivity_tolerance", "value": "180" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "Enables the health test that the Impala Catalog Server's process state is consistent with the role configuration", "display_name": "Impala Catalog Server Process Health Test", "name": "catalogserver_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_cmd_args_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_catalogserver_web_metric_collection", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_catalogserver_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enables the health test that verifies the Catalog Server is connected to the StateStore", "display_name": "Catalog Server Connectivity Health Test", "name": "catalogserver_connectivity_health_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_catalogserver_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server HDFS Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server HDFS Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hdfs_site_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the StateStore Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: StateStore Connectivity", "name": "role_health_suppression_catalogserver_connectivity", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Catalog Server Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Catalog Server Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "catalogserver_startup_tolerance", "value": "5" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "catalogserver_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Catalog Server", "name": "catalogd_java_opts", "value": "" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "Catalog Server Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "If true, loads catalog metadata in the background. If false, metadata is loaded lazily (on access). Only effective in CDH 5 and Impala 1.2.4 and higher.", "display_name": "Load Catalog in Background", "name": "load_catalog_in_background", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml for this role only.", "display_name": "Catalog Server HBase Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hbase_conf_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Breakpad Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Breakpad Dump Directory", "name": "role_config_suppression_minidump_path", "value": "false" }, { "desc": "Timeout for requests to the Hive Metastore Server from Catalog Server. Consider increasing this if you have tables with a lot of metadata and see timeout errors.", "display_name": "Catalog Server Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "3600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogserver_role_env_safety_valve", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Catalog Server command line flags. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_cmd_args_safety_valve", "value": null }, { "desc": "Maximum number of Breakpad dump files stored by Catalog Server Role.", "display_name": "Catalog Server Max Breakpad Dump Files", "name": "max_minidumps", "value": "9" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "CATALOGSERVER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_catalogserver_memory_rss_health", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml for this role only.", "display_name": "Catalog Server Hive Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hive_conf_safety_valve", "value": null }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above.", "display_name": "Catalog Server Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "Catalog Server Maximum Log Files", "name": "max_log_files", "value": "10" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-nfsgateway.json0000666000175100017510000006563613245514472025511 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_nfsgateway_role_env_safety_valve", "value": "false" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_nfsgateway_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_nfsgateway_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_nfsgateway_scm_health", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Log Directory parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Log Directory", "name": "role_config_suppression_nfsgateway_log_dir", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "NFS clients often reorder writes. As a result, sequential writes can arrive at the NFS Gateway in random order. This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. Please make sure this directory has enough space. For example, if the application uploads 10 files with each having 100MB, it is recommended that this directory have roughly 1GB of space in case write reorder happens (in the worst case) to every file.", "display_name": "Temporary Dump Directory", "name": "dfs_nfs3_dump_dir", "value": "/tmp/.hdfs-nfs" }, { "desc": "Enables the health test that the NFS Gateway's process state is consistent with the role configuration", "display_name": "NFS Gateway Process Health Test", "name": "nfsgateway_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NFS Gateway in Bytes", "name": "nfsgateway_java_heapsize", "value": "268435456" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "nfsgateway_config_safety_valve", "value": null }, { "desc": "The port number of the system portmap or rpcbind service. This configuration is used by Cloudera Manager to verify if the system portmap or rpcbind service is running before starting NFS Gateway role. Cloudera Manager does not manage the system portmap or rpcbind service.", "display_name": "Portmap (or Rpcbind) Port", "name": "nfs3_portmap_port", "value": "111" }, { "desc": "The NFS Gateway server port.", "display_name": "NFS Gateway Server Port", "name": "nfs3_server_port", "value": "2049" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NFS Gateway", "name": "nfsgateway_java_opts", "value": "" }, { "desc": "The maximum number of rolled log files to keep for NFS Gateway logs. Typically used by log4j or logback.", "display_name": "NFS Gateway Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Temporary Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Temporary Dump Directory Free Space Monitoring Percentage Thresholds", "name": "nfsgateway_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_nfsgateway_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "When computing the overall NFS Gateway health, consider the host's health.", "display_name": "NFS Gateway Host Health Test", "name": "nfsgateway_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Temporary Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Temporary Dump Directory", "name": "role_config_suppression_dfs_nfs3_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "By default, NFS Gateway exported directories can be mounted by any client. For better access control, update this property with a list of host names and access privileges separated by whitespace characters. Host name format can be a single host, a Java regular expression, or an IPv4 address. The access privilege uses rw to specify readwrite and ro to specify readonly access. If the access privilege is not provided, the default is read-only. Examples of host name format and access privilege: \"192.168.0.0/22 rw\", \"host.*.example.com\", \"host1.test.org ro\".", "display_name": "Allowed Hosts and Privileges", "name": "dfs_nfs_exports_allowed_hosts", "value": "* rw" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The port number of the mount daemon implemented inside the NFS Gateway server role.", "display_name": "NFS Gateway MountD Port", "name": "nfs3_mountd_port", "value": "4242" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NFS Gateway Environment Advanced Configuration Snippet (Safety Valve)", "name": "NFSGATEWAY_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NFS Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_nfsgateway_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "nfsgateway_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Hosts and Privileges parameter.", "display_name": "Suppress Parameter Validation: Allowed Hosts and Privileges", "name": "role_config_suppression_dfs_nfs_exports_allowed_hosts", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_nfsgateway_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Temporary Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Temporary Dump Directory Free Space", "name": "role_health_suppression_nfsgateway_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for NFS Gateway logs. Typically used by log4j or logback.", "display_name": "NFS Gateway Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory.", "display_name": "Temporary Dump Directory Free Space Monitoring Absolute Thresholds", "name": "nfsgateway_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NFS Gateway parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NFS Gateway", "name": "role_config_suppression_nfsgateway_java_opts", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_nfsgateway_unexpected_exits", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for NFS Gateway logs", "display_name": "NFS Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_nfsgateway_config_safety_valve", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Directory where NFS Gateway will place its log files.", "display_name": "NFS Gateway Log Directory", "name": "nfsgateway_log_dir", "value": "/var/log/hadoop-hdfs" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hive-server2-sentry-safety.xml0000666000175100017510000000065513245514472027370 0ustar zuulzuul00000000000000 hive.security.authorization.task.factory org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl hive.server2.session.hook org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook hive.sentry.conf.url file:///{{CMF_CONF_DIR}}/sentry-site.xml sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hive-site.xml0000666000175100017510000000303213245514472024121 0ustar zuulzuul00000000000000 hive.metastore.local false hive.metastore.uris thrift://test-master-001.novalocal:9083 hive.metastore.client.socket.timeout 300 hive.metastore.warehouse.dir /user/hive/warehouse hive.warehouse.subdir.inherit.perms true mapred.reduce.tasks -1 hive.exec.reducers.bytes.per.reducer 1073741824 hive.exec.copyfile.maxsize 33554432 hive.exec.reducers.max 999 hive.metastore.execute.setugi true hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.MemoryTokenStore hive.server2.enable.doAs true fs.hdfs.impl.disable.cache true hive.server2.use.SSL false sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hive-gateway.json0000666000175100017510000001222213245514472024770 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only, a string to be inserted into the client configuration for hive-site.xml.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hive_client_java_opts", "value": "-XX:MaxPermSize=512M -Djava.net.preferIPv4Stack=true" }, { "desc": "Timeout for requests to the Hive Metastore Server. Consider increasing this if you have tables with a lot of metadata and see timeout errors. Used by most Hive Metastore clients such as Hive CLI and HiveServer2, but not by Impala. Impala has a separately configured timeout.", "display_name": "Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "300" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hive" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hive-env.sh", "display_name": "Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh", "name": "hive_client_env_safety_valve", "value": null }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hive_client_java_heapsize", "value": "2147483648" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh parameter.", "display_name": "Suppress Parameter Validation: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh", "name": "role_config_suppression_hive_client_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hive_client_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/ks_indexer-service.json0000666000175100017510000004661613245514472026205 0ustar zuulzuul00000000000000[ { "desc": "Name of the HBase service that this Key-Value Store Indexer service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Whether to suppress the results of the Lily HBase Indexer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Lily HBase Indexer Health", "name": "service_health_suppression_ks_indexer_hbase_indexers_healthy", "value": "false" }, { "desc": "The password for the HBase Indexer TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HBase Indexer TLS/SSL Certificate Trust Store Password", "name": "keystore_indexer_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_keystore_indexer_sentry_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_ks_indexer_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Lily HBase Indexer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Lily HBase Indexer Count Validator", "name": "service_config_suppression_hbase_indexer_count_validator", "value": "false" }, { "desc": "Name of the Solr service that this Key-Value Store Indexer service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_keystore_indexer_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Morphlines File parameter.", "display_name": "Suppress Parameter Validation: Morphlines File", "name": "service_config_suppression_morphlines_conf_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Grok Dictionary File parameter.", "display_name": "Suppress Parameter Validation: Grok Dictionary File", "name": "service_config_suppression_grok_dictionary_conf_file", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. Sentry is supported only on CDH 5.4 or later deployments and requires authentication to be turned on for HBase Indexer.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "keystore_indexer_sentry_enabled", "value": "false" }, { "desc": "The health test thresholds of the overall Lily HBase Indexer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Lily HBase Indexers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Lily HBase Indexers falls below the critical threshold.", "display_name": "Healthy Lily HBase Indexer Monitoring Thresholds", "name": "ks_indexer_indexers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Sentry Validator", "name": "service_config_suppression_keystore_indexer_sentry_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Custom Mime-types File parameter.", "display_name": "Suppress Parameter Validation: Custom Mime-types File", "name": "service_config_suppression_custom_mimetypes_file", "value": "false" }, { "desc": "Text that goes into morphlines.conf file used by HBase Indexers. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service.", "display_name": "Morphlines File", "name": "morphlines_conf_file", "value": "SOLR_LOCATOR : {\n # Name of solr collection\n collection : collection\n \n # ZooKeeper ensemble\n zkHost : \"$ZK_HOST\" \n}\n\n\nmorphlines : [\n{\nid : morphline\nimportCommands : [\"org.kitesdk.**\", \"com.ngdata.**\"]\n\ncommands : [ \n {\n extractHBaseCells {\n mappings : [\n {\n inputColumn : \"data:*\"\n outputField : \"data\" \n type : string \n source : value\n }\n ]\n }\n }\n\n\n { logDebug { format : \"output record: {}\", args : [\"@{}\"] } }\n]\n}\n]\n" }, { "desc": "Authentication mechanism used by HBase Indexer.", "display_name": "HBase Indexer Secure Authentication", "name": "hbase_indexer_security_authentication", "value": "simple" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "keystore_indexer_sentry_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Text that goes verbatim into grok-dictionary.conf file used by HBase Indexers.", "display_name": "Grok Dictionary File", "name": "grok_dictionary_conf_file", "value": "USERNAME [a-zA-Z0-9._-]+\nUSER %{USERNAME}\nINT (?:[+-]?(?:[0-9]+))\nBASE10NUM (?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))\nNUMBER (?:%{BASE10NUM})\nBASE16NUM (?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))\nUUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}\n\n# Networking\nMAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})\nCISCOMAC (?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})\nWINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})\nCOMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})\nIP (?/(?>[\\w_%!$@:.,-]+|\\\\.)*)+\n#UNIXPATH (?/dev/pts/%{NONNEGINT})\nBSDTTY (?>/dev/tty[pq][a-z0-9])\nTTY (?:%{BSDTTY}|%{LINUXTTY})\nWINPATH (?>[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+\nURIPROTO [A-Za-z]+(\\+[A-Za-z+]+)?\nURIHOST %{IPORHOST}(?::%{POSINT:port})?\n# uripath comes loosely from RFC1738, but mostly from what Firefox\n# doesn't turn into %XX\nURIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=#%_\\-]*)+\n#URIPARAM \\?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?\nURIPARAM \\?[A-Za-z0-9$.+!*'|(){},~#%&/=:;_?\\-\\[\\]]*\nURIPATHPARAM %{URIPATH}(?:%{URIPARAM})?\nURI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?\n\n# Months: January, Feb, 3, 03, 12, December\nMONTH \\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b\nMONTHNUM (?:0?[1-9]|1[0-2])\nMONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\n\n# Days: Monday, Tue, Thu, etc...\nDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)\n\n# Years?\nYEAR (?>\\d\\d){1,2}\n# Time: HH:MM:SS\n#TIME \\d{2}:\\d{2}(?::\\d{2}(?:\\.\\d+)?)?\n# I'm still on the fence about using grok to perform the time match,\n# since it's probably slower.\n# TIME %{POSINT<24}:%{POSINT<60}(?::%{POSINT<60}(?:\\.%{POSINT})?)?\nHOUR (?:2[0123]|[01]?[0-9])\nMINUTE (?:[0-5][0-9])\n# '60' is a leap second in most time standards and thus is valid.\nSECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\nTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])\n# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)\nDATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}\nDATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}\nISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))\nISO8601_SECOND (?:%{SECOND}|60)\nTIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?\nDATE %{DATE_US}|%{DATE_EU}\nDATESTAMP %{DATE}[- ]%{TIME}\nTZ (?:[PMCE][SD]T)\nDATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}\nDATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}\n\n# Syslog Dates: Month Day HH:MM:SS\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}\nPROG (?:[\\w._/%-]+)\nSYSLOGPROG %{PROG:program}(?:\\[%{POSINT:pid}\\])?\nSYSLOGHOST %{IPORHOST}\nSYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>\nHTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}\n\n# Shortcuts\nQS %{QUOTEDSTRING}\n\n# Log formats\nSYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:\nCOMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}\n\n# Log Levels\nLOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_keystore_indexer_provider_resource", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "ks_indexer_env_safety_valve", "value": null }, { "desc": "Text that goes verbatim into custom-mimetypes.xml file used by HBase Indexers.", "display_name": "Custom Mime-types File", "name": "custom_mimetypes_file", "value": "\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access.", "display_name": "Sentry User to Group Mapping Class", "name": "keystore_indexer_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_keystore_indexer_truststore_file", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HBase Indexer might connect to. This is used when HBase Indexer is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HBase Indexer TLS/SSL Certificate Trust Store File", "name": "keystore_indexer_truststore_file", "value": null }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "keystore_indexer_provider_resource", "value": "/user/hbaseindexer/sentry/sentry-provider.ini" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/zookeeper-service.json0000666000175100017510000003113113245514472026037 0ustar zuulzuul00000000000000[ { "desc": "Configures the path of the root znode under which all canary updates are performed", "display_name": "ZooKeeper Canary Root Znode Path", "name": "zookeeper_canary_root_path", "value": "/cloudera_manager_zookeeper_canary" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "zookeeper" }, { "desc": "Amount of time, in ticks, to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they are dropped.", "display_name": "Synchronization Limit", "name": "syncLimit", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Server Count Validator", "name": "service_config_suppression_server_count_validator", "value": "false" }, { "desc": "When enabled, ZooKeeper auto purge feature retains this many most recent snapshots and the corresponding transaction logs in the dataDir and dataLogDir respectively and deletes the rest. Defaults to 5. Minimum value is 3.", "display_name": "Auto Purge Snapshots Retain Count", "name": "autopurgeSnapRetainCount", "value": "5" }, { "desc": "The time interval in hours for which the purge task has to be triggered. Set to a positive integer (1 and above) to enable the auto purging. Defaults to 24.", "display_name": "Auto Purge Time Interval", "name": "autopurgeInterval", "value": "24" }, { "desc": "Enable Kerberos authentication for ZooKeeper.", "display_name": "Enable Kerberos Authentication", "name": "enableSecurity", "value": "false" }, { "desc": "Whether to suppress the results of the ZooKeeper Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ZooKeeper Canary", "name": "service_health_suppression_zookeeper_canary_health", "value": "false" }, { "desc": "The number of snapshot files and corresponding transaction logs to keep when running the Cleanup command.", "display_name": "Cleanup Retain Count", "name": "cleanupRetainCount", "value": "5" }, { "desc": "Enables the health check that a client can connect to ZooKeeper and perform basic operations", "display_name": "ZooKeeper Canary Health Check", "name": "zookeeper_canary_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "zookeeper_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Canary Root Znode Path parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Canary Root Znode Path", "name": "service_config_suppression_zookeeper_canary_root_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_zookeeper_env_safety_valve", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "zookeeper" }, { "desc": "Configures the timeout used by the canary sessions with ZooKeeper servers", "display_name": "ZooKeeper Canary Session Timeout", "name": "zookeeper_canary_session_timeout", "value": "30000" }, { "desc": "The maximum size of the data that can be stored in a znode in bytes.", "display_name": "Jute Max Buffer", "name": "jute_maxbuffer", "value": "4194304" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "zookeeper" }, { "desc": "Whether to suppress the results of the Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Server Health", "name": "service_health_suppression_zookeeper_servers_healthy", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Amount of time, in ticks, to allow followers to connect and sync to a leader. Increase this value as needed, if the amount of data managed by ZooKeeper is large.", "display_name": "Initialization Limit", "name": "initLimit", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Configures the timeout used by the canary for connection establishment with ZooKeeper servers", "display_name": "ZooKeeper Canary Connection Timeout", "name": "zookeeper_canary_connection_timeout", "value": "10000" }, { "desc": "Configures the timeout used by the canary for ZooKeeper operations", "display_name": "ZooKeeper Canary Operation Timeout", "name": "zookeeper_canary_operation_timeout", "value": "30000" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The length of time, in milliseconds, of a single tick, which is the basic time unit used by ZooKeeper. A tick is used to regulate heartbeats and timeouts.", "display_name": "Tick Time", "name": "tickTime", "value": "2000" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "The health test thresholds of the overall Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Servers falls below the critical threshold.", "display_name": "Healthy Server Monitoring Thresholds", "name": "zookeeper_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the ZooKeeper Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: ZooKeeper Server Count Validator", "name": "service_config_suppression_zookeeper_odd_number_of_servers_validator", "value": "false" }, { "desc": "Automatically create data directories at startup, if they do not exist. Enabling this configuration should be used with care as it will suppress any errors in setup of data directories.", "display_name": "Enable auto-creation of data directories", "name": "zookeeper_datadir_autocreate", "value": "false" }, { "desc": "Whether the leader accepts client connections.", "display_name": "Leader Serves", "name": "leaderServes", "value": "yes" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/priority-one-confs.json0000666000175100017510000000320513245514472026145 0ustar zuulzuul00000000000000[ "dfs_block_size", "dfs_umaskmode", "dfs_webhdfs_enabled", "dfs_permissions", "dfs_replication", "io_compression_codecs", "io_sort_mb", "dfs_datanode_du_reserved", "dfs_datanode_failed_volumes_tolerated", "dfs_name_dir_restore", "fs_trash_interval", "dfs_safemode_min_datanodes", "dfs_safemode_extension", "dfs_access_time_precision", "yarn_acl_enable", "yarn_admin_acl", "yarn_log_aggregation_enable", "yarn_log_aggregation_retain_seconds", "mapreduce_jobhistory_max_age_ms", "mapreduce_jobhistory_cleaner_interval", "mapreduce_reduce_memory_mb", "mapreduce_reduce_java_opts", "mapreduce_map_memory_mb", "mapreduce_map_java_opts", "yarn_nodemanager_container_manager_thread_count", "yarn_nodemanager_delete_thread_count", "yarn_nodemanager_heartbeat_interval_ms", "yarn_nodemanager_localizer_cache_cleanup_interval_ms", "yarn_nodemanager_localizer_client_thread_count", "yarn_nodemanager_localizer_cache_target_size_mb", "yarn_nodemanager_localizer_fetch_thread_count", "yarn_nodemanager_log_retain_seconds", "yarn_nodemanager_resource_memory_mb", "yarn_resourcemanager_client_thread_count", "yarn_resourcemanager_scheduler_client_thread_count", "yarn_resourcemanager_admin_client_thread_count", "yarn_resourcemanager_amliveliness_monitor_interval_ms", "yarn_am_liveness_monitor_expiry_interval_ms", "yarn_resourcemanager_am_max_retries", "yarn_scheduler_minimum_allocation_mb", "yarn_scheduler_maximum_allocation_mb", "yarn_app_mapreduce_am_command_opts", "yarn_app_mapreduce_am_resource_mb" ] sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hive-service.json0000666000175100017510000014504013245514472024774 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "The number of partitions per RPC retrieved during Hive Metastore Sentry HDFS Sync Cache Initialization.", "display_name": "Number of Partitions per RPC on Initialization", "name": "sentry_hdfs_sync_metastore_cache_partitions_per_rpc", "value": "100" }, { "desc": "List of users that are allowed to bypass Sentry Authorization in the Hive metastore. These are usually service users that already ensure that all activity has been authorized, such as hive and impala. Only applies when Hive is using Sentry Service.", "display_name": "Bypass Sentry Authorization Users", "name": "sentry_metastore_service_users", "value": "hive,impala,hue,hdfs" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Sentry Validator", "name": "service_config_suppression_hive_sentry_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HiveServer2 Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HiveServer2 Count Validator", "name": "service_config_suppression_hiveserver2_count_validator", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hive" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HiveServer2 might connect to. This is used when HiveServer2 is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HiveServer2 TLS/SSL Certificate Trust Store File", "name": "hiveserver2_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Name parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Name", "name": "service_config_suppression_hive_metastore_database_name", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_service_config_safety_valve", "value": null }, { "desc": "Port number of Hive Metastore database", "display_name": "Hive Metastore Database Port", "name": "hive_metastore_database_port", "value": "3306" }, { "desc": "Smaller than this size, Hive uses a single-threaded copy; larger than this size, Hive uses DistCp.", "display_name": "Hive Copy Large File Size", "name": "hive_exec_copyfile_maxsize", "value": "33554432" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hive/audit" }, { "desc": "Name of Hive Metastore database", "display_name": "Hive Metastore Database Name", "name": "hive_metastore_database_name", "value": "metastore" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop user to group mapping or local groups defined in the policy file. Hadoop user to group mapping may be configured in the Cloudera Manager HDFS service configuration page under the Security section.", "display_name": "Sentry User to Group Mapping Class", "name": "hive_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using the Sentry service instead, add the Sentry service as a dependency to the Hive service. The Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry is supported only on CDH 4.4 or later deployments. Before enabling Sentry, read the requirements and configuration steps in Setting Up Hive Authorization with Sentry.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Bypass Metastore Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Bypass Metastore Validator", "name": "service_config_suppression_hive_bypass_metastore_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the Spark on YARN service that this Hive service instance depends on. If selected, Hive jobs can use the Spark execution engine instead of MapReduce2. Requires that Hive depends on YARN. See Configuring Hive on Spark for more information about Hive on Spark. In CDH lower than 5.7, Hive on Spark also requires setting Enable Hive on Spark to true.", "display_name": "Spark On YARN Service", "name": "spark_on_yarn_service", "value": null }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "The password for the HiveServer2 JKS keystore file.", "display_name": "HiveServer2 TLS/SSL Server JKS Keystore File Password", "name": "hiveserver2_keystore_password", "value": null }, { "desc": "Instead of talking to Hive Metastore Server for Metastore information, Hive clients will talk directly to the Metastore database.", "display_name": "Bypass Hive Metastore Server", "name": "hive_bypass_metastore_server", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Host parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Host", "name": "service_config_suppression_hive_metastore_database_host", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The maximum number of retries allowed during Hive Metastore Sentry HDFS Sync cache initialization.", "display_name": "Max Number of Retries on Initialization", "name": "sentry_hdfs_sync_metastore_cache_retry_max_num", "value": "1" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_service_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "hive_server2_sentry_safety_valve", "value": null }, { "desc": "Size per reducer. If the input size is 10GiB and this is set to 1GiB, Hive will use 10 reducers.", "display_name": "Hive Bytes Per Reducer", "name": "hive_bytes_per_reducer", "value": "67108864" }, { "desc": "The directory in which Hive lineage log files are written.", "display_name": "Hive Lineage Log Directory", "name": "lineage_event_log_dir", "value": "/var/log/hive/lineage" }, { "desc": "Encrypt communication between clients and HiveServer2 using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HiveServer2", "name": "hiveserver2_enable_ssl", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_hiveserver2_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_hive_server2_sentry_safety_valve", "value": "false" }, { "desc": "Password for Hive Metastore database", "display_name": "Hive Metastore Database Password", "name": "hive_metastore_database_password", "value": "" }, { "desc": "The password for the HiveServer2 TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HiveServer2 TLS/SSL Certificate Trust Store Password", "name": "hiveserver2_truststore_password", "value": null }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Hive events:\n

\n\n
    \n
  • userName: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • operation: the Hive operation being performed.
  • \n
  • databaseName: the databaseName for the operation.
  • \n
  • tableName: the tableName for the operation.
  • \n
\n\n

\nThe default Hive audit event filter discards HDFS directory events generated by\nHive jobs that reference the /tmp directory.\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default Hive audit event filter discards HDFS directory events \",\n \"generated by Hive jobs that reference the /tmp directory.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"operation\", \"match\" : \"QUERY\" },\n { \"name\" : \"objectType\", \"match\" : \"DFS_DIR\"},\n { \"name\" : \"resourcePath\", \"match\" : \"/tmp/hive-(?:.+)?/hive_(?:.+)?/-mr-.*\" }\n ]\n }\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the WebHCat Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: WebHCat Server Count Validator", "name": "service_config_suppression_webhcat_count_validator", "value": "false" }, { "desc": "Directory name where Hive Metastore's database is stored (only for Derby)", "display_name": "Hive Metastore Derby Path", "name": "hive_metastore_derby_path", "value": "/var/lib/hive/cloudera_manager/derby/metastore_db" }, { "desc": "Perform DataNucleus validation of metadata during startup. Note: when enabled, Hive will log DataNucleus warnings even though Hive will function normally.", "display_name": "Hive Metastore Database DataNucleus Metadata Validation", "name": "hive_metastore_database_datanucleus_metadata_validation", "value": "false" }, { "desc": "Name of the Sentry service that this Hive service instance depends on. If selected, Hive uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Max number of reducers to use. If the configuration parameter Hive Reduce Tasks is negative, Hive will limit the number of reducers to the value of this parameter.", "display_name": "Hive Max Reducers", "name": "hive_max_reducers", "value": "1099" }, { "desc": "Type of Hive Metastore database. Note that Derby is not recommended and Cloudera Impala does not support Derby.", "display_name": "Hive Metastore Database Type", "name": "hive_metastore_database_type", "value": "mysql" }, { "desc": "Whether to suppress the results of the HiveServer2 Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HiveServer2 Health", "name": "service_health_suppression_hive_hiveserver2s_healthy", "value": "false" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "hiveserver2_ldap_uri", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hive_service_env_safety_valve", "value": "false" }, { "desc": "Host name of Hive Metastore database", "display_name": "Hive Metastore Database Host", "name": "hive_metastore_database_host", "value": "localhost" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Prevent Metastore operations in the event of schema version incompatibility. Consider setting this to true to reduce probability of schema corruption during Metastore operations. Note that setting this property to true will also set datanucleus.autoCreateSchema property to false and datanucleus.fixedDatastore property to true. Any values set in Cloudera Manager for these properties will be overridden.", "display_name": "Strict Hive Metastore Schema Validation", "name": "hive_metastore_schema_verification", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hive on Spark Dependency Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive on Spark Dependency Validator", "name": "service_config_suppression_hive_on_spark_missing_dependency", "value": "false" }, { "desc": "This configuration overrides the value set for Hive Proxy User Groups configuration in HDFS service for use by Hive Metastore Server. Specify a comma-delimited list of groups that you want to allow access to Hive Metastore metadata and allow the Hive user to impersonate. A value of '*' allows all groups. The default value of empty inherits the value set for Hive Proxy User Groups configuration in the HDFS service.", "display_name": "Hive Metastore Access Control and Proxy User Groups Override", "name": "hive_proxy_user_groups_list", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_hive_sentry_provider_resource", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Warehouse Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Warehouse Directory", "name": "service_config_suppression_hive_warehouse_directory", "value": "false" }, { "desc": "Wait duration in milliseconds for each retry during Hive Metastore Sentry HDFS Sync Cache Initialization.", "display_name": "Retry Wait Time on Initialization", "name": "sentry_hdfs_sync_metastore_cache_retry_wait_duration_millis", "value": "1000" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "hive_core_site_safety_valve", "value": null }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "hive.metastore.client.socket.timeout60" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Bypass Sentry Authorization Users parameter.", "display_name": "Suppress Parameter Validation: Bypass Sentry Authorization Users", "name": "service_config_suppression_sentry_metastore_service_users", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_hiveserver2_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Replication Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hive_replication_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Auxiliary JARs Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Auxiliary JARs Directory", "name": "service_config_suppression_hive_aux_jars_path_dir", "value": "false" }, { "desc": "The health test thresholds of the overall WebHCat Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" WebHCat Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" WebHCat Servers falls below the critical threshold.", "display_name": "Healthy WebHCat Server Monitoring Thresholds", "name": "hive_webhcats_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml. Applies to all Hive Replication jobs.", "display_name": "Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_service_replication_config_safety_valve", "value": null }, { "desc": "This parameter is useful when authenticating against a non Active Directory server such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,<this parameter>. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server will look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "hiveserver2_ldap_basedn", "value": null }, { "desc": "If set to true, the Hive metastore will treat a problem with cache initialization as a fatal error.", "display_name": "Abort on Initialization Failure", "name": "sentry_hdfs_sync_metastore_cache_fail_on_partial_update", "value": "true" }, { "desc": "Whether Hive Metastore should try to use direct SQL queries instead of DataNucleus for certain read paths. This can improve metastore performance when fetching many partitions by orders of magnitude. In case of failure, execution will fall back to DataNucleus. This configuration is not supported and is disabled when Hive service is configured with Postgres.", "display_name": "Enable Direct SQL", "name": "hive_metastore_database_datanucleus_try_direct_sql", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Proxy Groups Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Proxy Groups Validator", "name": "service_config_suppression_hive_proxy_groups_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Hive warehouse directory is the location in HDFS where Hive's tables are stored. Note that Hive's default value for its warehouse directory is '/user/hive/warehouse'.", "display_name": "Hive Warehouse Directory", "name": "hive_warehouse_directory", "value": "/user/hive/warehouse" }, { "desc": "User for Hive Metastore database", "display_name": "Hive Metastore Database User", "name": "hive_metastore_database_user", "value": "hive" }, { "desc": "Whether to suppress the results of the WebHCat Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: WebHCat Server Health", "name": "service_health_suppression_hive_webhcats_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Derby Path parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Derby Path", "name": "service_config_suppression_hive_metastore_derby_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Metastore Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Metastore Server Count Validator", "name": "service_config_suppression_hivemetastore_count_validator", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Access Control and Proxy User Groups Override parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Access Control and Proxy User Groups Override", "name": "service_config_suppression_hive_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_hiveserver2_ldap_domain", "value": "false" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "hiveserver2_ldap_domain", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_hiveserver2_keystore_password", "value": "false" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "hiveserver2_enable_ldap_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_hive_core_site_safety_valve", "value": "false" }, { "desc": "Default number of reduce tasks per job. Usually set to a prime number close to the number of available hosts. Ignored when mapred.job.tracker is \"local\". Hadoop sets this to 1 by default, while Hive uses -1 as the default. When set to -1, Hive will automatically determine an appropriate number of reducers for each job.", "display_name": "Hive Reduce Tasks", "name": "hive_reduce_tasks", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Name for Sentry Authorization parameter.", "display_name": "Suppress Parameter Validation: Server Name for Sentry Authorization", "name": "service_config_suppression_hive_sentry_server", "value": "false" }, { "desc": "The health test thresholds of the overall Hive Metastore Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Hive Metastore Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Hive Metastore Servers falls below the critical threshold.", "display_name": "Healthy Hive Metastore Server Monitoring Thresholds", "name": "hive_hivemetastores_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The health test thresholds of the overall HiveServer2 health. The check returns \"Concerning\" health if the percentage of \"Healthy\" HiveServer2s falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" HiveServer2s falls below the critical threshold.", "display_name": "Healthy HiveServer2 Monitoring Thresholds", "name": "hive_hiveserver2s_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Automatically create or upgrade tables in the Hive Metastore database when needed. Consider setting this to false and managing the schema manually.", "display_name": "Auto Create and Upgrade Hive Metastore Database Schema", "name": "hive_metastore_database_auto_create_schema", "value": "false" }, { "desc": "The server name used when defining privilege rules in Sentry authorization. Sentry uses this name as an alias for the Hive service. It does not correspond to any physical server name.", "display_name": "Server Name for Sentry Authorization", "name": "hive_sentry_server", "value": "server1" }, { "desc": "Enable collection of lineage from the service's roles.", "display_name": "Enable Lineage Collection", "name": "navigator_lineage_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_hiveserver2_ldap_uri", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of Hive replication jobs.", "display_name": "Hive Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_replication_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "service_config_suppression_hive_service_replication_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "service_config_suppression_navigator_lineage_client_config_safety_valve", "value": "false" }, { "desc": "Name of the HBase service that this Hive service instance depends on.", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Directory containing auxiliary JARs used by Hive. This should be a directory location and not a classpath containing one or more JARs. This directory must be created and managed manually on hosts that run the Hive Metastore Server, HiveServer2, or the Hive CLI. The directory location is set in the environment as HIVE_AUX_JARS_PATH and will generally override the hive.aux.jars.path property set in XML files, even if hive.aux.jars.path is set in an advanced configuration snippet.", "display_name": "Hive Auxiliary JARs Directory", "name": "hive_aux_jars_path_dir", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database User parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database User", "name": "service_config_suppression_hive_metastore_database_user", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userNname, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Hive events:\n

\n\n
    \n
  • operation: the Hive operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • databaseName: the database affected by the operation.
  • \n
  • tableName: the table or view affected by the operation.
  • \n
  • objectType: the type of object affected by the operation.
  • \n
  • resourcePath: the path of the resource affected by the operation.
  • \n
\n\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Whether to suppress the results of the Hive Metastore Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Metastore Server Health", "name": "service_health_suppression_hive_hivemetastores_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_hiveserver2_keystore_path", "value": "false" }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "hive_sentry_provider_resource", "value": "/user/hive/sentry/sentry-provider.ini" }, { "desc": "The maximum size, in megabytes, per log file for Hive lineage logs. Typically used by log4j or logback.", "display_name": "Hive Maximum Lineage Log File Size", "name": "max_lineage_log_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_hiveserver2_ldap_basedn", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Hive service instance depends on.", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "The number of threads used during Hive Metastore Sentry HDFS Sync Cache Initialization.", "display_name": "Number of Threads on Initialization", "name": "sentry_hdfs_sync_metastore_cache_init_threads", "value": "10" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HiveServer2 is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HiveServer2 TLS/SSL Server JKS Keystore File Location", "name": "hiveserver2_keystore_path", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Hive Derby Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Derby Validator", "name": "service_config_suppression_hive_derby_validator", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hive" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hive" }, { "desc": "Let the table directories inherit the permission of the Warehouse or Database directory instead of being created with the permissions derived from dfs umask. This allows Impala to insert into tables created via Hive.", "display_name": "Hive Warehouse Subdirectories Inherit Permissions", "name": "hive_warehouse_subdir_inherit_perms", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Client TLS/SSL In Use With LDAP Authentication Validator configuration validator.", "display_name": "Suppress Configuration Validator: Client TLS/SSL In Use With LDAP Authentication Validator", "name": "service_config_suppression_hive_client_ssl_recommended_with_ldap_auth_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Password parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Password", "name": "service_config_suppression_hive_metastore_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Lineage Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Lineage Log Directory", "name": "service_config_suppression_lineage_event_log_dir", "value": "false" }, { "desc": "The number of tables per RPC retrieved during Hive Metastore Sentry HDFS Sync Cache Initialization.", "display_name": "Number of Tables per RPC on Initialization", "name": "sentry_hdfs_sync_metastore_cache_tables_per_rpc", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Concurrency Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Concurrency Configuration Validator", "name": "service_config_suppression_hive_concurrency_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Allows URIs when defining privileges in per-database policy files. Warning: Typically, this configuration should be disabled. Enabling it would allow database policy file owner (which is generally not Hive admin user) to grant load privileges to any directory with read access to Hive admin user, including databases controlled by other database policy files.", "display_name": "Allow URIs in Database Policy File", "name": "sentry_allow_uri_db_policyfile", "value": "false" }, { "desc": "In unsecure mode, setting this property to true will cause the Metastore Server to execute DFS operations using the client's reported user and group permissions. Cloudera Manager will set this for all clients and servers.", "display_name": "Set User and Group Information", "name": "hive_set_ugi", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.lineage.client.properties.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "navigator_lineage_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "service_config_suppression_hive_service_config_safety_valve", "value": "false" }, { "desc": "MapReduce jobs are run against this service.", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hbase-master.json0000666000175100017510000010367413245514472024765 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"IPC Server handler.*ClosedChannelException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"IPC Server Responder, call.*output error\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Daughter regiondir does not exist: .*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"File.*might still be open.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"File.*might still be open.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Moving table .+ state to enabled but was already enabled\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Received OPENED for region.*but region was in the state.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Log Directory parameter.", "display_name": "Suppress Parameter Validation: Master Log Directory", "name": "role_config_suppression_hbase_master_log_dir", "value": "false" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "master_startup_tolerance", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_master_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_master_host_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "When computing the overall Master health, consider the host's health.", "display_name": "Master Host Health Test", "name": "master_host_health_enabled", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Directory where Master will place its log files.", "display_name": "Master Log Directory", "name": "hbase_master_log_dir", "value": "/var/log/hbase" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "master_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Time period in seconds to reset long-running metrics (e.g. compactions). This is an HBase specific configuration.", "display_name": "Extended Period", "name": "hbase_metrics_extended_period", "value": "3600" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "master_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "master_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_master_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the HBase Master Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HBase Master Canary", "name": "role_health_suppression_master_canary_health", "value": "false" }, { "desc": "The port that the HBase Master binds to.", "display_name": "HBase Master Port", "name": "hbase_master_port", "value": "60000" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "master_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_master_log_directory_free_space", "value": "false" }, { "desc": "Number of pooled threads to handle the recovery of the region servers in the master.", "display_name": "RegionServer Recovery Threads", "name": "hbase_master_executor_serverops_threads", "value": "5" }, { "desc": "List of org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are loaded by default on the active HMaster process. For any implemented coprocessor methods, the listed classes will be called in order. After implementing your own MasterObserver, just put it in HBase's classpath and add the fully qualified class name here.", "display_name": "HBase Coprocessor Master Classes", "name": "hbase_coprocessor_master_classes", "value": "" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase Master in Bytes", "name": "hbase_master_java_heapsize", "value": "1073741824" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_master_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "master_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Enables the health test that the Master's process state is consistent with the role configuration", "display_name": "Master Process Health Test", "name": "master_scm_health_enabled", "value": "true" }, { "desc": "When true, HBase Master will bind to 0.0.0.0. Only available with CDH 4.3 and later.", "display_name": "HBase Master Bind to Wildcard Address", "name": "hbase_master_bind_to_wildcard_address", "value": "true" }, { "desc": "Enables the health test that a client can connect to the HBase Master", "display_name": "HBase Master Canary Health Test", "name": "master_canary_health_enabled", "value": "true" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Coprocessor Master Classes parameter.", "display_name": "Suppress Parameter Validation: HBase Coprocessor Master Classes", "name": "role_config_suppression_hbase_coprocessor_master_classes", "value": "false" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_master_gc_duration", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master Web UI Address parameter.", "display_name": "Suppress Parameter Validation: HBase Master Web UI Address", "name": "role_config_suppression_hbase_master_info_bindaddress", "value": "false" }, { "desc": "The port for the HBase Master web UI. Set to -1 to disable the HBase Master web UI.", "display_name": "HBase Master Web UI Port", "name": "hbase_master_info_port", "value": "60010" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_master_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Master Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_master_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_master_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase Master", "name": "hbase_master_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master Log Cleaner Plugins parameter.", "display_name": "Suppress Parameter Validation: HBase Master Log Cleaner Plugins", "name": "role_config_suppression_hbase_master_logcleaner_plugins", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Master logs. Typically used by log4j or logback.", "display_name": "Master Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Maximum time an HLog remains in the .oldlogdir directory until an HBase Master thread deletes it.", "display_name": "Maximum Time to Keep HLogs", "name": "hbase_master_logcleaner_ttl", "value": "60000" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Master Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase Master DNS Network Interface", "name": "role_config_suppression_hbase_master_dns_interface", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Master Environment Advanced Configuration Snippet (Safety Valve)", "name": "MASTER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_master_scm_health", "value": "false" }, { "desc": "Number of RPC Server instances spun up on HBase Master.", "display_name": "HBase Master Handler Count", "name": "hbase_master_handler_count", "value": "25" }, { "desc": "The address for the HBase Master web UI", "display_name": "HBase Master Web UI Address", "name": "hbase_master_info_bindAddress", "value": null }, { "desc": "A comma-separated list of LogCleanerDelegate(s) that are used in LogsCleaner. WAL/HLog cleaner(s) are called in order, so put the log cleaner that prunes the most log files in the front. To implement your own LogCleanerDelegate, add it to HBase's classpath and add the fully-qualified class name here. You should always add the above default log cleaners in the list, unless you have a special reason not to.", "display_name": "HBase Master Log Cleaner Plugins", "name": "hbase_master_logcleaner_plugins", "value": null }, { "desc": "The maximum size, in megabytes, per log file for Master logs. Typically used by log4j or logback.", "display_name": "Master Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_master_web_metric_collection", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The host name or IP address of the DNS name server which an HBase Master should use to determine the host name used for communication and display purposes.", "display_name": "HBase Master DNS Name Server", "name": "hbase_master_dns_nameserver", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase Master DNS Name Server", "name": "role_config_suppression_hbase_master_dns_nameserver", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Number of pooled threads to handle region opening in the master.", "display_name": "Region Opening Threads", "name": "hbase_master_executor_openregion_threads", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase Master parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase Master", "name": "role_config_suppression_hbase_master_java_opts", "value": "false" }, { "desc": "The name of the DNS network interface from which an HBase Master should report its IP address.", "display_name": "HBase Master DNS Network Interface", "name": "hbase_master_dns_interface", "value": null }, { "desc": "The minimum log level for Master logs", "display_name": "Master Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Master Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_master_role_env_safety_valve", "value": "false" }, { "desc": "Number of pooled threads to handle region closing in the master.", "display_name": "Region Closing Threads", "name": "hbase_master_executor_closeregion_threads", "value": "5" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/kms-service.json0000666000175100017510000001370013245514472024630 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "KMS_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Key Management Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Key Management Server Count Validator", "name": "service_config_suppression_kms_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_kms_service_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "kms" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "kms" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Authentication type for the KMS. Can either be \"simple\" or \"kerberos\".", "display_name": "Authentication Type", "name": "hadoop_kms_authentication_type", "value": "simple" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/ks_indexer-hbase_indexer.json0000666000175100017510000005372713245514472027346 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Kerberos principal used by the Lily HBase Indexer roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "hbase" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_hbase_indexer_process_groupname", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "ZooKeeper session timeout. Controls the amount of time the HBase Indexer will attempt to connect to ZooKeeper before timing out.", "display_name": "HBase Indexer ZooKeeper Session Timeout", "name": "hbase_indexer_zk_session_timeout", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Lily HBase Indexer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Lily HBase Indexer", "name": "role_config_suppression_hbase_indexer_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer Log Directory", "name": "role_config_suppression_hbase_indexer_log_dir", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_indexer_swap_memory_usage", "value": "false" }, { "desc": "HTTP port used by HBase Indexer.", "display_name": "HBase Indexer HTTP Port", "name": "hbase_indexer_http_port", "value": "11060" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbase_indexer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_indexer_host_health", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Lily HBase Indexer logs. Typically used by log4j or logback.", "display_name": "Lily HBase Indexer Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The user that the HBase Indexer process should run as.", "display_name": "System User", "name": "hbase_indexer_process_username", "value": "hbase" }, { "desc": "The group that the HBase Indexer process should run as.", "display_name": "System Group", "name": "hbase_indexer_process_groupname", "value": "hbase" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Lily HBase Indexer", "name": "hbase_indexer_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_indexer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_indexer_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_indexer_unexpected_exits", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only. A string to be inserted into hbase-indexer-site.xml for this role only.", "display_name": "Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml", "name": "hbase_indexer_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASE_INDEXER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_hbase_indexer_process_username", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Lily HBase Indexer logs. Typically used by log4j or logback.", "display_name": "Lily HBase Indexer Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where HBase Indexer will place its log files.", "display_name": "HBase Indexer Log Directory", "name": "hbase_indexer_log_dir", "value": "/var/log/hbase-solr" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_indexer_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml", "name": "role_config_suppression_hbase_indexer_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When computing the overall Lily HBase Indexer health, consider the host's health.", "display_name": "Lily HBase Indexer Host Health Test", "name": "hbase_indexer_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbase_indexer_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_indexer_log_directory_free_space", "value": "false" }, { "desc": "Enables the health test that the Lily HBase Indexer's process state is consistent with the role configuration", "display_name": "Lily HBase Indexer Process Health Test", "name": "hbase_indexer_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for Lily HBase Indexer logs", "display_name": "Lily HBase Indexer Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Lily HBase Indexer in Bytes", "name": "hbase_indexer_java_heapsize", "value": "1073741824" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hbase-hbaserestserver.json0000666000175100017510000007165513245514472026704 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase REST Server", "name": "hbase_restserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_hbase_restserver_keystore_file", "value": "false" }, { "desc": "Maximum size of the HBase REST Server thread pool. The server can process this number of concurrent requests. Setting this too high can lead to out of memory errors.", "display_name": "HBase REST Server Maximum Threads", "name": "hbase_restserver_threads_max", "value": "100" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbaserestserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Log Directory", "name": "role_config_suppression_hbase_restserver_log_dir", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HBase REST Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASERESTSERVER_role_env_safety_valve", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_restserver_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_hbase_restserver_keystore_password", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbaserestserver_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_restserver_config_safety_valve", "value": null }, { "desc": "The password for the HBase REST Server JKS keystore file.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore File Password", "name": "hbase_restserver_keystore_password", "value": null }, { "desc": "The maximum number of rolled log files to keep for HBase REST Server logs. Typically used by log4j or logback.", "display_name": "HBase REST Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_rest_server_scm_health", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase REST Server in Bytes", "name": "hbase_restserver_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_rest_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Encrypt communication between clients and HBase REST Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HBase REST Server", "name": "hbase_restserver_ssl_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Enables the health test that the HBase REST Server's process state is consistent with the role configuration", "display_name": "HBase REST Server Process Health Test", "name": "hbaserestserver_scm_health_enabled", "value": "true" }, { "desc": "The port that HBase REST Server binds to.", "display_name": "HBase REST Server Port", "name": "hbase_restserver_port", "value": "20550" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server DNS Name Server", "name": "role_config_suppression_hbase_restserver_dns_nameserver", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_rest_server_file_descriptor", "value": "false" }, { "desc": "If true, HBase REST Server Web UI will bind to a wildcard address (0.0.0.0). Otherwise it will bind to a host name. Only available in CDH 4.3 and later.", "display_name": "HBase REST Server Web UI Bind to Wildcard Address", "name": "hbase_restserver_info_bind_to_wildcard", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HBase REST Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase REST Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore File Location", "name": "hbase_restserver_keystore_file", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_rest_server_swap_memory_usage", "value": "false" }, { "desc": "When false, all HTTP methods are permitted (GET/PUT/POST/DELETE). When true, only GET is permitted.", "display_name": "Enable HBase REST Server Read Only Mode", "name": "hbase_restserver_readonly", "value": "false" }, { "desc": "The port that HBase REST Server Web UI binds to.", "display_name": "HBase REST Server Web UI Port", "name": "hbase_restserver_info_port", "value": "8085" }, { "desc": "Directory where HBase REST Server will place its log files.", "display_name": "HBase REST Server Log Directory", "name": "hbase_restserver_log_dir", "value": "/var/log/hbase" }, { "desc": "When computing the overall HBase REST Server health, consider the host's health.", "display_name": "HBase REST Server Host Health Test", "name": "hbaserestserver_host_health_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase REST Server is acting as a TLS/SSL server.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore Key Password", "name": "hbase_restserver_keystore_keypassword", "value": null }, { "desc": "The maximum size, in megabytes, per log file for HBase REST Server logs. Typically used by log4j or logback.", "display_name": "HBase REST Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The host name or IP address of the DNS name server which an HBase REST Server should use to determine the host name used for communication and display purposes.", "display_name": "HBase REST Server DNS Name Server", "name": "hbase_restserver_dns_nameserver", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase REST Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase REST Server", "name": "role_config_suppression_hbase_restserver_java_opts", "value": "false" }, { "desc": "HBase REST Server will bind to this address.", "display_name": "HBase REST Server Host Address", "name": "hbase_restserver_host", "value": "0.0.0.0" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Host Address parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Host Address", "name": "role_config_suppression_hbase_restserver_host", "value": "false" }, { "desc": "The name of the DNS network interface from which an HBase REST Server should report its IP address.", "display_name": "HBase REST Server DNS Network Interface", "name": "hbase_restserver_dns_interface", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server DNS Network Interface", "name": "role_config_suppression_hbase_restserver_dns_interface", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_rest_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_hbase_restserver_keystore_keypassword", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_rest_server_host_health", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Minimum size of the HBase REST Server thread pool. The server will maintain at least this number of threads in the pool at all times. The thread pool can grow up to the maximum size set by hbase.rest.threads.max.", "display_name": "HBase REST Server Minimum Threads", "name": "hbase_restserver_threads_min", "value": "2" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_rest_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for HBase REST Server logs", "display_name": "HBase REST Server Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/yarn-resourcemanager.json0000666000175100017510000012721113245514472026534 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "resourcemanager_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For applications that request containers on particular racks, the minimum time in milliseconds the Fair Scheduler waits before accepting a placement on another rack. Requires Fair Scheduler continuous scheduling to be enabled. If continuous scheduling is disabled, yarn.scheduler.fair.locality.threshold.rack should be used instead.", "display_name": "Fair Scheduler Rack Locality Delay", "name": "yarn_scheduler_fair_locality_delay_rack_ms", "value": "4000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_resourcemanager_config_safety_valve", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The expiry interval to wait until a NodeManager is considered dead.", "display_name": "NodeManager Monitor Expiry", "name": "yarn_nm_liveness_monitor_expiry_interval_ms", "value": "600000" }, { "desc": "The HTTP port of the ResourceManager web application.", "display_name": "ResourceManager Web Application HTTP Port", "name": "resourcemanager_webserver_port", "value": "8088" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The address of the applications manager interface in the ResourceManager.", "display_name": "ResourceManager Address", "name": "yarn_resourcemanager_address", "value": "8032" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "resourcemanager_web_metric_collection_enabled", "value": "true" }, { "desc": "The periodic interval that the ResourceManager will check whether ApplicationMasters is still alive.", "display_name": "ApplicationMaster Monitor Interval", "name": "yarn_resourcemanager_amliveliness_monitor_interval_ms", "value": "1000" }, { "desc": "Number of threads used to handle the ResourceManager admin interface.", "display_name": "Admin Client Thread Count", "name": "yarn_resourcemanager_admin_client_thread_count", "value": "1" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "If enabled, the ResourceManager binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind ResourceManager to Wildcard Address", "name": "yarn_rm_bind_wildcard", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_fair_scheduler_configuration", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_resource_manager_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The address of the admin interface in the ResourceManager.", "display_name": "Administration Address", "name": "yarn_resourcemanager_admin_address", "value": "8033" }, { "desc": "The periodic interval that the ResourceManager will check whether NodeManagers are still alive.", "display_name": "NodeManager Monitor Interval", "name": "yarn_resourcemanager_nm_liveness_monitor_interval_ms", "value": "1000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_resource_manager_swap_memory_usage", "value": "false" }, { "desc": "The maximum number of application attempts. This is a global setting for all ApplicationMasters. Each ApplicationMaster can specify its individual maximum through the API, but if the individual maximum is more than the global maximum, the ResourceManager overrides it.", "display_name": "ApplicationMaster Maximum Attempts", "name": "yarn_resourcemanager_am_max_retries", "value": "2" }, { "desc": "For advanced use only. A string to be inserted into nodes_exclude.txt for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt", "name": "rm_hosts_exclude_safety_valve", "value": null }, { "desc": "Enable continuous scheduling in the Fair Scheduler. When enabled, scheduling decisions are decoupled from NodeManager heartbeats, leading to faster resource allocations.", "display_name": "Enable Fair Scheduler Continuous Scheduling", "name": "yarn_scheduler_fair_continuous_scheduling_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_resource_manager_web_metric_collection", "value": "false" }, { "desc": "The periodic interval that the ResourceManager will check whether containers are still alive.", "display_name": "Container Monitor Interval", "name": "yarn_resourcemanager_container_liveness_monitor_interval_ms", "value": "600000" }, { "desc": "The maximum size, in megabytes, per log file for ResourceManager logs. Typically used by log4j or logback.", "display_name": "ResourceManager Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "resourcemanager_mapred_safety_valve", "value": null }, { "desc": "For applications that request containers on particular racks, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another rack. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. If not set, this means don't pass up any scheduling opportunities. Requires Fair Scheduler continuous scheduling to be disabled. If continuous scheduling is enabled, yarn.scheduler.fair.locality-delay-rack-ms should be used instead.", "display_name": "Fair Scheduler Rack Locality Threshold", "name": "resourcemanager_fair_scheduler_locality_threshold_rack", "value": null }, { "desc": "The class to use as the resource scheduler. FairScheduler is only supported in CDH 4.2.1 and later.", "display_name": "Scheduler Class", "name": "yarn_resourcemanager_scheduler_class", "value": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When enabled, any applications that were running on the cluster when the ResourceManager died will be recovered when the ResourceManager next starts. Note: If RM-HA is enabled, then this configuration is always enabled.", "display_name": "Enable ResourceManager Recovery", "name": "yarn_resourcemanager_recovery_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Maximum Attempts parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Maximum Attempts", "name": "role_config_suppression_yarn_resourcemanager_am_max_retries", "value": "false" }, { "desc": "The expiry interval to wait until an ApplicationMaster is considered dead.", "display_name": "ApplicationMaster Monitor Expiry", "name": "yarn_am_liveness_monitor_expiry_interval_ms", "value": "600000" }, { "desc": "Enter an XML string that represents the Capacity Scheduler configuration.", "display_name": "Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)", "name": "resourcemanager_capacity_scheduler_configuration", "value": "\n\n \n yarn.scheduler.capacity.root.queues\n default\n \n \n yarn.scheduler.capacity.root.capacity\n 100\n \n \n yarn.scheduler.capacity.root.default.capacity\n 100\n \n\n" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The largest amount of physical memory, in MiB, that can be requested for a container.", "display_name": "Container Memory Maximum", "name": "yarn_scheduler_maximum_allocation_mb", "value": "65536" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt", "name": "role_config_suppression_rm_hosts_exclude_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When enabled, if a pool's minimum share is not met for some period of time, the Fair Scheduler preempts applications in other pools. Preemption guarantees that production applications are not starved while also allowing the cluster to be used for experimental and research applications. To minimize wasted computation, the Fair Scheduler preempts the most recently launched applications.", "display_name": "Enable Fair Scheduler Preemption", "name": "resourcemanager_fair_scheduler_preemption", "value": "false" }, { "desc": "For applications that request containers on particular nodes, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another node. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. If not set, this means don't pass up any scheduling opportunities. Requires Fair Scheduler continuous scheduling to be disabled. If continuous scheduling is enabled, yarn.scheduler.fair.locality-delay-node-ms should be used instead.", "display_name": "Fair Scheduler Node Locality Threshold", "name": "resourcemanager_fair_scheduler_locality_threshold_node", "value": null }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "For applications that request containers on particular nodes, the minimum time in milliseconds the Fair Scheduler waits before accepting a placement on another node. Requires Fair Scheduler continuous scheduling to be enabled. If continuous scheduling is disabled, yarn.scheduler.fair.locality.threshold.node should be used instead.", "display_name": "Fair Scheduler Node Locality Delay", "name": "yarn_scheduler_fair_locality_delay_node_ms", "value": "2000" }, { "desc": "The minimum log level for ResourceManager logs", "display_name": "ResourceManager Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "resourcemanager_startup_tolerance_minutes", "value": "5" }, { "desc": "When enabled, ResourceManager has proxy user privileges.", "display_name": "Enable ResourceManager Proxy User Privileges", "name": "yarn_resourcemanager_proxy_user_privileges_enabled", "value": "true" }, { "desc": "An XML string that will be inserted verbatim into the Fair Scheduler allocations file. For CDH5, overrides the configuration set using the Pools configuration UI. For CDH4, this is the only way to configure the Fair Scheduler for YARN.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "resourcemanager_fair_scheduler_configuration", "value": null }, { "desc": "For advanced use only. A string to be inserted into nodes_allow.txt for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt", "name": "rm_hosts_allow_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_resource_manager_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_resource_manager_log_directory_free_space", "value": "false" }, { "desc": "The maximum number of completed applications that the ResourceManager keeps.", "display_name": "Max Completed Applications", "name": "yarn_resourcemanager_max_completed_applications", "value": "10000" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "resourcemanager_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_capacity_scheduler_configuration", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt", "name": "role_config_suppression_rm_hosts_allow_safety_valve", "value": "false" }, { "desc": "The address of the scheduler interface in the ResourceManager.", "display_name": "Scheduler Address", "name": "yarn_resourcemanager_scheduler_address", "value": "8030" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "resourcemanager_gc_duration_window", "value": "5" }, { "desc": "The timeout for the ResourceManager session with ZooKeeper. The session expires if the ZooKeeper ensemble does not hear from the ResourceManager within the specified timeout period (no heartbeat). Session expiration is managed by the ZooKeeper ensemble, not by the ResourceManager.", "display_name": "ZooKeeper Session Timeout", "name": "yarn_resourcemanager_zk_timeout_ms", "value": "60000" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_resource_manager_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "resourcemanager_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_resource_manager_gc_duration", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The smallest number of virtual CPU cores that can be requested for a container. If using the Capacity or FIFO scheduler (or any scheduler, prior to CDH 5), virtual core requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores Minimum", "name": "yarn_scheduler_minimum_allocation_vcores", "value": "1" }, { "desc": "If using the Fair Scheduler, memory requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 5.", "display_name": "Container Memory Increment", "name": "yarn_scheduler_increment_allocation_mb", "value": "512" }, { "desc": "When computing the overall ResourceManager health, consider the host's health.", "display_name": "ResourceManager Host Health Test", "name": "resourcemanager_host_health_enabled", "value": "true" }, { "desc": "When enabled, the Fair Scheduler will assign shares to individual apps based on their size, rather than providing an equal share to all apps regardless of size.", "display_name": "Fair Scheduler Size-Based Weight", "name": "resourcemanager_fair_scheduler_size_based_weight", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The address of the resource tracker interface in the ResourceManager.", "display_name": "Resource Tracker Address", "name": "yarn_resourcemanager_resource_tracker_address", "value": "8031" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_resource_manager_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Log Directory parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Log Directory", "name": "role_config_suppression_resource_manager_log_dir", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for ResourceManager", "name": "resource_manager_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "ResourceManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Enables multiple Fair Scheduler container assignments in one heartbeat, which improves cluster throughput when there are many small tasks to run.", "display_name": "Fair Scheduler Assign Multiple Tasks", "name": "resourcemanager_fair_scheduler_assign_multiple", "value": "true" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "resourcemanager_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_resourcemanager_mapred_safety_valve", "value": "false" }, { "desc": "Directory where ResourceManager will place its log files.", "display_name": "ResourceManager Log Directory", "name": "resource_manager_log_dir", "value": "/var/log/hadoop-yarn" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Number of threads to handle resource tracker calls.", "display_name": "Resource Tracker Thread Count", "name": "yarn_resourcemanager_resource_tracker_client_thread_count", "value": "50" }, { "desc": "If using the Fair Scheduler, virtual core requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 5.", "display_name": "Container Virtual CPU Cores Increment", "name": "yarn_scheduler_increment_allocation_vcores", "value": "1" }, { "desc": "The HTTPS port of the ResourceManager web application.", "display_name": "ResourceManager Web Application HTTPS Port (TLS/SSL)", "name": "resourcemanager_webserver_https_port", "value": "8090" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of ResourceManager in Bytes", "name": "resource_manager_java_heapsize", "value": "1073741824" }, { "desc": "The number of threads used to handle applications manager requests.", "display_name": "Client Thread Count", "name": "yarn_resourcemanager_client_thread_count", "value": "50" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_resource_manager_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for ResourceManager logs. Typically used by log4j or logback.", "display_name": "ResourceManager Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "ResourceManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "RESOURCEMANAGER_role_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The utilization threshold after which preemption kicks in. The utilization is computed as the maximum ratio of usage to capacity among all resources.", "display_name": "Fair Scheduler Preemption Utilization Threshold", "name": "yarn_scheduler_fair_preemption_cluster_utilization_threshold", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for ResourceManager parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for ResourceManager", "name": "role_config_suppression_resource_manager_java_opts", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The number of threads used to handle requests through the scheduler interface.", "display_name": "Scheduler Thread Count", "name": "yarn_resourcemanager_scheduler_client_thread_count", "value": "50" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "When set to true, the Fair Scheduler uses the username as the default pool name, in the event that a pool name is not specified. When set to false, all applications are run in a shared pool, called default.", "display_name": "Fair Scheduler User As Default Queue", "name": "resourcemanager_fair_scheduler_user_as_default_queue", "value": "true" }, { "desc": "Enables the health test that the ResourceManager's process state is consistent with the role configuration", "display_name": "ResourceManager Process Health Test", "name": "resourcemanager_scm_health_enabled", "value": "true" }, { "desc": "The largest number of virtual CPU cores that can be requested for a container. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores Maximum", "name": "yarn_scheduler_maximum_allocation_vcores", "value": "32" }, { "desc": "The smallest amount of physical memory, in MiB, that can be requested for a container. If using the Capacity or FIFO scheduler (or any scheduler, prior to CDH 5), memory requests will be rounded up to the nearest multiple of this number.", "display_name": "Container Memory Minimum", "name": "yarn_scheduler_minimum_allocation_mb", "value": "1024" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hue-hue_load_balancer.json0000666000175100017510000003764113245514472026600 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hue_load_balancer_scm_health", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Hue Load Balancer is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue Load Balancer TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_certificate_key", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_ssl_certificate_key", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hue_load_balancer_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hue_load_balancer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf parameter.", "display_name": "Suppress Parameter Validation: Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf", "name": "role_config_suppression_hue_load_balancer_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_ssl_certificate", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Load Balancer Environment Advanced Configuration Snippet (Safety Valve)", "name": "HUE_LOAD_BALANCER_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into httpd.conf for this role only. This can only add options to the configuration, and cannot override previously defined options.", "display_name": "Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf", "name": "hue_load_balancer_safety_valve", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hue_load_balancer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Port to use to connect to the Hue through the Load Balancer.", "display_name": "Hue Load Balancer Port", "name": "listen", "value": "8889" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hue_load_balancer_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hue_load_balancer_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Load Balancer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Load Balancer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hue_load_balancer_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Directory where Hue Load Balancer will place its log files.", "display_name": "Hue Load Balancer Log Directory", "name": "hue_load_balancer_log_dir", "value": "/var/log/hue-httpd" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When computing the overall Load Balancer health, consider the host's health.", "display_name": "Load Balancer Host Health Test", "name": "hue_load_balancer_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer Log Directory", "name": "role_config_suppression_hue_load_balancer_log_dir", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hue_load_balancer_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Enables the health test that the Load Balancer's process state is consistent with the role configuration", "display_name": "Load Balancer Process Health Test", "name": "hue_load_balancer_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hue_load_balancer_swap_memory_usage", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Hue Load Balancer is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue Load Balancer TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_certificate", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-datanode.json0000666000175100017510000012741713245514472025114 0ustar zuulzuul00000000000000[ { "desc": "Specifies the maximum number of threads to use for transferring data in and out of the DataNode.", "display_name": "Maximum Number of Transfer Threads", "name": "dfs_datanode_max_xcievers", "value": "4096" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Transceiver Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Transceiver Usage", "name": "role_health_suppression_data_node_transceivers_usage", "value": "false" }, { "desc": "In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This behavior will always be disabled for workloads that read only short sections of a block (e.g HBase random-IO workloads). This property is supported in CDH3u3 or later deployments.", "display_name": "Enable purging cache after reads", "name": "dfs_datanode_drop_cache_behind_reads", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether DataNodes should use DataNode hostnames when connecting to DataNodes for data transfer. This property is supported in CDH3u4 or later deployments.", "display_name": "Use DataNode Hostname", "name": "dfs_datanode_use_datanode_hostname", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for DataNode", "name": "datanode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Enables the health test that the DataNode's process state is consistent with the role configuration", "display_name": "DataNode Process Health Test", "name": "datanode_scm_health_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "DataNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of DataNode in Bytes", "name": "datanode_java_heapsize", "value": "1073741824" }, { "desc": "Directory where DataNode will place its log files.", "display_name": "DataNode Log Directory", "name": "datanode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_data_node_web_metric_collection", "value": "false" }, { "desc": "While reading block files, the DataNode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader's position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the DataNode will attempt to read ahead. A value of 0 disables this feature. This property is supported in CDH3u3 or later deployments.", "display_name": "Number of read ahead bytes", "name": "dfs_datanode_readahead_bytes", "value": "4194304" }, { "desc": "Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls how much DataNode volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis.", "display_name": "Available Space Policy Balanced Threshold", "name": "dfs_datanode_available_space_balanced_threshold", "value": "10737418240" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "datanode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a DataNode Data Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "DataNode Data Directory Free Space Monitoring Percentage Thresholds", "name": "datanode_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "datanode_pause_duration_window", "value": "5" }, { "desc": "In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This property is supported in CDH3u3 or later deployments.", "display_name": "Enable purging cache after writes", "name": "dfs_datanode_drop_cache_behind_writes", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory.", "display_name": "DataNode Data Directory Free Space Monitoring Absolute Thresholds", "name": "datanode_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Permissions for the directories on the local file system where the DataNode stores its blocks. The permissions must be octal. 755 and 700 are typical values.", "display_name": "DataNode Data Directory Permissions", "name": "dfs_datanode_data_dir_perm", "value": "700" }, { "desc": "Maximum amount of bandwidth that each DataNode can use for balancing. Specified in bytes per second.", "display_name": "DataNode Balancing Bandwidth", "name": "dfs_balance_bandwidthPerSec", "value": "10485760" }, { "desc": "The health test thresholds of transceivers usage in a DataNode. Specified as a percentage of the total configured number of transceivers.", "display_name": "DataNode Transceivers Usage Thresholds", "name": "datanode_transceivers_usage_thresholds", "value": "{\"critical\":\"95.0\",\"warning\":\"75.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory parameter.", "display_name": "Suppress Parameter Validation: DataNode Data Directory", "name": "role_config_suppression_dfs_data_dir_list", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_data_node_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 5, \"content\":\"Datanode registration failed\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Got a command from standby NN - ignoring command:.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Failed Volumes Tolerated Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Failed Volumes Tolerated Validator", "name": "role_config_suppression_datanode_failed_volumes_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Data Directory Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Data Directory Status", "name": "role_health_suppression_data_node_volume_failures", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the DataNode Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: DataNode Data Directory Free Space", "name": "role_health_suppression_datanode_data_directories_free_space", "value": "false" }, { "desc": "When computing the overall DataNode health, consider the host's health.", "display_name": "DataNode Host Health Test", "name": "datanode_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for DataNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for DataNode", "name": "role_config_suppression_datanode_java_opts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for DataNode logs. Typically used by log4j or logback.", "display_name": "DataNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "datanode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DateNode Plugins parameter.", "display_name": "Suppress Parameter Validation: DateNode Plugins", "name": "role_config_suppression_dfs_datanode_plugins_list", "value": "false" }, { "desc": "Whether to suppress the results of the NameNode Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Connectivity", "name": "role_health_suppression_data_node_ha_connectivity", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Port for the DataNode HTTP web UI. Combined with the DataNode's hostname to build its HTTP address.", "display_name": "DataNode HTTP Web UI Port", "name": "dfs_datanode_http_port", "value": "50075" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of free space in a DataNode. Specified as a percentage of the capacity on the DataNode.", "display_name": "DataNode Free Space Monitoring Thresholds", "name": "datanode_free_space_thresholds", "value": "{\"critical\":\"10.0\",\"warning\":\"20.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Port for DataNode's XCeiver Protocol. Combined with the DataNode's hostname to build its address.", "display_name": "DataNode Transceiver Port", "name": "dfs_datanode_port", "value": "50010" }, { "desc": "The health test thresholds of the number of blocks on a DataNode", "display_name": "DataNode Block Count Thresholds", "name": "datanode_block_count_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"500000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds of failed volumes in a DataNode.", "display_name": "DataNode Volume Failures Thresholds", "name": "datanode_volume_failures_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_data_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "The minimum log level for DataNode logs", "display_name": "DataNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_data_node_scm_health", "value": "false" }, { "desc": "DataNode Policy for picking which volume should get a new block. The Available Space policy is only available starting with CDH 4.3.", "display_name": "DataNode Volume Choosing Policy", "name": "dfs_datanode_volume_choosing_policy", "value": "org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory Permissions parameter.", "display_name": "Suppress Parameter Validation: DataNode Data Directory Permissions", "name": "role_config_suppression_dfs_datanode_data_dir_perm", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "datanode_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: DataNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: DataNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_datanode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_data_node_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Reserved Space Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Reserved Space Validator", "name": "role_config_suppression_datanode_reserved_space_validator", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "datanode_config_safety_valve", "value": null }, { "desc": "The maximum amount of memory a DataNode may use to cache data blocks in memory. Setting it to zero will disable caching.", "display_name": "Maximum Memory Used for Caching", "name": "dfs_datanode_max_locked_memory", "value": "4294967296" }, { "desc": "Enables the health test that verifies the DataNode is connected to the NameNode", "display_name": "DataNode Connectivity Health Test", "name": "datanode_connectivity_health_enabled", "value": "true" }, { "desc": "Minimum number of running threads for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Min Threadcount", "name": "dfs_thrift_threads_min", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_datanode_config_safety_valve", "value": "false" }, { "desc": "The base port where the secure DataNode web UI listens. Combined with the DataNode's hostname to build its secure web UI address.", "display_name": "Secure DataNode Web UI Port (TLS/SSL)", "name": "dfs_datanode_https_port", "value": "50475" }, { "desc": "The amount of time to wait for the DataNode to fully start up and connect to the NameNode before enforcing the connectivity check.", "display_name": "DataNode Connectivity Tolerance at Startup", "name": "datanode_connectivity_tolerance", "value": "180" }, { "desc": "Maximum number of running threads for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Max Threadcount", "name": "dfs_thrift_threads_max", "value": "20" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Free Space", "name": "role_health_suppression_data_node_free_space_remaining", "value": "false" }, { "desc": "If this configuration is enabled, the DataNode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. This property is supported in CDH3u3 or later deployments.", "display_name": "Enable immediate enqueuing of data to disk after writes", "name": "dfs_datanode_sync_behind_writes", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_data_node_swap_memory_usage", "value": "false" }, { "desc": "Comma-separated list of DataNode plug-ins to be activated. If one plug-in cannot be loaded, all the plug-ins are ignored.", "display_name": "DateNode Plugins", "name": "dfs_datanode_plugins_list", "value": "" }, { "desc": "Comma-delimited list of directories on the local file system where the DataNode stores HDFS block data. Typical values are /data/N/dfs/dn for N = 1, 2, 3.... In CDH 5.7 and higher, these directories can be optionally tagged with their storage types, e.g., [SSD]/data/1/dns/dn. HDFS supports the following storage types: [DISK], [SSD], [ARCHIVE], [RAM_DISK]. The default storage type of a directory will be [DISK] if it does not have a storage type tagged explicitly. These directories should be mounted using the noatime option, and the disks should be configured using JBOD. RAID is not recommended. Warning: Be very careful when modifying this property. Removing or changing entries can result in data loss. To hot swap drives in CDH 5.4 and higher, override the value of this property for the specific DataNode role instance that has the drive to be hot-swapped; do not modify the property value in the role group. See Configuring Hot Swap for DataNodes for more information.", "display_name": "DataNode Data Directory", "name": "dfs_data_dir_list", "value": null }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_data_node_pause_duration", "value": "false" }, { "desc": "The number of volumes that are allowed to fail before a DataNode stops offering service. By default, any volume failure will cause a DataNode to shutdown.", "display_name": "DataNode Failed Volumes Tolerated", "name": "dfs_datanode_failed_volumes_tolerated", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "If enabled, the DataNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind DataNode to Wildcard Address", "name": "dfs_datanode_bind_wildcard", "value": "false" }, { "desc": "The number of server threads for the DataNode.", "display_name": "Handler Count", "name": "dfs_datanode_handler_count", "value": "3" }, { "desc": "The maximum number of rolled log files to keep for DataNode logs. Typically used by log4j or logback.", "display_name": "DataNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "DataNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "DATANODE_role_env_safety_valve", "value": null }, { "desc": "Timeout in seconds for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Timeout", "name": "dfs_thrift_timeout", "value": "60" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_data_node_log_directory_free_space", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Block Count heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Block Count", "name": "role_health_suppression_data_node_block_count", "value": "false" }, { "desc": "Port for the various DataNode Protocols. Combined with the DataNode's hostname to build its IPC port address.", "display_name": "DataNode Protocol Port", "name": "dfs_datanode_ipc_port", "value": "50020" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: DataNode Log Directory", "name": "role_config_suppression_datanode_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of DataNode in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of DataNode in Bytes", "name": "role_config_suppression_datanode_java_heapsize", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Reserved space in bytes per volume for non Distributed File System (DFS) use.", "display_name": "Reserved Space for Non DFS Use", "name": "dfs_datanode_du_reserved", "value": "10737418240" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations.", "display_name": "Available Space Policy Balanced Preference", "name": "dfs_datanode_available_space_balanced_preference", "value": "0.75" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_data_node_unexpected_exits", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "datanode_web_metric_collection_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/impala-service.json0000666000175100017510000013554013245514472025310 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Impala Llama ApplicationMaster Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Llama ApplicationMaster Health", "name": "service_health_suppression_impala_llamas_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Bypass Hive Metastore Validator configuration validator.", "display_name": "Suppress Configuration Validator: Bypass Hive Metastore Validator", "name": "service_config_suppression_impala_bypass_hms_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "The user that this Impala's processes should run as (except Llama, which has its own user).", "display_name": "Impala System User (except Llama)", "name": "process_username", "value": "impala" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Impala events:\n

\n\n
    \n
  • userName: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • operation: the Impala operation being performed.
  • \n
  • databaseName: the databaseName for the operation.
  • \n
  • tableName: the tableName for the operation.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. Typically edited using the Rules configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Configuration Rules (Staged)", "name": "impala_schedule_rules_draft", "value": null }, { "desc": "Name of the HDFS service that this Impala service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Maximum amount of time (in milliseconds) that a request waits to be admitted before timing out. Must be a positive integer.", "display_name": "Admission Control Queue Timeout", "name": "admission_control_queue_timeout", "value": "60000" }, { "desc": "Use Dynamic Resource Pools to configure Impala admission control or RM for this Impala service. These features are only supported in Impala 1.3 or higher deployments.", "display_name": "Enable Dynamic Resource Pools", "name": "admission_control_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Local Path Access User Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Local Path Access User Validator", "name": "service_config_suppression_local_path_access_user_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Short-Circuit Read Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Short-Circuit Read Validator", "name": "service_config_suppression_impala_dfs_client_read_shortcircuit_validator", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using Sentry service instead, add Sentry service as a dependency to Impala service. Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry is supported only on Impala 1.1 or later deployments.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_impalad_sentry_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The time period over which to compute the assignment locality ratio. Specified in minutes.", "display_name": "Assignment Locality Monitoring Period", "name": "impala_assignment_locality_window", "value": "15" }, { "desc": "Timeout in milliseconds for all HBase RPCs made by Impala. Overrides configuration in HBase service.", "display_name": "HBase RPC Timeout", "name": "hbase_rpc_timeout", "value": "3000" }, { "desc": "Use debug build of Impala binaries when starting roles. Useful when performing diagnostic activities to get more information in the stacktrace or core dump.", "display_name": "Use Debug Build", "name": "use_debug_build", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Pattern parameter.", "display_name": "Suppress Parameter Validation: LDAP Pattern", "name": "service_config_suppression_ldap_bind_pattern", "value": "false" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Impala is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_private_key", "value": null }, { "desc": "Whether to suppress the results of the Impala Catalog Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Catalog Server Health", "name": "service_health_suppression_impala_catalogserver_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP Configuration Validator", "name": "service_config_suppression_impala_ldap_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala System Group (except Llama) parameter.", "display_name": "Suppress Parameter Validation: Impala System Group (except Llama)", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress the results of the Assignment Locality heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Assignment Locality", "name": "service_health_suppression_impala_assignment_locality", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Daemon Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Daemon Count Validator", "name": "service_config_suppression_impalad_count_validator", "value": "false" }, { "desc": "Specifies the set of authorized proxy users (users who can impersonate other users during authorization) and whom they are allowed to impersonate. Input is a semicolon-separated list of key=value pairs of authorized proxy users to the user(s) they can impersonate. These users are specified as a comma separated list of short usernames, or '*' to indicate all users. For example: joe=alice,bob;hue=*;admin=*. Only valid when Sentry is enabled.", "display_name": "Proxy User Configuration", "name": "impala_authorized_proxy_user_config", "value": "hue=*" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Catalog Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Catalog Server Count Validator", "name": "service_config_suppression_catalogserver_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL CA Certificate parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL CA Certificate", "name": "service_config_suppression_ssl_client_ca_certificate", "value": "false" }, { "desc": "Configures the maximum number of concurrently running queries for admission control when using a single pool. -1 indicates no limit and 0 indicates all incoming requests will be rejected. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Max Running Queries", "name": "admission_control_single_pool_max_requests", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_impala_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Llama Supported Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Llama Supported Validator", "name": "service_config_suppression_impala_llama_supported_validator", "value": "false" }, { "desc": "Name of YARN service to use for resource management integration between Impala and YARN. This service dependency and the existence of a Llama role is required for using said integration.", "display_name": "YARN Service for Resource Management", "name": "yarn_service", "value": null }, { "desc": "Name of the Sentry service that this Impala service instance depends on. If selected, Impala uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties parameter.", "display_name": "Suppress Parameter Validation: Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "service_config_suppression_navigator_lineage_client_config_safety_valve", "value": "false" }, { "desc": "Time in seconds before Impala Daemon or Catalog Server times out with the StateStore.", "display_name": "StateStoreSubscriber Timeout", "name": "statestore_subscriber_timeout", "value": "30" }, { "desc": "Configures the max memory of running queries for admission control when using a single pool. -1 or 0 indicates no limit. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Mem Limit", "name": "admission_control_single_pool_mem_limit", "value": "-1" }, { "desc": "Maximum number of HBase client retries for Impala. Used as a maximum for all operations such as fetching of the root region from the root RegionServer, getting a cell's value, and starting a row update. Overrides configuration in the HBase service.", "display_name": "Maximum HBase Client Retries", "name": "hbase_client_retries_number", "value": "3" }, { "desc": "Controls the aggregate metrics generated for Impala queries. The structure is a JSON list of the attributes to aggregate and the entities to aggregate to. For example, if the attributeName is 'hdfs_bytes_read' and the aggregationTargets is ['USER'] then the Service Monitor will create the metric 'impala_query_hdfs_bytes_read_rate' and, every ten minutes, will record the total hdfs bytes read for each user across all their Impala queries. By default it will also record the number of queries issues ('num_impala_queries_rate') for both users and pool. For a full list of the supported attributes see the Impala search page. Note that the valid aggregation targets are USER, YARN_POOL, and IMPALA (the service), and that these aggregate metrics can be viewed on both the reports and charts search pages.", "display_name": "Impala Query Aggregates", "name": "impala_query_aggregates", "value": "[\n {\n \"attributeName\": \"hdfs_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"thread_cpu_time\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"bytes_streamed\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cm_cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"query_duration\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"admission_wait\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"memory_accrual\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"thread_cpu_time\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"memory_spilled\",\n \"aggregationTargets\": [\"USER\", \"IMPALA_POOL_USER\", \"IMPALA_POOL\", \"IMPALA\", \"CLUSTER\"]\n }\n]\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Query Aggregates parameter.", "display_name": "Suppress Parameter Validation: Impala Query Aggregates", "name": "service_config_suppression_impala_query_aggregates", "value": "false" }, { "desc": "The health test thresholds of the overall Impala Llama ApplicationMaster health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Impala Llama ApplicationMasters falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Impala Llama ApplicationMasters falls below the critical threshold.", "display_name": "Healthy Impala Llama ApplicationMaster Monitoring Thresholds", "name": "impala_llamas_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala System User (except Llama) parameter.", "display_name": "Suppress Parameter Validation: Impala System User (except Llama)", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Proxy User Configuration parameter.", "display_name": "Suppress Parameter Validation: Proxy User Configuration", "name": "service_config_suppression_impala_authorized_proxy_user_config", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Daemon Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Health", "name": "service_health_suppression_impala_impalads_healthy", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Impala is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_server_certificate", "value": null }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Permissions Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Permissions Validator", "name": "service_config_suppression_short_circuit_reads_data_directory_permissions_validator", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. The value specified here is what gets deployed to all the hosts. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Configuration Rules (Deployed)", "name": "impala_schedule_rules", "value": "[]" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP Authentication", "name": "enable_ldap_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala StateStore Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala StateStore Count Validator", "name": "service_config_suppression_statestore_count_validator", "value": "false" }, { "desc": "Use Impala Admission Control to throttle Impala requests. Unless 'Enable Dynamic Resource Pools' is enabled, Impala uses a single, default pool that is configured using the Single Pool configurations below. These features are only supported in Impala 1.3 or higher deployments.", "display_name": "Enable Impala Admission Control", "name": "all_admission_control_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_ldap_domain", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_ldap_basedn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. Typically edited using the Pools configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Allocations (Staged)", "name": "impala_scheduled_allocations_draft", "value": null }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "impalad_sentry_safety_valve", "value": null }, { "desc": "Configures the maximum number of queued queries for admission control when using a single pool. -1 or 0 disables queuing, i.e. incoming requests are rejected if they can not be executed immediately. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Max Queued Queries", "name": "admission_control_single_pool_max_queued", "value": "200" }, { "desc": "The health test thresholds of the overall Impala Daemon health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Impala Daemons falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Impala Daemons falls below the critical threshold.", "display_name": "Healthy Impala Daemon Monitoring Thresholds", "name": "impala_impalads_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Enable HDFS Block Metadata API Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Enable HDFS Block Metadata API Configuration Validator", "name": "service_config_suppression_impala_hdfs_dfs_datanode_hdfs_blocks_metadata_enabled_set_validator", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Impala Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "impala_service_env_safety_valve", "value": null }, { "desc": "If checked and Impala is using Kerberos, Impala will use the hadoop.security.auth_to_local rules from HDFS configuration. The relevant HDFS configuration is derived from Additional Rules to Map Kerberos Principals to Short Names.", "display_name": "Use HDFS Rules to Map Kerberos Principals to Short Names", "name": "load_hdfs_auth_to_local_rules", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_impala_cmd_args_safety_valve", "value": "false" }, { "desc": "Number of minutes between reestablishing our ticket with the Kerberos server.", "display_name": "Kerberos Re-init Interval", "name": "kerberos_reinit_interval", "value": "60" }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with an LDAP server that was specified with ldap://. Not required when using an LDAP URL with prefix ldaps://, because that already specifies TLS. This option is also known as \"Use StartTLS\".", "display_name": "Enable LDAP TLS", "name": "enable_ldap_tls", "value": "false" }, { "desc": "When set, this parameter allows arbitrary mapping from usernames into a Distinguished Name (DN). The string specified must have a placeholder named \"#UID\" inside it, and that #UID is replaced with the username. For example, you could mimic the behavior of LDAP BaseDN by specifying \"uid=#UID,ou=People,dc=cloudera,dc=com\". When the username of \"mike\" comes in, it replaces the #UID and the result is \"uid=mike,ou=People,dc=cloudera,dc=com\". This option should be used when more control over the DN is needed. This parameter is mutually exclusive with LDAP Domain and LDAP BaseDN.", "display_name": "LDAP Pattern", "name": "ldap_bind_pattern", "value": null }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "LDAP URL", "name": "impala_ldap_uri", "value": null }, { "desc": "Controls which queries a non-admin user can see in the queries list view", "display_name": "Non-Admin Users Query List Visibility Settings", "name": "user_query_list_settings", "value": "ALL" }, { "desc": "The location on disk of the certificate, in PEM format, used to confirm the authenticity of SSL/TLS servers that the Impala daemons might connect to. Since the Impala daemons connect to each other, this should also include the CA certificate used to sign all the SSL/TLS Certificates. Without this parameter, SSL/TLS between Impala daemons will not be enabled.", "display_name": "Impala TLS/SSL CA Certificate", "name": "ssl_client_ca_certificate", "value": null }, { "desc": "Used to generate a core dump to get more information about an Impala crash. Unless otherwise configured systemwide using /proc/sys/kernel/core_pattern, the dump is generated in the 'current directory' of the Impala process (usually a subdirectory of the /var/run/cloudera-scm-agent/process directory). The core file can be very large.", "display_name": "Enable Core Dump", "name": "enable_core_dump", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Private Key Password", "name": "service_config_suppression_ssl_private_key_password", "value": "false" }, { "desc": "The password for the private key in the Impala TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Impala TLS/SSL Private Key Password", "name": "ssl_private_key_password", "value": null }, { "desc": "The minimum number of assignments that must occur during the test time period before the threshold values will be checked. Until this number of assignments have been observed in the test time period the health test will be disabled.", "display_name": "Assignment Locality Minimum Assignments", "name": "impala_assignment_locality_minimum", "value": "10" }, { "desc": "Enable collection of lineage from the service's roles.", "display_name": "Enable Lineage Collection", "name": "navigator_lineage_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Impala DSSD Short-Circuit Read with Kerberos Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala DSSD Short-Circuit Read with Kerberos Validator", "name": "service_config_suppression_impala_dssd_kerberos_scr_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Impala StateStore Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala StateStore Health", "name": "service_health_suppression_impala_statestore_health", "value": "false" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Allocations (Deployed)", "name": "impala_scheduled_allocations", "value": "{\"defaultFairSharePreemptionThreshold\":null,\"defaultFairSharePreemptionTimeout\":null,\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":null,\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":null,\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"allowPreemptionFrom\":null,\"fairSharePreemptionThreshold\":null,\"fairSharePreemptionTimeout\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null,\"type\":null}],\"schedulablePropertiesList\":[{\"impalaDefaultQueryMemLimit\":null,\"impalaDefaultQueryOptions\":null,\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"impalaQueueTimeout\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null,\"type\":null}],\"userMaxAppsDefault\":null,\"users\":[]}" }, { "desc": "When computing the overall IMPALA health, consider Impala Catalog Server's health", "display_name": "Impala Catalog Server Role Health Test", "name": "impala_catalogserver_health_enabled", "value": "true" }, { "desc": "Name of the HBase service that this Impala service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Server Private Key File (PEM Format)", "name": "service_config_suppression_ssl_private_key", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Impala events:\n

\n\n
    \n
  • operation: the Impala operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • databaseName: the database affected by the operation.
  • \n
  • tableName: the table affected by the operation.
  • \n
  • objectType: the type of object affected by the operation.
  • \n
  • privilege: the privilege associated with the operation.
  • \n
\n\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Name of the Hive service that this Impala service instance depends on", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CA Certificate File Validator configuration validator.", "display_name": "Suppress Configuration Validator: CA Certificate File Validator", "name": "service_config_suppression_pem_ca_cert_recommended_for_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Llama ApplicationMaster Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Llama ApplicationMaster Count Validator", "name": "service_config_suppression_llama_count_validator", "value": "false" }, { "desc": "This parameter is useful when authenticating against a non Active Directory server such as OpenLDAP. When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,<this parameter>. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server will look like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is mutually exclusive with Active Directory Domain.", "display_name": "LDAP BaseDN", "name": "ldap_baseDN", "value": null }, { "desc": "Name of the ZooKeeper service to use for leader election and fencing when Llama is configured for high availability. This service dependency is required when more than one Llama role is present.", "display_name": "ZooKeeper Service for Llama HA", "name": "zookeeper_service", "value": null }, { "desc": "The health test thresholds for the assignment locality health test. Specified as a percentage of total assignments.", "display_name": "Assignment Locality Ratio Thresholds", "name": "impala_assignment_locality_thresholds", "value": "{\"critical\":\"5.0\",\"warning\":\"80.0\"}" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "impala" }, { "desc": "The group that this Impala's processes should run as (except Llama, which has its own group).", "display_name": "Impala System Group (except Llama)", "name": "process_groupname", "value": "impala" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala TLS/SSL Server Certificate File (PEM Format)", "name": "service_config_suppression_ssl_server_certificate", "value": "false" }, { "desc": "Controls which queries admin users can see in the queries list view", "display_name": "Admin Users Query List Visibility Settings", "name": "admin_query_list_settings", "value": "ALL" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Short-Circuit Read Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Short-Circuit Read Validator", "name": "service_config_suppression_impala_dssd_shortcircuit_read_enable_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Encrypt communication between clients and Impala using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Impala", "name": "client_services_ssl_enabled", "value": "false" }, { "desc": "This parameter is useful when authenticating against an Active Directory server. This value is appended to all usernames before authenticating against AD. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to AD. If this field is unset, the username remains unaltered before being passed to AD.", "display_name": "Active Directory Domain", "name": "ldap_domain", "value": null }, { "desc": "When computing the overall IMPALA health, consider Impala StateStore's health", "display_name": "Impala StateStore Role Health Test", "name": "impala_statestore_health_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Impala Daemon command-line flags. Applies to all roles in this service. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "impala_cmd_args_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_impala_ldap_uri", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Sentry Validator", "name": "service_config_suppression_impala_sentry_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.lineage.client.properties.", "display_name": "Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "navigator_lineage_client_config_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hive-hivemetastore.json0000666000175100017510000006243113245514472026215 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Hive Metastore Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Hive Metastore Server", "name": "role_config_suppression_hive_metastore_java_opts", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "hivemetastore_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hivemetastore_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Metastore Max Message Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Metastore Max Message Size Validator", "name": "role_config_suppression_hive_metastore_max_message_size_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hivemetastore_log_directory_free_space", "value": "false" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "hivemetastore_pause_duration_window", "value": "5" }, { "desc": "Maximum number of worker threads in the Hive Metastore Server's thread pool", "display_name": "Max Hive Metastore Server Threads", "name": "hive_metastore_max_threads", "value": "100000" }, { "desc": "Enables the health test that checks that basic Hive Metastore operations succeed", "display_name": "Hive Metastore Canary Health Test", "name": "metastore_canary_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hivemetastore_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Hive Metastore Server", "name": "hive_metastore_java_opts", "value": "-XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Time-to-live in seconds for notifications present in NOTIFICATION_LOG. Only used when Enable Stored Notifications in Database is enabled.", "display_name": "Time-to-live for Database Notifications", "name": "hive_metastore_event_db_listener_timetolive", "value": "172800" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_metastore_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hivemetastore_swap_memory_usage", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Hive Metastore Server logs. Typically used by log4j or logback.", "display_name": "Hive Metastore Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hivemetastore_scm_health", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enable stored notifications of metadata changes. When enabled, each metadata change will be stored in NOTIFICATION_LOG.", "display_name": "Enable Stored Notifications in Database", "name": "hive_enable_db_notification", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "When computing the overall Hive Metastore Server health, consider the host's health.", "display_name": "Hive Metastore Server Host Health Test", "name": "hivemetastore_host_health_enabled", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Controls whether the Hive metrics subsystem is enabled for the role.", "display_name": "Enable Metrics Subsystem", "name": "hive_metrics_enabled", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Hive Metastore Server in Bytes", "name": "hive_metastore_java_heapsize", "value": "8589934592" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hivemetastore_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hivemetastore_heap_dump_directory_free_space", "value": "false" }, { "desc": "Port on which Hive Metastore Server will listen for connections.", "display_name": "Hive Metastore Server Port", "name": "hive_metastore_port", "value": "9083" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_metastore_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Hive Metastore Server logs. Typically used by log4j or logback.", "display_name": "Hive Metastore Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_hivemetastore_pause_duration", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Hive Metastore Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Metastore Canary", "name": "role_health_suppression_hivemetastore_canary_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hive_metrics_sample_file_location", "value": "false" }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hive/metrics-hivemetastore/metrics.log. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Location", "name": "hive_metrics_sample_file_location", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hivemetastore_unexpected_exits", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Logging Frequency", "name": "hive_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_metastore_config_safety_valve", "value": "false" }, { "desc": "Maximum message size Hive MetaStore will accept.", "display_name": "Max Message Size for Hive MetaStore", "name": "hive_metastore_server_max_message_size", "value": "104857600" }, { "desc": "Enables the health test that the Hive Metastore Server's process state is consistent with the role configuration", "display_name": "Hive Metastore Server Process Health Test", "name": "hivemetastore_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Log Directory", "name": "role_config_suppression_hive_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Minimum number of worker threads in the Hive Metastore Server's thread pool", "display_name": "Min Hive Metastore Server Threads", "name": "hive_metastore_min_threads", "value": "200" }, { "desc": "Directory where Hive Metastore Server will place its log files.", "display_name": "Hive Metastore Server Log Directory", "name": "hive_log_dir", "value": "/var/log/hive" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The delegation token store implementation class. Use DBTokenStore for Highly Available Metastore Configuration.", "display_name": "Hive Metastore Delegation Token Store", "name": "hive_metastore_delegation_token_store", "value": "org.apache.hadoop.hive.thrift.MemoryTokenStore" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The minimum log level for Hive Metastore Server logs", "display_name": "Hive Metastore Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_metastore_env_safety_valve", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hbase-service.json0000666000175100017510000013523113245514472025124 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hbase" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for HBase events:\n

\n
    \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • username: the user performing the action.
  • \n
  • tableName: the table affected by the operation.
  • \n
  • family: the column family affected by the operation.
  • \n
  • qualifier: the qualifier the operation.
  • \n
  • action: the action being performed.
  • \n
\n\n

\nThe default HBase audit event filter discards events that affect the internal\n-ROOT-, .META. and _acl_ tables.\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default HBase audit event filter discards events that affect the \",\n \"internal -ROOT-, .META. and _acl_ tables.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"tableName\", \"match\" : \"(?:-ROOT-|.META.|_acl_|hbase:meta|hbase:acl)\" }\n ]\n }\n ]\n}\n" }, { "desc": "Enables the canary that checks HBase region availability by scanning a row from every region.", "display_name": "HBase Region Health Canary", "name": "hbase_region_health_canary_enabled", "value": "true" }, { "desc": "Name of the HDFS service that this HBase service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Period of time, in milliseconds, to pause between connection retries to ZooKeeper. Used together with ${zookeeper.retries} in an exponential backoff fashion when making queries to ZooKeeper.", "display_name": "ZooKeeper Connection Retry Pause Duration", "name": "zookeeper_pause", "value": null }, { "desc": "When computing the overall HBase cluster health, consider the active HBase Master's health.", "display_name": "Active Master Health Test", "name": "hbase_master_health_enabled", "value": "true" }, { "desc": "Size of the threadpool used for hedged reads in hdfs clients. If a read from a block is slow, a parallel 'hedged' read will be started against a different block replica. The first one to return with a result is used while the other one is cancelled. This 'hedged' read feature helps rein in the outliers. A value of zero disables the feature.", "display_name": "HDFS Hedged Read Threadpool Size", "name": "hbase_server_dfs_client_hedged_read_threadpool_size", "value": "0" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hbase/audit" }, { "desc": "Use this to enable Http server usage on thrift, which is mainly needed for \"doAs\" functionality.", "display_name": "Enable HBase Thrift Http Server", "name": "hbase_thriftserver_http", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase User to Impersonate parameter.", "display_name": "Suppress Parameter Validation: HBase User to Impersonate", "name": "service_config_suppression_hbase_user_to_impersonate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the AWS S3 Secret Access Key for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: AWS S3 Secret Access Key for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_secret_access_key", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Timeout for all HBase RPCs in milliseconds.", "display_name": "RPC Timeout", "name": "hbase_rpc_timeout", "value": "60000" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "hbase_ssl_server_safety_valve", "value": null }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hbase_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "service_config_suppression_hbase_service_config_safety_valve", "value": "false" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "The number of times to retry connections to ZooKeeper. Used for reading and writing root region location. Used together with ${zookeeper.pause} in an exponential backoff fashion when making queries to ZooKeeper.", "display_name": "ZooKeeper Connection Retries", "name": "zookeeper_retries", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Timeout for graceful shutdown of this HBase service. Once this timeout is reached, any remaining running roles are abruptly shutdown. A value of 0 means no timeout.", "display_name": "Graceful Shutdown Timeout", "name": "hbase_graceful_stop_timeout", "value": "180" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_hbase_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The health test thresholds of the overall RegionServer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" RegionServers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" RegionServers falls below the critical threshold.", "display_name": "Healthy RegionServer Monitoring Thresholds", "name": "hbase_regionservers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress the results of the RegionServer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: RegionServer Health", "name": "service_health_suppression_hbase_region_servers_healthy", "value": "false" }, { "desc": "An alert is published if the HBase region health canary detects at least this many unhealthy regions. This setting takes precedence over the hbase_canary_alert_unhealthy_region_percent_threshold config.", "display_name": "HBase Canary Unhealthy Region Count Alert Threshold", "name": "hbase_canary_alert_unhealthy_region_count_threshold", "value": null }, { "desc": "Set to true to cause the hosting server (Master or RegionServer) to abort if a coprocessor throws a Throwable object that is not IOException or a subclass of IOException. Setting it to true might be useful in development environments where one wants to terminate the server as soon as possible to simplify coprocessor failure analysis.", "display_name": "HBase Coprocessor Abort on Error", "name": "hbase_coprocessor_abort_on_error", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_hbase_core_site_safety_valve", "value": "false" }, { "desc": "The user the management services impersonate when connecting to HBase. If no value is specified, the HBase superuser is used.", "display_name": "HBase User to Impersonate", "name": "hbase_user_to_impersonate", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Amazon S3 Path for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: Amazon S3 Path for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_path", "value": "false" }, { "desc": "Timeout (in ms) for the distributed log splitting manager to receive response from a worker.", "display_name": "SplitLog Manager Timeout", "name": "hbase_service_splitlog_manager_timeout", "value": "120000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Scheduler Pool for Remote Snapshots in AWS S3 parameter.", "display_name": "Suppress Parameter Validation: Scheduler Pool for Remote Snapshots in AWS S3", "name": "service_config_suppression_hbase_snapshot_s3_scheduler_pool", "value": "false" }, { "desc": "Allow indexing of tables in HBase by Lily HBase Indexer. Note: Replication must be enabled for indexing to work.", "display_name": "Enable Indexing", "name": "hbase_enable_indexing", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Superusers parameter.", "display_name": "Suppress Parameter Validation: HBase Superusers", "name": "service_config_suppression_hbase_superuser", "value": "false" }, { "desc": "The password for the HBase JKS keystore file.", "display_name": "HBase TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Ratio of Lily HBase Indexers used by each HBase RegionServer while doing replication.", "display_name": "Replication Source Ratio", "name": "hbase_replication_source_ratio", "value": "1.0" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "hbase_core_site_safety_valve", "value": null }, { "desc": "The timeout before injecting a snapshot timeout error when waiting for a snapshot completion.", "display_name": "HBase Master Snapshot Waiting Timeout", "name": "hbase_snapshot_master_timeoutMillis", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Authentication And Authorization Validation configuration validator.", "display_name": "Suppress Configuration Validator: HBase Authentication And Authorization Validation", "name": "service_config_suppression_hbase_authentication_and_authorization_validator", "value": "false" }, { "desc": "The maximum amount of time the Hbase RegionServer waits for a snapshot to complete.", "display_name": "HBase RegionServer Snapshot Timeout", "name": "hbase_snapshot_region_timeout", "value": "60000" }, { "desc": "If this is set to \"kerberos\", HBase REST Server will authenticate its clients. HBase Proxy User Hosts and Groups should be configured to allow specific users to access HBase through REST Server.", "display_name": "HBase REST Authentication", "name": "hbase_restserver_security_authentication", "value": "simple" }, { "desc": "Specifies the combined maximum allowed size of a KeyValue instance. This option configures an upper boundary for a single entry saved in a storage file. This option prevents a region from splitting if the data is too large. Set this option to a fraction of the maximum region size. To disable this check, use a value of zero or less.", "display_name": "Maximum Size of HBase Client KeyValue", "name": "hbase_client_keyvalue_maxsize", "value": "10485760" }, { "desc": "Enable snapshots. Disabling snapshots requires deletion of all snapshots before restarting the HBase master; the HBase master will not start if snapshots are disabled and snapshots exist.", "display_name": "Enable Snapshots", "name": "hbase_snapshot_enabled", "value": "true" }, { "desc": "Maximum number of client retries. Used as a maximum for all operations such as fetching of the root region from the root RegionServer, getting a cell's value, and starting a row update.", "display_name": "Maximum HBase Client Retries", "name": "hbase_client_retries_number", "value": "35" }, { "desc": "Set to true to use HBase Secure RPC Engine for remote procedure calls (RPC). This is only effective in simple authentication mode. Does not provide authentication for RPC calls, but provides user information in the audit logs. Changing this setting requires a restart of this and all dependent services and redeployment of client configurations, along with a restart of the Service Monitor management role.", "display_name": "HBase Secure RPC Engine", "name": "hbase_secure_rpc_engine", "value": "false" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "zookeeper.recovery.retry0zookeeper.recovery.retry.intervalmill3000hbase.zookeeper.recoverable.waittime1000zookeeper.session.timeout30000hbase.rpc.timeout10000hbase.client.retries.number1hbase.client.rpc.maxattempts1hbase.client.operation.timeout10000" }, { "desc": "Access key ID required to access Amazon S3 to store remote snapshots.", "display_name": "Amazon S3 Access Key ID for Remote Snapshots", "name": "hbase_snapshot_s3_access_key_id", "value": null }, { "desc": "Allow HBase tables to be replicated.", "display_name": "Enable Replication", "name": "hbase_enable_replication", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Amazon S3 Access Key ID for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: Amazon S3 Access Key ID for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_access_key_id", "value": "false" }, { "desc": "Whether to suppress the results of the HBase Master Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HBase Master Health", "name": "service_health_suppression_hbase_master_health", "value": "false" }, { "desc": "Enable HBase authorization.", "display_name": "HBase Secure Authorization", "name": "hbase_security_authorization", "value": "false" }, { "desc": "Period of time, in milliseconds, to pause between searches for work. Used as a sleep interval by service threads such as a META scanner and log roller.", "display_name": "HBase Server Thread Wake Frequency", "name": "hbase_server_thread_wakefrequency", "value": "10000" }, { "desc": "Write buffer size in bytes. A larger buffer requires more memory on both the client and the server because the server instantiates the passed write buffer to process it but reduces the number of remote procedure calls (RPC). To estimate the amount of server memory used, multiply the value of 'hbase.client.write.buffer' by the value of 'hbase.regionserver.handler.count'.", "display_name": "HBase Client Write Buffer", "name": "hbase_client_write_buffer", "value": "2097152" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Root Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Root Directory", "name": "service_config_suppression_hdfs_rootdir", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the RegionServer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: RegionServer Count Validator", "name": "service_config_suppression_regionserver_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the ZooKeeper Max Session Timeout Validator configuration validator.", "display_name": "Suppress Configuration Validator: ZooKeeper Max Session Timeout Validator", "name": "service_config_suppression_zookeeper_max_session_timeout_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode Rootserver parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode Rootserver", "name": "service_config_suppression_zookeeper_znode_rootserver", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode Parent parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode Parent", "name": "service_config_suppression_zookeeper_znode_parent", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Region Health Canary Exclude Tables parameter.", "display_name": "Suppress Parameter Validation: HBase Region Health Canary Exclude Tables", "name": "service_config_suppression_hbase_region_health_canary_exclude_tables", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HBase Proxy User Groups", "name": "service_config_suppression_hbase_proxy_user_groups_list", "value": "false" }, { "desc": "The root znode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file paths are configured with a relative path, so they will all go under this directory unless changed.", "display_name": "ZooKeeper Znode Parent", "name": "zookeeper_znode_parent", "value": "/hbase" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "AWS secret access key required to access S3 to store remote snapshots.", "display_name": "AWS S3 Secret Access Key for Remote Snapshots", "name": "hbase_snapshot_s3_secret_access_key", "value": null }, { "desc": "Number of rows to fetch when calling next on a scanner if it is not served from memory. Higher caching values enable faster scanners but require more memory and some calls of next may take longer when the cache is empty.", "display_name": "HBase Client Scanner Caching", "name": "hbase_client_scanner_caching", "value": "100" }, { "desc": "Tables to exclude in the HBase Region Health Canary which will scan a row from every region.", "display_name": "HBase Region Health Canary Exclude Tables", "name": "hbase_region_health_canary_exclude_tables", "value": "" }, { "desc": "The maximum amount of time the HBase master waits for a snapshot to complete.", "display_name": "HBase Master Snapshot Timeout", "name": "hbase_snapshot_master_timeout_millis", "value": "60000" }, { "desc": "Amazon S3 path where remote snapshots should be stored.", "display_name": "Amazon S3 Path for Remote Snapshots", "name": "hbase_snapshot_s3_path", "value": null }, { "desc": "Start a process to periodically check that RegionServer is alive when RegionServer is started. Note: This canary is different from the Cloudera Service Monitoring canary and is provided by the HBase service itself.", "display_name": "Enable HBase Canary", "name": "hbase_regionserver_enable_canary", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "true" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase is acting as a TLS/SSL server.", "display_name": "HBase TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Path to ZooKeeper Node holding root region location. This is written by the HBase Master and read by clients and RegionServers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, the root location is stored at /hbase/root-region-server.", "display_name": "ZooKeeper Znode Rootserver", "name": "zookeeper_znode_rootserver", "value": "root-region-server" }, { "desc": "When computing the overall HBase cluster health, consider the health of the backup HBase Masters.", "display_name": "Backup Masters Health Test", "name": "hbase_backup_masters_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "An alert is published if the HBase region health canary runs slowly.", "display_name": "HBase Region Health Canary Slow Run Alert Enabled", "name": "hbase_region_health_canary_slow_run_alert_enabled", "value": "true" }, { "desc": "Name of the scheduler pool to use for MR jobs created during export/import of remote snapshots in AWS S3.", "display_name": "Scheduler Pool for Remote Snapshots in AWS S3", "name": "hbase_snapshot_s3_scheduler_pool", "value": null }, { "desc": "Configure the type of encrypted communication to be used with RPC.", "display_name": "HBase Transport Security", "name": "hbase_rpc_protection", "value": "authentication" }, { "desc": "An alert is published if the HBase region health canary detects at least this percentage of total regions are unhealthy. This threshold is used if the explicit count is not set via the hbase_canary_alert_unhealthy_region_count_threshold config.", "display_name": "HBase Canary Unhealthy Region Percentage Alert Threshold", "name": "hbase_canary_alert_unhealthy_region_percent_threshold", "value": "0.1" }, { "desc": "Duration to wait before starting up a 'hedged' read.", "display_name": "HDFS Hedged Read Delay Threshold", "name": "hbase_server_dfs_client_hedged_read_threshold_millis", "value": "500" }, { "desc": "Enable TLS/SSL encryption for HBase web UIs.", "display_name": "Web UI TLS/SSL Encryption Enabled", "name": "hbase_hadoop_ssl_enabled", "value": "false" }, { "desc": "List of users or groups, who are allowed full privileges, regardless of stored ACLs, across the cluster. Only used when HBase security is enabled.", "display_name": "HBase Superusers", "name": "hbase_superuser", "value": "" }, { "desc": "Use this to allow proxy users on thrift gateway, which is mainly needed for \"doAs\" functionality.", "display_name": "Enable HBase Thrift Proxy Users", "name": "hbase_thriftserver_support_proxyuser", "value": "false" }, { "desc": "Maximum number of hlog entries to replicate in one go. If this is large, and a consumer takes a while to process the events, the HBase RPC call will time out.", "display_name": "Replication Batch Size", "name": "hbase_replication_source_nb_capacity", "value": "1000" }, { "desc": "Whether to suppress configuration warnings produced by the Master Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Master Count Validator", "name": "service_config_suppression_master_count_validator", "value": "false" }, { "desc": "ZooKeeper session timeout in milliseconds. HBase passes this to the ZooKeeper quorum as the suggested maximum time for a session. See http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions The client sends a requested timeout, the server responds with the timeout that it can give the client.", "display_name": "ZooKeeper Session Timeout", "name": "zookeeper_session_timeout", "value": "60000" }, { "desc": "Enable HBase row-level authorization.", "display_name": "HBase Row-Level Authorization", "name": "hbase_row_level_authorization", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HBase user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'. Note: This property is used only if HBase REST/Thrift Server Authentication is enabled.", "display_name": "HBase Proxy User Hosts", "name": "hbase_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HBase Proxy User Hosts", "name": "service_config_suppression_hbase_proxy_user_hosts_list", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the HBase user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'. Note: This property is used only if HBase REST/Thrift Server Authentication is enabled.", "display_name": "HBase Proxy User Groups", "name": "hbase_proxy_user_groups_list", "value": "*" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • username: treats the field value as a user name, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can used to compare HBase events:\n

\n\n
    \n
  • operation: the HBase operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • tableName: the name of the table affected by the operation.
  • \n
  • family: the column family affected by the operation.
  • \n
  • qualifier: the qualifier of the operation.
  • \n
\n\n

\nThe default event tracker for HBase services defines equality by comparing the\nusername, operation, table name, family, and qualifier of the events.\n

\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": "{\n \"comment\" : [\n \"The default event tracker for HBase services defines equality by \",\n \"comparing the username, operation, table name, family, and qualifier of \",\n \"the events.\"\n ],\n \"timeToLive\" : 60000,\n \"fields\" : [\n { \"type\": \"value\", \"name\" : \"tableName\" },\n { \"type\": \"value\", \"name\" : \"family\" },\n { \"type\": \"value\", \"name\" : \"qualifier\" },\n { \"type\": \"value\", \"name\" : \"operation\" },\n { \"type\": \"username\", \"name\" : \"username\" }\n ]\n}\n" }, { "desc": "The tolerance window that will be used in HBase service tests that depend on detection of the active HBase Master.", "display_name": "HBase Active Master Detection Window", "name": "hbase_active_master_detecton_window", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Thrift Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Thrift Server Count Validator", "name": "service_config_suppression_hbasethriftserver_count_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_service_config_safety_valve", "value": null }, { "desc": "Choose the authentication mechanism used by HBase.", "display_name": "HBase Secure Authentication", "name": "hbase_security_authentication", "value": "simple" }, { "desc": "A general client pause time value. Used mostly as a time period to wait before retrying operations such as a failed get or region lookup.", "display_name": "HBase Client Pause", "name": "hbase_client_pause", "value": "100" }, { "desc": "Name of the ZooKeeper service that this HBase service instance depends on.", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "If this is set, HBase Thrift Server authenticates its clients. HBase Proxy User Hosts and Groups should be configured to allow specific users to access HBase through Thrift Server.", "display_name": "HBase Thrift Authentication", "name": "hbase_thriftserver_security_authentication", "value": "none" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hbase" }, { "desc": "Whether to suppress configuration warnings produced by the HBase REST Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase REST Server Count Validator", "name": "service_config_suppression_hbaserestserver_count_validator", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hbase" }, { "desc": "The HDFS directory shared by HBase RegionServers.", "display_name": "HDFS Root Directory", "name": "hdfs_rootdir", "value": "/hbase" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hbase_service_env_safety_valve", "value": null }, { "desc": "Whether the step to reload regions back onto the original RegionServers should be skipped during rolling restart. This can be used to increase the speed of rolling restart or upgrade operations, but can result in regions being moved multiple times, decreasing performance for clients during rolling restart.", "display_name": "Skip Region Reload During Rolling Restart", "name": "hbase_skip_reload_during_rr", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HBase DSSD Short-Circuit Read with Kerberos Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase DSSD Short-Circuit Read with Kerberos Validator", "name": "service_config_suppression_hbase_dssd_kerberos_scr_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase TLS/SSL Server JKS Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-gateway.json0000666000175100017510000001403613245514472024766 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_hdfs_client_config_safety_valve", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hadoop" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Trash Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Trash Enabled Validator", "name": "role_config_suppression_hdfs_trash_disabled_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether HDFS clients will use the legacy block reader.", "display_name": "Use Legacy Blockreader", "name": "dfs_client_use_legacy_blockreader", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh", "display_name": "HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "hdfs_client_env_safety_valve", "value": null }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for hdfs-site.xml.", "display_name": "HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "hdfs_client_config_safety_valve", "value": null }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "false" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hbase_client_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Move deleted files to the trash so that they can be recovered if necessary. This client side configuration takes effect only if the HDFS service-wide trash is disabled (NameNode Filesystem Trash Interval set to 0) and is ignored otherwise. The trash is not automatically emptied when enabled with this configuration.", "display_name": "Use Trash", "name": "dfs_client_use_trash", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hdfs_client_java_heapsize", "value": "268435456" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "role_config_suppression_hdfs_client_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hbase_client_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/cdh_config.sh0000777000175100017510000000010113245514472024116 0ustar zuulzuul00000000000000#!/bin/bash tox -evenv -- python $(dirname $0)/cdh_config.py $* sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/zookeeper-server.json0000666000175100017510000011571213245514472025715 0ustar zuulzuul00000000000000[ { "desc": "The port to monitor for inter-server communication", "display_name": "Quorum Port", "name": "quorumPort", "value": "3181" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Log Directory parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Log Directory", "name": "role_config_suppression_zk_server_log_dir", "value": "false" }, { "desc": "Specifies the name of the user that has read-only privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Name of User with Read-Only access to the JMX Agent", "name": "jmx_passwd_file_readonly_user", "value": "monitorRole" }, { "desc": "Enables the health test that the Server's process state is consistent with the role configuration", "display_name": "Server Process Health Test", "name": "zookeeper_server_scm_health_enabled", "value": "true" }, { "desc": "The address (IPv4, IPv6, or hostname) to monitor for client connections. This is the address that clients attempt to connect to. This setting is optional, because by default, ZooKeeper binds in such a way that any connection to the client port for any address/interface/NIC on the server will be accepted.", "display_name": "Client Port Address", "name": "clientPortAddress", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Specifies the password of the user that has read-only privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Password of User with Read-Only Access to the JMX agent", "name": "jmx_passwd_file_readonly_user_password", "value": "MONITOR" }, { "desc": "Whether to suppress the results of the Quorum Membership heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Quorum Membership", "name": "role_health_suppression_zookeeper_server_quorum_membership", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Name of User with Read-Write Access to the JMX Agent parameter.", "display_name": "Suppress Parameter Validation: Name of User with Read-Write Access to the JMX Agent", "name": "role_config_suppression_jmx_passwd_file_readwrite_user", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health check thresholds of the weighted average size of the ZooKeeper Server connection count over a recent period. See ZooKeeper Server Connection Count Monitoring Period.", "display_name": "ZooKeeper Server Connection Count Thresholds", "name": "zookeeper_server_connection_count_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into zoo.cfg for this role only.", "display_name": "Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg", "name": "zookeeper_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Transaction Log Directory.", "display_name": "Transaction Log Directory Free Space Monitoring Absolute Thresholds", "name": "zookeeper_server_data_log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The port to monitor for client connections. This is the port that clients attempt to connect to.", "display_name": "Client Port", "name": "clientPort", "value": "2181" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg parameter.", "display_name": "Suppress Parameter Validation: Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg", "name": "role_config_suppression_zookeeper_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Password of User with Read-Only Access to the JMX agent parameter.", "display_name": "Suppress Parameter Validation: Password of User with Read-Only Access to the JMX agent", "name": "role_config_suppression_jmx_passwd_file_readonly_user_password", "value": "false" }, { "desc": "The port used by the ZooKeeper Server's RMI registry. This is required to enable JMX access through RMI which is required for Cloudera Manager ZooKeeper monitoring. This is added as \"-Dcom.sun.management.jmxremote.port\" to the ZooKeeper Server's JVM command line.", "display_name": "JMX Remote Port", "name": "server_jmx_agent_port", "value": "9010" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_zookeeper_server_gc_duration", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_zookeeper_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Unique identifier for each ZooKeeper server, typically starts at 1", "display_name": "ZooKeeper Server ID", "name": "serverId", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "zookeeper_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_zookeeper_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_zookeeper_server_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The disk location that ZooKeeper will use to store its transaction logs.", "display_name": "Transaction Log Directory", "name": "dataLogDir", "value": "/var/lib/zookeeper" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Data Directory.", "display_name": "Data Directory Free Space Monitoring Absolute Thresholds", "name": "zookeeper_server_data_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The maximum size, in megabytes, per log file for Server logs. Typically used by log4j or logback.", "display_name": "Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Password of user with read-write access to the JMX agent parameter.", "display_name": "Suppress Parameter Validation: Password of user with read-write access to the JMX agent", "name": "role_config_suppression_jmx_passwd_file_readwrite_user_password", "value": "false" }, { "desc": "The minimum session timeout, in milliseconds, that the ZooKeeper Server will allow the client to negotiate", "display_name": "Minimum Session Timeout", "name": "minSessionTimeout", "value": "4000" }, { "desc": "The percentage thresholds of the ratio of the maximum request latency to the maximum client-negotiable session timeout since the server was started.", "display_name": "Maximum Latency Monitoring Thresholds", "name": "zookeeper_server_max_latency_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"75.0\"}" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Enables authentication when interacting with the JMX agent on the ZooKeeper server.", "display_name": "Enable Authenticated Communication with the JMX Agent", "name": "enable_jmx_authentication", "value": "false" }, { "desc": "Enables the quorum membership check for this ZooKeeper Server.", "display_name": "Enable the Quorum Membership Check", "name": "zookeeper_server_quorum_membership_enabled", "value": "true" }, { "desc": "The port used by the ZooKeeper Server's RMI server to handle JMX RMI requests. This is added as \"-Dcom.sun.management.jmxremote.rmi.port=<port>\" to the ZooKeeper Server's JVM command line. This has an effect only in Oracle JDK 7u4 and higher. If the setting is left blank, the JMX Remote Port value is used. If set to 0 or -1, this setting is ignored. When this setting is not in effect, the JVM uses a random port for the RMI server.", "display_name": "JMX RMI Server Port", "name": "server_jmx_rmi_port", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_zookeeper_server_host_health", "value": "false" }, { "desc": "Directory where ZooKeeper will place its log files.", "display_name": "ZooKeeper Log Directory", "name": "zk_server_log_dir", "value": "/var/log/zookeeper" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Transaction Log Directory parameter.", "display_name": "Suppress Parameter Validation: Transaction Log Directory", "name": "role_config_suppression_datalogdir", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The minimum log level for Server logs", "display_name": "Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SERVER_role_env_safety_valve", "value": null }, { "desc": "The period to review when computing the moving average of the outstanding requests queue size. Specified in minutes.", "display_name": "ZooKeeper Server Outstanding Requests Monitoring Period", "name": "zookeeper_server_outstanding_requests_window", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Port Address parameter.", "display_name": "Suppress Parameter Validation: Client Port Address", "name": "role_config_suppression_clientportaddress", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Zookeeper Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Zookeeper Server", "name": "role_config_suppression_zk_server_java_opts", "value": "false" }, { "desc": "The disk location that ZooKeeper will use to store its database snapshots.", "display_name": "Data Directory", "name": "dataDir", "value": "/var/lib/zookeeper" }, { "desc": "The tolerance window that will be used in the detection of a ZooKeeper server's membership in a quorum. Specified in minutes.", "display_name": "Quorum Membership Detection Window", "name": "zookeeper_server_quorum_membership_detection_window", "value": "3" }, { "desc": "The health check thresholds of the weighted average size of the ZooKeeper Server outstanding requests queue over a recent period. See ZooKeeper Server Outstanding Requests Monitoring Period.", "display_name": "ZooKeeper Server Outstanding Requests Thresholds", "name": "zookeeper_server_outstanding_requests_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_server_role_env_safety_valve", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Specifies the name of the user that has read-write privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Name of User with Read-Write Access to the JMX Agent", "name": "jmx_passwd_file_readwrite_user", "value": "controlRole" }, { "desc": "The port to monitor for leadership election", "display_name": "Election Port", "name": "electionPort", "value": "4181" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The maximum number of concurrent connections (at the socket level) that a single client, identified by the IP address, may make to a single member of the ZooKeeper ensemble. This setting is used to prevent certain classes of DoS attacks, including file descriptor exhaustion. To remove the limit on concurrent connections, set this value to 0.", "display_name": "Maximum Client Connections", "name": "maxClientCnxns", "value": "60" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Zookeeper Server", "name": "zk_server_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Maximum Request Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Maximum Request Latency", "name": "role_health_suppression_zookeeper_server_max_latency", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_zookeeper_server_unexpected_exits", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "zookeeper_server_gc_duration_window", "value": "5" }, { "desc": "Specifies the password of the user that has read-write privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Password of user with read-write access to the JMX agent", "name": "jmx_passwd_file_readwrite_user_password", "value": "CONTROL" }, { "desc": "Enables the JMX agent on the ZooKeeper server. Turning this off on any of the ZooKeeper servers that are part of a service will prevent Cloudera Manager from being able to monitor that server and may affect the monitoring provided on the entire service.", "display_name": "Enable JMX Agent", "name": "enable_jmx_agent", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_zookeeper_server_scm_health", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Transaction Log Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Transaction Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Transaction Log Directory Free Space Monitoring Percentage Thresholds", "name": "zookeeper_server_data_log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Data Directory parameter.", "display_name": "Suppress Parameter Validation: Data Directory", "name": "role_config_suppression_datadir", "value": "false" }, { "desc": "Whether to suppress the results of the Outstanding Requests heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Outstanding Requests", "name": "role_health_suppression_zookeeper_server_outstanding_requests", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Name of User with Read-Only access to the JMX Agent parameter.", "display_name": "Suppress Parameter Validation: Name of User with Read-Only access to the JMX Agent", "name": "role_config_suppression_jmx_passwd_file_readonly_user", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "zookeeper_server_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of ZooKeeper Server in Bytes", "name": "zookeeper_server_java_heapsize", "value": "1073741824" }, { "desc": "The maximum number of rolled log files to keep for Server logs. Typically used by log4j or logback.", "display_name": "Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_zookeeper_server_log_directory_free_space", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Data Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Data Directory Free Space Monitoring Percentage Thresholds", "name": "zookeeper_server_data_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The period to review when computing the moving average of the connection count. Specified in minutes.", "display_name": "ZooKeeper Server Connection Count Monitoring Period", "name": "zookeeper_server_connection_count_window", "value": "3" }, { "desc": "When computing the overall Server health, consider the host's health.", "display_name": "Server Host Health Test", "name": "zookeeper_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The maximum session timeout, in milliseconds, that the ZooKeeper Server will allow the client to negotiate", "display_name": "Maximum Session Timeout", "name": "maxSessionTimeout", "value": "40000" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Transaction Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Transaction Log Directory Free Space", "name": "role_health_suppression_zookeeper_server_data_log_directory_free_space", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Data Directory Free Space", "name": "role_health_suppression_zookeeper_server_data_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Connection Count heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Connection Count", "name": "role_health_suppression_zookeeper_server_connection_count", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/spark-gateway.json0000666000175100017510000002135213245514472025161 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Data Serializer parameter.", "display_name": "Suppress Parameter Validation: Spark Data Serializer", "name": "role_config_suppression_spark_data_serializer", "value": "false" }, { "desc": "Python library paths to add to PySpark applications.", "display_name": "Extra Python Path", "name": "spark_python_path", "value": "" }, { "desc": "When dynamic allocation is enabled, time after which idle executors will be stopped.", "display_name": "Executor Idle Timeout", "name": "spark_dynamic_allocation_idle_timeout", "value": "60" }, { "desc": "Enables the external shuffle service. The external shuffle service preserves shuffle files written by executors so that the executors can be deallocated without losing work. Must be enabled if Enable Dynamic Allocation is enabled. Recommended and enabled by default for CDH 5.5 and higher.", "display_name": "Enable Shuffle Service", "name": "spark_shuffle_service_enabled", "value": "true" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/spark" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "51" }, { "desc": "The minimum log level for the Spark shell.", "display_name": "Shell Logging Threshold", "name": "spark_gateway_shell_logging_threshold", "value": "WARN" }, { "desc": "Enable dynamic allocation of executors in Spark applications.", "display_name": "Enable Dynamic Allocation", "name": "spark_dynamic_allocation_enabled", "value": "true" }, { "desc": "Which deploy mode to use by default. Can be overridden by users when launching applications.", "display_name": "Default Application Deploy Mode", "name": "spark_deploy_mode", "value": "client" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-env.sh.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "When dynamic allocation is enabled, timeout before requesting new executors when there are backlogged tasks.", "display_name": "Scheduler Backlog Timeout", "name": "spark_dynamic_allocation_scheduler_backlog_timeout", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Extra Python Path parameter.", "display_name": "Suppress Parameter Validation: Extra Python Path", "name": "role_config_suppression_spark_python_path", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-defaults.conf.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", "name": "spark-conf/spark-defaults.conf_client_config_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When dynamic allocation is enabled, timeout before requesting new executors after the initial backlog timeout has already expired. By default this is the same value as the initial backlog timeout.", "display_name": "Sustained Scheduler Backlog Timeout", "name": "spark_dynamic_allocation_sustained_scheduler_backlog_timeout", "value": null }, { "desc": "When dynamic allocation is enabled, maximum number of executors to allocate. By default, Spark relies on YARN to control the maximum number of executors for the application.", "display_name": "Maximum Executor Count", "name": "spark_dynamic_allocation_max_executors", "value": null }, { "desc": "When dynamic allocation is enabled, time after which idle executors with cached RDDs blocks will be stopped. By default, they're never stopped. This configuration is only available starting in CDH 5.5.", "display_name": "Caching Executor Idle Timeout", "name": "spark_dynamic_allocation_cached_idle_timeout", "value": null }, { "desc": "Write Spark application history logs to HDFS.", "display_name": "Enable History", "name": "spark_history_enabled", "value": "true" }, { "desc": "When dynamic allocation is enabled, minimum number of executors to keep alive while the application is running.", "display_name": "Minimum Executor Count", "name": "spark_dynamic_allocation_min_executors", "value": "0" }, { "desc": "Name of class implementing org.apache.spark.serializer.Serializer to use in Spark applications.", "display_name": "Spark Data Serializer", "name": "spark_data_serializer", "value": "org.apache.spark.serializer.KryoSerializer" }, { "desc": "Whether to allow users to kill running stages from the Spark Web UI.", "display_name": "Enable Kill From UI", "name": "spark_gateway_ui_kill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", "name": "role_config_suppression_spark-conf/spark-defaults.conf_client_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When dynamic allocation is enabled, number of executors to allocate when the application starts. By default, this is the same value as the minimum number of executors.", "display_name": "Initial Executor Count", "name": "spark_dynamic_allocation_initial_executors", "value": null }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "role_config_suppression_spark-conf/spark-env.sh_client_config_safety_valve", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-service.json0000666000175100017510000026531613245514472024776 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_core_site_safety_valve", "value": "false" }, { "desc": "Timeout in milliseconds for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations(). This value is only emitted for Impala.", "display_name": "HDFS File Block Storage Location Timeout", "name": "dfs_client_file_block_storage_locations_timeout", "value": "10000" }, { "desc": "Select an Object Store service to enable cloud storage support. Once enabled, HDFS will still act as the default FS; this additional storage will be available for use via direct references (fully-qualified URIs). Access to the object store is automatically extended to HDFS dependent services of type: Hive, Impala.", "display_name": "Object Store Service", "name": "object_store_service", "value": null }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Log and Query Redaction Policy parameter.", "display_name": "Suppress Parameter Validation: Log and Query Redaction Policy", "name": "service_config_suppression_redaction_policy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_shell_cmd_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the FailoverProxyProvider Class parameter.", "display_name": "Suppress Parameter Validation: FailoverProxyProvider Class", "name": "service_config_suppression_dfs_ha_proxy_provider", "value": "false" }, { "desc": "The domain to use for the HTTP cookie that stores the authentication token. In order for authentiation to work correctly across all Hadoop nodes' web-consoles the domain must be correctly set. Important: when using IP addresses, browsers ignore cookies with domain settings. For this setting to work properly all nodes in the cluster must be configured to generate URLs with hostname.domain names on it.", "display_name": "Hadoop HTTP Authentication Cookie Domain", "name": "hadoop_http_auth_cookie_domain", "value": "" }, { "desc": "The user that this service's processes should run as (except the HttpFS server, which has its own user)", "display_name": "System User", "name": "process_username", "value": "hdfs" }, { "desc": "Comma-delimited list of groups that you want to allow the Hue user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hue Proxy User Groups", "name": "hue_proxy_user_groups_list", "value": "*" }, { "desc": "The password for the TLS/SSL truststore.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Truststore Password", "name": "hadoop_group_mapping_ldap_keystore_passwd", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Groups", "name": "service_config_suppression_flume_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Failover Controller Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Failover Controller Count Validator", "name": "service_config_suppression_failovercontroller_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hue Proxy User Hosts", "name": "hue_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Groups", "name": "service_config_suppression_oozie_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Admin Users parameter.", "display_name": "Suppress Parameter Validation: Authorized Admin Users", "name": "service_config_suppression_hadoop_authorized_admin_users", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_replication_env_safety_valve", "value": "false" }, { "desc": "The service monitor will use these permissions to create the directory and files to test if the hdfs service is healthy. Permissions are specified using the 10-character unix-symbolic format e.g. '-rwxr-xr-x'.", "display_name": "HDFS Health Canary Directory Permissions", "name": "firehose_hdfs_canary_directory_permissions", "value": "-rwxrwxrwx" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hadoop-hdfs/audit" }, { "desc": "Class for user to group mapping (get groups for a given user).", "display_name": "Hadoop User Group Mapping Implementation", "name": "hadoop_security_group_mapping", "value": "org.apache.hadoop.security.ShellBasedUnixGroupsMapping" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop HTTP Authentication Cookie Domain parameter.", "display_name": "Suppress Parameter Validation: Hadoop HTTP Authentication Cookie Domain", "name": "service_config_suppression_hadoop_http_auth_cookie_domain", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the YARN user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "YARN Proxy User Groups", "name": "yarn_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Count Validator", "name": "service_config_suppression_namenode_count_validator", "value": "false" }, { "desc": "Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Oozie Proxy User Groups", "name": "oozie_proxy_user_groups_list", "value": "*" }, { "desc": "Comma-separated list of compression codecs that can be used in job or map compression.", "display_name": "Compression Codecs", "name": "io_compression_codecs", "value": "org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec" }, { "desc": "Comma-separated list of users authorized to used Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Users", "name": "hadoop_authorized_users", "value": "*" }, { "desc": "Quality of protection for secured RPC connections between NameNode and HDFS clients. For effective RPC protection, enable Kerberos authentication.", "display_name": "Hadoop RPC Protection", "name": "hadoop_rpc_protection", "value": "authentication" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_datanode_read_shortcircuit", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Checkpoint Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Checkpoint Configuration Validator", "name": "service_config_suppression_nameservice_checkpoint_configuration_validator", "value": "false" }, { "desc": "Distinguished name of the user to bind as. This is used to connect to LDAP/AD for searching user and group information. This may be left blank if the LDAP server supports anonymous binds.", "display_name": "Hadoop User Group Mapping LDAP Bind User Distinguished Name", "name": "hadoop_group_mapping_ldap_bind_user", "value": "" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Balancer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Balancer Count Validator", "name": "service_config_suppression_balancer_count_validator", "value": "false" }, { "desc": "Password for the TLS/SSL client truststore. Defines a cluster-wide default that can be overridden by individual services.", "display_name": "Cluster-Wide Default TLS/SSL Client Truststore Password", "name": "ssl_client_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Bind User Distinguished Name parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Bind User Distinguished Name", "name": "service_config_suppression_hadoop_group_mapping_ldap_bind_user", "value": "false" }, { "desc": "The password of the bind user.", "display_name": "Hadoop User Group Mapping LDAP Bind User Password", "name": "hadoop_group_mapping_ldap_bind_passwd", "value": "" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping Search Base parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping Search Base", "name": "service_config_suppression_hadoop_group_mapping_ldap_base", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Groups", "name": "service_config_suppression_yarn_proxy_user_groups_list", "value": "false" }, { "desc": "Enable/Disable the Log and Query Redaction Policy for this cluster.", "display_name": "Enable Log and Query Redaction", "name": "redaction_policy_enabled", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to all roles and client configurations in this HDFS service as well as all its dependent services. Any configs added here will be overridden by their default values in HDFS (which can be found in hdfs-default.xml).", "display_name": "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "core_site_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Authentication And Authorization Validation configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Authentication And Authorization Validation", "name": "service_config_suppression_hdfs_authentication_and_authorization_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Proxy User Groups", "name": "service_config_suppression_smon_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Local Path Access Users parameter.", "display_name": "Suppress Parameter Validation: DataNode Local Path Access Users", "name": "service_config_suppression_dfs_block_local_path_access_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue's Kerberos Principal Short Name parameter.", "display_name": "Suppress Parameter Validation: Hue's Kerberos Principal Short Name", "name": "service_config_suppression_hue_kerberos_principal_shortname", "value": "false" }, { "desc": "The default block size in bytes for new HDFS files. Note that this value is also used as the HBase Region Server HLog block size.", "display_name": "HDFS Block Size", "name": "dfs_block_size", "value": "134217728" }, { "desc": "The health test thresholds of the overall DataNode health. The check returns \"Concerning\" health if the percentage of \"Healthy\" DataNodes falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" DataNodes falls below the critical threshold.", "display_name": "Healthy DataNode Monitoring Thresholds", "name": "hdfs_datanodes_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Private Keys for SSH Fencing Strategy parameter.", "display_name": "Suppress Parameter Validation: Private Keys for SSH Fencing Strategy", "name": "service_config_suppression_dfs_ha_fencing_ssh_private_key_files", "value": "false" }, { "desc": "Enable WebHDFS interface", "display_name": "Enable WebHDFS", "name": "dfs_webhdfs_enabled", "value": "true" }, { "desc": "The short name of the Hue Kerberos principal. Normally, you do not need to specify this configuration. Cloudera Manager auto-configures this property so that Hue and Cloudera Manamgent Service work properly.", "display_name": "Hue's Kerberos Principal Short Name", "name": "hue_kerberos_principal_shortname", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Search Filter parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Search Filter", "name": "service_config_suppression_hadoop_group_mapping_ldap_group_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Groups", "name": "service_config_suppression_hue_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the JournalNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: JournalNode Count Validator", "name": "service_config_suppression_journalnode_count_validator", "value": "false" }, { "desc": "Typically, HDFS clients and servers communicate by opening sockets via an IP address. In certain networking configurations, it is preferable to open sockets after doing a DNS lookup on the hostname. Enable this property to open sockets after doing a DNS lookup on the hostname. This property is supported in CDH3u4 or later deployments.", "display_name": "Use DataNode Hostname", "name": "dfs_client_use_datanode_hostname", "value": "false" }, { "desc": "Enter a FailoverProxyProvider implementation to configure two URIs to connect to during fail-over. The first configured address is tried first, and on a fail-over event the other address is tried.", "display_name": "FailoverProxyProvider Class", "name": "dfs_ha_proxy_provider", "value": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "The search base for the LDAP connection. This is a distinguished name, and will typically be the root of the LDAP directory.", "display_name": "Hadoop User Group Mapping Search Base", "name": "hadoop_group_mapping_ldap_base", "value": "" }, { "desc": "If false, permission checking is turned off for files in HDFS.", "display_name": "Check HDFS Permissions", "name": "dfs_permissions", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the NFS Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NFS Gateway Count Validator", "name": "service_config_suppression_nfsgateway_count_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Corrupt Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Corrupt Blocks", "name": "service_health_suppression_hdfs_blocks_with_corrupt_replicas", "value": "false" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for HDFS events:\n

\n\n
    \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • command: the HDFS operation being performed.
  • \n
  • src: the source path for the operation.
  • \n
  • dest: the destination path for the operation.
  • \n
  • permissions: the permissions associated with the operation.
  • \n
\n\n

\nThe default HDFS audit event filter discards events generated by the internal\nCloudera and Hadoop users (cloudera-scm, dr.who, hbase, hive, impala, mapred,\nsolr, and spark), 'ls' actions performed by the hdfs user, operations in the\nCloudera Hive canary directory, and events that affect files in the /tmp\ndirectory.\"\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default HDFS audit event filter discards events generated by the internal \",\n \"Cloudera and Hadoop users (cloudera-scm, dr.who, hbase, hive, impala, \",\n \"mapred, solr, and spark), 'ls' actions performed by the hdfs user, \",\n \"operations in the Cloudera Hive canary directory, and events that affect \",\n \"files in the /tmp directory.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n {\n \"name\" : \"username\",\n \"match\" : \"(?:cloudera-scm|dr.who|hbase|hive|impala|mapred|solr|spark)(?:/.+)?\"\n }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n {\n \"name\" : \"username\",\n \"match\" : \"(?:hdfs)(?:/.+)?\"\n },\n {\n \"name\" : \"operation\",\n \"match\" : \"(?:listStatus|listCachePools|listCacheDirectives|getfileinfo)\"\n }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/user/hue/\\\\.cloudera_manager_hive_metastore_canary(?:/.*)?\" }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/user/hue/\\\\.Trash/Current/user/hue/\\\\.cloudera_manager_hive_metastore_canary(?:/.*)?\" }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/tmp(?:/.*)?\" }\n ]\n }\n ]\n}\n" }, { "desc": "Comma-separated list of groups authorized to used Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Groups", "name": "hadoop_authorized_groups", "value": "" }, { "desc": "If AES/CTR/NoPadding is chosen for the Data Transfer Encryption Algorithm, this specifies the length (in bits) of the AES key. When this parameter is changed, a full, non-rolling restart of the cluster must be performed.", "display_name": "Data Transfer Cipher Suite Key Strength", "name": "dfs_encrypt_data_transfer_cipher_keybits", "value": "256" }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted shuffle and encrypted web UIs. Applies to all configurations of daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Oozie Proxy User Hosts", "name": "oozie_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Authorization Provider Group parameter.", "display_name": "Suppress Parameter Validation: Sentry Authorization Provider Group", "name": "service_config_suppression_sentry_authorization_provider_hdfs_group", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Cloudera Service Monitor user to impersonate other users. The default '*' allows all hosts. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'.", "display_name": "Service Monitor Proxy User Hosts", "name": "smon_proxy_user_hosts_list", "value": "*" }, { "desc": "Comma-delimited list of groups that you want to allow the mapred user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Mapred Proxy User Groups", "name": "mapred_proxy_user_groups_list", "value": "*" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_service_env_safety_valve", "value": null }, { "desc": "For paths where authorization is enforced by Sentry Synchronization, file permissions will use this parameter as the group. This group should normally include the hive and impala users.", "display_name": "Sentry Authorization Provider Group", "name": "sentry_authorization_provider_hdfs_group", "value": "hive" }, { "desc": "Additional mapping rules that will be inserted before rules generated from the list of trusted realms and before the default rule. After changing this value and restarting the service, any services depending on this one must be restarted as well. The hadoop.security.auth_to_local property is configured using this information.", "display_name": "Additional Rules to Map Kerberos Principals to Short Names", "name": "extra_auth_to_local_rules", "value": null }, { "desc": "Password for the server keystore file used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "service_config_suppression_hadoop_policy_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the SecondaryNameNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: SecondaryNameNode Count Validator", "name": "service_config_suppression_secondarynamenode_count_validator", "value": "false" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "service_config_suppression_hdfs_service_config_safety_valve", "value": "false" }, { "desc": "Enables Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Note: This is effective only if Kerberos is enabled for the HDFS service.", "display_name": "Enable Kerberos Authentication for HTTP Web-Consoles", "name": "hadoop_secure_web_ui", "value": "false" }, { "desc": "For advanced use only. Key-value pairs (one on each line) to be inserted into the HDFS replication configuration for hadoop-env.sh.", "display_name": "HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "hdfs_replication_haoop_env_sh_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Secure Web UI Validator configuration validator.", "display_name": "Suppress Configuration Validator: Secure Web UI Validator", "name": "service_config_suppression_hadoop_secure_web_ui", "value": "false" }, { "desc": "Comma-delimited list of groups to allow the HDFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "HDFS Proxy User Groups", "name": "hdfs_proxy_user_groups_list", "value": "*" }, { "desc": "The service monitor will use this directory to create files to test if the hdfs service is healthy. The directory and files are created with permissions specified by 'HDFS Health Canary Directory Permissions'", "display_name": "HDFS Health Canary Directory", "name": "firehose_hdfs_canary_directory", "value": "/tmp/.cloudera_health_monitoring_canary_files" }, { "desc": "Comma-delimited list of groups that you want to allow the Hive user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hive Proxy User Groups", "name": "hive_proxy_user_groups_list", "value": "*" }, { "desc": "Comma-separated list of users authorized to perform admin operations on Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Admin Users", "name": "hadoop_authorized_admin_users", "value": "*" }, { "desc": "The health check thresholds of the number of missing blocks. Specified as a percentage of the total number of blocks.", "display_name": "Missing Block Monitoring Thresholds", "name": "hdfs_missing_blocks_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The amount of time after NameNode(s) start that the lack of an active NameNode will be tolerated. This is intended to allow either the auto-failover daemon to make a NameNode active, or a specifically issued failover command to take effect. This is an advanced option that does not often need to be changed.", "display_name": "NameNode Activation Startup Tolerance", "name": "hdfs_namenode_activation_startup_tolerance", "value": "180" }, { "desc": "

Note: Do not edit this property in the classic layout. Switch to the new layout to use preconfigured redaction rules and test your rules inline.

Use this property to define a list of rules to be followed for redacting sensitive information from log files and query strings. Click + to add a new redaction rule. You can choose one of the preconfigured rules or add a custom rule. When specifying a custom rule, the Search field should contain a regular expression that will be matched against the data. If a match is found, it is replaced by the contents of the Replace field.

Trigger is an optional field. It can be used to specify a simple string to be searched in the data. If the string is found, the redactor attempts to find a match for the Search regex. If no trigger is specified, redaction occurs by matching the Search regular expression. Use the Trigger field to enhance performance: simple string matching is faster than regular expression matching.

Test your rules by entering sample text into the Test Redaction Rules text box and clicking Test Redaction. If no rules match, the text you entered is returned unchanged.

", "display_name": "Log and Query Redaction Policy", "name": "redaction_policy", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Hosts", "name": "service_config_suppression_hue_proxy_user_hosts_list", "value": "false" }, { "desc": "Comma-delimited list of groups to allow the HttpFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "HttpFS Proxy User Groups", "name": "httpfs_proxy_user_groups_list", "value": "*" }, { "desc": "Allows the flume user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Flume Proxy User Groups", "name": "flume_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the mapred user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Mapred Proxy User Hosts", "name": "mapred_proxy_user_hosts_list", "value": "*" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "dfs.socket.timeout3000dfs.datanode.socket.write.timeout3000ipc.client.connect.max.retries1fs.permissions.umask-mode000" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Count Validator", "name": "service_config_suppression_datanode_count_validator", "value": "false" }, { "desc": "Path to the TLS/SSL client truststore file. Defines a cluster-wide default that can be overridden by individual services. This truststore must be in JKS format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. The contents of the truststore can be modified without restarting any roles. By default, changes to its contents are picked up within ten seconds. If not set, the default Java truststore is used to verify certificates.", "display_name": "Cluster-Wide Default TLS/SSL Client Truststore Location", "name": "ssl_client_truststore_location", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Enable TLS/SSL encryption for HDFS, MapReduce, and YARN web UIs, as well as encrypted shuffle for MapReduce and YARN.", "display_name": "Hadoop TLS/SSL Enabled", "name": "hdfs_hadoop_ssl_enabled", "value": "false" }, { "desc": "List of Kerberos realms that Hadoop services should trust. If empty, defaults to the default_realm property configured in the krb5.conf file. After changing this value and restarting the service, all services depending on this service must also be restarted. Adds mapping rules for each domain to the hadoop.security.auth_to_local property in core-site.xml.", "display_name": "Trusted Kerberos Realms", "name": "trusted_realms", "value": "" }, { "desc": "Whether to suppress the results of the Failover Controllers Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Failover Controllers Health", "name": "service_health_suppression_hdfs_failover_controllers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Proxy User Hosts", "name": "service_config_suppression_smon_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress the results of the DataNode Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: DataNode Health", "name": "service_health_suppression_hdfs_data_nodes_healthy", "value": "false" }, { "desc": "Enables the health check that verifies that the failover controllers associated with this service are healthy and running.", "display_name": "Failover Controllers Healthy", "name": "failover_controllers_healthy_enabled", "value": "true" }, { "desc": "The attribute of the group object that identifies the users that are members of the group. The default will usually be appropriate for any LDAP installation.", "display_name": "Hadoop User Group Mapping LDAP Group Membership Attribute", "name": "hadoop_group_mapping_ldap_member_attr", "value": "member" }, { "desc": "Comma separated list of users allowed to do short circuit read. A short circuit read allows a client co-located with the data to read HDFS file blocks directly from HDFS. If empty, will default to the DataNode process' user.", "display_name": "DataNode Local Path Access Users", "name": "dfs_block_local_path_access_user", "value": null }, { "desc": "The timeout, in milliseconds, to use with the Cloudera Manager agent-based fencer.", "display_name": "Timeout for Cloudera Manager Fencing Strategy", "name": "dfs_ha_fencing_cloudera_manager_timeout_millis", "value": "10000" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "Maximum bandwidth used for image transfer in bytes per second. This can help keep normal NameNode operations responsive during checkpointing. A default value of 0 indicates that throttling is disabled.", "display_name": "FsImage Transfer Bandwidth", "name": "dfs_image_transfer_bandwidthPerSec", "value": "0" }, { "desc": "The health check thresholds of the number of blocks that have at least one corrupt replica. Specified as a percentage of the total number of blocks.", "display_name": "Blocks With Corrupt Replicas Monitoring Thresholds", "name": "hdfs_blocks_with_corrupt_replicas_thresholds", "value": "{\"critical\":\"1.0\",\"warning\":\"0.5\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Groups", "name": "service_config_suppression_mapred_proxy_user_groups_list", "value": "false" }, { "desc": "The user the management services impersonates when connecting to HDFS. If no value is specified, the HDFS superuser is used.", "display_name": "HDFS User to Impersonate", "name": "hdfs_user_to_impersonate", "value": null }, { "desc": "File path to a jks-format truststore containing the TLS/SSL certificate used sign the LDAP server's certificate. Note that in previous releases this was erroneously referred to as a \"keystore\".", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Truststore", "name": "hadoop_group_mapping_ldap_keystore", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Synchronization Path Prefixes parameter.", "display_name": "Suppress Parameter Validation: Sentry Synchronization Path Prefixes", "name": "service_config_suppression_hdfs_sentry_sync_path_prefixes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-Wide Default TLS/SSL Client Truststore Password parameter.", "display_name": "Suppress Parameter Validation: Cluster-Wide Default TLS/SSL Client Truststore Password", "name": "service_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "The name of the group of superusers.", "display_name": "Superuser Group", "name": "dfs_permissions_supergroup", "value": "supergroup" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP URL parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP URL", "name": "service_config_suppression_hadoop_group_mapping_ldap_url", "value": "false" }, { "desc": "Allows the Cloudera Service Monitor user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "Service Monitor Proxy User Groups", "name": "smon_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Groups parameter.", "display_name": "Suppress Parameter Validation: Authorized Groups", "name": "service_config_suppression_hadoop_authorized_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Groups", "name": "service_config_suppression_httpfs_proxy_user_groups_list", "value": "false" }, { "desc": "The minimal block replication.", "display_name": "Minimal Block Replication", "name": "dfs_replication_min", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the UNIX Domain Socket path parameter.", "display_name": "Suppress Parameter Validation: UNIX Domain Socket path", "name": "service_config_suppression_dfs_domain_socket_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Hadoop Group Name parameter.", "display_name": "Suppress Parameter Validation: Shared Hadoop Group Name", "name": "service_config_suppression_hdfs_hadoop_group_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Single User Mode Overrides Validator configuration validator.", "display_name": "Suppress Configuration Validator: Single User Mode Overrides Validator", "name": "service_config_suppression_single_user_mode_override_validator", "value": "false" }, { "desc": "The maximal block replication.", "display_name": "Maximal Block Replication", "name": "dfs_replication_max", "value": "512" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "Enable authorization", "display_name": "Hadoop Secure Authorization", "name": "hadoop_security_authorization", "value": "false" }, { "desc": "The Key Management Server used by HDFS. This must be set to use encryption for data at rest.", "display_name": "KMS Service", "name": "kms_service", "value": null }, { "desc": "The attribute of the group object that identifies the group name. The default will usually be appropriate for all LDAP systems.", "display_name": "Hadoop User Group Mapping LDAP Group Name Attribute", "name": "hadoop_group_mapping_ldap_group_name_attr", "value": "cn" }, { "desc": "Enables DataNode support for the experimental DistributedFileSystem.getFileVBlockStorageLocations API. Applicable to CDH 4.1 and onwards.", "display_name": "Enable HDFS Block Metadata API", "name": "dfs_datanode_hdfs_blocks_metadata_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Groups", "name": "service_config_suppression_hdfs_proxy_user_groups_list", "value": "false" }, { "desc": "The tolerance window that will be used in HDFS service tests that depend on detection of the active NameNode.", "display_name": "Active NameNode Detection Window", "name": "hdfs_active_namenode_detecton_window", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Hosts", "name": "service_config_suppression_flume_proxy_user_hosts_list", "value": "false" }, { "desc": "Default block replication. The number of replications to make when the file is created. The default value is used if a replication number is not specified.", "display_name": "Replication Factor", "name": "dfs_replication", "value": "3" }, { "desc": "Comma-delimited list of groups that you want to allow the HTTP user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. This is used by WebHCat.", "display_name": "HTTP Proxy User Groups", "name": "HTTP_proxy_user_groups_list", "value": "*" }, { "desc": "The name of the system group shared by all the core Hadoop services.", "display_name": "Shared Hadoop Group Name", "name": "hdfs_hadoop_group_name", "value": "hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP User Search Filter parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP User Search Filter", "name": "service_config_suppression_hadoop_group_mapping_ldap_user_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS High Availability Fencing Methods parameter.", "display_name": "Suppress Parameter Validation: HDFS High Availability Fencing Methods", "name": "service_config_suppression_dfs_ha_fencing_methods", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Hosts", "name": "service_config_suppression_hive_proxy_user_hosts_list", "value": "false" }, { "desc": "The amount of time to wait for HDFS filesystem image transfer from NameNode to complete.", "display_name": "FsImage Transfer Timeout", "name": "dfs_image_transfer_timeout", "value": "60000" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS snapshot shell command.", "display_name": "HDFS Snapshot Shell Command Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_shell_cmd_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codecs parameter.", "display_name": "Suppress Parameter Validation: Compression Codecs", "name": "service_config_suppression_io_compression_codecs", "value": "false" }, { "desc": "An additional filter to use when searching for LDAP users. The default will usually be appropriate for Active Directory installations. If connecting to a generic LDAP server, ''sAMAccountName'' will likely be replaced with ''uid''. {0} is a special string used to denote where the username fits into the filter.", "display_name": "Hadoop User Group Mapping LDAP User Search Filter", "name": "hadoop_group_mapping_ldap_user_filter", "value": "(&(objectClass=user)(sAMAccountName={0}))" }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "List of fencing methods to use for service fencing. shell(./cloudera_manager_agent_fencer.py) is a fencing mechanism designed to use the Cloudera Manager agent. The sshfence method uses SSH. If using custom fencers (that may communicate with shared store, power units, or network switches), use the shell to invoke them.", "display_name": "HDFS High Availability Fencing Methods", "name": "dfs_ha_fencing_methods", "value": "shell(./cloudera_manager_agent_fencer.py)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Hosts", "name": "service_config_suppression_hdfs_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Health Canary Directory Permissions parameter.", "display_name": "Suppress Parameter Validation: HDFS Health Canary Directory Permissions", "name": "service_config_suppression_firehose_hdfs_canary_directory_permissions", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Heap Size Validator", "name": "service_config_suppression_nameservice_namenodes_heap_size_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Health Canary Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Health Canary Directory", "name": "service_config_suppression_firehose_hdfs_canary_directory", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS replication jobs.", "display_name": "HDFS Replication Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_replication_env_safety_valve", "value": null }, { "desc": "Enables the health check that a client can create, read, write, and delete files", "display_name": "HDFS Canary Health Check", "name": "hdfs_canary_health_enabled", "value": "true" }, { "desc": "Whether to serve logs over HTTP from HDFS web servers. This includes listing the logs directory at the /logs endpoint, which may be a security concern.", "display_name": "Serve logs over HTTP", "name": "http_logs_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP TLS/SSL Truststore Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP TLS/SSL Truststore Password", "name": "service_config_suppression_hadoop_group_mapping_ldap_keystore_passwd", "value": "false" }, { "desc": "Path on the DataNode's local file system to a UNIX domain socket used for communication between the DataNode and local HDFS clients. This socket is used for Short Circuit Reads. Only the HDFS System User and \"root\" should have write access to the parent directory and all of its ancestors. This property is supported in CDH 4.2 or later deployments.", "display_name": "UNIX Domain Socket path", "name": "dfs_domain_socket_path", "value": "/var/run/hdfs-sockets/dn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User's Home Directory parameter.", "display_name": "Suppress Parameter Validation: System User's Home Directory", "name": "service_config_suppression_hdfs_user_home_dir", "value": "false" }, { "desc": "Algorithm to encrypt data transfer between DataNodes and clients, and among DataNodes. If 3des or rc4 are chosen, the entire communication is encrypted with that algorithm. In CDH 5.4 and higher, if AES/CTR/NoPadding is chosen, 3des is used for the initial key exchange, and then AES/CTR/NoPadding is used for the transfer of data. This is the most secure option, and is recommended for clusters running CDH 5.4 or higher. It also requires that the \"openssl-devel\" package be installed on all machines in the cluster. When this parameter is changed, a full, nonrolling restart of the cluster must be performed.", "display_name": "Data Transfer Encryption Algorithm", "name": "dfs_encrypt_data_transfer_algorithm", "value": "rc4" }, { "desc": "The health check thresholds of the number of under-replicated blocks. Specified as a percentage of the total number of blocks.", "display_name": "Under-replicated Block Monitoring Thresholds", "name": "hdfs_under_replicated_blocks_thresholds", "value": "{\"critical\":\"40.0\",\"warning\":\"10.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "hdfs_service_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Hosts", "name": "service_config_suppression_yarn_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress the results of the HDFS Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HDFS Canary", "name": "service_health_suppression_hdfs_canary_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "service_config_suppression_hdfs_ssl_client_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HttpFS Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HttpFS Count Validator", "name": "service_config_suppression_httpfs_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. This is used by WebHCat.", "display_name": "HTTP Proxy User Hosts", "name": "HTTP_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress the results of the Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Free Space", "name": "service_health_suppression_hdfs_free_space_remaining", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Groups", "name": "service_config_suppression_http_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Membership Attribute parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Membership Attribute", "name": "service_config_suppression_hadoop_group_mapping_ldap_member_attr", "value": "false" }, { "desc": "Whether to suppress the results of the Under-Replicated Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Under-Replicated Blocks", "name": "service_health_suppression_hdfs_under_replicated_blocks", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • username: treats the field value as a user name, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare HDFS events:\n

\n\n
    \n
  • operation: the HDFS operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • src: the source path for the operation.
  • \n
  • dest: the destination path for the operation.
  • \n
  • permissions: the permissions associated with the operation.
  • \n
\n\n

\nThe default event tracker for HDFS services defines equality by comparing the\nusername, operation, and source path of the events.\n

\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": "{\n \"comment\" : [\n \"The default event tracker for HDFS services defines equality by \",\n \"comparing the username, operation, and source path of the events.\"\n ],\n \"timeToLive\" : 60000,\n \"fields\" : [\n { \"type\": \"value\", \"name\" : \"src\" },\n { \"type\": \"value\", \"name\" : \"operation\" },\n { \"type\": \"username\", \"name\" : \"username\" }\n ]\n}\n" }, { "desc": "Choose the authentication mechanism used by Hadoop", "display_name": "Hadoop Secure Authentication", "name": "hadoop_security_authentication", "value": "simple" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Replication Factor", "name": "service_config_suppression_dfs_replication", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "hadoop_policy_config_safety_valve", "value": null }, { "desc": "Enable encryption of data transfer between DataNodes and clients, and among DataNodes. For effective data transfer protection, enable Kerberos authentication and choose Privacy Quality of RPC Protection.", "display_name": "Enable Data Transfer Encryption", "name": "dfs_encrypt_data_transfer", "value": "false" }, { "desc": "Comma-delimited list of hosts that you want to allow the YARN user to impersonate. The default '*' allows all hosts. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'.", "display_name": "YARN Proxy User Hosts", "name": "yarn_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Bind User Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Bind User Password", "name": "service_config_suppression_hadoop_group_mapping_ldap_bind_passwd", "value": "false" }, { "desc": "When computing the overall HDFS cluster health, consider the active NameNode's health", "display_name": "Active NameNode Role Health Check", "name": "hdfs_namenode_health_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "hdfs_ssl_server_safety_valve", "value": null }, { "desc": "The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory.", "display_name": "System User's Home Directory", "name": "hdfs_user_home_dir", "value": "/var/lib/hadoop-hdfs" }, { "desc": "Whether to suppress the results of the Missing Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Missing Blocks", "name": "service_health_suppression_hdfs_missing_blocks", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "SASL protection mode for secured connections to the DataNodes when reading or writing data.", "display_name": "DataNode Data Transfer Protection", "name": "dfs_data_transfer_protection", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Users parameter.", "display_name": "Suppress Parameter Validation: Authorized Users", "name": "service_config_suppression_hadoop_authorized_users", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the flume user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Flume Proxy User Hosts", "name": "flume_proxy_user_hosts_list", "value": "*" }, { "desc": "

The URL of the LDAP server. The URL must be prefixed with ldap:// or ldaps://. The URL can optionally specify a custom port, for example: ldaps://ldap_server.example.com:1636. Note that usernames and passwords will be transmitted in the clear unless either an ldaps:// URL is used, or \"Enable LDAP TLS\" is turned on (where available). Also note that encryption must be in use between the client and this service for the same reason.

For more detail on the LDAP URL format, see RFC 2255 . A space-separated list of URLs can be entered; in this case the URLs will each be tried in turn until one replies.

", "display_name": "Hadoop User Group Mapping LDAP URL", "name": "hadoop_group_mapping_ldap_url", "value": null }, { "desc": "SSH connection timeout, in milliseconds, to use with the built-in sshfence fencer.", "display_name": "Timeout for SSH Fencing Strategy", "name": "dfs_ha_fencing_ssh_connect_timeout", "value": "30000" }, { "desc": "Enable automatic synchronization of HDFS ACLs with Sentry privileges. HDFS Access Control Lists and Check HDFS Permissions must be enabled when this feature is enabled. Use Sentry Synchronization Path Prefixes to define the HDFS regions where authorization is enforced using Sentry information. For more information, see Synchronizing HDFS ACLs and Sentry Authorization.", "display_name": "Enable Sentry Synchronization", "name": "hdfs_sentry_sync_enable", "value": "false" }, { "desc": "Comma-delimited list of hosts where you allow the HttpFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HttpFS Proxy User Hosts", "name": "httpfs_proxy_user_hosts_list", "value": "*" }, { "desc": "Name of the ZooKeeper service that this HDFS service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Mountpoints Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Mountpoints Validator", "name": "service_config_suppression_nameservice_mountpoints_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Admin Groups parameter.", "display_name": "Suppress Parameter Validation: Authorized Admin Groups", "name": "service_config_suppression_hadoop_authorized_admin_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Hosts", "name": "service_config_suppression_mapred_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Superuser Group parameter.", "display_name": "Suppress Parameter Validation: Superuser Group", "name": "service_config_suppression_dfs_permissions_supergroup", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Auto Failover Validator configuration validator.", "display_name": "Suppress Configuration Validator: Auto Failover Validator", "name": "service_config_suppression_auto_failover_validator", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hdfs" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Encryption Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Encryption Validator", "name": "service_config_suppression_hdfs_encryption_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Hosts", "name": "service_config_suppression_oozie_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Hosts", "name": "service_config_suppression_http_proxy_user_hosts_list", "value": "false" }, { "desc": "The group that this service's processes should run as (except the HttpFS server, which has its own group)", "display_name": "System Group", "name": "process_groupname", "value": "hdfs" }, { "desc": "ACLs (Access Control Lists) enhance the existing HDFS permission model to support controlling file access for arbitrary combinations of users and groups instead of a single owner, single group, and all other users. When ACLs are disabled, the NameNode rejects all attempts to set an ACL.", "display_name": "Enable Access Control Lists", "name": "dfs_namenode_acls_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Trusted Kerberos Realms parameter.", "display_name": "Suppress Parameter Validation: Trusted Kerberos Realms", "name": "service_config_suppression_trusted_realms", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the NFS High Availability Validator configuration validator.", "display_name": "Suppress Configuration Validator: NFS High Availability Validator", "name": "service_config_suppression_nfs_ha_validator", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Whether to suppress the results of the NameNode Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Health", "name": "service_health_suppression_hdfs_ha_namenode_health", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hive Proxy User Hosts", "name": "hive_proxy_user_hosts_list", "value": "*" }, { "desc": "Default umask for file and directory creation, specified in an octal value (with a leading 0)", "display_name": "Default Umask", "name": "dfs_umaskmode", "value": "022" }, { "desc": "The health check thresholds of free space in HDFS. Specified as a percentage of total HDFS capacity.", "display_name": "HDFS Free Space Monitoring Thresholds", "name": "hdfs_free_space_thresholds", "value": "{\"critical\":\"10.0\",\"warning\":\"20.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS User to Impersonate parameter.", "display_name": "Suppress Parameter Validation: HDFS User to Impersonate", "name": "service_config_suppression_hdfs_user_to_impersonate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Groups", "name": "service_config_suppression_hive_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Comma-separated list of groups authorized to perform admin operations on Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Admin Groups", "name": "hadoop_authorized_admin_groups", "value": "" }, { "desc": "An additional filter to use when searching for groups.", "display_name": "Hadoop User Group Mapping LDAP Group Search Filter", "name": "hadoop_group_mapping_ldap_group_filter", "value": "(objectClass=group)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Rules to Map Kerberos Principals to Short Names parameter.", "display_name": "Suppress Parameter Validation: Additional Rules to Map Kerberos Principals to Short Names", "name": "service_config_suppression_extra_auth_to_local_rules", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-Wide Default TLS/SSL Client Truststore Location parameter.", "display_name": "Suppress Parameter Validation: Cluster-Wide Default TLS/SSL Client Truststore Location", "name": "service_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Hosts", "name": "service_config_suppression_httpfs_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_hdfs_ssl_server_safety_valve", "value": "false" }, { "desc": "A list of path prefixes that define the HDFS regions where authorization is enforced using Sentry information. Only relevant when Sentry Synchronization is enabled.", "display_name": "Sentry Synchronization Path Prefixes", "name": "hdfs_sentry_sync_path_prefixes", "value": "/user/hive/warehouse" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Name Attribute parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Name Attribute", "name": "service_config_suppression_hadoop_group_mapping_ldap_group_name_attr", "value": "false" }, { "desc": "When computing the overall HDFS cluster health, consider the health of the standby NameNode.", "display_name": "Standby NameNode Health Check", "name": "hdfs_standby_namenodes_health_enabled", "value": "true" }, { "desc": "The SSH private key files to use with the built-in sshfence fencer. These are to be accessible to the hdfs user on the machines running the NameNodes.", "display_name": "Private Keys for SSH Fencing Strategy", "name": "dfs_ha_fencing_ssh_private_key_files", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "service_config_suppression_hdfs_replication_haoop_env_sh_safety_valve", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HDFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HDFS Proxy User Hosts", "name": "hdfs_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether or not to use TLS/SSL when connecting to the LDAP server.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Enabled", "name": "hadoop_group_mapping_ldap_use_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Redaction Policy Validator configuration validator.", "display_name": "Suppress Configuration Validator: Redaction Policy Validator", "name": "service_config_suppression_redaction_policy_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into ssl-client.xml. Applies cluster-wide, but can be overridden by individual services.", "display_name": "HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "hdfs_ssl_client_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP TLS/SSL Truststore parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP TLS/SSL Truststore", "name": "service_config_suppression_hadoop_group_mapping_ldap_keystore", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hbase-gateway.json0000666000175100017510000001505513245514472025126 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_client_config_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for hbase-site.xml.", "display_name": "HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_client_config_safety_valve", "value": null }, { "desc": "If a multiget operation is performed with 'Consistency.TIMELINE', the read RPC is sent to the primary region server first. After this timeout, a parallel RPC for secondary region replicas is also sent if the primary does not respond. After this, the result is returned from whichever RPC is finished first. If the response returns from the primary region replica, that the data is the most recent. Result.isStale() API has been added to inspect the staleness. If the result is from a secondary region, Result.isStale() is set to true.", "display_name": "HBase Client Multiget Timeout For Secondary Region Replicas", "name": "hbase_client_primaryCallTimeout_multiget", "value": "10" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hbase" }, { "desc": "Whether to enable interruption of RPC threads at the client. The default value of true enables primary RegionServers to access data from other regions' secondary replicas.", "display_name": "Enable Client RPC Threads Interruption", "name": "hbase_ipc_client_allowsInterrupt", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hbase-env.sh", "display_name": "HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh", "name": "hbase_client_env_safety_valve", "value": null }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh parameter.", "display_name": "Suppress Parameter Validation: HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh", "name": "role_config_suppression_hbase_client_env_safety_valve", "value": "false" }, { "desc": "If a get operation is performed with 'Consistency.TIMELINE', the read RPC is sent to the primary region server first. After this timeout, parallel RPC for secondary region replicas is also sent if the primary does not respond. After this, the result is returned from whichever RPC is finished first. If the response returns from the primary region replica, that data is the most recent. Result.isStale() API has been added to inspect the staleness. If the result is from a secondary region, Result.isStale() is set to true.", "display_name": "HBase Client Get Timeout For Secondary Region Replicas", "name": "hbase_client_primaryCallTimeout_get", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hbase_client_java_opts", "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hbase_client_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hbase_client_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/sqoop-sqoop_server.json0000666000175100017510000005724513245514472026302 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_sqoop_server_scm_health", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Password for Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Password", "name": "sqoop_repository_database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_sqoop_server_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties", "name": "role_config_suppression_sqoop_config_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Directory where Sqoop 2 Server will place its log files.", "display_name": "Sqoop 2 Server Log Directory", "name": "sqoop_log_dir", "value": "/var/log/sqoop2" }, { "desc": "Admin port of Sqoop 2 Server. (Note: This configuration only applies from CDH 4.3 onwards.)", "display_name": "Sqoop 2 Admin Port", "name": "sqoop_admin_port", "value": "8005" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum number of clients allowed to connect to the Sqoop 2 Server.", "display_name": "Maximum Client Connections", "name": "max_client_connections", "value": "10" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only. A string to be inserted into sqoop.properties for this role only.", "display_name": "Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties", "name": "sqoop_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Log Directory", "name": "role_config_suppression_sqoop_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_sqoop_server_unexpected_exits", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Sqoop 2 Server logs. Typically used by log4j or logback.", "display_name": "Sqoop 2 Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_sqoop_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_sqoop_server_file_descriptor", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SQOOP_SERVER_role_env_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "sqoop_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Password parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Password", "name": "role_config_suppression_sqoop_repository_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database User parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database User", "name": "role_config_suppression_sqoop_repository_database_user", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Sqoop 2 Server in Bytes", "name": "sqoop_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Directory where the Sqoop 2 Server places its metastore data. This is used only when Sqoop Repository Database Type is Derby.", "display_name": "Sqoop 2 Server Metastore Directory", "name": "sqoop_metastore_data_dir", "value": "/var/lib/sqoop2" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Sqoop 2 Server", "name": "sqoop_java_opts", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Type of Sqoop repository database.", "display_name": "Sqoop Repository Database Type", "name": "sqoop_repository_database_type", "value": "derby" }, { "desc": "User for Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database User", "name": "sqoop_repository_database_user", "value": "sa" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_sqoop_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Name of Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Name", "name": "sqoop_repository_database_name", "value": "sqoop" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Sqoop 2 Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Sqoop 2 Server", "name": "role_config_suppression_sqoop_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Metastore Directory parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Metastore Directory", "name": "role_config_suppression_sqoop_metastore_data_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Sqoop 2 Server logs. Typically used by log4j or logback.", "display_name": "Sqoop 2 Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Host parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Host", "name": "role_config_suppression_sqoop_repository_database_host", "value": "false" }, { "desc": "When computing the overall Sqoop 2 Server health, consider the host's health.", "display_name": "Sqoop 2 Server Host Health Test", "name": "sqoop_server_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Enables the health test that the Sqoop 2 Server's process state is consistent with the role configuration", "display_name": "Sqoop 2 Server Process Health Test", "name": "sqoop_server_scm_health_enabled", "value": "true" }, { "desc": "HTTP port of Sqoop 2 Server. (Note: This configuration only applies from CDH 4.3 onwards.)", "display_name": "Sqoop 2 HTTP Port", "name": "sqoop_http_port", "value": "12000" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_sqoop_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Host name of Sqoop repository database. If the port is non-default for your database type, use host:port notation. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Host", "name": "sqoop_repository_database_host", "value": "localhost" }, { "desc": "The minimum log level for Sqoop 2 Server logs", "display_name": "Sqoop 2 Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Name parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Name", "name": "role_config_suppression_sqoop_repository_database_name", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_sqoop_server_host_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/0000775000175100017510000000000013245515027022744 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/common/0000775000175100017510000000000013245515027024234 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/common/install_extjs0000666000175100017510000000115213245514476027052 0ustar zuulzuul00000000000000#!/bin/bash EXTJS_DESTINATION_DIR="/var/lib/oozie" EXTJS_DOWNLOAD_URL="http://tarballs.openstack.org/sahara/dist/common-artifacts/ext-2.2.zip" extjs_archive=/tmp/$(basename $EXTJS_DOWNLOAD_URL) if [ ! -n "$EXTJS_DESTINATION_DIR" ]; then if [ $test_only -eq 0 ]; then wget -O $extjs_archive $EXTJS_DOWNLOAD_URL mkdir -p $EXTJS_DESTINATION_DIR if [ -z "${EXTJS_NO_UNPACK:-}" ]; then unzip -d "$EXTJS_DESTINATION_DIR" $extjs_archive rm -f $extjs_archive else mv $extjs_archive $EXTJS_DESTINATION_DIR fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/common/add_jar0000666000175100017510000000132113245514476025551 0ustar zuulzuul00000000000000#!/usr/bin/env bash hadoop="2.6.0" HADOOP_SWIFT_JAR_NAME="hadoop-openstack-$hadoop-cdh$plugin_version.jar" HADOOP_SWIFT_BUILD_LOCATION="http://tarballs.openstack.org/sahara/dist/hadoop-openstack/master" SWIFT_LIB_URI="$HADOOP_SWIFT_BUILD_LOCATION/hadoop-openstack-${hadoop}.jar" HADOOP_SWIFT_JAR_NAME="hadoop-openstack.jar" if [ ! -f $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME ]; then if [ $test_only -eq 0 ]; then wget -O $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME $SWIFT_LIB_URI if [ $? -ne 0 ]; then echo -e "Could not download Swift Hadoop FS implementation.\nAborting" exit 1 fi chmod 0644 $hdfs_lib_dir/$HADOOP_SWIFT_JAR_NAME else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/common/install_cloudera0000666000175100017510000000130113245514472027503 0ustar zuulzuul00000000000000#!/bin/bash HADOOP_OPENSTACK_5_9_x_URL="https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-openstack/2.6.0-cdh$plugin_version/hadoop-openstack-2.6.0-cdh$plugin_version.jar" dest=/usr/lib/hadoop-mapreduce/hadoop-openstack.jar if [ ! -L "/usr/lib/oozie/oozie-sharelib-yarn.tar.gz" ]; then if [ $test_only -eq 0 ]; then wget -O $dest $HADOOP_OPENSTACK_5_9_x_URL ln -s /usr/lib/oozie/oozie-sharelib-yarn /usr/lib/oozie/oozie-sharelib-yarn.tar.gz ln -s /usr/lib/oozie/oozie-sharelib-mr1 /usr/lib/oozie/oozie-sharelib-mr1.tar.gz ln -s /usr/lib/oozie/oozie-sharelib-yarn.tar.gz /usr/lib/oozie/oozie-sharelib.tar.gz else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/common/unlimited_security_artifacts0000666000175100017510000000055513245514472032152 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -n /tmp/UnlimitedPolicy/ ]; then if [ $test_only -eq 0 ]; then mkdir /tmp/UnlimitedPolicy/ wget $unlimited_security_location/local_policy.jar -O /tmp/UnlimitedPolicy/local_policy.jar wget $unlimited_security_location/US_export_policy.jar -O /tmp/UnlimitedPolicy/US_export_policy.jar else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/image.yaml0000666000175100017510000000623313245514472024723 0ustar zuulzuul00000000000000arguments: java_distro: default: cloudera-jdk description: The distribution of Java to install. Defaults to openjdk. choices: - openjdk - oracle-java - cloudera-jdk plugin_version: default: 5.9.0 description: The distribution of CDH to install. Defaults to 5.9.0. hidden: True required: False choices: - 5.9.0 - 5.9.1 - 5.9.2 - 5.9.3 hdfs_lib_dir: default: /usr/lib/hadoop-mapreduce description: The path to HDFS_LIB_DIR. Default to /usr/lib/hadoop-mapreduce required: False unlimited_security_location: default: http://sahara-files.mirantis.com/kerberos-artifacts/ description: Place where UnlimitedSecurity polices are located required: False validators: - script: common/install_extjs - os_case: - centos: - package: wget - script: centos/wget_repo: env_vars: [plugin_version] - ubuntu: - script: ubuntu/wget_repo: env_vars: [plugin_version] - argument_case: argument_name: java_distro cases: openjdk: - any: - all: - package: java-1.8.0-openjdk-devel - all: - package: java-1.7.0-openjdk-devel cloudera-jdk: - all: - package: oracle-j2sdk1.7 - package: ntp - package: - cloudera-manager-agent - cloudera-manager-daemons - cloudera-manager-server - cloudera-manager-server-db-2 - package: - hadoop-hdfs-namenode - hadoop-hdfs-datanode - hadoop-hdfs-secondarynamenode - hadoop-mapreduce - hadoop-mapreduce-historyserver - hadoop-yarn-nodemanager - hadoop-yarn-resourcemanager - package: - hbase - hbase-solr - package: - hive-hcatalog - hive-metastore - hive-server2 - hive-webhcat-server - hue - package: - oozie - spark-core - os_case: - centos: - package: spark-history-server - package: zookeeper - package: unzip - package: flume-ng - package: hadoop-kms - package: - impala - impala-server - impala-state-store - impala-catalog - impala-shell - package: keytrustee-keyprovider - package: - sentry - solr-server - solr-doc - search - sqoop2 - package: - kafka - kafka-server - script: common/install_cloudera: env_vars: [plugin_version] - os_case: - centos: - script: centos/turn_off_services - ubuntu: - script: ubuntu/turn_off_services - script: common/add_jar: env_vars: [plugin_version, hdfs_lib_dir] - script: common/unlimited_security_artifacts: env_vars: [unlimited_security_location] - os_case: - centos: - package: - krb5-server - krb5-libs - krb5-workstation - rng-tools - package: iptables-services - script: centos/selinux_permissive - script: centos/disable_firewall - package: nmap-ncat - ubuntu: - package: - krb5-admin-server - libpam-krb5 - ldap-utils - krb5-user - rng-tools sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/centos/0000775000175100017510000000000013245515027024237 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/centos/turn_off_services0000666000175100017510000000147013245514472027716 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files oozie.service | grep 'enabled' | wc -l) if [ "$check" != "on" ]; then if [ $test_only -eq 0 ]; then for i in cloudera-scm-agent \ cloudera-scm-server \ cloudera-scm-server-db \ hadoop-hdfs-datanode \ hadoop-hdfs-namenode \ hadoop-hdfs-secondarynamenode \ hadoop-mapreduce-historyserver \ hadoop-yarn-nodemanager \ hadoop-yarn-resourcemanager \ hive-metastore \ hive-server2 \ hive-webhcat-server \ hue \ oozie \ postgresql \ impala-catalog \ impala-server \ impala-state-store \ solr-server \ spark-history-server do chkconfig $i off done else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/centos/selinux_permissive0000666000175100017510000000046113245514472030125 0ustar zuulzuul00000000000000#!/bin/bash check=$(cat /etc/selinux/config | grep "SELINUX=permissive" | wc -l) echo $check if [ $check -eq 0 ]; then if [ $test_only -eq 0 ]; then echo "SELINUX=permissive" > /etc/selinux/config echo "SELINUXTYPE=targeted" >> /etc/selinux/config else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/centos/wget_repo0000666000175100017510000000426613245514472026172 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -f /etc/yum.repos.d/cloudera-cdh5.repo ]; then if [ $test_only -eq 0 ]; then echo '[cloudera-cdh5]' > /etc/yum.repos.d/cloudera-cdh5.repo echo "name=Cloudera's Distribution for Hadoop, Version 5" >> /etc/yum.repos.d/cloudera-cdh5.repo echo "baseurl=http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/$plugin_version/" >> /etc/yum.repos.d/cloudera-cdh5.repo echo "gpgkey = http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-cdh5.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/cloudera-cdh5.repo echo '[cloudera-manager]' > /etc/yum.repos.d/cloudera-manager.repo echo 'name=Cloudera Manager' >> /etc/yum.repos.d/cloudera-manager.repo echo "baseurl=http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/$plugin_version/" >> /etc/yum.repos.d/cloudera-manager.repo echo "gpgkey = http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-manager.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/cloudera-manager.repo echo '[navigator-keytrustee]' > /etc/yum.repos.d/kms.repo echo "name=Cloudera's Distribution for navigator-Keytrustee, Version 5" >> /etc/yum.repos.d/kms.repo echo "baseurl=http://archive.cloudera.com/navigator-keytrustee5/redhat/7/x86_64/navigator-keytrustee/$plugin_version/" >> /etc/yum.repos.d/kms.repo echo "gpgkey = http://archive.cloudera.com/navigator-keytrustee5/redhat/7/x86_64/navigator-keytrustee/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/kms.repo echo 'gpgcheck = 1' >> /etc/yum.repos.d/kms.repo echo "[cloudera-kafka]" > /etc/yum.repos.d/cloudera-kafka.repo echo "name=Cloudera's Distribution for kafka, Version 2.0.2" >> /etc/yum.repos.d/cloudera-kafka.repo echo "baseurl=http://archive.cloudera.com/kafka/redhat/7/x86_64/kafka/2.0.2/" >> /etc/yum.repos.d/cloudera-kafka.repo echo "gpgkey = http://archive.cloudera.com/kafka/redhat/7/x86_64/kafka/RPM-GPG-KEY-cloudera" >> /etc/yum.repos.d/cloudera-kafka.repo echo "gpgcheck = 1" >> /etc/yum.repos.d/cloudera-kafka.repo yum clean all else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/centos/disable_firewall0000666000175100017510000000103513245514472027456 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files iptables.service | grep 'enabled' | wc -l) if [ $check -eq 1 ]; then if [ $test_only -eq 0 ]; then if type -p systemctl && [[ "$(systemctl --no-pager list-unit-files firewalld)" =~ 'enabled' ]]; then systemctl disable firewalld fi if type -p service; then service ip6tables save service iptables save chkconfig ip6tables off chkconfig iptables off fi else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/ubuntu/0000775000175100017510000000000013245515027024266 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/ubuntu/turn_off_services0000666000175100017510000000150013245514472027737 0ustar zuulzuul00000000000000#!/bin/bash check=$(systemctl --no-pager list-unit-files oozie.service | grep 'enabled' | wc -l) if [ "$check" != "on" ]; then if [ $test_only -eq 0 ]; then for i in cloudera-scm-agent \ cloudera-scm-server \ cloudera-scm-server-db \ hadoop-hdfs-datanode \ hadoop-hdfs-namenode \ hadoop-hdfs-secondarynamenode \ hadoop-mapreduce-historyserver \ hadoop-yarn-nodemanager \ hadoop-yarn-resourcemanager \ hive-metastore \ hive-server2 \ hive-webhcat-server \ hue \ oozie \ postgresql \ impala-catalog \ impala-server \ impala-state-store \ solr-server \ spark-history-server do update-rc.d -f $i remove done else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/images/ubuntu/wget_repo0000666000175100017510000000411013245514472026205 0ustar zuulzuul00000000000000#!/bin/bash if [ ! -f /etc/apt/sources.list.d/cdh5.list ]; then if [ $test_only -eq 0 ]; then # Add repository with postgresql package (it's dependency of cloudera packages) # Base image doesn't contain this repo echo "deb http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty universe multiverse main" >> /etc/apt/sources.list # Cloudera repositories echo "deb [arch=amd64] http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh$plugin_version contrib" > /etc/apt/sources.list.d/cdh5.list echo "deb-src http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh$plugin_version contrib" >> /etc/apt/sources.list.d/cdh5.list wget -qO - http://archive-primary.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key | apt-key add - echo "deb [arch=amd64] http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm$plugin_version contrib" > /etc/apt/sources.list.d/cm5.list echo "deb-src http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm$plugin_version contrib" >> /etc/apt/sources.list.d/cm5.list wget -qO - http://archive-primary.cloudera.com/cm5/ubuntu/trusty/amd64/cm/archive.key | apt-key add - wget -O /etc/apt/sources.list.d/kms.list http://archive.cloudera.com/navigator-keytrustee5/ubuntu/trusty/amd64/navigator-keytrustee/cloudera.list wget -qO - http://archive.cloudera.com/navigator-keytrustee5/ubuntu/trusty/amd64/navigator-keytrustee/archive.key | apt-key add - # add Kafka repository echo 'deb http://archive.cloudera.com/kafka/ubuntu/trusty/amd64/kafka/ trusty-kafka2.0.2 contrib' >> /etc/apt/sources.list wget -qO - https://archive.cloudera.com/kafka/ubuntu/trusty/amd64/kafka/archive.key | apt-key add - #change repository priority echo 'Package: zookeeper' >> /etc/apt/preferences.d/cloudera-pin echo 'Pin: origin "archive.cloudera.com"' >> /etc/apt/preferences.d/cloudera-pin echo 'Pin-Priority: 1001' >> /etc/apt/preferences.d/cloudera-pin apt-get update else exit 0 fi fi sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/kms-kms.json0000666000175100017510000007672113245514472023776 0ustar zuulzuul00000000000000[ { "desc": "Comma-delimited list of hosts where you want to allow the hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hue Proxy User Hosts", "name": "hadoop_kms_proxyuser_hue_hosts", "value": "*" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kms_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "When computing the overall Key Management Server health, consider the host's health.", "display_name": "Key Management Server Host Health Test", "name": "kms_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hive_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hue_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the KMS Blacklist Users parameter.", "display_name": "Suppress Parameter Validation: KMS Blacklist Users", "name": "role_config_suppression_kms_blacklist_users", "value": "false" }, { "desc": "Allows the httpfs superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HttpFS Proxy User Groups", "name": "hadoop_kms_proxyuser_httpfs_groups", "value": "*" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_flume_groups", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kms_kms_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JavaKeyStoreProvider Directory parameter.", "display_name": "Suppress Parameter Validation: JavaKeyStoreProvider Directory", "name": "role_config_suppression_hadoop_security_key_provider_dir", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the mapred user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Mapred Proxy User Hosts", "name": "hadoop_kms_proxyuser_mapred_hosts", "value": "*" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kms_kms_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_yarn_groups", "value": "false" }, { "desc": "Allows the hdfs superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HDFS Proxy User Groups", "name": "hadoop_kms_proxyuser_hdfs_groups", "value": "*" }, { "desc": "Port used to access the KMS' embedded Tomcat admin console.", "display_name": "KMS Admin Port", "name": "kms_admin_port", "value": "16001" }, { "desc": "A comma-separated list of users (no spaces) for whom to disallow access to key material. These users can still fetch key metadata and create encrypted encryption keys, but are unable to do any other KMS operations. Typically, HDFS superusers will be specified here.", "display_name": "KMS Blacklist Users", "name": "kms_blacklist_users", "value": "" }, { "desc": "Comma-delimited list of hosts where you want to allow the yarn user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "YARN Proxy User Hosts", "name": "hadoop_kms_proxyuser_yarn_hosts", "value": "*" }, { "desc": "The maximum size, in megabytes, per log file for Key Management Server logs. Typically used by log4j or logback.", "display_name": "Key Management Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Comma-delimited list of hosts where you want to allow the hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hive Proxy User Hosts", "name": "hadoop_kms_proxyuser_hive_hosts", "value": "*" }, { "desc": "The minimum log level for Key Management Server logs", "display_name": "Key Management Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_http_hosts", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Key Management Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hive_groups", "value": "false" }, { "desc": "Allows the mapred superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Mapred Proxy User Groups", "name": "hadoop_kms_proxyuser_mapred_groups", "value": "*" }, { "desc": "Maximum number of threads used to handle KMS requests.", "display_name": "KMS Max Threads", "name": "kms_max_threads", "value": "250" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Key Management Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "KMS_role_env_safety_valve", "value": null }, { "desc": "Comma-delimited list of hosts where you want to allow the hdfs user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HDFS Proxy User Hosts", "name": "hadoop_kms_proxyuser_hdfs_hosts", "value": "*" }, { "desc": "The log directory for log files of the role Key Management Server.", "display_name": "Key Management Server Log Directory", "name": "log_dir", "value": "/var/log/hadoop-kms" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Key Management Server TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kms_kms_file_descriptor", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "For advanced use only. A string to be inserted into kms-acls.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml", "name": "kms-acls.xml_role_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hue_groups", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Oozie Proxy User Hosts", "name": "hadoop_kms_proxyuser_oozie_hosts", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_hdfs_hosts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the httpfs user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HttpFS Proxy User Hosts", "name": "hadoop_kms_proxyuser_httpfs_hosts", "value": "*" }, { "desc": "Port used by clients to interact with the KMS.", "display_name": "KMS HTTP Port", "name": "kms_http_port", "value": "16000" }, { "desc": "Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Oozie Proxy User Groups", "name": "hadoop_kms_proxyuser_oozie_groups", "value": "*" }, { "desc": "Maximum heap size of the KMS.", "display_name": "KMS Heap Size", "name": "kms_heap_size", "value": "1073741824" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kms_kms_swap_memory_usage", "value": "false" }, { "desc": "Directory where configuration and binaries are staged before starting KMS. Does not normally need to be modified.", "display_name": "KMS Staging Directory", "name": "kms_staging_dir", "value": "/var/lib/hadoop-kms" }, { "desc": "Allows the hue superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hue Proxy User Groups", "name": "hadoop_kms_proxyuser_hue_groups", "value": "*" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_oozie_hosts", "value": "false" }, { "desc": "Allows the HTTP superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HTTP Proxy User Groups", "name": "hadoop_kms_proxyuser_HTTP_groups", "value": "*" }, { "desc": "For advanced use only. A string to be inserted into kms-site.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml", "name": "kms-site.xml_role_safety_valve", "value": null }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Key Management Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Key Management Server TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kms_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into core-site.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "core-site.xml_role_safety_valve", "value": null }, { "desc": "Allows the yarn superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "YARN Proxy User Groups", "name": "hadoop_kms_proxyuser_yarn_groups", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_http_groups", "value": "false" }, { "desc": "Allows the hive superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hive Proxy User Groups", "name": "hadoop_kms_proxyuser_hive_groups", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_yarn_hosts", "value": "false" }, { "desc": "Directory of the keystore file kms.keystore used by JavaKeyStoreProvider that backs the KMS.", "display_name": "JavaKeyStoreProvider Directory", "name": "hadoop_security_key_provider_dir", "value": "/var/lib/kms" }, { "desc": "Enables the health test that the Key Management Server's process state is consistent with the role configuration", "display_name": "Key Management Server Process Health Test", "name": "kms_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_flume_hosts", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Key Management Server logs. Typically used by log4j or logback.", "display_name": "Key Management Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "role_config_suppression_core-site.xml_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the KMS Staging Directory parameter.", "display_name": "Suppress Parameter Validation: KMS Staging Directory", "name": "role_config_suppression_kms_staging_dir", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HTTP Proxy User Hosts", "name": "hadoop_kms_proxyuser_HTTP_hosts", "value": "*" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_oozie_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_hdfs_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_httpfs_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Groups", "name": "role_config_suppression_hadoop_kms_proxyuser_mapred_groups", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kms_kms_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml", "name": "role_config_suppression_kms-acls.xml_role_safety_valve", "value": "false" }, { "desc": "Encrypt communication between clients and Key Management Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Key Management Server", "name": "ssl_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_httpfs_hosts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Key Management Server TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Allows the flume superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Flume Proxy User Groups", "name": "hadoop_kms_proxyuser_flume_groups", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Hosts", "name": "role_config_suppression_hadoop_kms_proxyuser_mapred_hosts", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The password for the Key Management Server JKS keystore file.", "display_name": "Key Management Server TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml", "name": "role_config_suppression_kms-site.xml_role_safety_valve", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the flume user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Flume Proxy User Hosts", "name": "hadoop_kms_proxyuser_flume_hosts", "value": "*" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hbase-regionserver.json0000666000175100017510000016642413245514472026206 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_regionserver_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the HBase RegionServer Multiple HDFS WAL with Replication Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase RegionServer Multiple HDFS WAL with Replication Validator", "name": "role_config_suppression_hbase_regionserver_multiwal_replication_validator", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Maximum number of HStoreFiles to compact per minor compaction.", "display_name": "Maximum Number of HStoreFiles Compaction", "name": "hbase_hstore_compaction_max", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: RegionServer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_regionserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase HRegion Major Compaction parameter.", "display_name": "Suppress Parameter Validation: HBase HRegion Major Compaction", "name": "role_config_suppression_hbase_hregion_majorcompaction", "value": "false" }, { "desc": "The address for the HBase RegionServer web UI", "display_name": "HBase RegionServer Web UI Address", "name": "hbase_regionserver_info_bindAddress", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Number of threads to use while loading and unloading regions to or from a RegionServer. Can be used to increase the speed of decommissioning or rolling restart operations.", "display_name": "Region Mover Threads", "name": "hbase_regionserver_regionmover_thread_count", "value": "1" }, { "desc": "The health test thresholds of the average size of the HBase RegionServer flush queue over a recent period. See HBase RegionServer Flush Queue Monitoring Period.", "display_name": "HBase RegionServer Flush Queue Monitoring Thresholds", "name": "regionserver_flush_queue_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "The amount of time to wait for the HBase Region Server to fully start up and connect to the HBase Master before enforcing the connectivity check.", "display_name": "HBase Region Server Connectivity Tolerance at Startup", "name": "regionserver_connectivity_tolerance", "value": "180" }, { "desc": "Whether to suppress the results of the Store File Index Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Store File Index Size", "name": "role_health_suppression_region_server_store_file_idx_size", "value": "false" }, { "desc": "The hashing algorithm for use in HashFunction. Two values are supported: 'murmur' (for MurmurHash) and 'jenkins' (for JenkinsHash).", "display_name": "HBase Hash Type", "name": "hbase_hash_type", "value": "murmur" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: RegionServer DNS Network Interface", "name": "role_config_suppression_hbase_regionserver_dns_interface", "value": "false" }, { "desc": "The port for the HBase RegionServer web UI. Set to -1 to disable RegionServer web UI.", "display_name": "HBase RegionServer Web UI port", "name": "hbase_regionserver_info_port", "value": "60030" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The period over which to compute the moving average of the compaction queue size.", "display_name": "HBase RegionServer Compaction Queue Monitoring Period", "name": "regionserver_compaction_queue_window", "value": "5" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase RegionServer", "name": "hbase_regionserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "RegionServer Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The health test thresholds of the latency that the RegionServer sees for HDFS read operations", "display_name": "HBase RegionServer HDFS Read Latency Thresholds", "name": "regionserver_read_latency_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"50.0\"}" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds of the size used by the HStoreFile index. Specified as a percentage of the total heap size.", "display_name": "Percentage of Heap Used by HStoreFile Index", "name": "regionserver_store_file_idx_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "Maximum number of Write-Ahead Log (WAL) files. This value multiplied by HDFS Block Size (dfs.blocksize) is the size of the WAL that will need to be replayed when a server crashes. This value is inversely proportional to the frequency of flushes to disk.", "display_name": "Maximum number of Write-Ahead Log (WAL) files", "name": "hbase_regionserver_maxlogs", "value": "32" }, { "desc": "List of coprocessors that are loaded by default on all tables. For any override coprocessor method, these classes will be called in order. After implementing your own coprocessor, just put it in HBase's classpath and add the fully qualified class name here. A coprocessor can also be loaded on demand by setting HTableDescriptor.", "display_name": "HBase Coprocessor Region Classes", "name": "hbase_coprocessor_region_classes", "value": "" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "regionserver_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_region_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_region_server_scm_health", "value": "false" }, { "desc": "The Hierarchical Storage Management policy that should be used by the RegionServer for the write-ahead-log. Using an SSD policy will have no effect unless HDFS HSM is configured to know which drives are SSDs. See Enabling HSM with HBase.", "display_name": "WAL HSM Storage Policy", "name": "hbase_regionserver_wal_storage_policy", "value": "NONE" }, { "desc": "Blocks writes if the size of the memstore increases to the value of 'hbase.hregion.block.memstore' multiplied by the value of 'hbase.hregion.flush.size' bytes. This setting is useful for preventing runaway memstore during spikes in update traffic. Without an upper-bound, memstore fills such that when it flushes, the resultant process of flushing files take a long time to compact or split, or worse, an \"out of memory\" error occurs.", "display_name": "HBase Memstore Block Multiplier", "name": "hbase_hregion_memstore_block_multiplier", "value": "2" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Sync the HLog to HDFS after this interval, in milliseconds, if it has not accumulated the number of HLog Entries specified to trigger a sync.", "display_name": "Synch Interval of HLog Entries", "name": "hbase_regionserver_optionallogflushinterval", "value": "1000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \"IPC Server handler.*took.*appending an edit to hlog.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"ABORTING region server serverName.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"DFSOutputStream ResponseProcessor exception.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Error Recovery for block blk.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Failed init\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Problem renewing lease for DFSClient.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"remote error telling master we are up\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Session.*for server.*closing socket connection and attempting reconnect.\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting .*compaction on region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_REGION_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^completed compaction on region (.+) after (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_REGION_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting compaction on (.+) in region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"COLUMN_FAMILY\", \"group1\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^completed compaction: regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Completed compaction: Request = regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^aborted compaction: regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_ABORTED\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Finished memstore flush of .+ for region (.+) in (.+), sequenceid=(.+), compaction requested=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DURATION\", \"group2\": \"SEQUENCE_ID\", \"group3\": \"COMPACTION_REQUESTED\" },\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Flush of region (.+) due to global heap pressure$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_DUE_TO_HEAP_PRESSURE\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"WARN\", \"content\": \"^Region (.+) has too many store files; delaying flush up to 90000ms$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_DELAYED_TOO_MANY_STORE_FILES\", \"attribute:SEVERITY\": \"CRITICAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting split of region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Running rollback/cleanup of failed split of (.+);.+$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_ABORTED\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Region split, hbase:meta updated, and report to master. Parent=(.+), new regions: (.+,.*,.+), (.+,.*,.+). Split took (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DAUGHTER_REGIONS\", \"group2\": \"DAUGHTER_REGIONS\", \"group3\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Region split, META updated, and report to master. Parent=(.+), new regions: (.+,.*,.+), (.+,.*,.+). Split took (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DAUGHTER_REGIONS\", \"group2\": \"DAUGHTER_REGIONS\", \"group3\": \"DURATION\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress the results of the Cluster Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Cluster Connectivity", "name": "role_health_suppression_region_server_master_connectivity", "value": "false" }, { "desc": "The host name or IP address of the DNS name server which a RegionServer should use to determine the host name used by the HBase Master for communication and display purposes.", "display_name": "RegionServer DNS Nameserver", "name": "hbase_regionserver_dns_nameserver", "value": null }, { "desc": "Number of threads for completing small compactions.", "display_name": "RegionServer Small Compactions Thread Count", "name": "hbase_regionserver_thread_compaction_small", "value": "1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds of the total size of RegionServer's memstores. Specified as a percentage of the configured upper limit. See Maximum Size of All Memstores in RegionServer.", "display_name": "HBase RegionServer Memstore Size Thresholds", "name": "regionserver_memstore_size_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Secure Bulk Load Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Secure Bulk Load Directory", "name": "role_config_suppression_hbase_bulkload_staging_dir", "value": "false" }, { "desc": "Percentage of maximum heap (-Xmx setting) to allocate to block cache used by HFile/StoreFile. To disable, set this value to 0 .", "display_name": "HFile Block Cache Size", "name": "hfile_block_cache_size", "value": "0.4" }, { "desc": "The maximum size, in megabytes, per log file for RegionServer logs. Typically used by log4j or logback.", "display_name": "RegionServer Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Memstore will be flushed to disk if size of the memstore exceeds this value in number of bytes. This value is checked by a thread that runs the frequency specified by hbase.server.thread.wakefrequency.", "display_name": "HBase Memstore Flush Size", "name": "hbase_hregion_memstore_flush_size", "value": "134217728" }, { "desc": "The HLog file reader implementation.", "display_name": "HLog Reader Implementation", "name": "hbase_regionserver_hlog_reader_impl", "value": null }, { "desc": "If this number of HStoreFiles in any one HStore is exceeded, then a compaction is run to rewrite all HStoreFiles files as one HStoreFile. (One HStoreFile is written per flush of memstore.) You can delay compaction by specifying a larger number, but the compaction will take longer when it does run. During a compaction, updates cannot be flushed to disk. Long compactions require memory sufficient to carry the logging of all updates across the duration of the compaction. If too large, clients timeout during compaction.", "display_name": "HStore Compaction Threshold", "name": "hbase_hstore_compactionThreshold", "value": "3" }, { "desc": "The period over which to compute the moving average of the HDFS sync latency of the HBase RegionServer.", "display_name": "HBase RegionServer HDFS Sync Latency Monitoring Period", "name": "regionserver_sync_latency_window", "value": "5" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_region_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Codecs parameter.", "display_name": "Suppress Parameter Validation: RegionServer Codecs", "name": "role_config_suppression_hbase_regionserver_codecs", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The maximum size byte array that should come from the MSLAB, in bytes.", "display_name": "Maximum Byte Array from MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_max_allocation", "value": "262144" }, { "desc": "The health test thresholds for the latency of HDFS write operations that the RegionServer detects", "display_name": "HBase RegionServer HDFS Sync Latency Thresholds", "name": "regionserver_sync_latency_thresholds", "value": "{\"critical\":\"5000.0\",\"warning\":\"500.0\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_region_server_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where RegionServer will place its log files.", "display_name": "RegionServer Log Directory", "name": "hbase_regionserver_log_dir", "value": "/var/log/hbase" }, { "desc": "The period of time that an HRegion will block updates after reaching the HStoreFile limit that is specified by 'hbase.hstore.blockingStoreFiles'. After this time has elapsed, the HRegion will stop blocking updates even if a compaction has not been completed.", "display_name": "HStore Blocking Wait Time", "name": "hbase_hstore_blockingWaitTime", "value": "90000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Flush Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Flush Queue Size", "name": "role_health_suppression_region_server_flush_queue", "value": "false" }, { "desc": "Timeout for Canary to perform its checks.", "display_name": "Canary Timeout", "name": "hbase_regionserver_canary_timeout", "value": "15" }, { "desc": "Duration between consecutive checks done by the Canary.", "display_name": "Canary Interval", "name": "hbase_regionserver_canary_interval", "value": "6" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Coprocessor Region Classes parameter.", "display_name": "Suppress Parameter Validation: HBase Coprocessor Region Classes", "name": "role_config_suppression_hbase_coprocessor_region_classes", "value": "false" }, { "desc": "The lease period, in milliseconds, for the HBase RegionServer. Clients must report in within this period or else they are considered dead.", "display_name": "HBase RegionServer Lease Period", "name": "hbase_regionserver_lease_period", "value": "60000" }, { "desc": "Period, in milliseconds, at which to roll the commit log.", "display_name": "HBase RegionServer Log Roll Period", "name": "hbase_regionserver_logroll_period", "value": "3600000" }, { "desc": "The minimum log level for RegionServer logs", "display_name": "RegionServer Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "regionserver_web_metric_collection_enabled", "value": "true" }, { "desc": "The name of the DNS Network Interface from which a RegionServer should report its IP address.", "display_name": "RegionServer DNS Network Interface", "name": "hbase_regionserver_dns_interface", "value": null }, { "desc": "The number of reservation blocks that are used to prevent unstable RegionServers caused by an OOME.", "display_name": "RegionServer Reservation Blocks", "name": "hbase_regionserver_nbreservationblocks", "value": "4" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HLog Reader Implementation parameter.", "display_name": "Suppress Parameter Validation: HLog Reader Implementation", "name": "role_config_suppression_hbase_regionserver_hlog_reader_impl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase RegionServer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase RegionServer", "name": "role_config_suppression_hbase_regionserver_java_opts", "value": "false" }, { "desc": "The implementation that should be used by the RegionServer for the write-ahead-log.", "display_name": "WAL Provider", "name": "hbase_regionserver_wal_provider", "value": "multiwal" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "regionserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The size of the chunks allocated by MSLAB, in bytes.", "display_name": "Chunk Size Allocated by MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_chunksize", "value": "2097152" }, { "desc": "Whether to suppress the results of the Memstore Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Memstore Size", "name": "role_health_suppression_region_server_memstore_size", "value": "false" }, { "desc": "If the memstores in a region are this size or larger when closing, run a pre-flush process to clear out memstores before putting up the region closed flag and taking the region offline. On close, a flush process is run under the close flag up to empty memory. During this time, the region is offline and no writes are taken. If the memstore content is large, the flush process could take a long time to complete. The pre-flush process cleans out the bulk of the memstore before putting up the close flag and taking the region offline, so that the flush process that runs under the close flag has little to do.", "display_name": "HBase Memstore Pre-close Flush Size", "name": "hbase_hregion_preclose_flush_size", "value": "5242880" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_regionserver_config_safety_valve", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: RegionServer Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period over which to compute the moving average of the HDFS read latency of the HBase RegionServer.", "display_name": "HBase RegionServer HDFS Read Latency Monitoring Period", "name": "regionserver_read_latency_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of HBase RegionServer in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of HBase RegionServer in Bytes", "name": "role_config_suppression_hbase_regionserver_java_heapsize", "value": "false" }, { "desc": "The directory for HBase secure bulk Load.", "display_name": "HBase Secure Bulk Load Directory", "name": "hbase_bulkload_staging_dir", "value": "/tmp/hbase-staging" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "An interface that is assignable to HRegionInterface. Used in HBase Client for opening a proxy to a remote HBase RegionServer.", "display_name": "HBase RegionServer Interface Class", "name": "hbase_regionserver_class", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase RegionServer Web UI Address parameter.", "display_name": "Suppress Parameter Validation: HBase RegionServer Web UI Address", "name": "role_config_suppression_hbase_regionserver_info_bindaddress", "value": "false" }, { "desc": "When true, HBase RegionServer will bind to 0.0.0.0. Only available in CDH 4.3 and later.", "display_name": "HBase RegionServer Bind to Wildcard Address", "name": "hbase_regionserver_bind_to_wildcard_address", "value": "true" }, { "desc": "The time, in milliseconds, between 'major' compactions of all HStoreFiles in a region. To disable automated major compactions, set this value to 0.", "display_name": "HBase HRegion Major Compaction", "name": "hbase_hregion_majorcompaction", "value": "604800000" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enable MemStore-Local Allocation Buffer (MSLAB) Allocation Scheme. Note: This feature is experimental in CDH3.", "display_name": "Enable MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_enabled", "value": "true" }, { "desc": "Enables the health test that the RegionServer is connected to the Master", "display_name": "HBase RegionServer to Master Connectivity Test", "name": "regionserver_master_connectivity_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": "32768" }, { "desc": "Limit for the number of regions after which no more region splitting should take place. This is not a hard limit for the number of regions but acts as a guideline for the RegionServer to stop splitting after a certain limit.", "display_name": "HBase Region Split Limit", "name": "hbase_regionserver_regionSplitLimit", "value": "2147483647" }, { "desc": "When using Multiple HDFS WAL as the WALProvider, sets how many write-ahead-logs each region server should run. Will result in this number of HDFS pipelines. Writes for a given Region only go to a single pipeline, spreading total RegionServer load.", "display_name": "Per-RegionServer Number of WAL Pipelines", "name": "hbase_regionserver_wal_pipelines", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HLog Writer Implementation parameter.", "display_name": "Suppress Parameter Validation: HLog Writer Implementation", "name": "role_config_suppression_hbase_regionserver_hlog_writer_impl", "value": "false" }, { "desc": "Enables the health test that the RegionServer's process state is consistent with the role configuration", "display_name": "RegionServer Process Health Test", "name": "regionserver_scm_health_enabled", "value": "true" }, { "desc": "Interval, in milliseconds, between messages from the RegionServer to the HBase Master. Use a high value such as 3000 for clusters that have more than 10 hosts.", "display_name": "HBase RegionServer Message Interval", "name": "hbase_regionserver_msginterval", "value": "3000" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "When memstores are being forced to flush to make room in memory, keep flushing until this amount is reached. If this amount is equal to 'hbase.regionserver.global.memstore.upperLimit', then minimum possible flushing will occur when updates are blocked due to memstore limiting.", "display_name": "Low Watermark for Memstore Flush", "name": "hbase_regionserver_global_memstore_lowerLimit", "value": "0.38" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Log Directory parameter.", "display_name": "Suppress Parameter Validation: RegionServer Log Directory", "name": "role_config_suppression_hbase_regionserver_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the RegionServer File Descriptor Limit Validator configuration validator.", "display_name": "Suppress Configuration Validator: RegionServer File Descriptor Limit Validator", "name": "role_config_suppression_regionserver_fd_limit_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_region_server_swap_memory_usage", "value": "false" }, { "desc": "Whether asynchronous WAL replication to the secondary region replicas is enabled.\u2028 If enabled, a replication peer named 'region_replica_replication' is created\u2028 that tails the logs and replicates the mutations to region replicas for tables that\u2028 have region replication > 1. Disabling this replication also\u2028 requires disabling the replication peer using shell or the ReplicationAdmin Java class.\u2028 Replication to secondary region replicas works over standard intercluster replication. If disabled explicitly, enable replication by setting 'hbase.replication' to true for this feature to work.\u2028", "display_name": "Enable Replication To Secondary Region Replicas", "name": "hbase_region_replica_replication_enabled", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "regionserver_gc_duration_window", "value": "5" }, { "desc": "Time period in seconds to reset long-running metrics (e.g. compactions). This is an HBase specific configuration.", "display_name": "Extended Period", "name": "hbase_metrics_extended_period", "value": "3600" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_region_server_host_health", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "RegionServer Environment Advanced Configuration Snippet (Safety Valve)", "name": "REGIONSERVER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Comma-separated list of codecs that the RegionServer requires to start. Use this setting to make sure that all RegionServers joining a cluster are installed with a particular set of codecs.", "display_name": "RegionServer Codecs", "name": "hbase_regionserver_codecs", "value": "" }, { "desc": "Maximum size of all memstores in a RegionServer before new updates are blocked and flushes are forced.", "display_name": "Maximum Size of All Memstores in RegionServer", "name": "hbase_regionserver_global_memstore_upperLimit", "value": "0.4" }, { "desc": "Number of handlers for processing priority requests in a RegionServer.", "display_name": "HBase RegionServer Meta-Handler Count", "name": "hbase_regionserver_metahandler_count", "value": "10" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_region_server_file_descriptor", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Compaction Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Compaction Queue Size", "name": "role_health_suppression_region_server_compaction_queue", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_region_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Jitter outer bound for major compactions.\u2028 On each RegionServer, the hbase.region.majorcompaction\u2028 interval is multiplied by a random fraction that is inside the bounds of this\u2028 maximum. This + or - product is added to when the next\u2028 major compaction is to run. Major compaction\u2028 should not occur on every RegionServer at the same time. The\u2028 smaller this number, the closer together the compactions.", "display_name": "HBase HRegion Major Compaction Jitter", "name": "hbase_hregion_majorcompaction_jitter", "value": "0.5" }, { "desc": "The period over which to compute the moving average of the flush queue size.", "display_name": "HBase RegionServer Flush Queue Monitoring Period", "name": "regionserver_flush_queue_window", "value": "5" }, { "desc": "Maximum HStoreFile size. If any one of a column families' HStoreFiles has grown to exceed this value, the hosting HRegion is split in two.", "display_name": "HBase Maximum File Size", "name": "hbase_hregion_max_filesize", "value": "10737418240" }, { "desc": "The health test thresholds of the weighted average size of the HBase RegionServer compaction queue over a recent period. See HBase RegionServer Compaction Queue Monitoring Period.", "display_name": "HBase RegionServer Compaction Queue Monitoring Thresholds", "name": "regionserver_compaction_queue_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_region_server_gc_duration", "value": "false" }, { "desc": "Read threadpool size used by the RegionServer HBase IPC Server.", "display_name": "RegionServer IPC Read Threadpool Size", "name": "hbase_ipc_server_read_threadpool_size", "value": "10" }, { "desc": "If there are more than this number of HStoreFiles in any one HStore, then updates are blocked for this HRegion until a compaction is completed, or until the value specified for 'hbase.hstore.blockingWaitTime' has been exceeded.", "display_name": "HStore Blocking Store Files", "name": "hbase_hstore_blockingStoreFiles", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "regionserver_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The maximum number of rolled log files to keep for RegionServer logs. Typically used by log4j or logback.", "display_name": "RegionServer Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The HLog file writer implementation.", "display_name": "HLog Writer Implementation", "name": "hbase_regionserver_hlog_writer_impl", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer DNS Nameserver parameter.", "display_name": "Suppress Parameter Validation: RegionServer DNS Nameserver", "name": "role_config_suppression_hbase_regionserver_dns_nameserver", "value": "false" }, { "desc": "The port that an HBase RegionServer binds to.", "display_name": "HBase RegionServer Port", "name": "hbase_regionserver_port", "value": "60020" }, { "desc": "Number of RPC Server instances spun up on RegionServers.", "display_name": "HBase RegionServer Handler Count", "name": "hbase_regionserver_handler_count", "value": "30" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase RegionServer in Bytes", "name": "hbase_regionserver_java_heapsize", "value": "4294967296" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase RegionServer Interface Class parameter.", "display_name": "Suppress Parameter Validation: HBase RegionServer Interface Class", "name": "role_config_suppression_hbase_regionserver_class", "value": "false" }, { "desc": "When computing the overall RegionServer health, consider the host's health.", "display_name": "RegionServer Host Health Test", "name": "regionserver_host_health_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/impala-llama.json0000666000175100017510000007277413245514472024747 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Kerberos principal used by the Impala Llama ApplicationMaster roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "llama" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_llama_fair_scheduler_safety_valve", "value": "false" }, { "desc": "Whether Llama should cache allocated resources on release.", "display_name": "Enable Resource Caching", "name": "llama_am_cache_enabled", "value": "true" }, { "desc": "When computing the overall Impala Llama ApplicationMaster health, consider the host's health.", "display_name": "Impala Llama ApplicationMaster Host Health Test", "name": "llama_host_health_enabled", "value": "true" }, { "desc": "The group that the Llama processes should run as.", "display_name": "Llama System Group", "name": "process_groupname", "value": "llama" }, { "desc": "Directory where Llama will place its log files.", "display_name": "Llama Log Directory", "name": "llama_log_dir", "value": "/var/log/impala-llama" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_llama_log_directory_free_space", "value": "false" }, { "desc": "Queues Llama ApplicationMaster should connect to at start up.", "display_name": "Core Queues", "name": "llama_am_core_queues", "value": "" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Port on which the Llama ApplicationMaster listens to HTTP requests.", "display_name": "Llama HTTP Port", "name": "llama_http_port", "value": "15001" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_llama_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_llama_host_health", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into llama-site.xml for this role only.", "display_name": "Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml", "name": "llama_config_valve", "value": null }, { "desc": "ACL for Impala ApplicationMaster clients. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. \"alice,bob users,wheel\". A special value of \"*\" means all users are allowed. These take effect only if security is enabled.", "display_name": "Client ACLs", "name": "llama_am_server_thrift_client_acl", "value": "*" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum amount of time the backed off reservations will be in 'backed off' state. The actual amount time is a random value between the minimum and the maximum.", "display_name": "Anti-Deadlock Maximum Delay", "name": "llama_am_gang_anti_deadlock_max_delay_ms", "value": "30000" }, { "desc": "Maximum number of retries for a client notification. After the maximum number of client notification retries has been reached without success the client is considered lost and all its reservations are released. A successful client notification resets the retries count.", "display_name": "Maximum Client Notification Retries", "name": "llama_am_server_thrift_client_notifier_max_retries", "value": "5" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Log Directory parameter.", "display_name": "Suppress Parameter Validation: Llama Log Directory", "name": "role_config_suppression_llama_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml", "name": "role_config_suppression_llama_config_valve", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "Port on which the Llama ApplicationMaster listens to administrative requests on its administrative Thrift interface.", "display_name": "Llama Thrift Admin Port", "name": "llama_am_server_thrift_admin_address", "value": "15002" }, { "desc": "The maximum number of rolled log files to keep for Impala Llama ApplicationMaster logs. Typically used by log4j or logback.", "display_name": "Impala Llama ApplicationMaster Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Timeout policy for resources being cached.", "display_name": "Resource Caching Idle Timeout", "name": "llama_am_cache_eviction_timeout_policy_idle_timeout_ms", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Administrative Interface ACLs parameter.", "display_name": "Suppress Parameter Validation: Administrative Interface ACLs", "name": "role_config_suppression_llama_am_server_thrift_admin_acl", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Impala Llama ApplicationMaster in Bytes", "name": "llama_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Minimum amount of time the backed off reservations will be in 'backed off' state. The actual amount time is a random value between the minimum and the maximum.", "display_name": "Anti-Deadlock Minimum Delay", "name": "llama_am_gang_anti_deadlock_min_delay_ms", "value": "10000" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_llama_file_descriptor", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Maximum number of threads used by the Llama ApplicationMaster auxiliary service uses for serving client requests.", "display_name": "Thrift Server Maximum Threads", "name": "llama_am_server_thrift_server_max_threads", "value": "50" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve)", "name": "LLAMA_role_env_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Interval of time without any new allocation that will trigger the Impala ApplicationMaster anti-deadlock logic.", "display_name": "Anti-Deadlock No Allocation Limit Interval", "name": "llama_am_gang_anti_deadlock_no_allocation_limit_ms", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Time in milliseconds after which Llama will discard its AM for a queue that has been empty of reservations. Does not apply to queues specified with the Core Queues property.", "display_name": "Queue Expiration Age", "name": "llama_am_queue_expire_ms", "value": "300000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client ACLs parameter.", "display_name": "Suppress Parameter Validation: Client ACLs", "name": "role_config_suppression_llama_am_server_thrift_client_acl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_llama_role_env_safety_valve", "value": "false" }, { "desc": "Minimum number of threads used by the Llama ApplicationMaster auxiliary service uses for serving client requests.", "display_name": "Thrift Server Minimum Threads", "name": "llama_am_server_thrift_server_min_threads", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Enter an XML string that will be inserted verbatim into the Fair Scheduler allocations file. Overrides the configuration set using the Pools configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "llama_fair_scheduler_safety_valve", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "llama_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Core Queues parameter.", "display_name": "Suppress Parameter Validation: Core Queues", "name": "role_config_suppression_llama_am_core_queues", "value": "false" }, { "desc": "If enabled, the Impala Llama ApplicationMaster binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind Impala Llama ApplicationMaster to Wildcard Address", "name": "llama_bind_wildcard", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama System Group parameter.", "display_name": "Suppress Parameter Validation: Llama System Group", "name": "role_config_suppression_process_groupname", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The maximum size, in megabytes, per log file for Impala Llama ApplicationMaster logs. Typically used by log4j or logback.", "display_name": "Impala Llama ApplicationMaster Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The minimum log level for Impala Llama ApplicationMaster logs", "display_name": "Impala Llama ApplicationMaster Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Llama Server", "name": "llama_java_opts", "value": "" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Llama ApplicationMaster heartbeat interval, in milliseconds. On each heartbeat the ApplicationMaster submits new reservations to YARN ResourceManager and gets updates from it.", "display_name": "AM Heartbeat Interval", "name": "llama_am_server_thrift_client_notifier_heartbeat_ms", "value": "5000" }, { "desc": "The user that the Llama process should run as.", "display_name": "Llama System User", "name": "process_username", "value": "llama" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_llama_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_llama_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Percentage of resources that will be backed off by the Impala ApplicationMaster anti-deadlock logic. Random reservations will be backed off until the percentage of backed off resources reaches this percentage.", "display_name": "Anti-Deadlock Backoff Percentage", "name": "llama_am_gang_anti_deadlock_backoff_percent", "value": "30" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Port on which the Llama ApplicationMaster serves its Thrift interface.", "display_name": "Llama Thrift Port", "name": "llama_port", "value": "15000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama System User parameter.", "display_name": "Suppress Parameter Validation: Llama System User", "name": "role_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_llama_swap_memory_usage", "value": "false" }, { "desc": "Whether to break resource requests into smaller requests of standard size before the resource cache. The sizes are taken from Yarn settings Container Memory Increment and Container Virtual CPU Cores Increment.", "display_name": "Enable Resource Cache Normalization", "name": "llama_am_resource_normalizing_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Socket time, in milliseconds, used Llama ApplicationMaster auxiliary service for all its server and client Thrift connections.", "display_name": "Thrift Transport Timeout", "name": "llama_am_server_thrift_transport_timeout_ms", "value": "60000" }, { "desc": "Enables the health test that the Impala Llama ApplicationMaster's process state is consistent with the role configuration", "display_name": "Impala Llama ApplicationMaster Process Health Test", "name": "llama_scm_health_enabled", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "ACL for Impala ApplicationMaster admins. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. \"alice,bob users,wheel\". A special value of \"*\" means all users are allowed. These take effect only if security is enabled.", "display_name": "Administrative Interface ACLs", "name": "llama_am_server_thrift_admin_acl", "value": "*" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Llama Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Llama Server", "name": "role_config_suppression_llama_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Client notification retry interval, in milliseconds.", "display_name": "Client Notification Retry Interval", "name": "llama_am_server_thrift_client_notifier_retry_interval_ms", "value": "5000" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/yarn-gateway.json0000666000175100017510000007072013245514472025015 0ustar zuulzuul00000000000000[ { "desc": "For MapReduce job outputs that are compressed, specify the compression codec to use. Will be part of generated client configuration.", "display_name": "Compression Codec of MapReduce Job Output", "name": "mapred_output_compression_codec", "value": "org.apache.hadoop.io.compress.DefaultCodec" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for mapred-site.xml.", "display_name": "MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "mapreduce_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Reduce Task Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: Reduce Task Java Opts Base", "name": "role_config_suppression_mapreduce_reduce_java_opts", "value": "false" }, { "desc": "Whether map tasks should attempt to use the optimized native implementation of the map-side output collector. This can improve performance of many jobs that are shuffle-intensive. Experimental in CDH 5.2.", "display_name": "Enable Optimized Map-side Output Collector", "name": "mapreduce_enable_native_map_output_collector", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Map Task Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Map Task Maximum Heap Size Validator", "name": "role_config_suppression_mapreduce_map_java_opts_max_heap_mapreduce_map_memory_mb_validator", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the ApplicationMaster Java Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: ApplicationMaster Java Maximum Heap Size Validator", "name": "role_config_suppression_yarn_app_mapreduce_am_max_heap_yarn_app_mapreduce_am_resource_mb_validator", "value": "false" }, { "desc": "The number of streams to merge at the same time while sorting files. That is, the number of sort heads to use during the merge sort on the reducer side. This determines the number of open file handles. Merging more files in parallel reduces merge sort iterations and improves run time by eliminating disk I/O. Note that merging more files in parallel uses more memory. If 'io.sort.factor' is set too high or the maximum JVM heap is set too low, excessive garbage collection will occur. The Hadoop default is 10, but Cloudera recommends a higher value. Will be part of generated client configuration.", "display_name": "I/O Sort Factor", "name": "io_sort_factor", "value": "64" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "92" }, { "desc": "Location to store the job history files of running jobs. This is a path on the host where the JobTracker is running.", "display_name": "Running Job History Location", "name": "hadoop_job_history_dir", "value": "/var/log/hadoop-mapreduce/history" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh", "display_name": "Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "mapreduce_client_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codec of MapReduce Job Output parameter.", "display_name": "Suppress Parameter Validation: Compression Codec of MapReduce Job Output", "name": "role_config_suppression_mapred_output_compression_codec", "value": "false" }, { "desc": "If enabled, multiple instances of some reduce tasks may be executed in parallel.", "display_name": "Reduce Tasks Speculative Execution", "name": "mapred_reduce_tasks_speculative_execution", "value": "false" }, { "desc": "The application framework to run jobs with. If not set, jobs will be run with the local job runner.", "display_name": "Application Framework", "name": "mapreduce_framework_name", "value": "yarn" }, { "desc": "Whether to suppress configuration warnings produced by the Job Submit Replication Validator configuration validator.", "display_name": "Suppress Configuration Validator: Job Submit Replication Validator", "name": "role_config_suppression_mapreduce_replication_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Redacted MapReduce Job Properties parameter.", "display_name": "Suppress Parameter Validation: Redacted MapReduce Job Properties", "name": "role_config_suppression_mapreduce_job_redacted_properties", "value": "false" }, { "desc": "Base sleep time between failover attempts. Used only if RM HA is enabled.", "display_name": "Client Failover Sleep Base Time", "name": "client_failover_sleep_base", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Running Job History Location parameter.", "display_name": "Suppress Parameter Validation: Running Job History Location", "name": "role_config_suppression_hadoop_job_history_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for yarn-site.xml.", "display_name": "YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_mapreduce_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Compression level for the codec used to compress MapReduce outputs. Default compression is a balance between speed and compression ratio.", "display_name": "Compression Level of Codecs", "name": "zlib_compress_level", "value": "DEFAULT_COMPRESSION" }, { "desc": "The maximum Java heap size, in bytes, of the map processes. This number will be formatted and concatenated with 'Map Task Java Opts Base' to pass to Hadoop.", "display_name": "Map Task Maximum Heap Size", "name": "mapreduce_map_java_opts_max_heap", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the Reduce Task Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Reduce Task Maximum Heap Size Validator", "name": "role_config_suppression_mapreduce_reduce_java_opts_max_heap_mapreduce_reduce_memory_mb_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Maximum Number of Attempts for MapReduce Jobs parameter.", "display_name": "Suppress Parameter Validation: Maximum Number of Attempts for MapReduce Jobs", "name": "role_config_suppression_mapreduce_am_max_attempts", "value": "false" }, { "desc": "The default number of parallel transfers run by reduce during the copy (shuffle) phase. This number should be between sqrt(nodes*number_of_map_slots_per_node) and nodes*number_of_map_slots_per_node/2. Will be part of generated client configuration.", "display_name": "Default Number of Parallel Transfers During Shuffle", "name": "mapred_reduce_parallel_copies", "value": "10" }, { "desc": "The maximum heap size, in bytes, of the Java MapReduce ApplicationMaster. This number will be formatted and concatenated with 'ApplicationMaster Java Opts Base' to pass to Hadoop.", "display_name": "ApplicationMaster Java Maximum Heap Size", "name": "yarn_app_mapreduce_am_max_heap", "value": "825955249" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codec of MapReduce Map Output parameter.", "display_name": "Suppress Parameter Validation: Compression Codec of MapReduce Map Output", "name": "role_config_suppression_mapred_map_output_compression_codec", "value": "false" }, { "desc": "The replication level for submitted job files.", "display_name": "Mapreduce Submit Replication", "name": "mapred_submit_replication", "value": "10" }, { "desc": "The total amount of memory buffer, in megabytes, to use while sorting files. Note that this memory comes out of the user JVM heap size (meaning total user JVM heap - this amount of memory = total user usable heap space. Note that Cloudera's default differs from Hadoop's default; Cloudera uses a bigger buffer by default because modern machines often have more RAM. The smallest value across all TaskTrackers will be part of generated client configuration.", "display_name": "I/O Sort Memory Buffer (MiB)", "name": "io_sort_mb", "value": "256" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_yarn_client_config_safety_valve", "value": "false" }, { "desc": "Whether to enable ubertask optimization, which runs \"sufficiently small\" jobs sequentially within a single JVM. \"Small\" is defined by the mapreduce.job.ubertask.maxmaps, mapreduce.job.ubertask.maxreduces, and mapreduce.job.ubertask.maxbytes settings.", "display_name": "Enable Ubertask Optimization", "name": "mapreduce_job_ubertask_enabled", "value": "false" }, { "desc": "The ratio of heap size to container size for both map and reduce tasks. The heap should be smaller than the container size to allow for some overhead of the JVM.", "display_name": "Heap to Container Size Ratio", "name": "mapreduce_job_heap_memory_mb_ratio", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_mapreduce_client_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The amount of physical memory, in MiB, allocated for each reduce task of a job. For versions before CDH 5.5, if not specified, by default it is set to 1024. For CDH 5.5 and higher, a value less than 128 is not supported but if it is specified as 0, the amount of physical memory to request is inferred from Reduce Task Maximum Heap Size and Heap to Container Size Ratio. If Reduce Task Maximum Heap Size is not specified, by default the amount of physical memory to request is set to 1024. This parameter has no effect prior to CDH 4.4.", "display_name": "Reduce Task Memory", "name": "mapreduce_reduce_memory_mb", "value": "0" }, { "desc": "The maximum permissible size of the split metainfo file. The JobTracker won't attempt to read split metainfo files bigger than the configured value. No limits if set to -1.", "display_name": "JobTracker MetaInfo Maxsize", "name": "mapreduce_jobtracker_split_metainfo_maxsize", "value": "10000000" }, { "desc": "Fraction of the number of map tasks in the job which should be completed before reduce tasks are scheduled for the job.", "display_name": "Number of Map Tasks to Complete Before Reduce Tasks", "name": "mapred_reduce_slowstart_completed_maps", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Classpath parameter.", "display_name": "Suppress Parameter Validation: MR Application Classpath", "name": "role_config_suppression_mapreduce_application_classpath", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Environment parameter.", "display_name": "Suppress Parameter Validation: MR Application Environment", "name": "role_config_suppression_mapreduce_admin_user_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Environment parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Environment", "name": "role_config_suppression_yarn_app_mapreduce_am_admin_user_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Framework Path parameter.", "display_name": "Suppress Parameter Validation: MR Application Framework Path", "name": "role_config_suppression_mapreduce_application_framework_path", "value": "false" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "mapreduce_client_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "The maximum Java heap size, in bytes, of the reduce processes. This number will be formatted and concatenated with 'Reduce Task Java Opts Base' to pass to Hadoop.", "display_name": "Reduce Task Maximum Heap Size", "name": "mapreduce_reduce_java_opts_max_heap", "value": "0" }, { "desc": "The physical memory requirement, in MiB, for the ApplicationMaster.", "display_name": "ApplicationMaster Memory", "name": "yarn_app_mapreduce_am_resource_mb", "value": "1024" }, { "desc": "A comma-separated list of job properties to redact in MapReduce.", "display_name": "Redacted MapReduce Job Properties", "name": "mapreduce_job_redacted_properties", "value": "fs.s3a.access.key,fs.s3a.secret.key" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "mapreduce_client_java_heapsize", "value": "825955249" }, { "desc": "The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string.", "display_name": "Mapreduce Task Timeout", "name": "mapred_task_timeout", "value": "600000" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Environment variables for the MapReduce ApplicationMaster. These settings can be overridden in the ApplicationMaster User Environment (yarn.app.mapreduce.am.env).", "display_name": "ApplicationMaster Environment", "name": "yarn_app_mapreduce_am_admin_user_env", "value": "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH" }, { "desc": "Limit on the number of counter groups allowed per job.", "display_name": "Job Counter Groups Limit", "name": "mapreduce_job_counter_groups_limit", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Map Task Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: Map Task Java Opts Base", "name": "role_config_suppression_mapreduce_map_java_opts", "value": "false" }, { "desc": "The amount of physical memory, in MiB, allocated for each map task of a job. For versions before CDH 5.5, if not specified, by default it is set to 1024. For CDH 5.5 and higher, a value less than 128 is not supported but if it is specified as 0, the amount of physical memory to request is inferred from Map Task Maximum Heap Size and Heap to Container Size Ratio. If Map Task Maximum Heap Size is not specified, by default the amount of physical memory to request is set to 1024.", "display_name": "Map Task Memory", "name": "mapreduce_map_memory_mb", "value": "0" }, { "desc": "Threshold for number of reduces, beyond which a job is considered too big for ubertask optimization. Note: As of CDH 5, MR2 does not support more than one reduce in an ubertask. (Zero is valid.)", "display_name": "Ubertask Maximum Reduces", "name": "mapreduce_job_ubertask_maxreduces", "value": "1" }, { "desc": "The number of virtual CPU cores for each reduce task of a job.", "display_name": "Reduce Task CPU Virtual Cores", "name": "mapreduce_reduce_cpu_vcores", "value": "1" }, { "desc": "For MapReduce map outputs that are compressed, specify the compression codec to use. Will be part of generated client configuration.", "display_name": "Compression Codec of MapReduce Map Output", "name": "mapred_map_output_compression_codec", "value": "org.apache.hadoop.io.compress.SnappyCodec" }, { "desc": "Classpaths to include for MapReduce applications. During evaluation, the string '{version}' in the value of this parameter will be replaced by the actual MapReduce version.", "display_name": "MR Application Classpath", "name": "mapreduce_application_classpath", "value": "$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH" }, { "desc": "Java command line arguments passed to the MapReduce ApplicationMaster.", "display_name": "ApplicationMaster Java Opts Base", "name": "yarn_app_mapreduce_am_command_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Maximum sleep time between failover attempts. Used only if RM HA is enabled.", "display_name": "Client Failover Sleep Max Time", "name": "client_failover_sleep_max", "value": "2000" }, { "desc": "Path for MapReduce application framework. During evaluation, the string '{version}' in the value of this parameter will be replaced by the actual MapReduce version.", "display_name": "MR Application Framework Path", "name": "mapreduce_application_framework_path", "value": "/user/yarn/mapreduce/mr-framework/{version}-mr-framework.tar.gz#mr-framework" }, { "desc": "Compress the output of MapReduce jobs. Will be part of generated client configuration.", "display_name": "Compress MapReduce Job Output", "name": "mapred_output_compress", "value": "false" }, { "desc": "For MapReduce job outputs that are compressed as SequenceFiles, you can select one of these compression type options: NONE, RECORD or BLOCK. Cloudera recommends BLOCK. Will be part of generated client configuration.", "display_name": "Compression Type of MapReduce Job Output", "name": "mapred_output_compression_type", "value": "BLOCK" }, { "desc": "Java opts for the map processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp pass a value of: \"-verbose:gc -Xloggc:/tmp/@taskid@.gc\". The configuration variable 'Map Task Memory' can be used to control the maximum memory of the map processes.", "display_name": "Map Task Java Opts Base", "name": "mapreduce_map_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Java Opts Base", "name": "role_config_suppression_yarn_app_mapreduce_am_command_opts", "value": "false" }, { "desc": "Additional execution environment entries for map and reduce task processes.", "display_name": "MR Application Environment", "name": "mapreduce_admin_user_env", "value": "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH" }, { "desc": "The soft limit in either the buffer or record collection buffers. When this limit is reached, a thread will begin to spill the contents to disk in the background. Note that this does not imply any chunking of data to the spill. A value less than 0.5 is not recommended. The syntax is in decimal units; the default is 80% and is formatted 0.8. Will be part of generated client configuration.", "display_name": "I/O Sort Spill Percent", "name": "io_sort_spill_percent", "value": "0.8" }, { "desc": "The default number of reduce tasks per job. Will be part of generated client configuration.", "display_name": "Default Number of Reduce Tasks per Job", "name": "mapred_reduce_tasks", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Limit on the number of counters allowed per job.", "display_name": "Job Counters Limit", "name": "mapreduce_job_counters_limit", "value": "120" }, { "desc": "The virtual CPU cores requirement, for the ApplicationMaster. This parameter has no effect prior to CDH 4.4.", "display_name": "ApplicationMaster Virtual CPU Cores", "name": "yarn_app_mapreduce_am_resource_cpu_vcores", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the I/O Sort Factor parameter.", "display_name": "Suppress Parameter Validation: I/O Sort Factor", "name": "role_config_suppression_io_sort_factor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "role_config_suppression_mapreduce_client_env_safety_valve", "value": "false" }, { "desc": "Size of buffer for read and write operations of SequenceFiles.", "display_name": "SequenceFile I/O Buffer Size", "name": "io_file_buffer_size", "value": "65536" }, { "desc": "If enabled, multiple instances of some map tasks may be executed in parallel.", "display_name": "Map Tasks Speculative Execution", "name": "mapred_map_tasks_speculative_execution", "value": "false" }, { "desc": "The number of virtual CPU cores allocated for each map task of a job. This parameter has no effect prior to CDH 4.4.", "display_name": "Map Task CPU Virtual Cores", "name": "mapreduce_map_cpu_vcores", "value": "1" }, { "desc": "Java opts for the reduce processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp pass a value of: \"-verbose:gc -Xloggc:/tmp/@taskid@.gc\". The configuration variable 'Reduce Task Memory' can be used to control the maximum memory of the reduce processes.", "display_name": "Reduce Task Java Opts Base", "name": "mapreduce_reduce_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "If enabled, uses compression on the map outputs before they are sent across the network. Will be part of generated client configuration.", "display_name": "Use Compression on Map Outputs", "name": "mapred_compress_map_output", "value": "true" }, { "desc": "The maximum number of application attempts for MapReduce jobs. The value of this parameter overrides ApplicationMaster Maximum Attempts for MapReduce jobs.", "display_name": "Maximum Number of Attempts for MapReduce Jobs", "name": "mapreduce_am_max_attempts", "value": "2" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Threshold for number of maps, beyond which a job is considered too big for ubertask optimization.", "display_name": "Ubertask Maximum Maps", "name": "mapreduce_job_ubertask_maxmaps", "value": "9" }, { "desc": "File format the AM will use when generating the .jhist file. Valid values are \"json\" for text output and \"binary\" for faster parsing.", "display_name": "JHist File Format", "name": "mapred_jobhistory_jhist_format", "value": "binary" }, { "desc": "Threshold for number of input bytes, beyond which a job is considered too big for ubertask optimization. If no value is specified, dfs.block.size is used as a default.", "display_name": "Ubertask Maximum Job Size", "name": "mapreduce_job_ubertask_maxbytes", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hue-hue_server.json0000666000175100017510000005514013245514472025332 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Top Banner Custom HTML parameter.", "display_name": "Suppress Parameter Validation: Top Banner Custom HTML", "name": "role_config_suppression_banner_html", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hue TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hue TLS/SSL Validator", "name": "role_config_suppression_hue_ssl_validator", "value": "false" }, { "desc": "Encrypt communication between clients and Hue using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Hue", "name": "ssl_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_ssl_private_key", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hue_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "Hue Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hue_server_hive_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Private Key Password", "name": "role_config_suppression_ssl_private_key_password", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hue_server_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Directory where Hue Server will place its log files.", "display_name": "Hue Server Log Directory", "name": "hue_server_log_dir", "value": "/var/log/hue" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hue_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hue Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hue_server_role_env_safety_valve", "value": "false" }, { "desc": "Timeout in seconds for Thrift calls to HiveServer2 and Impala.", "display_name": "HiveServer2 and Impala Thrift Connection Timeout", "name": "hs2_conn_timeout", "value": "120" }, { "desc": "An optional, custom one-line HTML code to display as a banner on top of all Hue Server web pages. Useful in displaying cluster identity of the Hue Server.", "display_name": "Top Banner Custom HTML", "name": "banner_html", "value": null }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_hue_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Server Log Directory", "name": "role_config_suppression_hue_server_log_dir", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "hue_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hue_server_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hue_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hue_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Location on HDFS where the jobsub examples and templates are stored.", "display_name": "Jobsub Examples and Templates Directory", "name": "hue_server_remote_data_dir", "value": "/user/hue/jobsub" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The path to the TLS/SSL file containing the certificate of the certificate authority (CA) and any intermediate certificates used to sign the server certificate. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format, and is usually created by concatenating together all of the appropriate root and intermediate certificates.", "display_name": "Hue TLS/SSL Server CA Certificate (PEM Format)", "name": "ssl_cacerts", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_ssl_certificate", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hue_server_host_health", "value": "false" }, { "desc": "When computing the overall Hue Server health, consider the host's health.", "display_name": "Hue Server Host Health Test", "name": "hue_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hue_server_file_descriptor", "value": "false" }, { "desc": "Port to use to connect to the Hue server.", "display_name": "Hue HTTP Port", "name": "hue_http_port", "value": "8888" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Jobsub Examples and Templates Directory parameter.", "display_name": "Suppress Parameter Validation: Jobsub Examples and Templates Directory", "name": "role_config_suppression_hue_server_remote_data_dir", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file.", "display_name": "Metrics Sample File Logging Frequency", "name": "hue_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Secret Key parameter.", "display_name": "Suppress Parameter Validation: Secret Key", "name": "role_config_suppression_secret_key", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Hue Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HUE_SERVER_role_env_safety_valve", "value": null }, { "desc": "For advanced use only. A string to be inserted into hue_safety_valve_server.ini for this role only.", "display_name": "Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini", "name": "hue_server_hue_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hue_metrics_sample_file_location", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue TLS/SSL Server Private Key File (PEM Format)", "name": "ssl_private_key", "value": null }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hue/metrics-hue_server/metrics.log.", "display_name": "Metrics Sample File Location", "name": "hue_metrics_sample_file_location", "value": null }, { "desc": "The password for the private key in the Hue TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Hue TLS/SSL Private Key Password", "name": "ssl_private_key_password", "value": null }, { "desc": "If enabled, the Hue server binds to the wildcard address (\"0.0.0.0\") for its ports.", "display_name": "Bind Hue Server to Wildcard Address", "name": "hue_server_bind_wildcard", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini parameter.", "display_name": "Suppress Parameter Validation: Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini", "name": "role_config_suppression_hue_server_hue_safety_valve", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "hue_server_web_metric_collection_enabled", "value": "true" }, { "desc": "Enables the health test that the Hue Server's process state is consistent with the role configuration", "display_name": "Hue Server Process Health Test", "name": "hue_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue TLS/SSL Server CA Certificate (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Hue TLS/SSL Server CA Certificate (PEM Format)", "name": "role_config_suppression_ssl_cacerts", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Hue is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Hue TLS/SSL Server Certificate File (PEM Format)", "name": "ssl_certificate", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hue Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hue_server_hive_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Random string used for secure hashing in the session store.", "display_name": "Secret Key", "name": "secret_key", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hive-metastore-sentry-safety.xml0000666000175100017510000000135213245514472027776 0ustar zuulzuul00000000000000 hive.metastore.client.impl org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient Sets custom Hive metastore client which Sentry uses to filter out metadata. hive.metastore.pre.event.listeners org.apache.sentry.binding.metastore.MetastoreAuthzBinding list of comma separated listeners for metastore events. hive.metastore.event.listeners org.apache.sentry.binding.metastore.SentryMetastorePostEventListener list of comma separated listeners for metastore, post events. sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hive-webhcat.json0000666000175100017510000005125013245514472024750 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_webhcat_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_webhcat_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_webhcat_log_directory_free_space", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "webhcat_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Port on which WebHCat Server will listen for connections.", "display_name": "WebHCat Server Port", "name": "hive_webhcat_address_port", "value": "50111" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_webhcat_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The maximum number of rolled log files to keep for WebHCat Server logs. Typically used by log4j or logback.", "display_name": "WebHCat Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Directory where WebHCat Server will place its log files.", "display_name": "WebHCat Server Log Directory", "name": "hcatalog_log_dir", "value": "/var/log/hcatalog" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of WebHCat Server in Bytes", "name": "hive_webhcat_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Log Directory", "name": "role_config_suppression_hcatalog_log_dir", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_webhcat_hive_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_webhcat_scm_health", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "WebHCat Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "WebHCat Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_webhcat_env_safety_valve", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for WebHCat Server", "name": "hive_webhcat_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Enables the health test that the WebHCat Server's process state is consistent with the role configuration", "display_name": "WebHCat Server Process Health Test", "name": "webhcat_scm_health_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_webhcat_swap_memory_usage", "value": "false" }, { "desc": "When computing the overall WebHCat Server health, consider the host's health.", "display_name": "WebHCat Server Host Health Test", "name": "webhcat_host_health_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into webhcat-site.xml for this role only.", "display_name": "WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml", "name": "hive_webhcat_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_webhcat_hive_config_safety_valve", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for WebHCat Server logs. Typically used by log4j or logback.", "display_name": "WebHCat Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_webhcat_host_health", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_webhcat_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for WebHCat Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for WebHCat Server", "name": "role_config_suppression_hive_webhcat_java_opts", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml", "name": "role_config_suppression_hive_webhcat_config_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for WebHCat Server logs", "display_name": "WebHCat Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/mapred-site.xml0000666000175100017510000001050113245514472024435 0ustar zuulzuul00000000000000 mapreduce.job.split.metainfo.maxsize 10000000 mapreduce.job.counters.max 120 mapreduce.output.fileoutputformat.compress false mapreduce.output.fileoutputformat.compress.type BLOCK mapreduce.output.fileoutputformat.compress.codec org.apache.hadoop.io.compress.DefaultCodec mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.compress true zlib.compress.level DEFAULT_COMPRESSION mapreduce.task.io.sort.factor 64 mapreduce.map.sort.spill.percent 0.8 mapreduce.reduce.shuffle.parallelcopies 10 mapreduce.task.timeout 600000 mapreduce.client.submit.file.replication 10 mapreduce.job.reduces 1 mapreduce.task.io.sort.mb 256 mapreduce.map.speculative false mapreduce.reduce.speculative false mapreduce.job.reduce.slowstart.completedmaps 0.8 mapreduce.jobhistory.address test-master-001.novalocal:10020 mapreduce.jobhistory.webapp.address test-master-001.novalocal:19888 mapreduce.jobhistory.webapp.https.address test-master-001.novalocal:19890 mapreduce.jobhistory.admin.address test-master-001.novalocal:10033 mapreduce.framework.name yarn yarn.app.mapreduce.am.staging-dir /user yarn.app.mapreduce.am.resource.mb 1024 yarn.app.mapreduce.am.resource.cpu-vcores 1 mapreduce.job.ubertask.enable false yarn.app.mapreduce.am.command-opts -Djava.net.preferIPv4Stack=true -Xmx825955249 mapreduce.map.java.opts -Djava.net.preferIPv4Stack=true -Xmx825955249 mapreduce.reduce.java.opts -Djava.net.preferIPv4Stack=true -Xmx825955249 yarn.app.mapreduce.am.admin.user.env LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH mapreduce.map.memory.mb 1024 mapreduce.map.cpu.vcores 1 mapreduce.reduce.memory.mb 1024 mapreduce.reduce.cpu.vcores 1 mapreduce.application.classpath $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH mapreduce.admin.user.env LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH mapreduce.shuffle.max.connections 80 sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/sqoop-service.json0000666000175100017510000001600113245514472025174 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_sqoop_env_safety_valve", "value": "false" }, { "desc": "When computing the overall SQOOP health, consider Sqoop 2 Server's health", "display_name": "Sqoop 2 Server Role Health Test", "name": "sqoop_sqoop_server_health_enabled", "value": "true" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "sqoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Sqoop 2 Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Sqoop 2 Server Count Validator", "name": "service_config_suppression_sqoop_server_count_validator", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "sqoop_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "sqoop2" }, { "desc": "Whether to suppress the results of the Sqoop 2 Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sqoop 2 Server Health", "name": "service_health_suppression_sqoop_sqoop_server_health", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "sqoop2" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "MapReduce jobs are run against this service.", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/solr-gateway.json0000666000175100017510000000437313245514472025024 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/solr" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/impala-impalad.json0000666000175100017510000012601613245514472025255 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Impala Daemon Scratch Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Scratch Directories Free Space", "name": "role_health_suppression_impalad_scratch_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Lineage Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Lineage Enabled Validator", "name": "role_config_suppression_impala_lineage_enabled_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemons Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Impala Daemons Load Balancer", "name": "role_config_suppression_impalad_load_balancer", "value": "false" }, { "desc": "The health test thresholds for failures encountered when monitoring queries within a recent period specified by the impala_query_monitoring_failure_window configuration for the role.", "display_name": "Query Monitoring Failures Thresholds", "name": "impala_query_monitoring_failure_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_hdfs_site_conf_safety_valve", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Impala Daemon command-line flags. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "impalad_cmd_args_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Impala Audit Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Audit Enabled Validator", "name": "role_config_suppression_impala_audit_enabled_validator", "value": "false" }, { "desc": "Directory where an Impala Daemon core dump is placed.", "display_name": "Impala Daemon Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/impalad" }, { "desc": "Port on which HiveServer2 client requests are served by Impala Daemons.", "display_name": "Impala Daemon HiveServer2 Port", "name": "hs2_port", "value": "21050" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_fair_scheduler_safety_valve", "value": "false" }, { "desc": "Maximum number of seconds that Impala attempts to register or re-register with Llama. If registration is unsuccessful, Impala cancels the action with an error, which could result in an impalad startup failure or a cancelled query. A setting of -1 seconds means try indefinitely.", "display_name": "Llama Registration Timeout Seconds", "name": "llama_registration_timeout_secs", "value": "30" }, { "desc": "When computing the overall Impala Daemon health, consider the host's health.", "display_name": "Impala Daemon Host Health Test", "name": "impalad_host_health_enabled", "value": "true" }, { "desc": "The directory in which Impala daemon audit event log files are written. If \"Impala Audit Event Generation\" property is enabled, Impala will generate its audit logs in this directory.", "display_name": "Impala Daemon Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/impalad/audit" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "A list of key-value pairs of additional query options to pass to the Impala Daemon command line, separated by ','.", "display_name": "Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve)", "name": "default_query_options", "value": "" }, { "desc": "Port on which Beeswax client requests are served by Impala Daemons.", "display_name": "Impala Daemon Beeswax Port", "name": "beeswax_port", "value": "21000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Enable or disable the Impala Daemon web server. This web server contains useful information about Impala Daemon.", "display_name": "Enable Impala Daemon Web Server", "name": "impalad_enable_webserver", "value": "true" }, { "desc": "An XML snippet to append to llama-site.xml for Impala Daemons. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve)", "name": "impala_llama_site_conf_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "impalad_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)", "name": "IMPALAD_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "Enables the health test that verifies the Impala Daemon is connected to the StateStore.", "display_name": "Impala Daemon Connectivity Health Test", "name": "impalad_connectivity_health_enabled", "value": "true" }, { "desc": "Enables the health check that determines if query collection for the Impala Daemon is successful.", "display_name": "Impala Daemon Query Collection Status Health Check", "name": "impala_query_monitoring_status_check_enabled", "value": "true" }, { "desc": "The amount of time at Impala Daemon startup allowed for the Impala Daemon to start accepting new queries for processing.", "display_name": "Impala Daemon Ready Status Startup Tolerance", "name": "impalad_ready_status_check_startup_tolerance", "value": "180" }, { "desc": "The maximum size (in queries) of the Impala Daemon audit event log file before a new one is created.", "display_name": "Impala Daemon Maximum Audit Log File Size", "name": "max_audit_event_log_file_size", "value": "5000" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_impalad_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Local UDF Library Dir parameter.", "display_name": "Suppress Parameter Validation: Local UDF Library Dir", "name": "role_config_suppression_local_library_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Server CA Certificate parameter.", "display_name": "Suppress Parameter Validation: LDAP Server CA Certificate", "name": "role_config_suppression_impalad_ldap_ca_certificate", "value": "false" }, { "desc": "The directory in which Impala daemon lineage log files are written. If \"Impala Lineage Generation\" property is enabled, Impala generates its lineage logs in this directory.", "display_name": "Impala Daemon Lineage Log Directory", "name": "lineage_event_log_dir", "value": "/var/log/impalad/lineage" }, { "desc": "The password for the private key in the Impala Daemon Webserver TLS/SSL Server Certificate and Private Key file. If left blank, the private key is not protected by a password.", "display_name": "Impala Daemon Webserver TLS/SSL Private Key Password", "name": "webserver_private_key_password_cmd", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_llama_site_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Scratch Directories parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Scratch Directories", "name": "role_config_suppression_scratch_dirs", "value": "false" }, { "desc": "Number of retry attempts the Impala Daemon will make when connecting to the Hive Metastore Server. By default, the Impala Daemon waits one second between retries.", "display_name": "Impala Daemon Hive Metastore Connection Retries", "name": "hive_metastore_connect_retries", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Breakpad Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Breakpad Dump Directory", "name": "role_config_suppression_minidump_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "User-defined function (UDF) libraries are copied from HDFS into this local directory.", "display_name": "Local UDF Library Dir", "name": "local_library_dir", "value": "/var/lib/impala/udfs" }, { "desc": "Number of seconds to wait between attempts during Llama registration.", "display_name": "Llama Registration Wait Seconds", "name": "llama_registration_wait_secs", "value": "3" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "impalad_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_impalad_unexpected_exits", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Impala Daemon logs. Typically used by log4j or logback.", "display_name": "Impala Daemon Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Username for Impala Daemon webserver authentication.", "display_name": "Impala Daemon Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Password for Impala Daemon webserver authentication.", "display_name": "Impala Daemon Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Directory where Impala Daemon will place its log files.", "display_name": "Impala Daemon Log Directory", "name": "log_dir", "value": "/var/log/impalad" }, { "desc": "An XML string to use verbatim for the contents of fair-scheduler.xml for Impala Daemons. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve)", "name": "impalad_fair_scheduler_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_cmd_args_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_default_query_options", "value": "false" }, { "desc": "Port on which ImpalaBackendService is exported.", "display_name": "Impala Daemon Backend Port", "name": "be_port", "value": "22000" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml for this role only.", "display_name": "Impala Daemon HBase Advanced Configuration Snippet (Safety Valve)", "name": "impalad_hbase_conf_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml for this role only.", "display_name": "Impala Daemon Hive Advanced Configuration Snippet (Safety Valve)", "name": "impala_hive_conf_safety_valve", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon HBase Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon HBase Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_hbase_conf_safety_valve", "value": "false" }, { "desc": "Directory for storing Impala Daemon Breakpad dumps.", "display_name": "Impala Daemon Breakpad Dump Directory", "name": "minidump_path", "value": "/var/log/impala-minidumps" }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above. ", "display_name": "Impala Daemon Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_impalad_memory_rss_health", "value": "false" }, { "desc": "Encrypt and verify the integrity of all data spilled to disk as part of a query. This feature is only supported for Impala 2.0 and higher and CDH 5.2 and higher (which includes Impala 2.0).", "display_name": "Disk Spill Encryption", "name": "disk_spill_encryption", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_impalad_swap_memory_usage", "value": "false" }, { "desc": "The timeout used by the Cloudera Manager Agent's query monitor when communicating with the Impala Daemon web server, specified in seconds.", "display_name": "Query Monitoring Timeout", "name": "executing_queries_timeout_seconds", "value": "5.0" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_impalad_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Hive Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Hive Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_hive_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Private Key Password", "name": "role_config_suppression_webserver_private_key_password_cmd", "value": "false" }, { "desc": "Enables audit event generation by Impala daemons. The audit log file will be placed in the directory specified by 'Impala Daemon Audit Log Directory' parameter.", "display_name": "Enable Impala Audit Event Generation", "name": "enable_audit_event_log", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Audit Log Directory", "name": "role_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_impalad_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_impalad_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Daemon Ready Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Ready Check", "name": "role_health_suppression_impalad_ready_status", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"any\"}" }, { "desc": "Enables the health check that determines if the Impala daemon is ready to process queries.", "display_name": "Impala Daemon Ready Status Health Check", "name": "impalad_ready_status_check_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "The polling period of the Impala query monitor in the Cloudera Manager Agent, specified in seconds. If set to zero, query monitoring is disabled.", "display_name": "Query Monitoring Period", "name": "query_monitoring_period_seconds", "value": "1.0" }, { "desc": "Whether to suppress the results of the Query Monitoring Status Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Query Monitoring Status Check", "name": "role_health_suppression_impalad_query_monitoring_status", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_impalad_log_directory_free_space", "value": "false" }, { "desc": "The path to the TLS/SSL file containing the server certificate key used for TLS/SSL. Used when Impala Daemon Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala Daemon Webserver TLS/SSL Server Certificate File (PEM Format)", "name": "webserver_certificate_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Impala Daemon Scratch Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Impala Daemon Scratch Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Impala Daemon Scratch Directories Free Space Monitoring Percentage Thresholds", "name": "impalad_scratch_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Maximum number of query results a client may request to be cached on a per-query basis to support restarting fetches. This option guards against unreasonably large result caches requested by clients. Requests exceeding this maximum will be rejected.", "display_name": "Result Cache Maximum Size", "name": "impalad_result_cache_max_size", "value": "100000" }, { "desc": "Whether to suppress the results of the StateStore Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: StateStore Connectivity", "name": "role_health_suppression_impalad_connectivity", "value": "false" }, { "desc": "Abort Impala startup if there are improper configs or running on unsupported hardware.", "display_name": "Abort on Config Error", "name": "abort_on_config_error", "value": "true" }, { "desc": "Directories where Impala Daemon will write data such as spilling information to disk to free up memory. This can potentially be large amounts of data.", "display_name": "Impala Daemon Scratch Directories", "name": "scratch_dirs", "value": null }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "Impala Daemon Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "Enables lineage generation by Impala daemons. The lineage log file is placed in the directory specified by the 'Impala Daemon Lineage Log Directory' parameter.", "display_name": "Enable Impala Lineage Generation", "name": "enable_lineage_log", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_impalad_host_health", "value": "false" }, { "desc": "The maximum size (in entries) of the Impala daemon lineage log file before a new one is created.", "display_name": "Impala Daemon Maximum Lineage Log File Size", "name": "max_lineage_log_file_size", "value": "5000" }, { "desc": "The path to the TLS/SSL file containing the private key used for TLS/SSL. Used when Impala Daemon Webserver is acting as a TLS/SSL server. The certificate file must be in PEM format.", "display_name": "Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "webserver_private_key_file", "value": null }, { "desc": "The location on disk of the certificate, in .pem format, used to confirm the authenticity of the LDAP server certificate. This is the Certificate Authority (CA) certificate, and it was used to sign the LDAP server certificate. If not set, all certificates are trusted, which means that an attacker could potentially intercept otherwise encrypted usernames and passwords.", "display_name": "LDAP Server CA Certificate", "name": "impalad_ldap_ca_certificate", "value": null }, { "desc": "Timeout for requests to the Hive Metastore Server from Impala. Consider increasing this if you have tables with a lot of metadata and see timeout errors.", "display_name": "Impala Daemon Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "3600" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The period to review when computing query monitoring failures.", "display_name": "Monitoring Period For Query Monitoring Failures", "name": "impala_query_monitoring_failure_window", "value": "5" }, { "desc": "Maximum number of Breakpad dump files stored by Impala Daemon Role.", "display_name": "Impala Daemon Max Breakpad Dump Files", "name": "max_minidumps", "value": "9" }, { "desc": "Port where StateStoreSubscriberService is running.", "display_name": "StateStoreSubscriber Service Port", "name": "state_store_subscriber_port", "value": "23000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Webserver TLS/SSL Server Private Key File (PEM Format)", "name": "role_config_suppression_webserver_private_key_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Lineage Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Lineage Log Directory", "name": "role_config_suppression_lineage_event_log_dir", "value": "false" }, { "desc": "Address of the load balancer used for Impala daemons. Should be specified in host:port format. If this is specified and Kerberos is enabled, Cloudera Manager adds a principal for 'impala/<load_balancer_host>@<realm>' to the keytab for all Impala daemons.", "display_name": "Impala Daemons Load Balancer", "name": "impalad_load_balancer", "value": null }, { "desc": "Maximum number of times a request to reserve, expand, or release resources is attempted until the request is cancelled.", "display_name": "Llama Maximum Request Attempts", "name": "llama_max_request_attempts", "value": "5" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The minimum log level for Impala Daemon logs", "display_name": "Impala Daemon Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Port where Impala debug web server runs.", "display_name": "Impala Daemon HTTP Server Port", "name": "impalad_webserver_port", "value": "25000" }, { "desc": "Enables the health test that the Impala Daemon's process state is consistent with the role configuration", "display_name": "Impala Daemon Process Health Test", "name": "impalad_scm_health_enabled", "value": "true" }, { "desc": "Memory limit in bytes for Impala Daemon, enforced by the daemon itself. If reached, queries running on the Impala Daemon may be killed. Leave it blank to let Impala pick its own limit. Use a value of -1 B to specify no limit.", "display_name": "Impala Daemon Memory Limit", "name": "impalad_memory_limit", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of time to wait for the Impala Daemon to fully start up and connect to the StateStore before enforcing the connectivity check.", "display_name": "Impala Daemon Connectivity Tolerance at Startup", "name": "impalad_connectivity_tolerance", "value": "180" }, { "desc": "Port where Llama notification callback should be started", "display_name": "Llama Callback Port", "name": "llama_callback_port", "value": "28000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Impala Daemon Scratch Directories.", "display_name": "Impala Daemon Scratch Directories Free Space Monitoring Absolute Thresholds", "name": "impalad_scratch_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml for this role only.", "display_name": "Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve)", "name": "impala_hdfs_site_conf_safety_valve", "value": null }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "impalad_web_metric_collection_enabled", "value": "true" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "Impala Maximum Log Files", "name": "max_log_files", "value": "10" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-namenode.json0000666000175100017510000015545413245514472025125 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Checkpoint Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Checkpoint Status", "name": "role_health_suppression_name_node_ha_checkpoint_age", "value": "false" }, { "desc": "Name of the journal located on each JournalNode filesystem.", "display_name": "Quorum-based Storage Journal name", "name": "dfs_namenode_quorum_journal_name", "value": null }, { "desc": "Timeout when starting a new edit segment with JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Start Segment Timeout", "name": "dfs_qjournal_start_segment_timeout_ms", "value": "20000" }, { "desc": "Indicate whether or not to avoid reading from stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Stale DataNodes are moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.", "display_name": "Avoid Reading Stale DataNode", "name": "dfs_namenode_avoid_read_stale_datanode", "value": "false" }, { "desc": "The base port where the DFS NameNode web UI listens. If the port number is 0, then the server starts on a free port. Combined with the NameNode's hostname to build its HTTP address.", "display_name": "NameNode Web UI Port", "name": "dfs_http_port", "value": "50070" }, { "desc": "Enables the health test that the NameNode is not in safemode", "display_name": "NameNode Safemode Health Test", "name": "namenode_safe_mode_enabled", "value": "true" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Timeout when accepting recovery of an edit segment from JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Accept Recovery Timeout", "name": "dfs_qjournal_accept_recovery_timeout_ms", "value": "120000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_name_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "namenode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health check thresholds of the NameNode's RPC latency.", "display_name": "NameNode RPC Latency Thresholds", "name": "namenode_rpc_latency_thresholds", "value": "{\"critical\":\"5000.0\",\"warning\":\"1000.0\"}" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "namenode_pause_duration_window", "value": "5" }, { "desc": "Timeout when writing edits to a JournalNode. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Write Transactions Timeout", "name": "dfs_qjournal_write_txns_timeout_ms", "value": "20000" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the RPC Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: RPC Latency", "name": "role_health_suppression_name_node_rpc_latency", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_name_node_file_descriptor", "value": "false" }, { "desc": "The maximum number of outgoing replication threads a node can have at one time. This limit is waived for the highest priority replications. Configure dfs.namenode.replication.max-streams-hard-limit to set the absolute limit, including the highest-priority replications.", "display_name": "Maximum Number of Replication Threads on a DataNode", "name": "dfs_namenode_replication_max_streams", "value": "20" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When the ratio of number stale DataNodes to total DataNodes marked is greater than this ratio, permit writing to stale nodes to prevent causing hotspots.", "display_name": "Write Stale DataNode Ratio", "name": "dfs_namenode_write_stale_datanode_ratio", "value": "0.5" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "namenode_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_name_node_swap_memory_usage", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the NameNode Data Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Data Directories Free Space", "name": "role_health_suppression_name_node_data_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Filesystem Trash Interval On Validator configuration validator.", "display_name": "Suppress Configuration Validator: Filesystem Trash Interval On Validator", "name": "role_config_suppression_fs_trash_interval_minimum_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_namenode_config_safety_valve", "value": "false" }, { "desc": "Enables the health test that the NameNode's process state is consistent with the role configuration", "display_name": "NameNode Process Health Test", "name": "namenode_scm_health_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: NameNode Log Directory", "name": "role_config_suppression_namenode_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Plugins parameter.", "display_name": "Suppress Parameter Validation: NameNode Plugins", "name": "role_config_suppression_dfs_namenode_plugins_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Nameservice parameter.", "display_name": "Suppress Parameter Validation: NameNode Nameservice", "name": "role_config_suppression_dfs_federation_namenode_nameservice", "value": "false" }, { "desc": "Enables the health test of the rolling metadata upgrade status of the NameNode. This covers rolling metadata upgrades. Nonrolling metadata upgrades are covered in a separate health test.", "display_name": "HDFS Rolling Metadata Upgrade Status Health Test", "name": "namenode_rolling_upgrade_status_enabled", "value": "true" }, { "desc": "Mount points that are mapped to this NameNode's nameservice.", "display_name": "Mount Points", "name": "nameservice_mountpoints", "value": "/" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "namenode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"},\n {\"alert\": false, \"rate\": 1, \"threshold\":\"INFO\", \"content\":\"Triggering checkpoint.*\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NameNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NameNode", "name": "role_config_suppression_namenode_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Edits Directories parameter.", "display_name": "Suppress Parameter Validation: NameNode Edits Directories", "name": "role_config_suppression_dfs_namenode_edits_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt", "name": "role_config_suppression_namenode_hosts_exclude_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Service Handler Count Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Service Handler Count Minimum Validator", "name": "role_config_suppression_dfs_namenode_service_handler_count_minimum_validator", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Timeout when preparing recovery of an edit segment with JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Prepare Recovery Timeout", "name": "dfs_qjournal_prepare_recovery_timeout_ms", "value": "120000" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NameNode in Bytes", "name": "namenode_java_heapsize", "value": "4294967296" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the JournalNode Sync Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JournalNode Sync Status", "name": "role_health_suppression_name_node_journal_node_sync_status", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for NameNode logs. Typically used by log4j or logback.", "display_name": "NameNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Determines where on the local file system the NameNode should store the name table (fsimage). For redundancy, enter a comma-delimited list of directories to replicate the name table in all of the directories. Typical values are /data/N/dfs/nn where N=1..3.", "display_name": "NameNode Data Directories", "name": "dfs_name_dir_list", "value": null }, { "desc": "Whether to suppress the results of the Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Upgrade Status", "name": "role_health_suppression_name_node_upgrade_status", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Edits Directory parameter.", "display_name": "Suppress Parameter Validation: Shared Edits Directory", "name": "role_config_suppression_dfs_namenode_shared_edits_dir", "value": "false" }, { "desc": "Default time interval for marking a DataNode as \"stale\". If the NameNode has not received heartbeat messages from a DataNode for more than this time interval, the DataNode is marked and treated as \"stale\" by default.", "display_name": "Stale DataNode Time Interval", "name": "dfs_namenode_stale_datanode_interval", "value": "30000" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_name_node_pause_duration", "value": "false" }, { "desc": "The port where the NameNode runs the HDFS protocol. Combined with the NameNode's hostname to build its address.", "display_name": "NameNode Port", "name": "namenode_port", "value": "8020" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The period to review when computing the moving average of the NameNode's RPC latency.", "display_name": "NameNode RPC Latency Monitoring Window", "name": "namenode_rpc_latency_window", "value": "5" }, { "desc": "Optional port for the service-rpc address which can be used by HDFS daemons instead of sharing the RPC address used by the clients.", "display_name": "NameNode Service RPC Port", "name": "dfs_namenode_servicerpc_address", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Quorum-based Storage Journal name parameter.", "display_name": "Suppress Parameter Validation: Quorum-based Storage Journal name", "name": "role_config_suppression_dfs_namenode_quorum_journal_name", "value": "false" }, { "desc": "The health check thresholds for the number of out-of-sync JournalNodes for this NameNode.", "display_name": "NameNode Out-Of-Sync JournalNodes Thresholds", "name": "namenode_out_of_sync_journal_nodes_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories.", "display_name": "NameNode Data Directories Free Space Monitoring Absolute Thresholds", "name": "namenode_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The absolute maximum number of outgoing replication threads a given node can have at one time. The regular limit (dfs.namenode.replication.max-streams) is waived for highest-priority block replications. Highest replication priority is for blocks that are at a very high risk of loss if the disk or server on which they remain fails. These are usually blocks with only one copy, or blocks with zero live copies but a copy in a node being decommissioned. dfs.namenode.replication.max-streams-hard-limit provides a limit on the total number of outgoing replication threads, including threads of all priorities.", "display_name": "Hard Limit on the Number of Replication Threads on a Datanode", "name": "dfs_namenode_replication_max_streams_hard_limit", "value": "40" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "NAMENODE_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where NameNode will place its log files.", "display_name": "NameNode Log Directory", "name": "namenode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Enables the health test of the metadata upgrade status of the NameNode. This covers nonrolling metadata upgrades. Rolling metadata upgrades are covered in a separate health test.", "display_name": "HDFS Metadata Upgrade Status Health Test", "name": "namenode_upgrade_status_enabled", "value": "true" }, { "desc": "Nameservice of this NameNode. The Nameservice represents the interface to this NameNode and its High Availability partner. The Nameservice also represents the namespace associated with a federated NameNode.", "display_name": "NameNode Nameservice", "name": "dfs_federation_namenode_nameservice", "value": null }, { "desc": "The minimum log level for NameNode logs", "display_name": "NameNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.replication.min. Enter a value less than or equal to 0 to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safemode permanent.", "display_name": "Safemode Threshold Percentage", "name": "dfs_safemode_threshold_pct", "value": "0.999" }, { "desc": "Directories on the local file system to store the NameNode edits. If not set, the edits are stored in the NameNode's Data Directories. The value of this configuration is automatically generated to be the Quorum-based Storage URI if there are JournalNodes and this NameNode is not Highly Available.", "display_name": "NameNode Edits Directories", "name": "dfs_namenode_edits_dir", "value": null }, { "desc": "If set to false and if one of the replicas of the NameNode storage fails, such as temporarily failure of NFS, this directory is not used until the NameNode restarts. If enabled, failed storage is re-checked on every checkpoint and, if it becomes valid, the NameNode will try to restore the edits and fsimage.", "display_name": "Restore NameNode Directories at Checkpoint Time", "name": "dfs_name_dir_restore", "value": "false" }, { "desc": "Timeout when creating new epoch number with JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode New Epoch Timeout", "name": "dfs_qjournal_new_epoch_timeout_ms", "value": "120000" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "namenode_startup_tolerance", "value": "5" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NameNode", "name": "namenode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the Java Heap Size of NameNode in Bytes Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: Java Heap Size of NameNode in Bytes Minimum Validator", "name": "role_config_suppression_namenode_java_heapsize_minimum_validator", "value": "false" }, { "desc": "Enable Automatic Failover to maintain High Availability. Requires a ZooKeeper service and a High Availability NameNode partner.", "display_name": "Enable Automatic Failover", "name": "autofailover_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NameNode Data Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NameNode Data Directories Free Space Monitoring Percentage Thresholds", "name": "namenode_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Timeout when finalizing current edit segment with JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Finalize Segment Timeout", "name": "dfs_qjournal_finalize_segment_timeout_ms", "value": "120000" }, { "desc": "For advanced use only. A string to be inserted into dfs_hosts_exclude.txt for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt", "name": "namenode_hosts_exclude_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Safe Mode Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Safe Mode Status", "name": "role_health_suppression_name_node_safe_mode", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_namenode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Rolling Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Rolling Upgrade Status", "name": "role_health_suppression_name_node_rolling_upgrade_status", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "This determines the percentage amount of block invalidations (deletes) to do over a single DataNode heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DataNode.", "display_name": "Invalidate Work Percentage Per Iteration", "name": "dfs_namenode_invalidate_work_pct_per_iteration", "value": "0.32" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "namenode_web_metric_collection_enabled", "value": "true" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_name_node_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The time between two periodic file system checkpoints.", "display_name": "Filesystem Checkpoint Period", "name": "fs_checkpoint_period", "value": "3600" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When computing the overall NameNode health, consider the host's health.", "display_name": "NameNode Host Health Test", "name": "namenode_host_health_enabled", "value": "true" }, { "desc": "The access time for HDFS file is precise upto this value. Setting the value of 0 disables access times for HDFS. When using the NFS Gateway role, make sure this property is enabled.", "display_name": "Access Time Precision", "name": "dfs_access_time_precision", "value": "3600000" }, { "desc": "Minimum number of running threads for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Min Threadcount", "name": "dfs_thrift_threads_min", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topology Script File Name parameter.", "display_name": "Suppress Parameter Validation: Topology Script File Name", "name": "role_config_suppression_topology_script_file_name", "value": "false" }, { "desc": "The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired.", "display_name": "Filesystem Checkpoint Transaction Threshold", "name": "fs_checkpoint_txns", "value": "1000000" }, { "desc": "Maximum number of running threads for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Max Threadcount", "name": "dfs_thrift_threads_max", "value": "20" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Handler Count Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Handler Count Minimum Validator", "name": "role_config_suppression_dfs_namenode_handler_count_minimum_validator", "value": "false" }, { "desc": "Full path to a custom topology script on the host file system. The topology script is used to determine the rack location of nodes. If left blank, a topology script will be provided that uses your hosts' rack information, visible in the \"Hosts\" page.", "display_name": "Topology Script File Name", "name": "topology_script_file_name", "value": null }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The minimum log level for NameNode block state change log messages. Setting this to WARN or higher greatly reduces the amount of log output related to block state changes.", "display_name": "NameNode Block State Change Logging Threshold", "name": "namenode_blockstatechange_log_threshold", "value": "INFO" }, { "desc": "Indicate whether or not to avoid writing to stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Writes avoid using stale DataNodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of DataNodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads.", "display_name": "Avoid Writing Stale DataNode", "name": "dfs_namenode_avoid_write_stale_datanode", "value": "false" }, { "desc": "The health test thresholds of failed status directories in a NameNode.", "display_name": "NameNode Directory Failures Thresholds", "name": "namenode_directory_failures_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_name_node_unexpected_exits", "value": "false" }, { "desc": "Number of minutes between trash checkpoints. Also controls the number of minutes after which a trash checkpoint directory is deleted. To disable the trash feature, enter 0.", "display_name": "Filesystem Trash Interval", "name": "fs_trash_interval", "value": "1440" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "namenode_config_safety_valve", "value": null }, { "desc": "This determines the total amount of block transfers to begin in parallel at a DataNode for replication, when such a command list is being sent over a DataNode heartbeat by the NameNode. The actual number is obtained by multiplying this value by the total number of live nodes in the cluster. The result number is the number of blocks to transfer immediately, per DataNode heartbeat.", "display_name": "Replication Work Multiplier Per Iteration", "name": "dfs_namenode_replication_work_multiplier_per_iteration", "value": "10" }, { "desc": "The maximum number of rolled log files to keep for NameNode logs. Typically used by log4j or logback.", "display_name": "NameNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Determines extension of safemode in milliseconds after the threshold level is reached.", "display_name": "Safemode Extension", "name": "dfs_safemode_extension", "value": "30000" }, { "desc": "Timeout in seconds for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Timeout", "name": "dfs_thrift_timeout", "value": "60" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The number of server threads for the NameNode used for service calls. Only used when NameNode Service RPC Port is configured.", "display_name": "NameNode Service Handler Count", "name": "dfs_namenode_service_handler_count", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt", "name": "role_config_suppression_namenode_hosts_allow_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Data Directories parameter.", "display_name": "Suppress Parameter Validation: NameNode Data Directories", "name": "role_config_suppression_dfs_name_dir_list", "value": "false" }, { "desc": "The health test thresholds of the number of transactions since the last HDFS namespace checkpoint. Specified as a percentage of the configured checkpointing transaction limit.", "display_name": "Filesystem Checkpoint Transactions Monitoring Thresholds", "name": "namenode_checkpoint_transactions_thresholds", "value": "{\"critical\":\"400.0\",\"warning\":\"200.0\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_name_node_web_metric_collection", "value": "false" }, { "desc": "The health test thresholds of the age of the HDFS namespace checkpoint. Specified as a percentage of the configured checkpoint interval.", "display_name": "Filesystem Checkpoint Age Monitoring Thresholds", "name": "namenode_checkpoint_age_thresholds", "value": "{\"critical\":\"400.0\",\"warning\":\"200.0\"}" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_name_node_scm_health", "value": "false" }, { "desc": "The base port where the secure NameNode web UI listens.", "display_name": "Secure NameNode Web UI Port (TLS/SSL)", "name": "dfs_https_port", "value": "50470" }, { "desc": "For advanced use only. A string to be inserted into dfs_hosts_allow.txt for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt", "name": "namenode_hosts_allow_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mount Points parameter.", "display_name": "Suppress Parameter Validation: Mount Points", "name": "role_config_suppression_nameservice_mountpoints", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "If enabled, the NameNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind NameNode to Wildcard Address", "name": "namenode_bind_wildcard", "value": "false" }, { "desc": "Comma-separated list of NameNode plug-ins to be activated. If one plug-in cannot be loaded, all the plug-ins are ignored.", "display_name": "NameNode Plugins", "name": "dfs_namenode_plugins_list", "value": "" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_name_node_host_health", "value": "false" }, { "desc": "Directory on a shared storage device, such as a Quorum-based Storage URI or a local directory that is an NFS mount from a NAS, to store the NameNode edits. The value of this configuration is automatically generated to be the Quourm Journal URI if there are JournalNodes and this NameNode is Highly Available.", "display_name": "Shared Edits Directory", "name": "dfs_namenode_shared_edits_dir", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The number of server threads for the NameNode.", "display_name": "NameNode Handler Count", "name": "dfs_namenode_handler_count", "value": "30" }, { "desc": "Timeout when selecting input streams on JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Select Input Streams Timeout", "name": "dfs_qjournal_select_input_streams_timeout_ms", "value": "20000" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Timeout when getting current states from JournalNodes. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Get State Timeout", "name": "dfs_qjournal_get_journal_state_timeout_ms", "value": "120000" }, { "desc": "Whether to suppress the results of the Name Directory Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Name Directory Status", "name": "role_health_suppression_name_node_directory_failures", "value": "false" }, { "desc": "Specifies the number of DataNodes that must be live before the name node exits safemode. Enter a value less than or equal to 0 to take the number of live DataNodes into account when deciding whether to remain in safemode during startup. Values greater than the number of DataNodes in the cluster will make safemode permanent.", "display_name": "Safemode Minimum DataNodes", "name": "dfs_safemode_min_datanodes", "value": "0" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/spark-service.json0000666000175100017510000002266713245514472025172 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Spark Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_spark_on_yarn_service_env_safety_valve", "value": "false" }, { "desc": "Enable whether the Spark communication protocols do authentication using a shared secret. If using Spark2, ensure that value of this property is the same in both services.", "display_name": "Spark Authentication", "name": "spark_authenticate", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark History Location (HDFS) parameter.", "display_name": "Suppress Parameter Validation: Spark History Location (HDFS)", "name": "service_config_suppression_spark_history_log_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into spark-conf/spark-env.sh. Applies to configurations of all roles in this service except client configuration.", "display_name": "Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_service_safety_valve", "value": null }, { "desc": "The location of Spark application history logs in HDFS. Changing this value will not move existing logs to the new location.", "display_name": "Spark History Location (HDFS)", "name": "spark_history_log_dir", "value": "/user/spark/applicationHistory" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark JAR Location (HDFS) parameter.", "display_name": "Suppress Parameter Validation: Spark JAR Location (HDFS)", "name": "service_config_suppression_spark_jar_hdfs_path", "value": "false" }, { "desc": "The location of the Spark JAR in HDFS. If left blank, Cloudera Manager will use the Spark JAR installed on the cluster nodes.", "display_name": "Spark JAR Location (HDFS)", "name": "spark_jar_hdfs_path", "value": "" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "spark" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Spark Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "SPARK_ON_YARN_service_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Name of the YARN (MR2 Included) service that this Spark service instance depends on", "display_name": "YARN (MR2 Included) Service", "name": "yarn_service", "value": null }, { "desc": "The port the Spark Shuffle Service listens for fetch requests. If using Spark2, ensure that value of this property is the same in both services.", "display_name": "Spark Shuffle Service Port", "name": "spark_shuffle_service_port", "value": "7337" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "service_config_suppression_spark-conf/spark-env.sh_service_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the History Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: History Server Count Validator", "name": "service_config_suppression_spark_yarn_history_server_count_validator", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-secondarynamenode.json0000666000175100017510000007625113245514472027032 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"},\n {\"alert\": false, \"rate\": 1, \"threshold\":\"INFO\", \"content\":\"Triggering checkpoint.*\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_secondary_name_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Secondary NameNode", "name": "secondarynamenode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "secondarynamenode_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories.", "display_name": "HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds", "name": "secondarynamenode_checkpoint_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Secondary NameNode in Bytes", "name": "secondary_namenode_java_heapsize", "value": "4294967296" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Nameservice parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Nameservice", "name": "role_config_suppression_dfs_secondarynamenode_nameservice", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_secondary_name_node_scm_health", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "secondarynamenode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_secondary_name_node_web_metric_collection", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Enables the health test that the SecondaryNameNode's process state is consistent with the role configuration", "display_name": "SecondaryNameNode Process Health Test", "name": "secondarynamenode_scm_health_enabled", "value": "true" }, { "desc": "The maximum number of rolled log files to keep for SecondaryNameNode logs. Typically used by log4j or logback.", "display_name": "SecondaryNameNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The time between two periodic file system checkpoints.", "display_name": "Filesystem Checkpoint Period", "name": "fs_checkpoint_period", "value": "3600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Checkpoint Directories parameter.", "display_name": "Suppress Parameter Validation: HDFS Checkpoint Directories", "name": "role_config_suppression_fs_checkpoint_dir_list", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "secondarynamenode_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_secondary_name_node_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "secondarynamenode_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Secondary NameNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Secondary NameNode", "name": "role_config_suppression_secondarynamenode_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_secondary_name_node_gc_duration", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "SECONDARYNAMENODE_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_secondary_name_node_unexpected_exits", "value": "false" }, { "desc": "If enabled, the SecondaryNameNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind SecondaryNameNode to Wildcard Address", "name": "secondary_namenode_bind_wildcard", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired.", "display_name": "Filesystem Checkpoint Transaction Threshold", "name": "fs_checkpoint_txns", "value": "1000000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_secondarynamenode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the HDFS Checkpoint Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HDFS Checkpoint Directories Free Space", "name": "role_health_suppression_secondary_name_node_checkpoint_directories_free_space", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_secondary_name_node_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Log Directory", "name": "role_config_suppression_secondarynamenode_log_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "HDFS Checkpoint Directories Free Space Monitoring Percentage Thresholds", "name": "secondarynamenode_checkpoint_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When computing the overall SecondaryNameNode health, consider the host's health.", "display_name": "SecondaryNameNode Host Health Test", "name": "secondarynamenode_host_health_enabled", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for SecondaryNameNode logs. Typically used by log4j or logback.", "display_name": "SecondaryNameNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_secondarynamenode_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "secondarynamenode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The base port where the secure SecondaryNameNode web UI listens.", "display_name": "Secure SecondaryNameNode Web UI Port (TLS/SSL)", "name": "dfs_secondary_https_port", "value": "50495" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_secondary_name_node_log_directory_free_space", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "secondarynamenode_gc_duration_window", "value": "5" }, { "desc": "Determines where on the local file system the HDFS SecondaryNameNode should store the temporary images to merge. For redundancy, enter a comma-delimited list of directories to replicate the image in all of the directories. Typical values are /data/N/dfs/snn for N = 1, 2, 3...", "display_name": "HDFS Checkpoint Directories", "name": "fs_checkpoint_dir_list", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_secondary_name_node_file_descriptor", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Nameservice of this SecondaryNameNode", "display_name": "SecondaryNameNode Nameservice", "name": "dfs_secondarynamenode_nameservice", "value": null }, { "desc": "The minimum log level for SecondaryNameNode logs", "display_name": "SecondaryNameNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Directory where SecondaryNameNode will place its log files.", "display_name": "SecondaryNameNode Log Directory", "name": "secondarynamenode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "The SecondaryNameNode HTTP port. If the port is 0, then the server starts on a free port. Combined with the SecondaryNameNode's hostname to build its HTTP address.", "display_name": "SecondaryNameNode Web UI Port", "name": "dfs_secondary_http_port", "value": "50090" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/yarn-jobhistory.json0000666000175100017510000007571213245514472025556 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Kerberos principal used by the JobHistory Server roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "mapred" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message. If a log message matches multiple rules, the first matching rule will be used.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:

  • {\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}
    This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.
  • {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"exceptiontype\": \".*\"}, {\"alert\": true, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"ERROR\"}
    In this example, an event generated may not be promoted to alert if an exception is in the ERROR log message, as the first rule with alert = false will match.
", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for JobHistory Server", "name": "mr2_jobhistory_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_jobhistory_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Enables the health test that the JobHistory Server's process state is consistent with the role configuration", "display_name": "JobHistory Server Process Health Test", "name": "jobhistory_scm_health_enabled", "value": "true" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_jobhistory_gc_duration", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "JobHistory Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of JobHistory Server in Bytes", "name": "mr2_jobhistory_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_jobhistory_scm_health", "value": "false" }, { "desc": "When computing the overall JobHistory Server health, consider the host's health.", "display_name": "JobHistory Server Host Health Test", "name": "jobhistory_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_history_process_groupname", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for JobHistory Server logs. Typically used by log4j or logback.", "display_name": "JobHistory Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_jobhistory_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_jobhistory_host_health", "value": "false" }, { "desc": "Directory where JobHistory Server will place its log files.", "display_name": "JobHistory Server Log Directory", "name": "mr2_jobhistory_log_dir", "value": "/var/log/hadoop-mapreduce" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_jobhistory_swap_memory_usage", "value": "false" }, { "desc": "Time interval for history cleaner to check for files to delete. Files are only deleted if they are older than mapreduce.jobhistory.max-age-ms.", "display_name": "Job History Files Cleaner Interval", "name": "mapreduce_jobhistory_cleaner_interval", "value": "86400000" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "jobhistory_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "jobhistory_gc_duration_window", "value": "5" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_jobhistory_config_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MapReduce ApplicationMaster Staging Root Directory parameter.", "display_name": "Suppress Parameter Validation: MapReduce ApplicationMaster Staging Root Directory", "name": "role_config_suppression_yarn_app_mapreduce_am_staging_dir", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "jobhistory_config_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "JobHistory Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "JOBHISTORY_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Log Directory", "name": "role_config_suppression_mr2_jobhistory_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_jobhistory_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_jobhistory_file_descriptor", "value": "false" }, { "desc": "The port of the MapReduce JobHistory Server. Together with the hostname of the JobHistory role, forms the address.", "display_name": "MapReduce JobHistory Server Port", "name": "mapreduce_jobhistory_address", "value": "10020" }, { "desc": "The group that the JobHistory Server process should run as.", "display_name": "System Group", "name": "history_process_groupname", "value": "hadoop" }, { "desc": "Job history files older than this time duration will deleted when the history cleaner runs.", "display_name": "Job History Files Maximum Age", "name": "mapreduce_jobhistory_max_age_ms", "value": "604800000" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for JobHistory Server logs. Typically used by log4j or logback.", "display_name": "JobHistory Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_history_process_username", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "jobhistory_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The HTTP port of the MapReduce JobHistory Server web application. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Web Application HTTP Port", "name": "mapreduce_jobhistory_webapp_address", "value": "19888" }, { "desc": "The root HDFS directory of the staging area for users' MR2 jobs; for example /user. The staging directories are always named after the user.", "display_name": "MapReduce ApplicationMaster Staging Root Directory", "name": "yarn_app_mapreduce_am_staging_dir", "value": "/user" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "jobhistory_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The minimum log level for JobHistory Server logs", "display_name": "JobHistory Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The port of the MapReduce JobHistory Server administrative interface. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Server Admin Interface Port", "name": "mapreduce_jobhistory_admin_address", "value": "10033" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "jobhistory_mapred_safety_valve", "value": null }, { "desc": "If enabled, the JobHistory Server binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind JobHistory Server to Wildcard Address", "name": "yarn_jobhistory_bind_wildcard", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JobHistory Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for JobHistory Server", "name": "role_config_suppression_mr2_jobhistory_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_jobhistory_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The user that the JobHistory Server process should run as.", "display_name": "System User", "name": "history_process_username", "value": "mapred" }, { "desc": "The HTTPS port of the MapReduce JobHistory Server web application. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Web Application HTTPS Port (TLS/SSL)", "name": "mapreduce_jobhistory_webapp_https_address", "value": "19890" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_jobhistory_mapred_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "jobhistory_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_jobhistory_web_metric_collection", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/flume-service.json0000666000175100017510000002436513245514472025157 0ustar zuulzuul00000000000000[ { "desc": "Name of the Hbase service that this Flume service instance depends on", "display_name": "Hbase Service", "name": "hbase_service", "value": null }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Flume might connect to. This is used when Flume is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Flume TLS/SSL Certificate Trust Store File", "name": "flume_truststore_file", "value": null }, { "desc": "Whether to suppress the results of the Agent Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Agent Health", "name": "service_health_suppression_flume_agents_healthy", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "flume" }, { "desc": "Name of the HDFS service that this Flume service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Agent Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Agent Count Validator", "name": "service_config_suppression_agent_count_validator", "value": "false" }, { "desc": "Name of the Solr service that this Flume service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Flume Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "flume_env_safety_valve", "value": null }, { "desc": "Sets the maximum number of Flume components that will be returned under Flume Metric Details. Increasing this value will negatively impact the interactive performance of the Flume Metrics Details page.", "display_name": "Maximum displayed Flume metrics components", "name": "flume_context_groups_request_limit", "value": "1000" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "flume" }, { "desc": "The health test thresholds of the overall Agent health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Agents falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Agents falls below the critical threshold.", "display_name": "Healthy Agent Monitoring Thresholds", "name": "flume_agents_healthy_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"95.0\"}" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "flume" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Flume TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_flume_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "The password for the Flume TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Flume TLS/SSL Certificate Trust Store Password", "name": "flume_truststore_password", "value": null }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Flume TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_flume_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Flume Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_flume_env_safety_valve", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/kafka-service.json0000666000175100017510000005267413245514472025130 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Controlled Shutdown Maximum Attempts parameter.", "display_name": "Suppress Parameter Validation: Controlled Shutdown Maximum Attempts", "name": "service_config_suppression_controlled.shutdown.max.retries", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_kafka_service_env_safety_valve", "value": "false" }, { "desc": "The default number of partitions for automatically created topics.", "display_name": "Default Number of Partitions", "name": "num.partitions", "value": "1" }, { "desc": "The amount of time to retain delete messages for log compacted topics. Once a consumer has seen an original message you need to ensure it also sees the delete message. If you removed the delete message too quickly, this might not happen. As a result there is a configurable delete retention time.", "display_name": "Log Compaction Delete Record Retention Time", "name": "log.cleaner.delete.retention.ms", "value": "604800000" }, { "desc": "Enables auto creation of topics on the server. If this is set to true, then attempts to produce, consume, or fetch metadata for a non-existent topic automatically create the topic with the default replication factor and number of partitions.", "display_name": "Topic Auto Creation", "name": "auto.create.topics.enable", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Number of threads used to replicate messages from leaders. Increasing this value increases the degree of I/O parallelism in the follower broker.", "display_name": "Number of Replica Fetchers", "name": "num.replica.fetchers", "value": "1" }, { "desc": "Enables Kafka monitoring.", "display_name": "Enable Kafka Monitoring (Note: Requires Kafka-1.3.0 parcel or higher)", "name": "monitoring.enabled", "value": "true" }, { "desc": "If automatic leader rebalancing is enabled, the controller tries to balance leadership for partitions among the brokers by periodically returning leadership for each partition to the preferred replica, if it is available.", "display_name": "Enable Automatic Leader Rebalancing", "name": "auto.leader.rebalance.enable", "value": "true" }, { "desc": "Number of unsuccessful controlled shutdown attempts before executing an unclean shutdown. For example, the default value of 3 means that the system will attempt a controlled shutdown 3 times before executing an unclean shutdown.", "display_name": "Controlled Shutdown Maximum Attempts", "name": "controlled.shutdown.max.retries", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "The number of partitions for the offset commit topic. Since changing this after deployment is currently unsupported, we recommend using a higher setting for production (for example, 100-200).", "display_name": "Offset Commit Topic Number of Partitions", "name": "offsets.topic.num.partitions", "value": "50" }, { "desc": "If a follower has not sent any fetch requests, nor has it consumed up to the leader's log end offset during this time, the leader removes the follower from the ISR set.", "display_name": "Allowed Replica Time Lag", "name": "replica.lag.time.max.ms", "value": "10000" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Replica Maximum Fetch Size parameter.", "display_name": "Suppress Parameter Validation: Replica Maximum Fetch Size", "name": "service_config_suppression_replica.fetch.max.bytes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The replication factor for the offset commit topic. A higher setting is recommended in order to ensure higher availability (for example, 3 or 4) . If the offsets topic is created when there are fewer brokers than the replication factor, then the offsets topic is created with fewer replicas.", "display_name": "Offset Commit Topic Replication Factor", "name": "offsets.topic.replication.factor", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the Kafka Broker Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kafka Broker Count Validator", "name": "service_config_suppression_kafka_broker_count_validator", "value": "false" }, { "desc": "Controls how frequently the log cleaner will attempt to clean the log. This ratio bounds the maximum space wasted in the log by duplicates. For example, at 0.5 at most 50% of the log could be duplicates. A higher ratio will mean fewer, more efficient cleanings but will mean more wasted space in the log.", "display_name": "Log Cleaner Clean Ratio", "name": "log.cleaner.min.cleanable.ratio", "value": "0.5" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "kafka" }, { "desc": "Enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so might result in data loss.", "display_name": "Enable Unclean Leader Election", "name": "unclean.leader.election.enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Default Replication Factor", "name": "service_config_suppression_default.replication.factor", "value": "false" }, { "desc": "Enables controlled shutdown of the broker. If enabled, the broker moves all leaders on it to other brokers before shutting itself down. This reduces the unavailability window during shutdown.", "display_name": "Enable Controlled Shutdown", "name": "controlled.shutdown.enable", "value": "true" }, { "desc": "The frequency with which to check for leader imbalance.", "display_name": "Leader Imbalance Check Interval", "name": "leader.imbalance.check.interval.seconds", "value": "300" }, { "desc": "The maximum number of bytes to fetch for each partition in fetch requests replicas send to the leader. This value should be larger than message.max.bytes.", "display_name": "Replica Maximum Fetch Size", "name": "replica.fetch.max.bytes", "value": "1048576" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Session Timeout parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Session Timeout", "name": "service_config_suppression_zookeeper.session.timeout.ms", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of Replica Fetchers parameter.", "display_name": "Suppress Parameter Validation: Number of Replica Fetchers", "name": "service_config_suppression_num.replica.fetchers", "value": "false" }, { "desc": "Any producer distinguished by clientId will get throttled if it produces more bytes than this value per-second. Only respected by Kafka 2.0 or later.", "display_name": "Default Producer Quota", "name": "quota.producer.default", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Kafka Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_service_env_safety_valve", "value": null }, { "desc": "The maximum size of a message that the server can receive. It is important that this property be in sync with the maximum fetch size the consumers use, or else an unruly producer could publish messages too large for consumers to consume.", "display_name": "Maximum Message Size", "name": "message.max.bytes", "value": "1000000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Leader Imbalance Allowed Per Broker parameter.", "display_name": "Suppress Parameter Validation: Leader Imbalance Allowed Per Broker", "name": "service_config_suppression_leader.imbalance.per.broker.percentage", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Enables the log cleaner to compact topics with cleanup.policy=compact on this cluster.", "display_name": "Enable Log Compaction", "name": "log.cleaner.enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Minimum Number of Replicas in ISR parameter.", "display_name": "Suppress Parameter Validation: Minimum Number of Replicas in ISR", "name": "service_config_suppression_min.insync.replicas", "value": "false" }, { "desc": "ZNode in ZooKeeper that should be used as a root for this Kafka cluster.", "display_name": "ZooKeeper Root", "name": "zookeeper.chroot", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the Kafka MirrorMaker Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kafka MirrorMaker Count Validator", "name": "service_config_suppression_kafka_mirror_maker_count_validator", "value": "false" }, { "desc": "The number of messages written to a log partition before triggering an fsync on the log. Setting this lower syncs data to disk more often, but has a major impact on performance. We recommend use of replication for durability rather than depending on single-server fsync; however, this setting can be used to be extra certain. If used in conjunction with log.flush.interval.ms, the log is flushed when either criteria is met.", "display_name": "Log Flush Message Interval", "name": "log.flush.interval.messages", "value": null }, { "desc": "The number of background threads to use for log cleaning.", "display_name": "Number of Log Cleaner Threads", "name": "log.cleaner.threads", "value": "1" }, { "desc": "Enable Kerberos authentication for this KAFKA service.", "display_name": "Enable Kerberos Authentication", "name": "kerberos.auth.enable", "value": "false" }, { "desc": "List of metric reporter class names. HTTP reporter is included by default.", "display_name": "List of Metric Reporters", "name": "kafka.metrics.reporters", "value": "nl.techop.kafka.KafkaHttpMetricsReporter" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Root parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Root", "name": "service_config_suppression_zookeeper.chroot", "value": "false" }, { "desc": "The frequency, in ms, with which the log flusher checks whether any log is eligible to be flushed to disk.", "display_name": "Log Flush Scheduler Interval", "name": "log.flush.scheduler.interval.ms", "value": null }, { "desc": "The minimum number of replicas in the in-sync replica needed to satisfy a produce request where required.acks=-1 (that is, all).", "display_name": "Minimum Number of Replicas in ISR", "name": "min.insync.replicas", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Offset Commit Topic Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Offset Commit Topic Replication Factor", "name": "service_config_suppression_offsets.topic.replication.factor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Leader Imbalance Check Interval parameter.", "display_name": "Suppress Parameter Validation: Leader Imbalance Check Interval", "name": "service_config_suppression_leader.imbalance.check.interval.seconds", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "If the server fails to send a heartbeat to ZooKeeper within this period of time, it is considered dead. If set too low, ZooKeeper might falsely consider a server dead; if set too high, ZooKeeper might take too long to recognize a dead server.", "display_name": "ZooKeeper Session Timeout", "name": "zookeeper.session.timeout.ms", "value": "6000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Maximum Message Size parameter.", "display_name": "Suppress Parameter Validation: Maximum Message Size", "name": "service_config_suppression_message.max.bytes", "value": "false" }, { "desc": "Enables topic deletion using admin tools. When delete topic is disabled, deleting topics through the admin tools has no effect.", "display_name": "Enable Delete Topic", "name": "delete.topic.enable", "value": "true" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "If a replica falls more than this number of messages behind the leader, the leader removes the follower from the ISR and treats it as dead. This property is deprecated in Kafka 1.4.0; higher versions use only replica.lag.time.max.ms.", "display_name": "Allowed Replica Message Lag", "name": "replica.lag.max.messages", "value": "4000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Number of Partitions parameter.", "display_name": "Suppress Parameter Validation: Default Number of Partitions", "name": "service_config_suppression_num.partitions", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Kafka service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The total memory used for log deduplication across all cleaner threads. This memory is statically allocated and will not cause GC problems.", "display_name": "Log Cleaner Deduplication Buffer Size", "name": "log.cleaner.dedupe.buffer.size", "value": "134217728" }, { "desc": "Any consumer distinguished by clientId/consumer group will get throttled if it fetches more bytes than this value per-second. Only respected by Kafka 2.0 or later.", "display_name": "Default Consumer Quota", "name": "quota.consumer.default", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Offset Commit Topic Number of Partitions parameter.", "display_name": "Suppress Parameter Validation: Offset Commit Topic Number of Partitions", "name": "service_config_suppression_offsets.topic.num.partitions", "value": "false" }, { "desc": "The default replication factor for automatically created topics.", "display_name": "Default Replication Factor", "name": "default.replication.factor", "value": "1" }, { "desc": "The maximum time between fsync calls on the log. If used in conjuction with log.flush.interval.messages, the log is flushed when either criteria is met.", "display_name": "Log Flush Time Interval", "name": "log.flush.interval.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the List of Metric Reporters parameter.", "display_name": "Suppress Parameter Validation: List of Metric Reporters", "name": "service_config_suppression_kafka.metrics.reporters", "value": "false" }, { "desc": "The percentage of leader imbalance allowed per broker. The controller rebalances leadership if this ratio goes above the configured value per broker.", "display_name": "Leader Imbalance Allowed Per Broker", "name": "leader.imbalance.per.broker.percentage", "value": "10" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/hdfs-httpfs.json0000666000175100017510000006371213245514472024642 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "httpfs_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_httpfs_scm_health", "value": "false" }, { "desc": "Kerberos principal used by the HttpFS roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "httpfs" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_httpfs_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_httpfs_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_httpfs_host_health", "value": "false" }, { "desc": "Address of the load balancer used for HttpFS roles. Should be specified in host:port format. Note: Changing this property will regenerate Kerberos keytabs for all HttpFS roles.", "display_name": "HttpFS Load Balancer", "name": "httpfs_load_balancer", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Log Directory parameter.", "display_name": "Suppress Parameter Validation: HttpFS Log Directory", "name": "role_config_suppression_httpfs_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_httpfs_unexpected_exits", "value": "false" }, { "desc": "The user that the HttpFS server process should run as.", "display_name": "System User", "name": "httpfs_process_username", "value": "httpfs" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The group that the HttpFS server process should run as.", "display_name": "System Group", "name": "httpfs_process_groupname", "value": "httpfs" }, { "desc": "When computing the overall HttpFS health, consider the host's health.", "display_name": "HttpFS Host Health Test", "name": "httpfs_host_health_enabled", "value": "true" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HttpFS logs. Typically used by log4j or logback.", "display_name": "HttpFS Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HttpFS Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Enables the health test that the HttpFS's process state is consistent with the role configuration", "display_name": "HttpFS Process Health Test", "name": "httpfs_scm_health_enabled", "value": "true" }, { "desc": "The password for the HttpFS JKS keystore file.", "display_name": "HttpFS TLS/SSL Server JKS Keystore File Password", "name": "httpfs_https_keystore_password", "value": null }, { "desc": "The password for the HttpFS TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HttpFS TLS/SSL Certificate Trust Store Password", "name": "httpfs_https_truststore_password", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_httpfs_https_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Signature Secret parameter.", "display_name": "Suppress Parameter Validation: Signature Secret", "name": "role_config_suppression_hdfs_httpfs_signature_secret", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml", "name": "role_config_suppression_httpfs_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_httpfs_https_keystore_password", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HttpFS Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_httpfs_role_env_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HttpFS parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HttpFS", "name": "role_config_suppression_httpfs_java_opts", "value": "false" }, { "desc": "Directory where HttpFS will place its log files.", "display_name": "HttpFS Log Directory", "name": "httpfs_log_dir", "value": "/var/log/hadoop-httpfs" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HttpFS Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_httpfs_process_groupname", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into httpfs-site.xml for this role only.", "display_name": "HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml", "name": "httpfs_config_safety_valve", "value": null }, { "desc": "The maximum size, in megabytes, per log file for HttpFS logs. Typically used by log4j or logback.", "display_name": "HttpFS Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags, PermGen, or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HttpFS", "name": "httpfs_java_opts", "value": "" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_httpfs_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HttpFS Environment Advanced Configuration Snippet (Safety Valve)", "name": "HTTPFS_role_env_safety_valve", "value": null }, { "desc": "The secret to use for signing client authentication tokens.", "display_name": "Signature Secret", "name": "hdfs_httpfs_signature_secret", "value": "hadoop httpfs secret" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HttpFS is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HttpFS TLS/SSL Server JKS Keystore File Location", "name": "httpfs_https_keystore_file", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HttpFS in Bytes", "name": "httpfs_java_heapsize", "value": "268435456" }, { "desc": "The port for the administration interface.", "display_name": "Administration Port", "name": "hdfs_httpfs_admin_port", "value": "14001" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HttpFS might connect to. This is used when HttpFS is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HttpFS TLS/SSL Certificate Trust Store File", "name": "httpfs_https_truststore_file", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The port where the REST interface to HDFS is available. The REST interface is served over HTTPS if TLS/SSL is enabled for HttpFS, or over HTTP otherwise.", "display_name": "REST Port", "name": "hdfs_httpfs_http_port", "value": "14000" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_httpfs_https_keystore_file", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_httpfs_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_httpfs_https_truststore_password", "value": "false" }, { "desc": "The minimum log level for HttpFS logs", "display_name": "HttpFS Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Encrypt communication between clients and HttpFS using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HttpFS", "name": "httpfs_use_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Load Balancer parameter.", "display_name": "Suppress Parameter Validation: HttpFS Load Balancer", "name": "role_config_suppression_httpfs_load_balancer", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_httpfs_process_username", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/oozie-service.json0000666000175100017510000003405113245514472025165 0ustar zuulzuul00000000000000[ { "desc": "Namespace used by this Oozie service in ZooKeeper when High Availability is enabled.", "display_name": "ZooKeeper Namespace", "name": "oozie_zookeeper_namespace", "value": "oozie" }, { "desc": "Name of the ZooKeeper service that this Oozie service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "The number of threads that Oozie will use for the Upload ShareLib command.", "display_name": "Oozie Upload ShareLib Command Concurrency Level", "name": "oozie_upload_sharelib_concurrency", "value": "8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Database Dump File parameter.", "display_name": "Suppress Parameter Validation: Database Dump File", "name": "service_config_suppression_database_dump_file", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "oozie" }, { "desc": "The timeout in seconds used for the Oozie Upload ShareLib command. When the value is zero, there is no timeout for the command.", "display_name": "Oozie Upload ShareLib Command Timeout", "name": "oozie_upload_sharelib_cmd_timeout", "value": "270" }, { "desc": "URL of the JMS Broker used by the Oozie service in JMS integration is enabled.", "display_name": "JMS Broker", "name": "oozie_jms_broker", "value": "tcp://localhost:61616" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ShareLib Root Directory parameter.", "display_name": "Suppress Parameter Validation: ShareLib Root Directory", "name": "service_config_suppression_oozie_sharelib_rootdir", "value": "false" }, { "desc": "Encrypt communication between clients and Oozie using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Oozie", "name": "oozie_use_ssl", "value": "false" }, { "desc": "List of event listeners used by the Oozie service. Listeners needed for JMS or SLA integration are automatically emitted if they are enabled.", "display_name": "Oozie Event Listeners", "name": "oozie_event_listeners", "value": "" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Oozie Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Oozie Server Health", "name": "service_health_suppression_oozie_oozie_servers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_oozie_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JMS Broker parameter.", "display_name": "Suppress Parameter Validation: JMS Broker", "name": "service_config_suppression_oozie_jms_broker", "value": "false" }, { "desc": "Whether to configure Oozie properties needed for SLA integration", "display_name": "Enable SLA Integration", "name": "oozie_use_sla", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "oozie" }, { "desc": "File where the database gets dumped to or loaded from.", "display_name": "Database Dump File", "name": "database_dump_file", "value": "/tmp/oozie_database_dump.zip" }, { "desc": "Root of the directory in HDFS where the Oozie ShareLibs are stored. The libraries are stored in the share/lib subdirectory under the specified root directory.", "display_name": "ShareLib Root Directory", "name": "oozie_sharelib_rootdir", "value": "/user/oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Event Listeners parameter.", "display_name": "Suppress Parameter Validation: Oozie Event Listeners", "name": "service_config_suppression_oozie_event_listeners", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Oozie Load Balancer", "name": "service_config_suppression_oozie_load_balancer", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "oozie" }, { "desc": "Whether to configure Oozie properties needed for JMS integration", "display_name": "Enable JMS Integration", "name": "oozie_use_jms", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Oozie Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "oozie_env_safety_valve", "value": null }, { "desc": "A list of credential class mappings for CredentialsProvider.", "display_name": "Oozie Credential Classes", "name": "oozie_credential_classes", "value": "hcat=org.apache.oozie.action.hadoop.HCatCredentials,hbase=org.apache.oozie.action.hadoop.HbaseCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Address of the load balancer used if Oozie HA is enabled. Should be specified in host:port format.", "display_name": "Oozie Load Balancer", "name": "oozie_load_balancer", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the Spark on Yarn service that this Oozie service instance depends on", "display_name": "Spark on Yarn Service", "name": "spark_on_yarn_service", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The health test thresholds of the overall Oozie Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Oozie Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Oozie Servers falls below the critical threshold.", "display_name": "Healthy Oozie Server Monitoring Thresholds", "name": "oozie_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Credential Classes parameter.", "display_name": "Suppress Parameter Validation: Oozie Credential Classes", "name": "service_config_suppression_oozie_credential_classes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Namespace parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Namespace", "name": "service_config_suppression_oozie_zookeeper_namespace", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Default timeout for a Coordinator Action input check (in minutes) for a Coordinator Job.", "display_name": "Coordinator Action Input Check Default Timeout", "name": "oozie_service_coord_normal_default_timeout", "value": "120" }, { "desc": "Name of the Hive service that this Oozie service instance depends on. This is used to configure Oozie HCat integration.", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Oozie Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Oozie Server Count Validator", "name": "service_config_suppression_oozie_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Service to run MapReduce jobs against", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null }, { "desc": "Coordinator Job Lookup trigger command is scheduled at this interval (in seconds).", "display_name": "Coordinator Job Lookup Interval", "name": "oozie_service_coord_lookup_interval", "value": "300" }, { "desc": "Use ACLs on Znode while a secure ZooKeeper is used for Oozie High Availability. Note: This config is not emitted if ZooKeeper is not secure.", "display_name": "Use ACLs on Znode", "name": "oozie_zk_secure", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_9_0/resources/kafka-kafka_broker.json0000666000175100017510000006744113245514472026107 0ustar zuulzuul00000000000000[ { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The port to give out to producers, consumers, and other brokers to use in establishing connections. This only needs to be set if this port is different from the port the server should bind to.", "display_name": "Advertised Port", "name": "advertised.port", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Broker Java Options parameter.", "display_name": "Suppress Parameter Validation: Additional Broker Java Options", "name": "role_config_suppression_broker_java_opts", "value": "false" }, { "desc": "If set, this is the hostname given out to producers, consumers, and other brokers to use in establishing connections. Never set this property at the group level; it should always be overriden on instance level.", "display_name": "Advertised Host", "name": "advertised.host.name", "value": null }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Metric Report Host parameter.", "display_name": "Suppress Parameter Validation: HTTP Metric Report Host", "name": "role_config_suppression_kafka.http.metrics.host", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kafka_kafka_broker_scm_health", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka Broker might connect to. This is used when Kafka Broker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Kafka Broker TLS/SSL Certificate Trust Store File", "name": "ssl_client_truststore_location", "value": "" }, { "desc": "Port the HTTP metric reporter listens on.", "display_name": "HTTP Metric Report Port", "name": "kafka.http.metrics.port", "value": "24042" }, { "desc": "Kafka broker port.", "display_name": "TCP Port", "name": "port", "value": "9092" }, { "desc": "The log for a topic partition is stored as a directory of segment files. This setting controls the size to which a segment file can grow before a new segment is rolled over in the log. This value should be larger than message.max.bytes.", "display_name": "Segment File Size", "name": "log.segment.bytes", "value": "1073741824" }, { "desc": "For advanced use only. A string to be inserted into kafka-monitoring.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties", "name": "kafka-monitoring.properties_role_safety_valve", "value": null }, { "desc": "The frequency, in milliseconds, that the log cleaner checks whether any log segment is eligible for deletion, per retention policies.", "display_name": "Data Retention Check Interval", "name": "log.retention.check.interval.ms", "value": "300000" }, { "desc": "The maximum number of rolled log files to keep for Kafka Broker logs. Typically used by log4j or logback.", "display_name": "Kafka Broker Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Advertised Host parameter.", "display_name": "Suppress Parameter Validation: Advertised Host", "name": "role_config_suppression_advertised.host.name", "value": "false" }, { "desc": "The minimum log level for Kafka Broker logs", "display_name": "Kafka Broker Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "When computing the overall Kafka Broker health, consider the host's health.", "display_name": "Kafka Broker Host Health Test", "name": "kafka_broker_host_health_enabled", "value": "true" }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in megabytes. Kafka does not generally require setting large heap sizes. It is better to let the file system cache utilize the available memory.", "display_name": "Java Heap Size of Broker", "name": "broker_max_heap_size", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kafka_kafka_broker_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "The number of I/O threads that the server uses for executing requests. You should have at least as many threads as you have disks.", "display_name": "Number of I/O Threads", "name": "num.io.threads", "value": "8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Protocol to be used for inter-broker communication.", "display_name": "Inter Broker Protocol", "name": "security.inter.broker.protocol", "value": "PLAINTEXT" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties", "name": "role_config_suppression_ssl.properties_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "For advanced use only. A string to be inserted into kafka.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties", "name": "kafka.properties_role_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out (in hours). Secondary to the log.retention.ms property. The special value of -1 is interpreted as unlimited. This property is deprecated in Kafka 1.4.0. Use log.retention.ms.", "display_name": "Data Retention Hours", "name": "log.retention.hours", "value": "168" }, { "desc": "Kafka broker secure port.", "display_name": "TLS/SSL Port", "name": "ssl_port", "value": "9093" }, { "desc": "The amount of data to retain in the log for each topic-partition. This is the limit per partition: multiply by the number of partitions to get the total data retained for the topic. The special value of -1 is interpreted as unlimited. If both log.retention.ms and log.retention.bytes are set, a segment is deleted when either limit is exceeded.", "display_name": "Data Retention Size", "name": "log.retention.bytes", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Host the HTTP metric reporter binds to.", "display_name": "HTTP Metric Report Host", "name": "kafka.http.metrics.host", "value": "0.0.0.0" }, { "desc": "The maximum size, in megabytes, per log file for Kafka Broker logs. Typically used by log4j or logback.", "display_name": "Kafka Broker Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties", "name": "role_config_suppression_kafka-monitoring.properties_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kafka_broker_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_BROKER_role_env_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Port for JMX.", "display_name": "JMX Port", "name": "jmx_port", "value": "9393" }, { "desc": "The log directory for log files of the role Kafka Broker.", "display_name": "Kafka Broker Log Directory", "name": "log_dir", "value": "/var/log/kafka" }, { "desc": "The maximum time before a new log segment is rolled out. This property is used in Cloudera Kafka 1.4.0 and later in place of log.roll.hours.", "display_name": "Data Log Roll Time", "name": "log.roll.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties", "name": "role_config_suppression_kafka.properties_role_safety_valve", "value": "false" }, { "desc": "The password that protects the private key contained in the JKS keystore used when Kafka Broker is acting as a TLS/SSL server.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Kafka Broker Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kafka_kafka_broker_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Data Directories parameter.", "display_name": "Suppress Parameter Validation: Data Directories", "name": "role_config_suppression_log.dirs", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out. If both log.retention.ms and log.retention.bytes are set, a segment is deleted when either limit is exceeded. The special value of -1 is interpreted as unlimited. This property is used in Kafka 1.4.0 and later in place of log.retention.hours.", "display_name": "Data Retention Time", "name": "log.retention.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out (in hours). This property is deprecated in Cloudera Kafka 1.4.0; use log.roll.ms.", "display_name": "Data Log Roll Hours", "name": "log.roll.hours", "value": "168" }, { "desc": "These arguments are passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags are passed here.", "display_name": "Additional Broker Java Options", "name": "broker_java_opts", "value": "-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka Broker is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kafka_kafka_broker_host_health", "value": "false" }, { "desc": "A list of one or more directories in which Kafka data is stored. Each new partition created is placed in the directory that currently has the fewest partitions. Each directory should be on its own separate drive.", "display_name": "Data Directories", "name": "log.dirs", "value": "/var/local/kafka/data" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Segment File Size parameter.", "display_name": "Suppress Parameter Validation: Segment File Size", "name": "role_config_suppression_log.segment.bytes", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Broker ID parameter.", "display_name": "Suppress Parameter Validation: Broker ID", "name": "role_config_suppression_broker.id", "value": "false" }, { "desc": "The password for the Kafka Broker JKS keystore file.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kafka_kafka_broker_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kafka_broker_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into ssl.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties", "name": "ssl.properties_role_safety_valve", "value": null }, { "desc": "Encrypt communication between clients and Kafka Broker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Kafka Broker", "name": "ssl_enabled", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "ID uniquely identifying each broker. Never set this property at the group level; it should always be overridden on instance level.", "display_name": "Broker ID", "name": "broker.id", "value": null }, { "desc": "Maximum number of connections allowed from each IP address.", "display_name": "Maximum Connections per IP Address", "name": "max.connections.per.ip", "value": null }, { "desc": "Enables the health test that the Kafka Broker's process state is consistent with the role configuration", "display_name": "Kafka Broker Process Health Test", "name": "kafka_broker_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of I/O Threads parameter.", "display_name": "Suppress Parameter Validation: Number of I/O Threads", "name": "role_config_suppression_num.io.threads", "value": "false" }, { "desc": "The password for the Kafka Broker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Kafka Broker TLS/SSL Certificate Trust Store Password", "name": "ssl_client_truststore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of Broker parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of Broker", "name": "role_config_suppression_broker_max_heap_size", "value": "false" }, { "desc": "Client authentication mode for SSL connections. Default is none, could be set to \"required\", i.e., client authentication is required or to \"requested\", i.e., client authentication is requested and client without certificates can still connect.", "display_name": "SSL Client Authentication", "name": "ssl.client.auth", "value": "none" }, { "desc": "Authenticate a SASL connection with zookeeper, if Kerberos authentication is enabled. It also allows a broker to set SASL ACL on zookeeper nodes which locks these nodes down so that only kafka broker can modify.", "display_name": "Authenticate Zookeeper Connection", "name": "authenticate.zookeeper.connection", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/cloudera_utils.py0000666000175100017510000010620613245514472022066 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools import six from sahara import context from sahara.i18n import _ from sahara.plugins.cdh.client import api_client from sahara.plugins.cdh.client import services from sahara.plugins.cdh import db_helper as dh from sahara.plugins.cdh import plugin_utils from sahara.plugins.cdh import validation from sahara.plugins import exceptions as ex from sahara.plugins import kerberos from sahara.swift import swift_helper from sahara.topology import topology_helper as t_helper from sahara.utils import cluster_progress_ops as cpo from sahara.utils import configs as s_cfg from sahara.utils import poll_utils from sahara.utils import xmlutils HDFS_SERVICE_TYPE = 'HDFS' YARN_SERVICE_TYPE = 'YARN' OOZIE_SERVICE_TYPE = 'OOZIE' HIVE_SERVICE_TYPE = 'HIVE' HUE_SERVICE_TYPE = 'HUE' SPARK_SERVICE_TYPE = 'SPARK_ON_YARN' ZOOKEEPER_SERVICE_TYPE = 'ZOOKEEPER' HBASE_SERVICE_TYPE = 'HBASE' FLUME_SERVICE_TYPE = 'FLUME' SENTRY_SERVICE_TYPE = 'SENTRY' SOLR_SERVICE_TYPE = 'SOLR' SQOOP_SERVICE_TYPE = 'SQOOP' KS_INDEXER_SERVICE_TYPE = 'KS_INDEXER' IMPALA_SERVICE_TYPE = 'IMPALA' KMS_SERVICE_TYPE = 'KMS' KAFKA_SERVICE_TYPE = 'KAFKA' def cloudera_cmd(f): @functools.wraps(f) def wrapper(*args, **kwargs): for cmd in f(*args, **kwargs): result = cmd.wait() if not result.success: if result.children is not None: for c in result.children: if not c.success: raise ex.HadoopProvisionError(c.resultMessage) else: raise ex.HadoopProvisionError(result.resultMessage) return wrapper class ClouderaUtils(object): CM_DEFAULT_USERNAME = 'admin' CM_DEFAULT_PASSWD = 'admin' CM_API_VERSION = 8 HDFS_SERVICE_NAME = 'hdfs01' YARN_SERVICE_NAME = 'yarn01' OOZIE_SERVICE_NAME = 'oozie01' HIVE_SERVICE_NAME = 'hive01' HUE_SERVICE_NAME = 'hue01' SPARK_SERVICE_NAME = 'spark_on_yarn01' ZOOKEEPER_SERVICE_NAME = 'zookeeper01' HBASE_SERVICE_NAME = 'hbase01' FLUME_SERVICE_NAME = 'flume01' SOLR_SERVICE_NAME = 'solr01' SQOOP_SERVICE_NAME = 'sqoop01' KS_INDEXER_SERVICE_NAME = 'ks_indexer01' IMPALA_SERVICE_NAME = 'impala01' SENTRY_SERVICE_NAME = 'sentry01' KMS_SERVICE_NAME = 'kms01' KAFKA_SERVICE_NAME = 'kafka01' NAME_SERVICE = 'nameservice01' def __init__(self): self.pu = plugin_utils.AbstractPluginUtils() self.validator = validation.Validator self.c_helper = None def get_api_client_by_default_password(self, cluster): manager_ip = self.pu.get_manager(cluster).management_ip return api_client.ApiResource(manager_ip, username=self.CM_DEFAULT_USERNAME, password=self.CM_DEFAULT_PASSWD, version=self.CM_API_VERSION) def get_api_client(self, cluster, api_version=None): manager_ip = self.pu.get_manager(cluster).management_ip cm_password = dh.get_cm_password(cluster) version = self.CM_API_VERSION if not api_version else api_version return api_client.ApiResource(manager_ip, username=self.CM_DEFAULT_USERNAME, password=cm_password, version=version) def update_cloudera_password(self, cluster): api = self.get_api_client_by_default_password(cluster) user = api.get_user(self.CM_DEFAULT_USERNAME) user.password = dh.get_cm_password(cluster) api.update_user(user) def get_cloudera_cluster(self, cluster): api = self.get_api_client(cluster) return api.get_cluster(cluster.name) @cloudera_cmd def start_cloudera_cluster(self, cluster): cm_cluster = self.get_cloudera_cluster(cluster) yield cm_cluster.start() @cloudera_cmd def stop_cloudera_cluster(self, cluster): cm_cluster = self.get_cloudera_cluster(cluster) yield cm_cluster.stop() def start_instances(self, cluster): self.start_cloudera_cluster(cluster) @cpo.event_wrapper(True, step=_("Delete instances"), param=('cluster', 1)) def delete_instances(self, cluster, instances): api = self.get_api_client(cluster) cm_cluster = self.get_cloudera_cluster(cluster) hosts = api.get_all_hosts(view='full') hostsnames_to_deleted = [i.fqdn() for i in instances] for host in hosts: if host.hostname in hostsnames_to_deleted: cm_cluster.remove_host(host.hostId) api.delete_host(host.hostId) @cpo.event_wrapper( True, step=_("Decommission nodes"), param=('cluster', 1)) def decommission_nodes(self, cluster, process, decommission_roles, roles_to_delete=None): service = self.get_service_by_role(process, cluster) service.decommission(*decommission_roles).wait() # not all roles should be decommissioned if roles_to_delete: decommission_roles.extend(roles_to_delete) for role_name in decommission_roles: service.delete_role(role_name) @cpo.event_wrapper( True, step=_("Refresh DataNodes"), param=('cluster', 1)) def refresh_datanodes(self, cluster): self._refresh_nodes(cluster, 'DATANODE', self.HDFS_SERVICE_NAME) @cpo.event_wrapper( True, step=_("Refresh YARNNodes"), param=('cluster', 1)) def refresh_yarn_nodes(self, cluster): self._refresh_nodes(cluster, 'NODEMANAGER', self.YARN_SERVICE_NAME) @cloudera_cmd def _refresh_nodes(self, cluster, process, service_name): cm_cluster = self.get_cloudera_cluster(cluster) service = cm_cluster.get_service(service_name) nds = [n.name for n in service.get_roles_by_type(process)] for nd in nds: for st in service.refresh(nd): yield st @cpo.event_wrapper( True, step=_("Restart stale services"), param=('cluster', 1)) @cloudera_cmd def restart_stale_services(self, cluster): cm_cluster = self.get_cloudera_cluster(cluster) yield cm_cluster.restart( restart_only_stale_services=True, redeploy_client_configuration=True) @cpo.event_wrapper(True, step=_("Deploy configs"), param=('cluster', 1)) @cloudera_cmd def deploy_configs(self, cluster): cm_cluster = self.get_cloudera_cluster(cluster) yield cm_cluster.deploy_client_config() def update_configs(self, instances): # instances non-empty cpo.add_provisioning_step( instances[0].cluster_id, _("Update configs"), len(instances)) with context.ThreadGroup() as tg: for instance in instances: tg.spawn("update-configs-%s" % instance.instance_name, self._update_configs, instance) context.sleep(1) @cpo.event_wrapper(True) @cloudera_cmd def _update_configs(self, instance): for process in instance.node_group.node_processes: process = self.pu.convert_role_showname(process) service = self.get_service_by_role(process, instance=instance) yield service.deploy_client_config(self.pu.get_role_name(instance, process)) def get_mgmt_service(self, cluster): api = self.get_api_client(cluster) cm = api.get_cloudera_manager() mgmt_service = cm.get_service() return mgmt_service @cloudera_cmd def restart_mgmt_service(self, cluster): service = self.get_mgmt_service(cluster) yield service.restart() @cloudera_cmd def start_service(self, service): yield service.start() @cloudera_cmd def stop_service(self, service): yield service.stop() @cloudera_cmd def start_roles(self, service, *role_names): for role in service.start_roles(*role_names): yield role @cpo.event_wrapper( True, step=_("Create mgmt service"), param=('cluster', 1)) def create_mgmt_service(self, cluster): api = self.get_api_client(cluster) cm = api.get_cloudera_manager() setup_info = services.ApiServiceSetupInfo() manager = self.pu.get_manager(cluster) hostname = manager.fqdn() processes = ['SERVICEMONITOR', 'HOSTMONITOR', 'EVENTSERVER', 'ALERTPUBLISHER'] for proc in processes: setup_info.add_role_info(self.pu.get_role_name(manager, proc), proc, hostname) cm.create_mgmt_service(setup_info) cm.hosts_start_roles([hostname]) def get_service_by_role(self, role, cluster=None, instance=None): if cluster: cm_cluster = self.get_cloudera_cluster(cluster) elif instance: cm_cluster = self.get_cloudera_cluster(instance.cluster) else: raise ValueError(_("'cluster' or 'instance' argument missed")) if role in ['NAMENODE', 'DATANODE', 'SECONDARYNAMENODE', 'HDFS_GATEWAY']: return cm_cluster.get_service(self.HDFS_SERVICE_NAME) elif role in ['RESOURCEMANAGER', 'NODEMANAGER', 'JOBHISTORY', 'YARN_GATEWAY']: return cm_cluster.get_service(self.YARN_SERVICE_NAME) elif role in ['OOZIE_SERVER']: return cm_cluster.get_service(self.OOZIE_SERVICE_NAME) elif role in ['HIVESERVER2', 'HIVEMETASTORE', 'WEBHCAT']: return cm_cluster.get_service(self.HIVE_SERVICE_NAME) elif role in ['HUE_SERVER']: return cm_cluster.get_service(self.HUE_SERVICE_NAME) elif role in ['SPARK_YARN_HISTORY_SERVER']: return cm_cluster.get_service(self.SPARK_SERVICE_NAME) elif role in ['SERVER']: return cm_cluster.get_service(self.ZOOKEEPER_SERVICE_NAME) elif role in ['MASTER', 'REGIONSERVER']: return cm_cluster.get_service(self.HBASE_SERVICE_NAME) elif role in ['AGENT']: return cm_cluster.get_service(self.FLUME_SERVICE_NAME) elif role in ['SENTRY_SERVER']: return cm_cluster.get_service(self.SENTRY_SERVICE_NAME) elif role in ['SQOOP_SERVER']: return cm_cluster.get_service(self.SQOOP_SERVICE_NAME) elif role in ['SOLR_SERVER']: return cm_cluster.get_service(self.SOLR_SERVICE_NAME) elif role in ['HBASE_INDEXER']: return cm_cluster.get_service(self.KS_INDEXER_SERVICE_NAME) elif role in ['CATALOGSERVER', 'STATESTORE', 'IMPALAD', 'LLAMA']: return cm_cluster.get_service(self.IMPALA_SERVICE_NAME) elif role in ['KMS']: return cm_cluster.get_service(self.KMS_SERVICE_NAME) elif role in ['JOURNALNODE']: return cm_cluster.get_service(self.HDFS_SERVICE_NAME) elif role in ['YARN_STANDBYRM']: return cm_cluster.get_service(self.YARN_SERVICE_NAME) elif role in ['KAFKA_BROKER']: return cm_cluster.get_service(self.KAFKA_SERVICE_NAME) else: raise ValueError( _("Process %(process)s is not supported by CDH plugin") % {'process': role}) @cpo.event_wrapper( True, step=_("First run cluster"), param=('cluster', 1)) @cloudera_cmd def first_run(self, cluster): cm_cluster = self.get_cloudera_cluster(cluster) yield cm_cluster.first_run() @cpo.event_wrapper(True, step=_("Create services"), param=('cluster', 1)) def create_services(self, cluster): api = self.get_api_client(cluster) cm_cluster = api.create_cluster(cluster.name, fullVersion=cluster.hadoop_version) if len(self.pu.get_zookeepers(cluster)) > 0: cm_cluster.create_service(self.ZOOKEEPER_SERVICE_NAME, ZOOKEEPER_SERVICE_TYPE) cm_cluster.create_service(self.HDFS_SERVICE_NAME, HDFS_SERVICE_TYPE) cm_cluster.create_service(self.YARN_SERVICE_NAME, YARN_SERVICE_TYPE) cm_cluster.create_service(self.OOZIE_SERVICE_NAME, OOZIE_SERVICE_TYPE) if self.pu.get_hive_metastore(cluster): cm_cluster.create_service(self.HIVE_SERVICE_NAME, HIVE_SERVICE_TYPE) if self.pu.get_hue(cluster): cm_cluster.create_service(self.HUE_SERVICE_NAME, HUE_SERVICE_TYPE) if self.pu.get_spark_historyserver(cluster): cm_cluster.create_service(self.SPARK_SERVICE_NAME, SPARK_SERVICE_TYPE) if self.pu.get_hbase_master(cluster): cm_cluster.create_service(self.HBASE_SERVICE_NAME, HBASE_SERVICE_TYPE) if len(self.pu.get_flumes(cluster)) > 0: cm_cluster.create_service(self.FLUME_SERVICE_NAME, FLUME_SERVICE_TYPE) if self.pu.get_sentry(cluster): cm_cluster.create_service(self.SENTRY_SERVICE_NAME, SENTRY_SERVICE_TYPE) if len(self.pu.get_solrs(cluster)) > 0: cm_cluster.create_service(self.SOLR_SERVICE_NAME, SOLR_SERVICE_TYPE) if self.pu.get_sqoop(cluster): cm_cluster.create_service(self.SQOOP_SERVICE_NAME, SQOOP_SERVICE_TYPE) if len(self.pu.get_hbase_indexers(cluster)) > 0: cm_cluster.create_service(self.KS_INDEXER_SERVICE_NAME, KS_INDEXER_SERVICE_TYPE) if self.pu.get_catalogserver(cluster): cm_cluster.create_service(self.IMPALA_SERVICE_NAME, IMPALA_SERVICE_TYPE) if self.pu.get_kms(cluster): cm_cluster.create_service(self.KMS_SERVICE_NAME, KMS_SERVICE_TYPE) if len(self.pu.get_kafka_brokers(cluster)) > 0: cm_cluster.create_service(self.KAFKA_SERVICE_NAME, KAFKA_SERVICE_TYPE) def _agents_connected(self, instances, api): hostnames = [i.fqdn() for i in instances] hostnames_to_manager = [h.hostname for h in api.get_all_hosts('full')] for hostname in hostnames: if hostname not in hostnames_to_manager: return False return True @cpo.event_wrapper(True, step=_("Await agents"), param=('cluster', 1)) def _await_agents(self, cluster, instances, timeout_config): api = self.get_api_client(instances[0].cluster) poll_utils.plugin_option_poll( cluster, self._agents_connected, timeout_config, _("Await Cloudera agents"), 5, { 'instances': instances, 'api': api}) def await_agents(self, cluster, instances): self._await_agents(cluster, instances, self.c_helper.AWAIT_AGENTS_TIMEOUT) @cpo.event_wrapper( True, step=_("Configure services"), param=('cluster', 1)) def configure_services(self, cluster): cm_cluster = self.get_cloudera_cluster(cluster) if len(self.pu.get_zookeepers(cluster)) > 0: zookeeper = cm_cluster.get_service(self.ZOOKEEPER_SERVICE_NAME) zookeeper.update_config(self._get_configs(ZOOKEEPER_SERVICE_TYPE, cluster=cluster)) hdfs = cm_cluster.get_service(self.HDFS_SERVICE_NAME) hdfs.update_config(self._get_configs(HDFS_SERVICE_TYPE, cluster=cluster)) yarn = cm_cluster.get_service(self.YARN_SERVICE_NAME) yarn.update_config(self._get_configs(YARN_SERVICE_TYPE, cluster=cluster)) oozie = cm_cluster.get_service(self.OOZIE_SERVICE_NAME) oozie.update_config(self._get_configs(OOZIE_SERVICE_TYPE, cluster=cluster)) if self.pu.get_hive_metastore(cluster): hive = cm_cluster.get_service(self.HIVE_SERVICE_NAME) hive.update_config(self._get_configs(HIVE_SERVICE_TYPE, cluster=cluster)) if self.pu.get_hue(cluster): hue = cm_cluster.get_service(self.HUE_SERVICE_NAME) hue.update_config(self._get_configs(HUE_SERVICE_TYPE, cluster=cluster)) if self.pu.get_spark_historyserver(cluster): spark = cm_cluster.get_service(self.SPARK_SERVICE_NAME) spark.update_config(self._get_configs(SPARK_SERVICE_TYPE, cluster=cluster)) if self.pu.get_hbase_master(cluster): hbase = cm_cluster.get_service(self.HBASE_SERVICE_NAME) hbase.update_config(self._get_configs(HBASE_SERVICE_TYPE, cluster=cluster)) if len(self.pu.get_flumes(cluster)) > 0: flume = cm_cluster.get_service(self.FLUME_SERVICE_NAME) flume.update_config(self._get_configs(FLUME_SERVICE_TYPE, cluster=cluster)) if self.pu.get_sentry(cluster): sentry = cm_cluster.get_service(self.SENTRY_SERVICE_NAME) sentry.update_config(self._get_configs(SENTRY_SERVICE_TYPE, cluster=cluster)) if len(self.pu.get_solrs(cluster)) > 0: solr = cm_cluster.get_service(self.SOLR_SERVICE_NAME) solr.update_config(self._get_configs(SOLR_SERVICE_TYPE, cluster=cluster)) if self.pu.get_sqoop(cluster): sqoop = cm_cluster.get_service(self.SQOOP_SERVICE_NAME) sqoop.update_config(self._get_configs(SQOOP_SERVICE_TYPE, cluster=cluster)) if len(self.pu.get_hbase_indexers(cluster)) > 0: ks_indexer = cm_cluster.get_service(self.KS_INDEXER_SERVICE_NAME) ks_indexer.update_config( self._get_configs(KS_INDEXER_SERVICE_TYPE, cluster=cluster)) if self.pu.get_catalogserver(cluster): impala = cm_cluster.get_service(self.IMPALA_SERVICE_NAME) impala.update_config(self._get_configs(IMPALA_SERVICE_TYPE, cluster=cluster)) if self.pu.get_kms(cluster): kms = cm_cluster.get_service(self.KMS_SERVICE_NAME) kms.update_config(self._get_configs(KMS_SERVICE_TYPE, cluster=cluster)) if len(self.pu.get_kafka_brokers(cluster)) > 0: kafka = cm_cluster.get_service(self.KAFKA_SERVICE_NAME) kafka.update_config(self._get_configs(KAFKA_SERVICE_TYPE, cluster=cluster)) def configure_instances(self, instances, cluster=None): # instances non-empty cpo.add_provisioning_step( instances[0].cluster_id, _("Configure instances"), len(instances)) for inst in instances: self.configure_instance(inst, cluster) def get_roles_list(self, node_processes): current = set(node_processes) extra_roles = { 'YARN_GATEWAY': ["YARN_NODEMANAGER"], 'HDFS_GATEWAY': ['HDFS_NAMENODE', 'HDFS_DATANODE', "HDFS_SECONDARYNAMENODE"] } for extra_role in six.iterkeys(extra_roles): valid_processes = extra_roles[extra_role] for valid in valid_processes: if valid in current: current.add(extra_role) break return list(current) def get_role_type(self, process): mapper = { 'YARN_GATEWAY': 'GATEWAY', 'HDFS_GATEWAY': 'GATEWAY', } return mapper.get(process, process) @cpo.event_wrapper(True) def configure_instance(self, instance, cluster=None): roles_list = self.get_roles_list(instance.node_group.node_processes) for role in roles_list: self._add_role(instance, role, cluster) def _add_role(self, instance, process, cluster): if process in ['CLOUDERA_MANAGER', 'HDFS_JOURNALNODE', 'YARN_STANDBYRM']: return process = self.pu.convert_role_showname(process) service = self.get_service_by_role(process, instance=instance) role_type = self.get_role_type(process) role = service.create_role(self.pu.get_role_name(instance, process), role_type, instance.fqdn()) role.update_config(self._get_configs(process, cluster, instance=instance)) @cloudera_cmd def restart_service(self, process, instance): service = self.get_service_by_role(process, instance=instance) yield service.restart() def update_role_config(self, instance, process): process = self.pu.convert_role_showname(process) service = self.get_service_by_role(process, instance=instance) api = self.get_api_client(instance.cluster) hosts = api.get_all_hosts(view='full') ihost_id = None for host in hosts: if instance.fqdn() == host.hostname: ihost_id = host.hostId break role_type = self.get_role_type(process) roles = service.get_roles_by_type(role_type) for role in roles: if role.hostRef.hostId == ihost_id: role.update_config( self._get_configs(role_type, instance=instance)) self.restart_service(process, instance) @cloudera_cmd def import_admin_credentials(self, cm, username, password): yield cm.import_admin_credentials(username, password) @cloudera_cmd def configure_for_kerberos(self, cluster): api = self.get_api_client(cluster, api_version=11) cluster = api.get_cluster(cluster.name) yield cluster.configure_for_kerberos() def push_kerberos_configs(self, cluster): manager = self.pu.get_manager(cluster) kdc_host = kerberos.get_kdc_host(cluster, manager) security_realm = kerberos.get_realm_name(cluster) username = "%s@%s" % (kerberos.get_admin_principal(cluster), kerberos.get_realm_name(cluster)) password = kerberos.get_server_password(cluster) api = self.get_api_client(cluster) cm = api.get_cloudera_manager() cm.update_config({'SECURITY_REALM': security_realm, 'KDC_HOST': kdc_host}) self.import_admin_credentials(cm, username, password) self.configure_for_kerberos(cluster) self.deploy_configs(cluster) def configure_rack_awareness(self, cluster): if t_helper.is_data_locality_enabled(): self._configure_rack_awareness(cluster) @cpo.event_wrapper( True, step=_("Configure rack awareness"), param=('cluster', 1)) def _configure_rack_awareness(self, cluster): api = self.get_api_client(cluster) topology = t_helper.generate_topology_map( cluster, is_node_awareness=False) for host in api.get_all_hosts(): host.rackId = topology[host.ipAddress] host.put_host() def full_cluster_stop(self, cluster): self.stop_cloudera_cluster(cluster) mgmt = self.get_mgmt_service(cluster) self.stop_service(mgmt) def full_cluster_start(self, cluster): self.start_cloudera_cluster(cluster) mgmt = self.get_mgmt_service(cluster) self.start_service(mgmt) def get_cloudera_manager_info(self, cluster): mng = self.pu.get_manager(cluster) info = { 'Cloudera Manager': { 'Web UI': 'http://%s:7180' % mng.get_ip_or_dns_name(), 'Username': 'admin', 'Password': dh.get_cm_password(cluster) } } return info @cpo.event_wrapper( True, step=_("Enable NameNode HA"), param=('cluster', 1)) @cloudera_cmd def enable_namenode_ha(self, cluster): standby_nn = self.pu.get_secondarynamenode(cluster) standby_nn_host_name = standby_nn.fqdn() jns = self.pu.get_jns(cluster) jn_list = [] for index, jn in enumerate(jns): jn_host_name = jn.fqdn() jn_list.append({'jnHostId': jn_host_name, 'jnName': 'JN%i' % index, 'jnEditsDir': '/dfs/jn' }) cm_cluster = self.get_cloudera_cluster(cluster) hdfs = cm_cluster.get_service(self.HDFS_SERVICE_NAME) nn = hdfs.get_roles_by_type('NAMENODE')[0] yield hdfs.enable_nn_ha(active_name=nn.name, standby_host_id=standby_nn_host_name, nameservice=self.NAME_SERVICE, jns=jn_list ) @cpo.event_wrapper( True, step=_("Enable ResourceManager HA"), param=('cluster', 1)) @cloudera_cmd def enable_resourcemanager_ha(self, cluster): new_rm = self.pu.get_stdb_rm(cluster) new_rm_host_name = new_rm.fqdn() cm_cluster = self.get_cloudera_cluster(cluster) yarn = cm_cluster.get_service(self.YARN_SERVICE_NAME) yield yarn.enable_rm_ha(new_rm_host_id=new_rm_host_name) def _get_configs(self, service, cluster=None, instance=None): def get_hadoop_dirs(mount_points, suffix): return ','.join([x + suffix for x in mount_points]) all_confs = {} if cluster: zk_count = self.validator.get_inst_count(cluster, 'ZOOKEEPER_SERVER') hbm_count = self.validator.get_inst_count(cluster, 'HBASE_MASTER') snt_count = self.validator.get_inst_count(cluster, 'SENTRY_SERVER') ks_count =\ self.validator.get_inst_count(cluster, 'KEY_VALUE_STORE_INDEXER') kms_count = self.validator.get_inst_count(cluster, 'KMS') imp_count =\ self.validator.get_inst_count(cluster, 'IMPALA_CATALOGSERVER') hive_count = self.validator.get_inst_count(cluster, 'HIVE_METASTORE') slr_count = self.validator.get_inst_count(cluster, 'SOLR_SERVER') sqp_count = self.validator.get_inst_count(cluster, 'SQOOP_SERVER') core_site_safety_valve = '' if self.pu.c_helper.is_swift_enabled(cluster): configs = swift_helper.get_swift_configs() confs = {c['name']: c['value'] for c in configs} core_site_safety_valve = xmlutils.create_elements_xml(confs) all_confs = { 'HDFS': { 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME if zk_count else '', 'dfs_block_local_path_access_user': 'impala' if imp_count else '', 'kms_service': self.KMS_SERVICE_NAME if kms_count else '', 'core_site_safety_valve': core_site_safety_valve }, 'HIVE': { 'mapreduce_yarn_service': self.YARN_SERVICE_NAME, 'sentry_service': self.SENTRY_SERVICE_NAME if snt_count else '', 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME if zk_count else '' }, 'OOZIE': { 'mapreduce_yarn_service': self.YARN_SERVICE_NAME, 'hive_service': self.HIVE_SERVICE_NAME if hive_count else '', 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME if zk_count else '' }, 'YARN': { 'hdfs_service': self.HDFS_SERVICE_NAME, 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME if zk_count else '' }, 'HUE': { 'hive_service': self.HIVE_SERVICE_NAME, 'oozie_service': self.OOZIE_SERVICE_NAME, 'sentry_service': self.SENTRY_SERVICE_NAME if snt_count else '', 'solr_service': self.SOLR_SERVICE_NAME if slr_count else '', 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME if zk_count else '', 'hbase_service': self.HBASE_SERVICE_NAME if hbm_count else '', 'impala_service': self.IMPALA_SERVICE_NAME if imp_count else '', 'sqoop_service': self.SQOOP_SERVICE_NAME if sqp_count else '' }, 'SPARK_ON_YARN': { 'yarn_service': self.YARN_SERVICE_NAME }, 'HBASE': { 'hdfs_service': self.HDFS_SERVICE_NAME, 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME, 'hbase_enable_indexing': 'true' if ks_count else 'false', 'hbase_enable_replication': 'true' if ks_count else 'false' }, 'FLUME': { 'hdfs_service': self.HDFS_SERVICE_NAME, 'solr_service': self.SOLR_SERVICE_NAME if slr_count else '', 'hbase_service': self.HBASE_SERVICE_NAME if hbm_count else '' }, 'SENTRY': { 'hdfs_service': self.HDFS_SERVICE_NAME, 'sentry_server_config_safety_valve': ( self.c_helper.SENTRY_IMPALA_CLIENT_SAFETY_VALVE if imp_count else '') }, 'SOLR': { 'hdfs_service': self.HDFS_SERVICE_NAME, 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME }, 'SQOOP': { 'mapreduce_yarn_service': self.YARN_SERVICE_NAME }, 'KS_INDEXER': { 'hbase_service': self.HBASE_SERVICE_NAME, 'solr_service': self.SOLR_SERVICE_NAME }, 'IMPALA': { 'hdfs_service': self.HDFS_SERVICE_NAME, 'hbase_service': self.HBASE_SERVICE_NAME if hbm_count else '', 'hive_service': self.HIVE_SERVICE_NAME, 'sentry_service': self.SENTRY_SERVICE_NAME if snt_count else '', 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME if zk_count else '' } } hive_confs = { 'HIVE': { 'hive_metastore_database_type': 'postgresql', 'hive_metastore_database_host': self.pu.get_manager(cluster).internal_ip, 'hive_metastore_database_port': '7432', 'hive_metastore_database_password': dh.get_hive_db_password(cluster) } } hue_confs = { 'HUE': { 'hue_webhdfs': self.pu.get_role_name( self.pu.get_namenode(cluster), 'NAMENODE') } } sentry_confs = { 'SENTRY': { 'sentry_server_database_type': 'postgresql', 'sentry_server_database_host': self.pu.get_manager(cluster).internal_ip, 'sentry_server_database_port': '7432', 'sentry_server_database_password': dh.get_sentry_db_password(cluster) } } kafka_confs = { 'KAFKA': { 'zookeeper_service': self.ZOOKEEPER_SERVICE_NAME } } all_confs = s_cfg.merge_configs(all_confs, hue_confs) all_confs = s_cfg.merge_configs(all_confs, hive_confs) all_confs = s_cfg.merge_configs(all_confs, sentry_confs) all_confs = s_cfg.merge_configs(all_confs, kafka_confs) all_confs = s_cfg.merge_configs(all_confs, cluster.cluster_configs) if instance: snt_count = self.validator.get_inst_count(instance.cluster, 'SENTRY_SERVER') paths = instance.storage_paths() instance_default_confs = { 'NAMENODE': { 'dfs_name_dir_list': get_hadoop_dirs(paths, '/fs/nn') }, 'SECONDARYNAMENODE': { 'fs_checkpoint_dir_list': get_hadoop_dirs(paths, '/fs/snn') }, 'DATANODE': { 'dfs_data_dir_list': get_hadoop_dirs(paths, '/fs/dn'), 'dfs_datanode_data_dir_perm': 755, 'dfs_datanode_handler_count': 30 }, 'NODEMANAGER': { 'yarn_nodemanager_local_dirs': get_hadoop_dirs(paths, '/yarn/local'), 'container_executor_allowed_system_users': "nobody,impala,hive,llama,hdfs,yarn,mapred," "spark,oozie", "container_executor_banned_users": "bin" }, 'SERVER': { 'maxSessionTimeout': 60000 }, 'HIVESERVER2': { 'hiveserver2_enable_impersonation': 'false' if snt_count else 'true', 'hive_hs2_config_safety_valve': ( self.c_helper.HIVE_SERVER2_SENTRY_SAFETY_VALVE if snt_count else '') }, 'HIVEMETASTORE': { 'hive_metastore_config_safety_valve': ( self.c_helper.HIVE_METASTORE_SENTRY_SAFETY_VALVE if snt_count else '') } } ng_user_confs = self.pu.convert_process_configs( instance.node_group.node_configs) all_confs = s_cfg.merge_configs(all_confs, ng_user_confs) all_confs = s_cfg.merge_configs(all_confs, instance_default_confs) return all_confs.get(service, {}) sahara-8.0.0/sahara/plugins/cdh/__init__.py0000666000175100017510000000000013245514472020570 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_5_0/0000775000175100017510000000000013245515026017460 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_5_0/deploy.py0000666000175100017510000001260613245514472021341 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ from sahara.plugins.cdh import commands as cmd from sahara.plugins.cdh import deploy as common_deploy from sahara.plugins.cdh.v5_5_0 import cloudera_utils as cu from sahara.plugins import utils as gu from sahara.service.edp import hdfs_helper as h from sahara.utils import cluster_progress_ops as cpo CU = cu.ClouderaUtilsV550() PACKAGES = common_deploy.PACKAGES def configure_cluster(cluster): instances = gu.get_instances(cluster) if not cmd.is_pre_installed_cdh(CU.pu.get_manager(cluster).remote()): CU.pu.configure_os(instances) CU.pu.install_packages(instances, PACKAGES) CU.pu.start_cloudera_agents(instances) CU.pu.start_cloudera_manager(cluster) CU.update_cloudera_password(cluster) CU.configure_rack_awareness(cluster) CU.await_agents(cluster, instances) CU.create_mgmt_service(cluster) CU.create_services(cluster) CU.configure_services(cluster) CU.configure_instances(instances, cluster) CU.deploy_configs(cluster) @cpo.event_wrapper( True, step=_("Start roles: NODEMANAGER, DATANODE"), param=('cluster', 0)) def _start_roles(cluster, instances): for instance in instances: if 'HDFS_DATANODE' in instance.node_group.node_processes: hdfs = CU.get_service_by_role('DATANODE', instance=instance) CU.start_roles(hdfs, CU.pu.get_role_name(instance, 'DATANODE')) if 'YARN_NODEMANAGER' in instance.node_group.node_processes: yarn = CU.get_service_by_role('NODEMANAGER', instance=instance) CU.start_roles(yarn, CU.pu.get_role_name(instance, 'NODEMANAGER')) def scale_cluster(cluster, instances): if not instances: return if not cmd.is_pre_installed_cdh(instances[0].remote()): CU.pu.configure_os(instances) CU.pu.install_packages(instances, PACKAGES) CU.pu.start_cloudera_agents(instances) CU.await_agents(cluster, instances) CU.configure_rack_awareness(cluster) CU.configure_instances(instances, cluster) CU.update_configs(instances) common_deploy.prepare_scaling_kerberized_cluster( cluster, CU, instances) CU.pu.configure_swift(cluster, instances) _start_roles(cluster, instances) CU.refresh_datanodes(cluster) CU.refresh_yarn_nodes(cluster) CU.restart_stale_services(cluster) def decommission_cluster(cluster, instances): dns = [] dns_to_delete = [] nms = [] nms_to_delete = [] for i in instances: if 'HDFS_DATANODE' in i.node_group.node_processes: dns.append(CU.pu.get_role_name(i, 'DATANODE')) dns_to_delete.append( CU.pu.get_role_name(i, 'HDFS_GATEWAY')) if 'YARN_NODEMANAGER' in i.node_group.node_processes: nms.append(CU.pu.get_role_name(i, 'NODEMANAGER')) nms_to_delete.append( CU.pu.get_role_name(i, 'YARN_GATEWAY')) if dns: CU.decommission_nodes( cluster, 'DATANODE', dns, dns_to_delete) if nms: CU.decommission_nodes( cluster, 'NODEMANAGER', nms, nms_to_delete) CU.delete_instances(cluster, instances) CU.refresh_datanodes(cluster) CU.refresh_yarn_nodes(cluster) CU.restart_stale_services(cluster) @cpo.event_wrapper(True, step=_("Prepare cluster"), param=('cluster', 0)) def _prepare_cluster(cluster): if CU.pu.get_oozie(cluster): CU.pu.install_extjs(cluster) if CU.pu.get_hive_metastore(cluster): CU.pu.configure_hive(cluster) if CU.pu.get_sentry(cluster): CU.pu.configure_sentry(cluster) @cpo.event_wrapper( True, step=_("Finish cluster starting"), param=('cluster', 0)) def _finish_cluster_starting(cluster): if CU.pu.get_hive_metastore(cluster): CU.pu.put_hive_hdfs_xml(cluster) server = CU.pu.get_hbase_master(cluster) if CU.pu.c_helper.is_hbase_common_lib_enabled(cluster) and server: with server.remote() as r: h.create_hbase_common_lib(r) if CU.pu.get_flumes(cluster): flume = CU.get_service_by_role('AGENT', cluster) CU.start_service(flume) def start_cluster(cluster): _prepare_cluster(cluster) CU.first_run(cluster) CU.pu.configure_swift(cluster) if len(CU.pu.get_jns(cluster)) > 0: CU.enable_namenode_ha(cluster) # updating configs for NameNode role on needed nodes CU.update_role_config(CU.pu.get_secondarynamenode(cluster), 'HDFS_NAMENODE') if CU.pu.get_stdb_rm(cluster): CU.enable_resourcemanager_ha(cluster) # updating configs for ResourceManager on needed nodes CU.update_role_config(CU.pu.get_stdb_rm(cluster), 'YARN_STANDBYRM') _finish_cluster_starting(cluster) common_deploy.setup_kerberos_for_cluster(cluster, CU) def get_open_ports(node_group): ports = common_deploy.get_open_ports(node_group) return ports sahara-8.0.0/sahara/plugins/cdh/v5_5_0/validation.py0000666000175100017510000000137113245514472022174 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_5_0 import plugin_utils as pu from sahara.plugins.cdh import validation class ValidatorV550(validation.Validator): PU = pu.PluginUtilsV550() sahara-8.0.0/sahara/plugins/cdh/v5_5_0/versionhandler.py0000666000175100017510000000257013245514472023067 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import abstractversionhandler as avm from sahara.plugins.cdh.v5_5_0 import cloudera_utils from sahara.plugins.cdh.v5_5_0 import config_helper from sahara.plugins.cdh.v5_5_0 import deploy from sahara.plugins.cdh.v5_5_0 import edp_engine from sahara.plugins.cdh.v5_5_0 import plugin_utils from sahara.plugins.cdh.v5_5_0 import validation class VersionHandler(avm.BaseVersionHandler): def __init__(self): super(VersionHandler, self).__init__() self.config_helper = config_helper.ConfigHelperV550() self.cloudera_utils = cloudera_utils.ClouderaUtilsV550() self.plugin_utils = plugin_utils.PluginUtilsV550() self.deploy = deploy self.edp_engine = edp_engine self.validation = validation.ValidatorV550() sahara-8.0.0/sahara/plugins/cdh/v5_5_0/config_helper.py0000666000175100017510000000737013245514472022653 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import config_helper as c_h from sahara.plugins import provisioning as p from sahara.utils import files as f class ConfigHelperV550(c_h.ConfigHelper): path_to_config = 'plugins/cdh/v5_5_0/resources/' CDH5_UBUNTU_REPO = ( 'deb [arch=amd64] http://archive.cloudera.com/cdh5' '/ubuntu/trusty/amd64/cdh trusty-cdh5.5.0 contrib' '\ndeb-src http://archive.cloudera.com/cdh5/ubuntu' '/trusty/amd64/cdh trusty-cdh5.5.0 contrib') DEFAULT_CDH5_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/cdh5/ubuntu' '/trusty/amd64/cdh/archive.key') CM5_UBUNTU_REPO = ( 'deb [arch=amd64] http://archive.cloudera.com/cm5' '/ubuntu/trusty/amd64/cm trusty-cm5.5.0 contrib' '\ndeb-src http://archive.cloudera.com/cm5/ubuntu' '/trusty/amd64/cm trusty-cm5.5.0 contrib') DEFAULT_CM5_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/cm5/ubuntu' '/trusty/amd64/cm/archive.key') CDH5_CENTOS_REPO = ( '[cloudera-cdh5]' '\nname=Cloudera\'s Distribution for Hadoop, Version 5' '\nbaseurl=http://archive.cloudera.com/cdh5/redhat/6' '/x86_64/cdh/5.5.0/' '\ngpgkey = http://archive.cloudera.com/cdh5/redhat/6' '/x86_64/cdh/RPM-GPG-KEY-cloudera' '\ngpgcheck = 1') CM5_CENTOS_REPO = ( '[cloudera-manager]' '\nname=Cloudera Manager' '\nbaseurl=http://archive.cloudera.com/cm5/redhat/6' '/x86_64/cm/5.5.0/' '\ngpgkey = http://archive.cloudera.com/cm5/redhat/6' '/x86_64/cm/RPM-GPG-KEY-cloudera' '\ngpgcheck = 1') KEY_TRUSTEE_UBUNTU_REPO_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/ubuntu/trusty/amd64/navigator-' 'keytrustee/cloudera.list') DEFAULT_KEY_TRUSTEE_UBUNTU_REPO_KEY_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/ubuntu/trusty/amd64/navigator-' 'keytrustee/archive.key') KEY_TRUSTEE_CENTOS_REPO_URL = ( 'http://archive.cloudera.com/navigator-' 'keytrustee5/redhat/6/x86_64/navigator-' 'keytrustee/navigator-keytrustee5.repo') DEFAULT_SWIFT_LIB_URL = ( 'https://repository.cloudera.com/artifactory/repo/org' '/apache/hadoop/hadoop-openstack/2.6.0-cdh5.5.0' '/hadoop-openstack-2.6.0-cdh5.5.0.jar') SWIFT_LIB_URL = p.Config( 'Hadoop OpenStack library URL', 'general', 'cluster', priority=1, default_value=DEFAULT_SWIFT_LIB_URL, description=("Library that adds Swift support to CDH. The file" " will be downloaded by VMs.")) HIVE_SERVER2_SENTRY_SAFETY_VALVE = f.get_file_text( path_to_config + 'hive-server2-sentry-safety.xml') HIVE_METASTORE_SENTRY_SAFETY_VALVE = f.get_file_text( path_to_config + 'hive-metastore-sentry-safety.xml') SENTRY_IMPALA_CLIENT_SAFETY_VALVE = f.get_file_text( path_to_config + 'sentry-impala-client-safety.xml') def __init__(self): super(ConfigHelperV550, self).__init__() self.priority_one_confs = self._load_json( self.path_to_config + 'priority-one-confs.json') self._init_all_ng_plugin_configs() sahara-8.0.0/sahara/plugins/cdh/v5_5_0/cloudera_utils.py0000666000175100017510000000203413245514472023055 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import cloudera_utils as cu from sahara.plugins.cdh.v5_5_0 import config_helper from sahara.plugins.cdh.v5_5_0 import plugin_utils as pu from sahara.plugins.cdh.v5_5_0 import validation class ClouderaUtilsV550(cu.ClouderaUtils): def __init__(self): cu.ClouderaUtils.__init__(self) self.pu = pu.PluginUtilsV550() self.validator = validation.ValidatorV550 self.c_helper = config_helper.ConfigHelperV550() sahara-8.0.0/sahara/plugins/cdh/v5_5_0/__init__.py0000666000175100017510000000000013245514472021565 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_5_0/plugin_utils.py0000666000175100017510000000146413245514472022563 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import plugin_utils as pu from sahara.plugins.cdh.v5_5_0 import config_helper class PluginUtilsV550(pu.AbstractPluginUtils): def __init__(self): self.c_helper = config_helper.ConfigHelperV550() sahara-8.0.0/sahara/plugins/cdh/v5_5_0/edp_engine.py0000666000175100017510000000367013245514472022143 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import confighints_helper as ch_helper from sahara.plugins.cdh import edp_engine from sahara.plugins.cdh.v5_5_0 import cloudera_utils as cu from sahara.service.edp.oozie import engine as oozie_engine from sahara.utils import edp class EdpOozieEngine(edp_engine.EdpOozieEngine): def __init__(self, cluster): super(EdpOozieEngine, self).__init__(cluster) self.cloudera_utils = cu.ClouderaUtilsV550() @staticmethod def get_possible_job_config(job_type): if edp.compare_job_type(job_type, edp.JOB_TYPE_HIVE): return {'job_config': ch_helper.get_possible_hive_config_from( 'plugins/cdh/v5_5_0/resources/hive-site.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_MAPREDUCE_STREAMING): return {'job_config': ch_helper.get_possible_mapreduce_config_from( 'plugins/cdh/v5_5_0/resources/mapred-site.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_PIG): return {'job_config': ch_helper.get_possible_pig_config_from( 'plugins/cdh/v5_5_0/resources/mapred-site.xml')} return oozie_engine.OozieJobEngine.get_possible_job_config(job_type) class EdpSparkEngine(edp_engine.EdpSparkEngine): edp_base_version = "5.5.0" sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/0000775000175100017510000000000013245515027021473 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/sentry-sentry_server.json0000666000175100017510000004615313245514472026640 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When computing the overall Sentry Server health, consider the host's health.", "display_name": "Sentry Server Host Health Test", "name": "sentry_server_host_health_enabled", "value": "true" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_sentry_server_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Sentry Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "sentry_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Sentry Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Sentry Server", "name": "role_config_suppression_sentry_server_java_opts", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Log Directory", "name": "role_config_suppression_sentry_server_log_dir", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Sentry Server", "name": "sentry_server_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_sentry_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Sentry Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Sentry Server in Bytes", "name": "sentry_server_java_heapsize", "value": "1073741824" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_sentry_env_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Sentry Server logs. Typically used by log4j or logback.", "display_name": "Sentry Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_sentry_server_host_health", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_sentry_server_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "RPC port number of Sentry Server.", "display_name": "Sentry Server RPC Port", "name": "sentry_service_server_rpc_port", "value": "8038" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_sentry_server_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Directory where Sentry Server will place its log files.", "display_name": "Sentry Server Log Directory", "name": "sentry_server_log_dir", "value": "/var/log/sentry" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The maximum number of rolled log files to keep for Sentry Server logs. Typically used by log4j or logback.", "display_name": "Sentry Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "sentry_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_sentry_server_swap_memory_usage", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for Sentry Server logs", "display_name": "Sentry Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_sentry_server_scm_health", "value": "false" }, { "desc": "Enables the health test that the Sentry Server's process state is consistent with the role configuration", "display_name": "Sentry Server Process Health Test", "name": "sentry_server_scm_health_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/solr-service.json0000666000175100017510000006515713245514472025026 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "solr" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Solr events:\n

\n\n
    \n
  • operation: the Solr operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "Choose the authentication mechanism used by Solr.", "display_name": "Solr Secure Authentication", "name": "solr_security_authentication", "value": "simple" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Solr events:\n

\n\n
    \n
  • operation: the Solr operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "SOLR Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_solr_sentry_safety_valve", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Solr might connect to. This is used when Solr is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Solr TLS/SSL Certificate Trust Store File", "name": "solr_https_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URI parameter.", "display_name": "Suppress Parameter Validation: LDAP URI", "name": "service_config_suppression_solr_ldap_uri", "value": "false" }, { "desc": "When set, this value is appended to all usernames before authenticating with the LDAP server. For example, if this parameter is set to \"my.domain.com\", and the user authenticating to the Solr daemon is \"mark\", then \"mark@my.domain.com\" is passed to the LDAP server. If this field is not set, the username remains unaltered before being passed to the LDAP server. This parameter is mutually exclusive with LDAP BaseDN.", "display_name": "LDAP Domain", "name": "ldap_domain", "value": null }, { "desc": "The password for the Solr TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Solr TLS/SSL Certificate Trust Store Password", "name": "solr_https_truststore_password", "value": null }, { "desc": "The password for the Solr JKS keystore file.", "display_name": "Solr TLS/SSL Server JKS Keystore File Password", "name": "solr_https_keystore_password", "value": null }, { "desc": "Name of the HDFS service that this Search service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "If Solr does not respond on its web URL within this time interval, the Catalina process is killed.", "display_name": "Solrd Watchdog Timeout", "name": "solrd_watchdog_timeout", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_solr_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Data Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Data Directory", "name": "service_config_suppression_hdfs_data_dir", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Search service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_solr_https_truststore_file", "value": "false" }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with the LDAP server.", "display_name": "Enable LDAP TLS", "name": "solr_ldap_enable_starttls", "value": "false" }, { "desc": "HDFS directory used for storage by this Solr service.", "display_name": "HDFS Data Directory", "name": "hdfs_data_dir", "value": "/solr" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP Secure URI and Start TLS Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP Secure URI and Start TLS Validator", "name": "service_config_suppression_solr_ldaps_or_tls_validator", "value": "false" }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "sentry_solr_provider_resource", "value": "/user/solr/sentry/sentry-provider.ini" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/solr/audit" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_solr_https_keystore_password", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "solr_hdfs_site_safety_valve", "value": null }, { "desc": "The health test thresholds of the overall Solr Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Solr Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Solr Servers falls below the critical threshold.", "display_name": "Healthy Solr Server Monitoring Thresholds", "name": "solr_solr_servers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_sentry_solr_provider_resource", "value": "false" }, { "desc": "When checked, LDAP-based authentication for users is enabled.", "display_name": "Enable LDAP", "name": "solr_enable_ldap_auth", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "solr" }, { "desc": "Whether to suppress configuration warnings produced by the Solr Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Solr Server Count Validator", "name": "service_config_suppression_solr_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_solr_core_site_safety_valve", "value": "false" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access.", "display_name": "Sentry User to Group Mapping Class", "name": "solr_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "solr_core_site_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_solr_https_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "solr_env_safety_valve", "value": null }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. Sentry is supported only on Search 1.1 or later and CDH 5 or later deployments and requires authentication to be turned on for Solr..", "display_name": "Enable Sentry Authorization", "name": "solr_sentry_enabled", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "solr" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Solr Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "solr_sentry_safety_valve", "value": null }, { "desc": "Enable the background watchdog thread that can kill Catalina process if Solr is not responsive.", "display_name": "Enable Solrd Watchdog", "name": "solrd_enable_watchdog", "value": "true" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Solr Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "service_config_suppression_solr_hdfs_site_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress the results of the Solr Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Solr Server Health", "name": "service_health_suppression_solr_solr_servers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "The URI of the LDAP server to use if LDAP authentication is enabled. The URI must be prefixed with ldap:// or ldaps://. Usernames and passwords are transmitted in the clear unless an \"ldaps://\" URI is specified (or LDAP TLS is enabled). The URI can optionally specify the port; for example, ldaps://ldap_server.example.com:636.", "display_name": "LDAP URI", "name": "solr_ldap_uri", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode", "name": "service_config_suppression_zookeeper_znode", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Solr is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Solr TLS/SSL Server JKS Keystore File Location", "name": "solr_https_keystore_file", "value": null }, { "desc": "ZooKeeper znode used to store information about this Solr service.", "display_name": "ZooKeeper Znode", "name": "zookeeper_znode", "value": "/solr" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_solr_ldap_basedn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Solr TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_solr_https_keystore_file", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP TLS Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP TLS Validator", "name": "service_config_suppression_solr_ldap_tls_validator", "value": "false" }, { "desc": "Encrypt communication between clients and Solr using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)). Additional manual steps must be performed; see Enabling TLS/SSL for Solr.", "display_name": "Enable TLS/SSL for Solr", "name": "solr_use_ssl", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "When set, this parameter is used to convert the username to the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,X. For example, if this parameter is set to \"ou=People,dc=example,dc=com\", and the username passed in is \"mark\", the resulting authentication passed to the LDAP server looks like \"uid=mark,ou=People,dc=example,dc=com\". This parameter is generally most useful when authenticating against an OpenLDAP server. This parameter is mutually exclusive with LDAP Domain.", "display_name": "LDAP BaseDN", "name": "solr_ldap_basedn", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Domain parameter.", "display_name": "Suppress Parameter Validation: LDAP Domain", "name": "service_config_suppression_ldap_domain", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/flume-agent.json0000666000175100017510000012217113245514472024603 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Used to select an agent configuration to use from flume.conf. Multiple agents may share the same agent name, in which case they will be assigned the same agent configuration.", "display_name": "Agent Name", "name": "agent_name", "value": "tier1" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Agent Log Directory parameter.", "display_name": "Suppress Parameter Validation: Flume Agent Log Directory", "name": "role_config_suppression_flume_agent_log_dir", "value": "false" }, { "desc": "When computing the overall Agent health, consider the host's health.", "display_name": "Agent Host Health Test", "name": "flume_agent_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "agent_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The maximum size, in megabytes, per log file for Agent logs. Typically used by log4j or logback.", "display_name": "Agent Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_flume_agent_swap_memory_usage", "value": "false" }, { "desc": "Text that goes verbatim into grok-dictionary.conf file used by the Flume-NG Solr sink.", "display_name": "Grok Dictionary File", "name": "agent_grok_dictionary_conf_file", "value": "USERNAME [a-zA-Z0-9._-]+\nUSER %{USERNAME}\nINT (?:[+-]?(?:[0-9]+))\nBASE10NUM (?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))\nNUMBER (?:%{BASE10NUM})\nBASE16NUM (?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))\nUUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}\n\n# Networking\nMAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})\nCISCOMAC (?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})\nWINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})\nCOMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})\nIP (?/(?>[\\w_%!$@:.,-]+|\\\\.)*)+\n#UNIXPATH (?/dev/pts/%{NONNEGINT})\nBSDTTY (?>/dev/tty[pq][a-z0-9])\nTTY (?:%{BSDTTY}|%{LINUXTTY})\nWINPATH (?>[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+\nURIPROTO [A-Za-z]+(\\+[A-Za-z+]+)?\nURIHOST %{IPORHOST}(?::%{POSINT:port})?\n# uripath comes loosely from RFC1738, but mostly from what Firefox\n# doesn't turn into %XX\nURIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=#%_\\-]*)+\n#URIPARAM \\?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?\nURIPARAM \\?[A-Za-z0-9$.+!*'|(){},~#%&/=:;_?\\-\\[\\]]*\nURIPATHPARAM %{URIPATH}(?:%{URIPARAM})?\nURI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?\n\n# Months: January, Feb, 3, 03, 12, December\nMONTH \\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b\nMONTHNUM (?:0?[1-9]|1[0-2])\nMONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\n\n# Days: Monday, Tue, Thu, etc...\nDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)\n\n# Years?\nYEAR (?>\\d\\d){1,2}\n# Time: HH:MM:SS\n#TIME \\d{2}:\\d{2}(?::\\d{2}(?:\\.\\d+)?)?\n# I'm still on the fence about using grok to perform the time match,\n# since it's probably slower.\n# TIME %{POSINT<24}:%{POSINT<60}(?::%{POSINT<60}(?:\\.%{POSINT})?)?\nHOUR (?:2[0123]|[01]?[0-9])\nMINUTE (?:[0-5][0-9])\n# '60' is a leap second in most time standards and thus is valid.\nSECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\nTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])\n# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)\nDATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}\nDATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}\nISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))\nISO8601_SECOND (?:%{SECOND}|60)\nTIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?\nDATE %{DATE_US}|%{DATE_EU}\nDATESTAMP %{DATE}[- ]%{TIME}\nTZ (?:[PMCE][SD]T)\nDATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}\nDATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}\n\n# Syslog Dates: Month Day HH:MM:SS\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}\nPROG (?:[\\w._/%-]+)\nSYSLOGPROG %{PROG:program}(?:\\[%{POSINT:pid}\\])?\nSYSLOGHOST %{IPORHOST}\nSYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>\nHTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}\n\n# Shortcuts\nQS %{QUOTEDSTRING}\n\n# Log formats\nSYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:\nCOMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}\n\n# Log Levels\nLOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Name parameter.", "display_name": "Suppress Parameter Validation: Agent Name", "name": "role_config_suppression_agent_name", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_flume_agent_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Morphlines File parameter.", "display_name": "Suppress Parameter Validation: Morphlines File", "name": "role_config_suppression_agent_morphlines_conf_file", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "agent_web_metric_collection_enabled", "value": "true" }, { "desc": "Verbatim contents of flume.conf. Multiple agents may be configured from the same configuration file; the Agent Name setting can be overridden to select which agent configuration to use for each agent.

To integrate with a secured cluster, you can use the substitution strings \"$KERBEROS_PRINCIPAL\" and \"$KERBEROS_KEYTAB\", which will be replaced by the principal name and the keytab path respectively.", "display_name": "Configuration File", "name": "agent_config_file", "value": "# Please paste flume.conf here. Example:\n\n# Sources, channels, and sinks are defined per\n# agent name, in this case 'tier1'.\ntier1.sources = source1\ntier1.channels = channel1\ntier1.sinks = sink1\n\n# For each source, channel, and sink, set\n# standard properties.\ntier1.sources.source1.type = netcat\ntier1.sources.source1.bind = 127.0.0.1\ntier1.sources.source1.port = 9999\ntier1.sources.source1.channels = channel1\ntier1.channels.channel1.type = memory\ntier1.sinks.sink1.type = logger\ntier1.sinks.sink1.channel = channel1\n\n# Other properties are specific to each type of\n# source, channel, or sink. In this case, we\n# specify the capacity of the memory channel.\ntier1.channels.channel1.capacity = 100\n" }, { "desc": "Text that goes into morphlines.conf file used by the Flume-NG Solr sink. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service.", "display_name": "Morphlines File", "name": "agent_morphlines_conf_file", "value": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Application configuration file in HOCON format (Human-Optimized Config Object Notation). \n# HOCON syntax is defined at http://github.com/typesafehub/config/blob/master/HOCON.md\n# and also used by Akka (http://www.akka.io) and Play (http://www.playframework.org/).\n# For more examples see http://doc.akka.io/docs/akka/2.1.2/general/configuration.html\n\n# morphline.conf example file\n# this is a comment\n\n# Specify server locations in a SOLR_LOCATOR variable; used later in variable substitutions:\nSOLR_LOCATOR : {\n # Name of solr collection\n collection : collection1\n \n # ZooKeeper ensemble\n zkHost : \"$ZK_HOST\"\n \n # Relative or absolute path to a directory containing conf/solrconfig.xml and conf/schema.xml\n # If this path is uncommented it takes precedence over the configuration stored in ZooKeeper. \n # solrHomeDir : \"example/solr/collection1\"\n \n # The maximum number of documents to send to Solr per network batch (throughput knob)\n # batchSize : 100\n}\n\n# Specify an array of one or more morphlines, each of which defines an ETL \n# transformation chain. A morphline consists of one or more (potentially \n# nested) commands. A morphline is a way to consume records (e.g. Flume events, \n# HDFS files or blocks), turn them into a stream of records, and pipe the stream \n# of records through a set of easily configurable transformations on it's way to \n# Solr (or a MapReduceIndexerTool RecordWriter that feeds via a Reducer into Solr).\nmorphlines : [\n {\n # Name used to identify a morphline. E.g. used if there are multiple morphlines in a \n # morphline config file\n id : morphline1 \n \n # Import all morphline commands in these java packages and their subpackages.\n # Other commands that may be present on the classpath are not visible to this morphline.\n importCommands : [\"org.kitesdk.**\", \"org.apache.solr.**\"]\n \n commands : [ \n { \n # Parse Avro container file and emit a record for each avro object\n readAvroContainer {\n # Optionally, require the input record to match one of these MIME types:\n # supportedMimeTypes : [avro/binary]\n \n # Optionally, use a custom Avro schema in JSON format inline:\n # schemaString : \"\"\"\"\"\"\n \n # Optionally, use a custom Avro schema file in JSON format:\n # schemaFile : /path/to/syslog.avsc\n }\n } \n \n { \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # extractAvroPaths is a command that uses zero or more avro path expressions to extract \n # values from an Avro object. Each expression consists of a record output field name (on \n # the left side of the colon ':') as well as zero or more path steps (on the right hand \n # side), each path step separated by a '/' slash. Avro arrays are traversed with the '[]'\n # notation.\n #\n # The result of a path expression is a list of objects, each of which is added to the \n # given record output field.\n # \n # The path language supports all Avro concepts, including nested structures, records, \n # arrays, maps, unions, etc, as well as a flatten option that collects the primitives in \n # a subtree into a flat list.\n extractAvroPaths {\n flatten : false\n paths : { \n id : /id \n text : /text \n user_friends_count : /user_friends_count\n user_location : /user_location\n user_description : /user_description\n user_statuses_count : /user_statuses_count\n user_followers_count : /user_followers_count\n user_name : /user_name\n user_screen_name : /user_screen_name\n created_at : /created_at\n retweet_count : /retweet_count\n retweeted : /retweeted\n in_reply_to_user_id : /in_reply_to_user_id\n source : /source\n in_reply_to_status_id : /in_reply_to_status_id\n media_url_https : /media_url_https\n expanded_url : /expanded_url\n }\n }\n }\n \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # convert timestamp field to native Solr timestamp format\n # e.g. 2012-09-06T07:14:34Z to 2012-09-06T07:14:34.000Z\n {\n convertTimestamp {\n field : created_at\n inputFormats : [\"yyyy-MM-dd'T'HH:mm:ss'Z'\", \"yyyy-MM-dd\"]\n inputTimezone : America/Los_Angeles\n# outputFormat : \"yyyy-MM-dd'T'HH:mm:ss.SSSZ\" \n outputTimezone : UTC\n }\n }\n \n # Consume the output record of the previous command and pipe another record downstream.\n #\n # Command that sanitizes record fields that are unknown to Solr schema.xml by either \n # deleting them (renameToPrefix is absent or a zero length string), or by moving them to a\n # field prefixed with the given renameToPrefix (e.g. renameToPrefix = \"ignored_\" to use \n # typical dynamic Solr fields).\n #\n # Recall that Solr throws an exception on any attempt to load a document that contains a \n # field that isn't specified in schema.xml.\n {\n sanitizeUnknownSolrFields {\n # Location from which to fetch Solr schema\n solrLocator : ${SOLR_LOCATOR}\n \n # renameToPrefix : \"ignored_\"\n }\n } \n \n # log the record at DEBUG level to SLF4J\n { logDebug { format : \"output record: {}\", args : [\"@{}\"] } } \n \n # load the record into a SolrServer or MapReduce SolrOutputFormat.\n { \n loadSolr {\n solrLocator : ${SOLR_LOCATOR}\n }\n }\n ]\n }\n]\n" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Agent Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Custom Mime-types File parameter.", "display_name": "Suppress Parameter Validation: Custom Mime-types File", "name": "role_config_suppression_agent_custom_mimetypes_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Home Directory parameter.", "display_name": "Suppress Parameter Validation: Flume Home Directory", "name": "role_config_suppression_agent_home_dir", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Plugin directories parameter.", "display_name": "Suppress Parameter Validation: Plugin directories", "name": "role_config_suppression_agent_plugin_dirs", "value": "false" }, { "desc": "Enables the health test that the Agent's process state is consistent with the role configuration", "display_name": "Agent Process Health Test", "name": "flume_agent_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_flume_agent_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_flume_agent_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "flume_agent_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_flume_agent_file_descriptor", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The port on which the Flume web server listens for requests.", "display_name": "HTTP Port", "name": "agent_http_port", "value": "41414" }, { "desc": "Home directory for Flume user. The File Channel uses paths for checkpoint and data directories that are within the user home.", "display_name": "Flume Home Directory", "name": "agent_home_dir", "value": "/var/lib/flume-ng" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Agent Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Agent in Bytes", "name": "agent_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Configuration File parameter.", "display_name": "Suppress Parameter Validation: Configuration File", "name": "role_config_suppression_agent_config_file", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Disables import of ZooKeeper configuration from the HBase classpath. This prevents zoo.cfg from overriding hbase-site.xml for Zookeeper quorum information. This option is only supported on CDH 4.4 or later deployments.", "display_name": "HBase sink prefer hbase-site.xml over Zookeeper config", "name": "agent_disable_zoo_cfg", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Agent Environment Advanced Configuration Snippet (Safety Valve)", "name": "AGENT_role_env_safety_valve", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Directory where Flume Agent will place its log files.", "display_name": "Flume Agent Log Directory", "name": "flume_agent_log_dir", "value": "/var/log/flume-ng" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Flume Agent parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Flume Agent", "name": "role_config_suppression_flume_agent_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Grok Dictionary File parameter.", "display_name": "Suppress Parameter Validation: Grok Dictionary File", "name": "role_config_suppression_agent_grok_dictionary_conf_file", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_flume_agent_log_directory_free_space", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Text that goes verbatim into custom-mimetypes.xml file used by the Flume-NG Solr sink.", "display_name": "Custom Mime-types File", "name": "agent_custom_mimetypes_file", "value": "\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here. Note that Flume agent only uses options that start with -D and -X (including -XX).", "display_name": "Java Configuration Options for Flume Agent", "name": "flume_agent_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_flume_agent_scm_health", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Agent logs. Typically used by log4j or logback.", "display_name": "Agent Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_flume_agent_host_health", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Agent Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Agent Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_agent_role_env_safety_valve", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "List of Flume plugin directories. This overrides the default Flume plugin directory.", "display_name": "Plugin directories", "name": "agent_plugin_dirs", "value": "/usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for Agent logs", "display_name": "Agent Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/sentry-impala-client-safety.xml0000666000175100017510000000063613245514472027561 0ustar zuulzuul00000000000000 sentry.service.client.server.rpc-port 3893 sentry.service.client.server.rpc-address hostname sentry.service.client.server.rpc-connection-timeout 200000 sentry.service.security.mode none sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-failovercontroller.json0000666000175100017510000005556213245514472027245 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where Failover Controller will place its log files.", "display_name": "Failover Controller Log Directory", "name": "failover_controller_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The RPC timeout for the HA health monitor.", "display_name": "HA Health Monitor RPC Timeout", "name": "ha_health_monitor_rpc_timeout_ms", "value": "45000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "fc_config_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hdfs_failovercontroller_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hdfs_failovercontroller_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_failovercontroller_role_env_safety_valve", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hdfs_failovercontroller_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Log Directory parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Log Directory", "name": "role_config_suppression_failover_controller_log_dir", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Failover Controller logs. Typically used by log4j or logback.", "display_name": "Failover Controller Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Failover Controller Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_fc_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Failover Controller in Bytes", "name": "failover_controller_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hdfs_failovercontroller_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hdfs_failovercontroller_scm_health", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hdfs_failovercontroller_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Failover Controller Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Failover Controller Environment Advanced Configuration Snippet (Safety Valve)", "name": "FAILOVERCONTROLLER_role_env_safety_valve", "value": null }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hdfs_failovercontroller_file_descriptor", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Failover Controller parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Failover Controller", "name": "role_config_suppression_failover_controller_java_opts", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "failovercontroller_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When computing the overall Failover Controller health, consider the host's health.", "display_name": "Failover Controller Host Health Test", "name": "failovercontroller_host_health_enabled", "value": "true" }, { "desc": "Enables the health test that the Failover Controller's process state is consistent with the role configuration", "display_name": "Failover Controller Process Health Test", "name": "failovercontroller_scm_health_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Failover Controller", "name": "failover_controller_java_opts", "value": "" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The maximum number of rolled log files to keep for Failover Controller logs. Typically used by log4j or logback.", "display_name": "Failover Controller Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The minimum log level for Failover Controller logs", "display_name": "Failover Controller Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/oozie-oozie_server.json0000666000175100017510000011402313245514472026232 0ustar zuulzuul00000000000000[ { "desc": "The period over which to compute the moving average of the callable queue size.", "display_name": "Oozie Server Callable Queue Monitoring Period", "name": "oozie_server_callable_queue_window", "value": "5" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "oozie_server_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Comma-separated list of ActionService executor extension classes. Only action types with associated executors can be used in workflows. For CDH 5.4 and higher, this parameter is used only to specify additional classes for workflows. All executor extension classes included in that release will be added automatically and do not need to be specified.", "display_name": "Oozie ActionService Executor Extension Classes", "name": "oozie_executor_extension_classes", "value": "" }, { "desc": "Comma-separated list of Oozie plug-ins to be activated. If one plugin cannot be loaded, all the plugins are ignored.", "display_name": "Oozie Server Plugins", "name": "oozie_plugins_list", "value": "" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Password parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Password", "name": "role_config_suppression_oozie_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Plugins parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Plugins", "name": "role_config_suppression_oozie_plugins_list", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_oozie_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_oozie_https_keystore_password", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "oozie_server_web_metric_collection_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Oozie Server", "name": "oozie_java_opts", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Host parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Host", "name": "role_config_suppression_oozie_email_smtp_host", "value": "false" }, { "desc": "Completed workflow jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Workflow Jobs", "name": "purgeservice_older_than", "value": "30" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_oozie_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Oozie Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie SchemaService Workflow Extension Schemas parameter.", "display_name": "Suppress Parameter Validation: Oozie SchemaService Workflow Extension Schemas", "name": "role_config_suppression_oozie_workflow_extension_schemas", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The SMTP server port to use for Oozie email action", "display_name": "Oozie Email Action SMTP Port", "name": "oozie_email_smtp_prt", "value": "25" }, { "desc": "Whether to suppress the results of the Callable Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Callable Queue Size", "name": "role_health_suppression_oozie_server_callablequeue_size_health", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Data Directory parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Data Directory", "name": "role_config_suppression_oozie_data_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_oozie_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_oozie_server_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action From Address parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action From Address", "name": "role_config_suppression_oozie_email_from_address", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Log Directory", "name": "role_config_suppression_oozie_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Enable SMTP authentication for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Enabled", "name": "oozie_email_smtp_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_oozie_https_keystore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml", "name": "role_config_suppression_oozie_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enables the health test that the Oozie Server's process state is consistent with the role configuration", "display_name": "Oozie Server Process Health Test", "name": "oozie_server_scm_health_enabled", "value": "true" }, { "desc": "SMTP password for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Password", "name": "oozie_email_smtp_password", "value": null }, { "desc": "Maximum concurrency for a given callable type. Each command is a callable type: submit, start, run, etc. Each action type is a callable type: MapReduce, SSH, sub-workflow, etc. All commands that use action executors (action-start, action-end. etc.) use the action type as the callable type.", "display_name": "Maximum concurrency for a given callable type", "name": "oozie_service_callablequeueservice_callable_concurrency", "value": "10" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "If true, enables the Oozie Server web console. ExtJS 2.2 zip archive must be extracted to /var/lib/oozie on the same host as the Oozie Server.", "display_name": "Enable Oozie Server Web Console", "name": "oozie_web_console", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "oozie_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Username for connecting to the database used by Oozie Server. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database User", "name": "oozie_database_user", "value": "sa" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_oozie_server_pause_duration", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_oozie_server_file_descriptor", "value": "false" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "oozie_server_pause_duration_window", "value": "5" }, { "desc": "The from address to be used for mailing all emails for Oozie email action", "display_name": "Oozie Email Action From Address", "name": "oozie_email_from_address", "value": "oozie@localhost" }, { "desc": "Port of Oozie Server", "display_name": "Oozie HTTP Port", "name": "oozie_http_port", "value": "11000" }, { "desc": "Whether to suppress the results of the Oozie Server Shared Library Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Oozie Server Shared Library Check", "name": "role_health_suppression_oozie_server_shared_lib_version_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where Oozie Server will place its log files.", "display_name": "Oozie Server Log Directory", "name": "oozie_log_dir", "value": "/var/log/oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Oozie Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Oozie Server", "name": "role_config_suppression_oozie_java_opts", "value": "false" }, { "desc": "Completed bundle jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Bundle Jobs", "name": "purgeservice_bundle_older_than", "value": "7" }, { "desc": "Maximum callable queue size", "display_name": "Maximum Callable Queue Size", "name": "oozie_service_callablequeueservice_queue_size", "value": "10000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The password for the Oozie JKS keystore file.", "display_name": "Oozie TLS/SSL Server JKS Keystore File Password", "name": "oozie_https_keystore_password", "value": null }, { "desc": "Completed coordinator jobs older than this value, in days, will be purged by the PurgeService.", "display_name": "Days to Keep Completed Coordinator Jobs", "name": "purgeservice_coord_older_than", "value": "7" }, { "desc": "Workflow Status metrics collection interval.", "display_name": "Workflow Status Metrics Collection Interval", "name": "oozie_job_metric_collection_interval", "value": "1" }, { "desc": "The minimum log level for Oozie Server logs", "display_name": "Oozie Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_oozie_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Name parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Name", "name": "role_config_suppression_oozie_database_name", "value": "false" }, { "desc": "When computing the overall Oozie Server health, consider the host's health.", "display_name": "Oozie Server Host Health Test", "name": "oozie_server_host_health_enabled", "value": "true" }, { "desc": "Password for connecting to the database used by Oozie Server. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database Password", "name": "oozie_database_password", "value": "" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie ActionService Executor Extension Classes parameter.", "display_name": "Suppress Parameter Validation: Oozie ActionService Executor Extension Classes", "name": "role_config_suppression_oozie_executor_extension_classes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds of the weighted average size of the Oozie Server callable queue over a recent period. See also Oozie Server Callable Queue Monitoring Period.", "display_name": "Oozie Server Callable Queue Monitoring Threshold", "name": "oozie_server_callable_queue_threshold", "value": "{\"critical\":\"95.0\",\"warning\":\"80.0\"}" }, { "desc": "Directory where the Oozie Server places its data. Only applicable when using Derby as the database type.", "display_name": "Oozie Server Data Directory", "name": "oozie_data_dir", "value": "/var/lib/oozie/data" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_oozie_server_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Oozie is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Oozie TLS/SSL Server JKS Keystore File Location", "name": "oozie_https_keystore_file", "value": "/var/lib/oozie/.keystore" }, { "desc": "Comma-separated list of SchemaService workflow extension schemas for additional action types. From CDH 5.4 and higher, this parameter is used only to specify additional schemas for workflows. All schemas included in that release will be added automatically and do not need to be specified.", "display_name": "Oozie SchemaService Workflow Extension Schemas", "name": "oozie_workflow_extension_schemas", "value": "" }, { "desc": "Port of the Oozie Server when using TLS/SSL.", "display_name": "Oozie HTTPS Port", "name": "oozie_https_port", "value": "11443" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to use the Codehale-based metrics for instrumentation. Enabling this disables the 'instrumentation' REST endpoint and enables the 'metrics' REST endpoint (<hostname:port>/v2/admin/metrics).", "display_name": "Enable The Metrics Instrumentation Service", "name": "oozie_use_metric_instrumentation", "value": "true" }, { "desc": "Type of the database used by Oozie Server.", "display_name": "Oozie Server Database Type", "name": "oozie_database_type", "value": "derby" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database Host parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database Host", "name": "role_config_suppression_oozie_database_host", "value": "false" }, { "desc": "Whether to purge completed workflows and their corresponding coordinator actions of long-running coordinator jobs if the completed workflow jobs are older than the value specified in oozie.service.PurgeService.older.than.", "display_name": "Enable Purge for Long-Running Coordinator Jobs", "name": "purgeservice_purge_old_coord_action", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_oozie_https_truststore_password", "value": "false" }, { "desc": "The SMTP server host to use for Oozie email action", "display_name": "Oozie Email Action SMTP Host", "name": "oozie_email_smtp_host", "value": "localhost" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_oozie_server_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Oozie TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_oozie_https_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Authentication Password parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Authentication Password", "name": "role_config_suppression_oozie_email_smtp_password", "value": "false" }, { "desc": "SMTP username for Oozie email action", "display_name": "Oozie Email Action SMTP Authentication Username", "name": "oozie_email_smtp_username", "value": null }, { "desc": "The password for the Oozie TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Oozie TLS/SSL Certificate Trust Store Password", "name": "oozie_https_truststore_password", "value": null }, { "desc": "Number of threads used for executing callables", "display_name": "Number Threads For Executing Callables", "name": "oozie_service_callablequeueservice_threads", "value": "50" }, { "desc": "The maximum number of rolled log files to keep for Oozie Server logs. Typically used by log4j or logback.", "display_name": "Oozie Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "720" }, { "desc": "The admin port Oozie server runs.", "display_name": "Oozie Admin Port", "name": "oozie_admin_port", "value": "11001" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Server Database User parameter.", "display_name": "Suppress Parameter Validation: Oozie Server Database User", "name": "role_config_suppression_oozie_database_user", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Oozie might connect to. This is used when Oozie is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Oozie TLS/SSL Certificate Trust Store File", "name": "oozie_https_truststore_file", "value": null }, { "desc": "If true, enables version check for Oozie Server and installed shared libraries.", "display_name": "Enable Oozie Server Shared Libraries Version Check", "name": "oozie_server_shared_lib_version_check_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "oozie_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Name of the database used by Oozie Server.", "display_name": "Oozie Server Database Name", "name": "oozie_database_name", "value": "oozie" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_oozie_server_swap_memory_usage", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "For advanced use only. A string to be inserted into oozie-site.xml for this role only.", "display_name": "Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml", "name": "oozie_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Email Action SMTP Authentication Username parameter.", "display_name": "Suppress Parameter Validation: Oozie Email Action SMTP Authentication Username", "name": "role_config_suppression_oozie_email_smtp_username", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Oozie Server in Bytes", "name": "oozie_java_heapsize", "value": "1073741824" }, { "desc": "Hostname of the database used by Oozie Server. If the port is non-default for your database type, use host:port notation. Does not apply if you are using Derby as the database type.", "display_name": "Oozie Server Database Host", "name": "oozie_database_host", "value": "localhost" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Oozie Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "OOZIE_SERVER_role_env_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/cdh_config.py0000666000175100017510000000741513245514472024144 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json import six from sahara.plugins.cdh.client import api_client # -- cm config -- cm_address = 'localhost' cm_port = 7180 cm_username = 'admin' cm_password = 'admin' hdfs_service_name = 'hdfs01' yarn_service_name = 'yarn01' oozie_service_name = 'oozie01' hive_service_name = 'hive01' hue_service_name = 'hue01' spark_service_name = 'spark_on_yarn01' zookeeper_service_name = 'zookeeper01' hbase_service_name = 'hbase01' flume_service_name = 'flume01' sqoop_service_name = 'sqoop01' solr_service_name = 'solr01' ks_indexer_service_name = 'ks_indexer01' impala_service_name = 'impala01' sentry_service_name = 'sentry01' def get_cm_api(): return api_client.ApiResource(cm_address, server_port=cm_port, username=cm_username, password=cm_password) def get_cluster(api): return api.get_all_clusters()[0] def process_service(service, service_name): for role_cfgs in service.get_all_role_config_groups(): role_cm_cfg = role_cfgs.get_config(view='full') role_cfg = parse_config(role_cm_cfg) role_name = role_cfgs.roleType.lower() write_cfg(role_cfg, '%s-%s.json' % (service_name, role_name)) service_cm_cfg = service.get_config(view='full')[0] service_cfg = parse_config(service_cm_cfg) write_cfg(service_cfg, '%s-service.json' % service_name) def parse_config(config): cfg = [] for name, value in six.iteritems(config): p = { 'name': value.name, 'value': value.default, 'display_name': value.displayName, 'desc': value.description } cfg.append(p) return cfg def write_cfg(cfg, file_name): to_write = json.dumps(cfg, sort_keys=True, indent=4, separators=(',', ': ')) with open(file_name, 'w') as f: f.write(to_write) def main(): client = get_cm_api() cluster = get_cluster(client) hdfs = cluster.get_service(hdfs_service_name) process_service(hdfs, 'hdfs') yarn = cluster.get_service(yarn_service_name) process_service(yarn, 'yarn') oozie = cluster.get_service(oozie_service_name) process_service(oozie, 'oozie') hive = cluster.get_service(hive_service_name) process_service(hive, 'hive') hue = cluster.get_service(hue_service_name) process_service(hue, 'hue') spark = cluster.get_service(spark_service_name) process_service(spark, 'spark') zookeeper = cluster.get_service(zookeeper_service_name) process_service(zookeeper, 'zookeeper') hbase = cluster.get_service(hbase_service_name) process_service(hbase, 'hbase') flume = cluster.get_service(flume_service_name) process_service(flume, 'flume') sqoop = cluster.get_service(sqoop_service_name) process_service(sqoop, 'sqoop') solr = cluster.get_service(solr_service_name) process_service(solr, 'solr') ks_indexer = cluster.get_service(ks_indexer_service_name) process_service(ks_indexer, 'ks_indexer') impala = cluster.get_service(impala_service_name) process_service(impala, 'impala') sentry = cluster.get_service(sentry_service_name) process_service(sentry, 'sentry') if __name__ == '__main__': main() sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/impala-statestore.json0000666000175100017510000005211113245514472026031 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SSL/TLS Certificate for Statestore Webserver parameter.", "display_name": "Suppress Parameter Validation: SSL/TLS Certificate for Statestore Webserver", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_statestore_heap_dump_directory_free_space", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "statestore_web_metric_collection_enabled", "value": "true" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "statestore_startup_tolerance", "value": "5" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to StateStore command line flags.", "display_name": "Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "statestore_cmd_args_safety_valve", "value": null }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "statestore_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "StateStore Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala StateStore Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala StateStore Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_statestore_role_env_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "When computing the overall Impala StateStore health, consider the host's health.", "display_name": "Impala StateStore Host Health Test", "name": "statestore_host_health_enabled", "value": "true" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_statestore_scm_health", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Username for Statestore webserver authentication.", "display_name": "Statestore Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Password for Statestore webserver authentication.", "display_name": "Statestore Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Maximum number of tasks allowed to be pending at the thread manager underlying the StateStore Thrift server (0 allows infinitely many pending tasks)", "display_name": "Maximum StateStore Pending Tasks", "name": "state_store_pending_task_count_max", "value": "0" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_statestore_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Port where StateStore debug web server runs.", "display_name": "StateStore HTTP Server Port", "name": "statestore_webserver_port", "value": "25010" }, { "desc": "Directory where StateStore will place its log files.", "display_name": "StateStore Log Directory", "name": "log_dir", "value": "/var/log/statestore" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_statestore_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_statestore_unexpected_exits", "value": "false" }, { "desc": "Port where StateStoreService is exported.", "display_name": "StateStore Service Port", "name": "state_store_port", "value": "24000" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Statestore Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_statestore_cmd_args_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: StateStore Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "Number of worker threads for the thread manager underlying the StateStore Thrift server.", "display_name": "StateStore Worker Threads", "name": "state_store_num_server_worker_threads", "value": "4" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the StateStore Log Directory parameter.", "display_name": "Suppress Parameter Validation: StateStore Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_statestore_memory_rss_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Statestore Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_statestore_file_descriptor", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above. ", "display_name": "StateStore Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The maximum size, in megabytes, per log file for Impala StateStore logs. Typically used by log4j or logback.", "display_name": "Impala StateStore Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Enables the health test that the Impala StateStore's process state is consistent with the role configuration", "display_name": "Impala StateStore Process Health Test", "name": "statestore_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_statestore_log_directory_free_space", "value": "false" }, { "desc": "Directory where a StateStore core dump is placed.", "display_name": "StateStore Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/statestore" }, { "desc": "Local path to the certificate presented by the StateStore debug webserver. This file must be in .pem format. If empty, webserver SSL/TLS support is not enabled.", "display_name": "SSL/TLS Certificate for Statestore Webserver", "name": "webserver_certificate_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Statestore Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Statestore Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "Enable/Disable StateStore web server. This web server contains useful information about StateStore daemon.", "display_name": "Enable StateStore Web Server", "name": "statestore_enable_webserver", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala StateStore Environment Advanced Configuration Snippet (Safety Valve)", "name": "STATESTORE_role_env_safety_valve", "value": null }, { "desc": "The minimum log level for Impala StateStore logs", "display_name": "Impala StateStore Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "StateStore Maximum Log Files", "name": "max_log_files", "value": "10" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "statestore_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_statestore_swap_memory_usage", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/kafka-kafka_mirror_maker.json0000666000175100017510000007023613245514472027304 0ustar zuulzuul00000000000000[ { "desc": "The maximum size, in megabytes, per log file for Kafka MirrorMaker logs. Typically used by log4j or logback.", "display_name": "Kafka MirrorMaker Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Stop the entire mirror maker when a send failure occurs.", "display_name": "Abort on Send Failure", "name": "abort.on.send.failure", "value": "true" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Maximum number of bytes that can be buffered between producer and consumer. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Queue Size", "name": "queue.byte.size", "value": "100000000" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kafka_kafka_mirror_maker_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kafka_mirror_maker_role_env_safety_valve", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Only required if source Kafka cluster requires client authentication.", "display_name": "Source Kafka Cluster's Client Auth", "name": "source.ssl.client.auth", "value": "false" }, { "desc": "When computing the overall Kafka MirrorMaker health, consider the host's health.", "display_name": "Kafka MirrorMaker Host Health Test", "name": "kafka_mirror_maker_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties", "name": "role_config_suppression_ssl_client.properties_role_safety_valve", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Kafka MirrorMaker logs. Typically used by log4j or logback.", "display_name": "Kafka MirrorMaker Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topic Blacklist parameter.", "display_name": "Suppress Parameter Validation: Topic Blacklist", "name": "role_config_suppression_blacklist", "value": "false" }, { "desc": "The minimum log level for Kafka MirrorMaker logs", "display_name": "Kafka MirrorMaker Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Offset commit interval in milliseconds.", "display_name": "Offset Commit Interval", "name": "offset.commit.interval.ms", "value": "60000" }, { "desc": "For advanced use only. A string to be inserted into mirror_maker_producers.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties", "name": "mirror_maker_producers.properties_role_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kafka_kafka_mirror_maker_host_health", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kafka_mirror_maker_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties", "name": "role_config_suppression_ssl_server.properties_role_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The password for the Kafka MirrorMaker JKS keystore file.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Destination Broker List parameter.", "display_name": "Suppress Parameter Validation: Destination Broker List", "name": "role_config_suppression_bootstrap.servers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Number of producer instances. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Number of Producers", "name": "num.producers", "value": "1" }, { "desc": "For advanced use only. A string to be inserted into ssl_client.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_client.properties", "name": "ssl_client.properties_role_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_MIRROR_MAKER_role_env_safety_valve", "value": null }, { "desc": "Name of the consumer group used by MirrorMaker. When multiple role instances are configured with the same topics and same group ID, the role instances load-balance replication for the topics. When multiple role instances are configured with the same topics but different group ID, each role instance replicates all the events for those topics - this can be used to replicate the source cluster into multiple destination clusters.", "display_name": "Consumer Group ID", "name": "group.id", "value": "cloudera_mirrormaker" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kafka_kafka_mirror_maker_swap_memory_usage", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Consumer Group ID parameter.", "display_name": "Suppress Parameter Validation: Consumer Group ID", "name": "role_config_suppression_group.id", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Consumer Rebalance Listener Arguments parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Consumer Rebalance Listener Arguments", "name": "role_config_suppression_consumer.rebalance.listener.args", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka MirrorMaker might connect to. This is used when Kafka MirrorMaker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Kafka MirrorMaker TLS/SSL Certificate Trust Store File", "name": "ssl_client_truststore_location", "value": "" }, { "desc": "Enables the health test that the Kafka MirrorMaker's process state is consistent with the role configuration", "display_name": "Kafka MirrorMaker Process Health Test", "name": "kafka_mirror_maker_scm_health_enabled", "value": "true" }, { "desc": "Number of messages that are buffered between producer and consumer. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Message Queue Size", "name": "queue.size", "value": "10000" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "A consumer rebalance listener of type ConsumerRebalanceListener to be invoked when MirrorMaker's consumer rebalances.", "display_name": "MirrorMaker Consumer Rebalance Listener", "name": "consumer.rebalance.listener", "value": "" }, { "desc": "Run with MirrorMaker settings that eliminate potential loss of data. This impacts performance, but is highly recommended. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Avoid Data Loss", "name": "no.data.loss", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties", "name": "role_config_suppression_mirror_maker_producers.properties_role_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into mirror_maker_consumers.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties", "name": "mirror_maker_consumers.properties_role_safety_valve", "value": null }, { "desc": "Protocol to be used for communication with source kafka cluster.", "display_name": "Source Kafka Cluster's Security Protocol", "name": "source.security.protocol", "value": "PLAINTEXT" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Consumer Rebalance Listener parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Consumer Rebalance Listener", "name": "role_config_suppression_consumer.rebalance.listener", "value": "false" }, { "desc": "Protocol to be used for communication with destination kafka cluster.", "display_name": "Destination Kafka Cluster's Security Protocol", "name": "destination.security.protocol", "value": "PLAINTEXT" }, { "desc": "Regular expression that represents a set of topics to mirror. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used.", "display_name": "Topic Whitelist", "name": "whitelist", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kafka_kafka_mirror_maker_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topic Whitelist parameter.", "display_name": "Suppress Parameter Validation: Topic Whitelist", "name": "role_config_suppression_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Port for JMX.", "display_name": "JMX Port", "name": "jmx_port", "value": "9394" }, { "desc": "Arguments used by MirrorMaker message handler.", "display_name": "MirrorMaker Message Handler Arguments", "name": "message.handler.args", "value": "" }, { "desc": "List of brokers on destination cluster. This should be more than one, for high availability, but there's no need to list all brokers.", "display_name": "Destination Broker List", "name": "bootstrap.servers", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Source Broker List parameter.", "display_name": "Suppress Parameter Validation: Source Broker List", "name": "role_config_suppression_source.bootstrap.servers", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kafka_kafka_mirror_maker_file_descriptor", "value": "false" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka MirrorMaker is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Message Handler parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Message Handler", "name": "role_config_suppression_message.handler", "value": "false" }, { "desc": "Only required if destination Kafka cluster requires client authentication.", "display_name": "Destination Kafka Cluster's Client Auth", "name": "destination.ssl.client.auth", "value": "false" }, { "desc": "A MirrorMaker message handler of type MirrorMakerMessageHandler that will process every record in-between producer and consumer.", "display_name": "MirrorMaker Message Handler", "name": "message.handler", "value": "" }, { "desc": "List of brokers on source cluster. This should be more than one, for high availability, but there's no need to list all brokers.", "display_name": "Source Broker List", "name": "source.bootstrap.servers", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties", "name": "role_config_suppression_mirror_maker_consumers.properties_role_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "For advanced use only. A string to be inserted into ssl_server.properties for this role only.", "display_name": "Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for ssl_server.properties", "name": "ssl_server.properties_role_safety_valve", "value": null }, { "desc": "Number of consumer threads.", "display_name": "Number of Consumer Threads", "name": "num.streams", "value": "1" }, { "desc": "The password that protects the private key contained in the JKS keystore used when Kafka MirrorMaker is acting as a TLS/SSL server.", "display_name": "Kafka MirrorMaker TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": "" }, { "desc": "Encrypt communication between clients and Kafka MirrorMaker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Kafka MirrorMaker", "name": "ssl_enabled", "value": "false" }, { "desc": "Regular expression that represents a set of topics to avoid mirroring. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used. WARNING: Does not work with Kafka 2.0 or later.", "display_name": "Topic Blacklist", "name": "blacklist", "value": "" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Arguments used by MirrorMaker consumer rebalance listener.", "display_name": "MirrorMaker Consumer Rebalance Listener Arguments", "name": "consumer.rebalance.listener.args", "value": "" }, { "desc": "The log directory for log files of the role Kafka MirrorMaker.", "display_name": "Kafka MirrorMaker Log Directory", "name": "log_dir", "value": "/var/log/kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MirrorMaker Message Handler Arguments parameter.", "display_name": "Suppress Parameter Validation: MirrorMaker Message Handler Arguments", "name": "role_config_suppression_message.handler.args", "value": "false" }, { "desc": "The password for the Kafka MirrorMaker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Kafka MirrorMaker TLS/SSL Certificate Trust Store Password", "name": "ssl_client_truststore_password", "value": "" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hue-kt_renewer.json0000666000175100017510000003322113245514472025320 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_kt_renewer_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kt_renewer_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Ticket Renewer Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kerberos Ticket Renewer Log Directory", "name": "role_config_suppression_kt_renewer_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kt_renewer_unexpected_exits", "value": "false" }, { "desc": "Interval in seconds with which Hue's Kerberos ticket will get renewed.", "display_name": "Hue Keytab Renewal Interval", "name": "keytab_reinit_frequency", "value": "3600" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kt_renewer_host_health", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve)", "name": "KT_RENEWER_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When computing the overall Kerberos Ticket Renewer health, consider the host's health.", "display_name": "Kerberos Ticket Renewer Host Health Test", "name": "kt_renewer_host_health_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kt_renewer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Enables the health test that the Kerberos Ticket Renewer's process state is consistent with the role configuration", "display_name": "Kerberos Ticket Renewer Process Health Test", "name": "kt_renewer_scm_health_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Directory where Kerberos Ticket Renewer will place its log files.", "display_name": "Kerberos Ticket Renewer Log Directory", "name": "kt_renewer_log_dir", "value": "/var/log/hue" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kerberos Ticket Renewer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kt_renewer_role_env_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kt_renewer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_kt_renewer_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kt_renewer_scm_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/solr-solr_server.json0000666000175100017510000006340713245514472025727 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_solr_server_web_metric_collection", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Solr Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Solr Server", "name": "role_config_suppression_solr_java_opts", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Enable caching of HDFS blocks in Solr. There is one block cache per Solr collection. configured to use off-heap memory, Maximum Off-Heap Memory must be set high enough to account for all block caches.", "display_name": "HDFS Block Cache", "name": "solr_hdfs_blockcache_enabled", "value": "true" }, { "desc": "When computing the overall Solr Server health, consider the host's health.", "display_name": "Solr Server Host Health Test", "name": "solr_server_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Admin port of the Solr Server.", "display_name": "Solr Admin Port", "name": "solr_admin_port", "value": "8984" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "solr_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Solr Server in Bytes", "name": "solr_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Solr Server API Liveness heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Solr Server API Liveness", "name": "role_health_suppression_solr_core_status_collection_health", "value": "false" }, { "desc": "The maximum number of request processing threads to be created by Solr server, which determines the maximum number of simultaneous requests that can be handled.", "display_name": "Solr Max Connector Threads", "name": "solr_max_connector_thread", "value": "10000" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_solr_server_swap_memory_usage", "value": "false" }, { "desc": "Enables the health test that the Solr Server's process state is consistent with the role configuration", "display_name": "Solr Server Process Health Test", "name": "solr_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_solr_server_host_health", "value": "false" }, { "desc": "The health test thresholds on the duration of the Solr Server API request.", "display_name": "Solr Server API Liveness Request Duration", "name": "solr_core_status_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Solr Server Log Directory", "name": "role_config_suppression_solr_log_dir", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of Solr Server in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of Solr Server in Bytes", "name": "role_config_suppression_solr_java_heapsize", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Directory where Solr Server will place its log files.", "display_name": "Solr Server Log Directory", "name": "solr_log_dir", "value": "/var/log/solr" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Solr Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Solr Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SOLR_SERVER_role_env_safety_valve", "value": null }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather status of Solr Cores from the Solr Server with a simple API request.", "display_name": "Solr Server API Liveness", "name": "solr_core_status_collection_health_enabled", "value": "true" }, { "desc": "Use off-heap memory when caching HDFS blocks in Solr.", "display_name": "HDFS Block Cache Off-Heap Memory", "name": "solr_hdfs_blockcache_direct_memory_allocation", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "HTTPS port of Solr Server.", "display_name": "Solr HTTPS port", "name": "solr_https_port", "value": "8985" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Data Directory parameter.", "display_name": "Suppress Parameter Validation: Solr Data Directory", "name": "role_config_suppression_solr_data_dir", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Solr Server logs. Typically used by log4j or logback.", "display_name": "Solr Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_solr_server_gc_duration", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory on local file system where Solr Server keeps the configurations for collections.", "display_name": "Solr Data Directory", "name": "solr_data_dir", "value": "/var/lib/solr" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "solr_server_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Solr Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_solr_server_role_env_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Number of slabs per block cache. The size of the cache is 8 KB (the block size) times the number of blocks per slab times the number of slabs.", "display_name": "HDFS Block Cache Number of Slabs", "name": "solr_hdfs_blockcache_slab_count", "value": "1" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_solr_server_unexpected_exits", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Solr Server logs. Typically used by log4j or logback.", "display_name": "Solr Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Number of blocks per cache slab. The size of the cache is 8 KB (the block size) times the number of blocks per slab times the number of slabs.", "display_name": "HDFS Block Cache Blocks per Slab", "name": "solr_hdfs_blockcache_blocksperbank", "value": "16384" }, { "desc": "Address of the load balancer, specified in host:port format.", "display_name": "Solr Load Balancer", "name": "solr_load_balancer", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_solr_server_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_solr_server_log_directory_free_space", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "solr_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_solr_server_file_descriptor", "value": "false" }, { "desc": "Maximum amount of off-heap memory in bytes that may be allocated by the Java process. Passed to Java -XX:MaxDirectMemorySize. If unset, defaults to the size of the heap.", "display_name": "Java Direct Memory Size of Solr Server in Bytes", "name": "solr_java_direct_memory_size", "value": "1073741824" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "solr_server_gc_duration_window", "value": "5" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "solr_server_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Solr Server", "name": "solr_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The minimum log level for Solr Server logs", "display_name": "Solr Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_solr_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "HTTP port of Solr Server.", "display_name": "Solr HTTP Port", "name": "solr_http_port", "value": "8983" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Solr Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Solr Load Balancer", "name": "role_config_suppression_solr_load_balancer", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hbase-hbasethriftserver.json0000666000175100017510000007251513245514472027217 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbasethriftserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase Thrift Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase Thrift Server", "name": "role_config_suppression_hbase_thriftserver_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_thrift_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server DNS Network Interface", "name": "role_config_suppression_hbase_thriftserver_dns_interface", "value": "false" }, { "desc": "The \"core size\" of the thread pool. New threads are created on every connection until this many threads are created.", "display_name": "HBase Thrift Server Min Worker Threads", "name": "hbase_thriftserver_min_worker_threads", "value": "200" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_thrift_server_host_health", "value": "false" }, { "desc": "The port that HBase Thrift Server binds to.", "display_name": "HBase Thrift Server Port", "name": "hbase_thriftserver_port", "value": "9090" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Bind Address parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Bind Address", "name": "role_config_suppression_hbase_thriftserver_bindaddress", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_thrift_server_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Log Directory", "name": "role_config_suppression_hbase_thriftserver_log_dir", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "When computing the overall HBase Thrift Server health, consider the host's health.", "display_name": "HBase Thrift Server Host Health Test", "name": "hbasethriftserver_host_health_enabled", "value": "true" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase Thrift Server over HTTP is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location", "name": "hbase_thriftserver_http_keystore_file", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase Thrift Server in Bytes", "name": "hbase_thriftserver_java_heapsize", "value": "1073741824" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The password for the HBase Thrift Server over HTTP JKS keystore file.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password", "name": "hbase_thriftserver_http_keystore_password", "value": null }, { "desc": "If true, HBase Thrift Server Web UI will bind to a wildcard address (0.0.0.0). Otherwise it will bind to a host name. Only available in CDH 4.3 and later.", "display_name": "HBase Thrift Server Web UI Bind to Wildcard Address", "name": "hbase_thriftserver_info_bind_to_wildcard", "value": "true" }, { "desc": "Enables the health test that the HBase Thrift Server's process state is consistent with the role configuration", "display_name": "HBase Thrift Server Process Health Test", "name": "hbasethriftserver_scm_health_enabled", "value": "true" }, { "desc": "Address to bind the HBase Thrift Server to. When using the THsHaServer or the TNonblockingServer, always binds to 0.0.0.0 irrespective of this configuration value.", "display_name": "HBase Thrift Server Bind Address", "name": "hbase_thriftserver_bindaddress", "value": "0.0.0.0" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Type of HBase Thrift Server.", "display_name": "HBase Thrift Server Type", "name": "hbase_thriftserver_type", "value": "threadpool" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_hbase_thriftserver_http_keystore_password", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HBase Thrift Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASETHRIFTSERVER_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HBase Thrift Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_thrift_server_log_directory_free_space", "value": "false" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase Thrift Server over HTTP is acting as a TLS/SSL server.", "display_name": "HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password", "name": "hbase_thriftserver_http_keystore_keypassword", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_hbase_thriftserver_http_keystore_file", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase Thrift Server", "name": "hbase_thriftserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Use framed transport. When using the THsHaServer or TNonblockingServer, framed transport is always used irrespective of this configuration value.", "display_name": "Enable HBase Thrift Server Framed Transport", "name": "hbase_thriftserver_framed", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for HBase Thrift Server logs. Typically used by log4j or logback.", "display_name": "HBase Thrift Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The port that HBase Thrift Server Web UI binds to.", "display_name": "HBase Thrift Server Web UI Port", "name": "hbase_thriftserver_info_port", "value": "9095" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server over HTTP TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_hbase_thriftserver_http_keystore_keypassword", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_thrift_server_swap_memory_usage", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The host name or IP address of the DNS name server which an HBase Thrift Server should use to determine the host name used for communication and display purposes.", "display_name": "HBase Thrift Server DNS Name Server", "name": "hbase_thriftserver_dns_nameserver", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server DNS Name Server", "name": "role_config_suppression_hbase_thriftserver_dns_nameserver", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HBase Thrift Server logs. Typically used by log4j or logback.", "display_name": "HBase Thrift Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_thrift_server_scm_health", "value": "false" }, { "desc": "The name of the DNS network interface from which an HBase Thrift Server should report its IP address.", "display_name": "HBase Thrift Server DNS Network Interface", "name": "hbase_thriftserver_dns_interface", "value": null }, { "desc": "Use the TCompactProtocol instead of the default TBinaryProtocol. TCompactProtocol is a binary protocol that is more compact than the default and typically more efficient.", "display_name": "Enable HBase Thrift Server Compact Protocol", "name": "hbase_thriftserver_compact", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Directory where HBase Thrift Server will place its log files.", "display_name": "HBase Thrift Server Log Directory", "name": "hbase_thriftserver_log_dir", "value": "/var/log/hbase" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_thriftserver_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_thrift_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbasethriftserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Thrift Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_thriftserver_config_safety_valve", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for HBase Thrift Server logs", "display_name": "HBase Thrift Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Encrypt communication between clients and HBase Thrift Server over HTTP using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HBase Thrift Server over HTTP", "name": "hbase_thriftserver_http_use_ssl", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-journalnode.json0000666000175100017510000007211713245514472025645 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "JournalNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "JOURNALNODE_role_env_safety_valve", "value": null }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_journal_node_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Fsync Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Fsync Latency", "name": "role_health_suppression_journal_node_fsync_latency", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: JournalNode Log Directory", "name": "role_config_suppression_journalnode_log_dir", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of JournalNode in Bytes", "name": "journalNode_java_heapsize", "value": "268435456" }, { "desc": "If enabled, the JournalNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind JournalNode to Wildcard Address", "name": "journalnode_bind_wildcard", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "journalnode_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_journal_node_scm_health", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "journalnode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_journal_node_host_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Enables the health test that the JournalNode's process state is consistent with the role configuration", "display_name": "JournalNode Process Health Test", "name": "journalnode_scm_health_enabled", "value": "true" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JournalNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_journal_node_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_jn_config_safety_valve", "value": "false" }, { "desc": "Port for the JournalNode's RPC. Combined with the JournalNode's hostname to build its RPC address.", "display_name": "JournalNode RPC Port", "name": "dfs_journalnode_rpc_port", "value": "8485" }, { "desc": "Directory on the local file system where NameNode edits are written.", "display_name": "JournalNode Edits Directory", "name": "dfs_journalnode_edits_dir", "value": null }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory.", "display_name": "JournalNode Edits Directory Free Space Monitoring Absolute Thresholds", "name": "journalnode_edits_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_journal_node_swap_memory_usage", "value": "false" }, { "desc": "Enables the health check that verifies the active NameNode's sync status to the JournalNode", "display_name": "Active NameNode Sync Status Health Check", "name": "journalnode_sync_status_enabled", "value": "true" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "When computing the overall JournalNode health, consider the host's health.", "display_name": "JournalNode Host Health Test", "name": "journalnode_host_health_enabled", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JournalNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for JournalNode", "name": "role_config_suppression_journalnode_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Sync Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sync Status", "name": "role_health_suppression_journal_node_sync_status", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The base port where the secure JournalNode web UI listens. Combined with the JournalNode's hostname to build its secure web UI address.", "display_name": "Secure JournalNode Web UI Port (TLS/SSL)", "name": "dfs_journalnode_https_port", "value": "8481" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "journalnode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JournalNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_journalnode_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "JournalNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for JournalNode fsync latency.", "display_name": "JournalNode Fsync Latency Thresholds", "name": "journalnode_fsync_latency_thresholds", "value": "{\"critical\":\"3000.0\",\"warning\":\"1000.0\"}" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "JournalNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "jn_config_safety_valve", "value": null }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JournalNode Edits Directory parameter.", "display_name": "Suppress Parameter Validation: JournalNode Edits Directory", "name": "role_config_suppression_dfs_journalnode_edits_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_journal_node_web_metric_collection", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for JournalNode logs. Typically used by log4j or logback.", "display_name": "JournalNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "journalnode_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_journal_node_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "journalnode_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's JournalNode Edits Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a JournalNode Edits Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "JournalNode Edits Directory Free Space Monitoring Percentage Thresholds", "name": "journalnode_edits_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_journal_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "Directory where JournalNode will place its log files.", "display_name": "JournalNode Log Directory", "name": "journalnode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "Whether to suppress the results of the JournalNode Edits Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JournalNode Edits Directory Free Space", "name": "role_health_suppression_journal_node_edits_directory_free_space", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for JournalNode", "name": "journalNode_java_opts", "value": "" }, { "desc": "The maximum number of rolled log files to keep for JournalNode logs. Typically used by log4j or logback.", "display_name": "JournalNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The amount of time at JournalNode startup allowed for the active NameNode to get in sync with the JournalNode.", "display_name": "Active NameNode Sync Status Startup Tolerance", "name": "journalnode_sync_status_startup_tolerance", "value": "180" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Port for the JournalNode HTTP web UI. Combined with the JournalNode hostname to build its HTTP address.", "display_name": "JournalNode HTTP Port", "name": "dfs_journalnode_http_port", "value": "8480" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for JournalNode logs", "display_name": "JournalNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_journal_node_gc_duration", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/yarn-service.json0000666000175100017510000010537513245514472025015 0ustar zuulzuul00000000000000[ { "desc": "ACL that determines which users and groups can submit and kill applications in any pool, and can issue commands on ResourceManager roles.", "display_name": "Admin ACL", "name": "yarn_admin_acl", "value": "*" }, { "desc": "The health test thresholds of the overall NodeManager health. The check returns \"Concerning\" health if the percentage of \"Healthy\" NodeManagers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" NodeManagers falls below the critical threshold.", "display_name": "Healthy NodeManager Monitoring Thresholds", "name": "yarn_nodemanagers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Name of the ZooKeeper service that this YARN service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. Typically edited using the Pools configuration UI.", "display_name": "Fair Scheduler Allocations", "name": "yarn_fs_scheduled_allocations", "value": "{\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":null,\"fairSharePreemptionTimeout\":null,\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":null,\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null}],\"schedulablePropertiesList\":[{\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null}],\"userMaxAppsDefault\":null,\"users\":[]}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Controls which applications non-admin users can see in the applications list view", "display_name": "Non-Admin Users Applications List Visibility Settings", "name": "user_application_list_settings", "value": "ALL" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "service_config_suppression_yarn_service_replication_config_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "yarn_core_site_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the ResourceManager Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: ResourceManager Count Validator", "name": "service_config_suppression_resourcemanager_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Application Aggregates parameter.", "display_name": "Suppress Parameter Validation: YARN Application Aggregates", "name": "service_config_suppression_yarn_application_aggregates", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Client Truststore File Password parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Client Truststore File Password", "name": "service_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted shuffle and encrypted web UIs. Applies to all configurations of daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "service_config_suppression_yarn_hadoop_policy_config_safety_valve", "value": "false" }, { "desc": "Entries to add to the classpaths of YARN applications.", "display_name": "YARN Application Classpath", "name": "yarn_application_classpath", "value": "$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*" }, { "desc": "For advanced use only, a string to be inserted into yarn-site.xml. Applies to all HDFS Replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_service_replication_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "service_config_suppression_mapreduce_service_replication_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the JobHistory Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: JobHistory Server Count Validator", "name": "service_config_suppression_jobhistory_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_yarn_core_site_safety_valve", "value": "false" }, { "desc": "UNIX user that containers run as when Linux-container-executor is used in nonsecure mode.", "display_name": "UNIX User for Nonsecure Mode with Linux Container Executor", "name": "yarn_nodemanager_linux_container_executor_nonsecure_mode_local_user", "value": "nobody" }, { "desc": "Whether YARN creates a cgroup per container, thereby isolating the CPU usage of containers. When set, yarn.nodemanager.linux-container-executor.resources-handler.class is configured to org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler. The host (in Cloudera Manager) must have cgroups enabled. The number of shares allocated to all YARN containers is configured by adjusting the CPU shares value of the Node Manager in the Resource Management configuration group.", "display_name": "Use CGroups for Resource Management", "name": "yarn_service_cgroups", "value": "false" }, { "desc": "Name of the HDFS service that this YARN service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "For advanced use only, a string to be inserted into mapred-site.xml. Applies to all HDFS Replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "mapreduce_service_replication_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the ResourceManager Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ResourceManager Health", "name": "service_health_suppression_yarn_resourcemanagers_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the UNIX User for Nonsecure Mode with Linux Container Executor parameter.", "display_name": "Suppress Parameter Validation: UNIX User for Nonsecure Mode with Linux Container Executor", "name": "service_config_suppression_yarn_nodemanager_linux_container_executor_nonsecure_mode_local_user", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "yarn_service_env_safety_valve", "value": null }, { "desc": "The tolerance window used in YARN service tests that depend on detection of the active ResourceManager.", "display_name": "Active ResourceManager Detection Window", "name": "yarn_active_resourcemanager_detecton_window", "value": "3" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "yarn_ssl_server_safety_valve", "value": null }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. Typically edited using the Rules configuration UI.", "display_name": "Fair Scheduler Configuration Rules", "name": "yarn_fs_schedule_rules", "value": "[]" }, { "desc": "Whether to suppress the results of the JobHistory Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JobHistory Server Health", "name": "service_health_suppression_yarn_jobhistory_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Llama Proxy User Hosts", "name": "service_config_suppression_llama_proxy_user_hosts_list", "value": "false" }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "Enables Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Note: This is effective only if Kerberos is enabled for the HDFS service.", "display_name": "Enable Kerberos Authentication for HTTP Web-Consoles", "name": "hadoop_secure_web_ui", "value": "false" }, { "desc": "Password for the server keystore file used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "The amount of time after ResourceManager(s) start that the lack of an active ResourceManager will be tolerated. This is an advanced option that does not often need to be changed.", "display_name": "ResourceManager Activation Startup Tolerance", "name": "yarn_resourcemanager_activation_startup_tolerance", "value": "180" }, { "desc": "When computing the overall YARN service health, whether to consider the active ResourceManager's health.", "display_name": "Active ResourceManager Role Health Check", "name": "yarn_resourcemanagers_health_enabled", "value": "true" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "yarn" }, { "desc": "When set to true, pools specified in applications but not explicitly configured, are created at runtime with default settings. When set to false, applications specifying pools not explicitly configured run in a pool named default. This setting applies when an application explicitly specifies a pool and when the application runs in a pool named with the username associated with the application.", "display_name": "Allow Undeclared Pools", "name": "yarn_scheduler_fair_allow_undeclared_pools", "value": "true" }, { "desc": "When computing the overall YARN health, consider JobHistory Server's health", "display_name": "JobHistory Server Role Health Test", "name": "yarn_jobhistoryserver_health_enabled", "value": "true" }, { "desc": "Path to the truststore file used when roles of this service act as TLS/SSL clients. Overrides the cluster-wide default truststore location set in HDFS. This truststore must be in JKS format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. The contents of the truststore can be modified without restarting any roles. By default, changes to its contents are picked up within ten seconds. If not set, the default Java truststore is used to verify certificates.", "display_name": "TLS/SSL Client Truststore File Location", "name": "ssl_client_truststore_location", "value": null }, { "desc": "How long to keep aggregation logs before deleting them.", "display_name": "Log Aggregation Retention Period", "name": "yarn_log_aggregation_retain_seconds", "value": "604800" }, { "desc": "When computing the overall YARN service health, whether to consider the health of the standby ResourceManager.", "display_name": "Standby ResourceManager Health Check", "name": "yarn_standby_resourcemanager_health_enabled", "value": "true" }, { "desc": "Cluster ID used when ResourceManager is Highly Available.", "display_name": "ResourceManager HA Cluster ID", "name": "yarn_rm_ha_cluster_id", "value": "yarnRM" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager HA Cluster ID parameter.", "display_name": "Suppress Parameter Validation: ResourceManager HA Cluster ID", "name": "service_config_suppression_yarn_rm_ha_cluster_id", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin ACL parameter.", "display_name": "Suppress Parameter Validation: Admin ACL", "name": "service_config_suppression_yarn_admin_acl", "value": "false" }, { "desc": "Whether users and groups specified in Admin ACL should be checked for authorization to perform admin operations.", "display_name": "Enable ResourceManager ACLs", "name": "yarn_acl_enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service MapReduce Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_yarn_service_mapred_safety_valve", "value": "false" }, { "desc": "The group that this service's processes should run as. (Except the Job History Server, which has its own group)", "display_name": "System Group", "name": "process_groupname", "value": "hadoop" }, { "desc": "For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "yarn_hadoop_policy_config_safety_valve", "value": null }, { "desc": "Comma-delimited list of hosts where you want to allow the Llama (AM for Impala) user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Llama Proxy User Hosts", "name": "llama_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_yarn_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Application Classpath parameter.", "display_name": "Suppress Parameter Validation: YARN Application Classpath", "name": "service_config_suppression_yarn_application_classpath", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Password for the TLS/SSL client truststore. Overrides the cluster-wide default truststore password set in HDFS.", "display_name": "TLS/SSL Client Truststore File Password", "name": "ssl_client_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into yarn-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_service_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "The user that this service's processes should run as. (Except the Job History Server, which has its own user)", "display_name": "System User", "name": "process_username", "value": "yarn" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Controls which applications an admin user can see in the applications list view", "display_name": "Admin Users Applications List Visibility Settings", "name": "admin_application_list_settings", "value": "ALL" }, { "desc": "Whether YARN uses the Linux Container Executor both in secure (Kerberos) and insecure (not Kerberos) environments. Cgroups enforcement only works when the Linux Container Executor is used.", "display_name": "Always Use Linux Container Executor", "name": "yarn_service_lce_always", "value": "false" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the Llama (AM for Impala) user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Llama Proxy User Groups", "name": "llama_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the NodeManager Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NodeManager Count Validator", "name": "service_config_suppression_nodemanager_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into ssl-client.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "yarn_ssl_client_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the NodeManager Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Health", "name": "service_health_suppression_yarn_node_managers_healthy", "value": "false" }, { "desc": "Controls the aggregate metrics generated for YARN applications. The structure is a JSON list of the attributes to aggregate and the entities to aggregate to. For example, if the attributeName is 'maps_completed' and the aggregationTargets is ['USER'] then the Service Monitor will create the metric 'yarn_application_maps_completed_rate' and, every ten minutes, will record the total maps completed for each user across all their YARN applications. By default it will also record the number of applications submitted ('apps_submitted_rate') for both users and pool. For a full list of the supported attributes see the YARN search page. Note that the valid aggregation targets are USER, YARN_POOL, and YARN (the service), and that these aggregate metrics can be viewed on both the reports and charts search pages.", "display_name": "YARN Application Aggregates", "name": "yarn_application_aggregates", "value": "[\n {\n \"attributeName\": \"maps_total\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"reduces_total\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"mb_millis_maps\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"mb_millis_reduces\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"vcores_millis_maps\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"vcores_millis_reduces\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"file_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"file_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cm_cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"application_duration\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"YARN\", \"CLUSTER\"]\n }\n]\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Client Truststore File Location parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Client Truststore File Location", "name": "service_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory.", "display_name": "System User's Home Directory", "name": "hdfs_user_home_dir", "value": "/var/lib/hadoop-yarn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User's Home Directory parameter.", "display_name": "Suppress Parameter Validation: System User's Home Directory", "name": "service_config_suppression_hdfs_user_home_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_yarn_service_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into mapred-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)", "name": "yarn_service_mapred_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "service_config_suppression_yarn_service_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to enable log aggregation", "display_name": "Enable Log Aggregation", "name": "yarn_log_aggregation_enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Llama Proxy User Groups", "name": "service_config_suppression_llama_proxy_user_groups_list", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml parameter.", "display_name": "Suppress Parameter Validation: YARN (MR2 Included) Service Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "service_config_suppression_yarn_ssl_client_safety_valve", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hive-hiveserver2.json0000666000175100017510000012010013245514472025562 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Downloaded Resources Directory.", "display_name": "Hive Downloaded Resources Directory Free Space Monitoring Absolute Thresholds", "name": "hiveserver2_downloaded_resources_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Local directory where Hive stores jars downloaded for remote file systems (HDFS). If not specified, Hive uses a default location.", "display_name": "Hive Downloaded Resources Directory", "name": "hiveserver2_downloaded_resources_dir", "value": null }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "hiveserver2_pause_duration_window", "value": "5" }, { "desc": "Minimum number of worker threads in HiveServer2's thread pool", "display_name": "Min HiveServer2 Threads", "name": "hiveserver2_min_threads", "value": "5" }, { "desc": "When enabled, Spark will add and remove executors dynamically to Hive jobs. This is done based on the workload.", "display_name": "Enable Dynamic Executor Allocation", "name": "hiveserver2_spark_dynamic_allocation_enabled", "value": "true" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The check interval for session/operation timeout, in milliseconds, which can be disabled by setting to zero or a negative value. ", "display_name": "Session Check Interval", "name": "hiveserver2_session_check_interval", "value": "3600000" }, { "desc": "The maximum percentage of heap to be used for hash in ReduceSink operator for Top-K selection. 0 means the optimization is disabled. Accepted values are between 0 and 1.", "display_name": "Maximum ReduceSink Top-K Memory Usage", "name": "hiveserver2_limit_pushdown_memory_usage", "value": "0.1" }, { "desc": "Whether to try sorted merge bucket (SMB) join.", "display_name": "Hive Optimize Sorted Merge Bucket Join", "name": "hiveserver2_optimize_bucketmapjoin_sortedmerge", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Number of Spark executors assigned to each application. This should not be set when Dynamic Executor Allocation is enabled.", "display_name": "Spark Executors Per Application", "name": "hiveserver2_spark_executor_instances", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_hs2_config_safety_valve", "value": "false" }, { "desc": "Some select queries can be converted to a single FETCH task instead of a MapReduce task, minimizing latency. A value of none disables all conversion, minimal converts simple queries such as SELECT * and filter on partition columns, and more converts SELECT queries including FILTERS.", "display_name": "Fetch Task Query Conversion", "name": "hiveserver2_fetch_task_conversion", "value": "minimal" }, { "desc": "Portion of total memory used in map-side partial aggregation. When exceeded, the partially aggregated results will be flushed from the map task to the reducers.", "display_name": "Ratio of Memory Usage for Map-Side Aggregation", "name": "hiveserver2_map_aggr_hash_memory_ratio", "value": "0.5" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hiveserver2_host_health", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Upper bound on the number of executors used by the application at any given time. This is used by dynamic executor allocation.", "display_name": "Upper Bound on Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_max_executors", "value": "2147483647" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HiveServer2 in Bytes", "name": "hiveserver2_java_heapsize", "value": "4294967296" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "When the number of ReduceSink operators after merging is less than this number, the ReduceDeDuplication optimization will be disabled.", "display_name": "Mininum Reducers for ReduceDeDuplication Optimization", "name": "hiveserver2_optimize_reducededuplication_min_reducer", "value": "4" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hiveserver2_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The desired file size after merging. This should be larger than hive.merge.smallfiles.avgsize.", "display_name": "Desired File Size After Merging", "name": "hiveserver2_merge_size_per_task", "value": "268435456" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hiveserver2_log_directory_free_space", "value": "false" }, { "desc": "HiveServer2 will impersonate the beeline client user when talking to other services such as MapReduce and HDFS.", "display_name": "HiveServer2 Enable Impersonation", "name": "hiveserver2_enable_impersonation", "value": "true" }, { "desc": "Enable optimization that checks if a query can be answered using statistics. If so, answers the query using only statistics stored in metastore.", "display_name": "Enable Stats Optimization", "name": "hiveserver2_compute_query_using_stats", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hive_metrics_sample_file_location", "value": "false" }, { "desc": "Enable optimization that converts common join into MapJoin based on input file size.", "display_name": "Enable MapJoin Optimization", "name": "hiveserver2_enable_mapjoin", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hiveserver2_swap_memory_usage", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Local Scratch Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Hive Local Scratch Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Hive Local Scratch Directory Free Space Monitoring Percentage Thresholds", "name": "hiveserver2_exec_local_scratch_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "In vectorized group-by, the number of row entries added to the hash table before re-checking average variable size for memory usage estimation.", "display_name": "Vectorized GroupBy Check Interval", "name": "hiveserver2_vectorized_groupby_checkinterval", "value": "4096" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Local Scratch Directory.", "display_name": "Hive Local Scratch Directory Free Space Monitoring Absolute Thresholds", "name": "hiveserver2_exec_local_scratch_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_hs2_config_safety_valve", "value": null }, { "desc": "Ratio between 0.0 and 1.0 of entries in the vectorized group-by aggregation hash that is flushed when the memory threshold is exceeded.", "display_name": "Vectorized GroupBy Flush Ratio", "name": "hiveserver2_vectorized_groupby_flush_ratio", "value": "0.1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HiveServer2 parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HiveServer2", "name": "role_config_suppression_hiveserver2_java_opts", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When computing the overall HiveServer2 health, consider the host's health.", "display_name": "HiveServer2 Host Health Test", "name": "hiveserver2_host_health_enabled", "value": "true" }, { "desc": "Maximum size of each Spark driver's Java heap memory when Hive is running on Spark.", "display_name": "Spark Driver Maximum Java Heap Size", "name": "hiveserver2_spark_driver_memory", "value": "268435456" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hiveserver2_scm_health", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Local Directory where Hive stores jars and data when performing a MapJoin optimization. If not specified, Hive uses a default location.", "display_name": "Hive Local Scratch Directory", "name": "hiveserver2_exec_local_scratchdir", "value": null }, { "desc": "Initial number of executors used by the application at any given time. This is required if the dynamic executor allocation feature is enabled.", "display_name": "Initial Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_initial_executors", "value": "1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Merge small files at the end of a map-only job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Map-Only Job", "name": "hiveserver2_merge_mapfiles", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Lower bound on the number of executors used by the application at any given time. This is used by dynamic executor allocation.", "display_name": "Lower Bound on Number of Executors", "name": "hiveserver2_spark_dynamic_allocation_min_executors", "value": "1" }, { "desc": "The number of rows with the same key value to be cached in memory per SMB-joined table.", "display_name": "Hive SMB Join Cache Rows", "name": "hiveserver2_smbjoin_cache_rows", "value": "10000" }, { "desc": "Maximum number of worker threads in HiveServer2's thread pool", "display_name": "Max HiveServer2 Threads", "name": "hiveserver2_max_threads", "value": "100" }, { "desc": "The minimum log level for HiveServer2 logs", "display_name": "HiveServer2 Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "This is the amount of extra off-heap memory that can be requested from YARN, per executor process. This, together with spark.executor.memory, is the total memory that YARN can use to create JVM for an executor process.", "display_name": "Spark Executor Memory Overhead", "name": "hiveserver2_spark_yarn_executor_memory_overhead", "value": "26" }, { "desc": "Enable optimization that vectorizes query execution by streamlining operations by processing a block of 1024 rows at a time.", "display_name": "Enable Vectorization Optimization", "name": "hiveserver2_vectorized_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_hs2_env_safety_valve", "value": "false" }, { "desc": "Number of cores per Spark executor.", "display_name": "Spark Executor Cores", "name": "hiveserver2_spark_executor_cores", "value": "1" }, { "desc": "Address of the load balancer used for HiveServer2 roles, specified in host:port format. If port is not specified, the port used by HiveServer2 is used. Note: Changing this property regenerates Kerberos keytabs for all HiveServer2 roles.", "display_name": "HiveServer2 Load Balancer", "name": "hiverserver2_load_balancer", "value": null }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hive/metrics-hivemetastore/metrics.log. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Location", "name": "hive_metrics_sample_file_location", "value": null }, { "desc": "Port on which HiveServer2 will listen for connections.", "display_name": "HiveServer2 Port", "name": "hs2_thrift_address_port", "value": "10000" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HiveServer2", "name": "hiveserver2_java_opts", "value": "-XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HiveServer2 Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_hs2_env_safety_valve", "value": null }, { "desc": "Maximum size of each Spark executor's Java heap memory when Hive is running on Spark.", "display_name": "Spark Executor Maximum Java Heap Size", "name": "hiveserver2_spark_executor_memory", "value": "268435456" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hiveserver2_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Remove extra map-reduce jobs if the data is already clustered by the same key, eliminating the need to repartition the dataset again.", "display_name": "Enable ReduceDeDuplication Optimization", "name": "hiveserver2_optimize_reducededuplication", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Hive Local Scratch Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Local Scratch Directory Free Space", "name": "role_health_suppression_hiveserver2_exec_local_scratch_directory_free_space", "value": "false" }, { "desc": "Whether to vectorize the reduce side of query execution.", "display_name": "Enable Reduce-Side Vectorization", "name": "hiveserver2_vectorized_reduce_enabled", "value": "false" }, { "desc": "Merge small files at the end of a map-reduce job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Map-Reduce Job", "name": "hiveserver2_merge_mapredfiles", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Logging Frequency", "name": "hive_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "Enables the health test that the HiveServer2's process state is consistent with the role configuration", "display_name": "HiveServer2 Process Health Test", "name": "hiveserver2_scm_health_enabled", "value": "true" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Directory where HiveServer2 will place its log files.", "display_name": "HiveServer2 Log Directory", "name": "hive_log_dir", "value": "/var/log/hive" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Operation will be closed when not accessed for this duration of time, in milliseconds; disable by setting to zero. For a positive value, checked for operations in terminal state only (FINISHED, CANCELED, CLOSED, ERROR). For a negative value, checked for all of the operations regardless of state.", "display_name": "Idle Operation Timeout", "name": "hiveserver2_idle_operation_timeout", "value": "259200000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Downloaded Resources Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Downloaded Resources Directory", "name": "role_config_suppression_hiveserver2_downloaded_resources_dir", "value": "false" }, { "desc": "When dynamic partition is enabled, reducers keep only one record writer at all times, which lowers the memory pressure on reducers.", "display_name": "Enable Sorted Dynamic Partition Optimizer", "name": "hiveserver2_optimize_sort_dynamic_partition", "value": "false" }, { "desc": "Controls whether the Hive metrics subsystem is enabled for the role.", "display_name": "Enable Metrics Subsystem", "name": "hive_metrics_enabled", "value": "true" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "hiveserver2_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hiveserver2_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HiveServer2 logs. Typically used by log4j or logback.", "display_name": "HiveServer2 Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "If Hive auto convert join is on, and the sum of the size for n-1 of the tables/partitions for a n-way join is smaller than the specified size, the join is directly converted to a MapJoin (there is no conditional task).", "display_name": "Hive Auto Convert Join Noconditional Size", "name": "hiveserver2_auto_convert_join_noconditionaltask_size", "value": "20971520" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_hiveserver2_pause_duration", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Log Directory parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Log Directory", "name": "role_config_suppression_hive_log_dir", "value": "false" }, { "desc": "When enabled, HiveServer2 logs EXPLAIN EXTENDED output for every query at INFO log4j level.", "display_name": "Enable Explain Logging", "name": "hiveserver2_enable_explain_output", "value": "true" }, { "desc": "Whether to suppress the results of the Hive Downloaded Resources Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Downloaded Resources Directory Free Space", "name": "role_health_suppression_hiveserver2_downloaded_resources_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Directory in HDFS where Hive writes intermediate data between MapReduce jobs. If not specified, Hive uses a default location.", "display_name": "Hive HDFS Scratch Directory", "name": "hiveserver2_exec_scratchdir", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 Load Balancer parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 Load Balancer", "name": "role_config_suppression_hiverserver2_load_balancer", "value": "false" }, { "desc": "Session will be closed when not accessed for this duration of time, in milliseconds; disable by setting to zero or a negative value.", "display_name": "Idle Session Timeout", "name": "hiveserver2_idle_session_timeout", "value": "604800000" }, { "desc": "Merge small files at the end of a Spark job. When enabled, a map-only job is created to merge the files in the destination table/partitions.", "display_name": "Enable Merging Small Files - Spark Job", "name": "hiveserver2_merge_sparkfiles", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for HiveServer2 logs. Typically used by log4j or logback.", "display_name": "HiveServer2 Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hiveserver2_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Enable map-side partial aggregation, which cause the mapper to generate fewer rows. This reduces the data to be sorted and distributed to reducers.", "display_name": "Enable Map-Side Aggregation", "name": "hiveserver2_map_aggr", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Local Scratch Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Local Scratch Directory", "name": "role_config_suppression_hiveserver2_exec_local_scratchdir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive HDFS Scratch Directory parameter.", "display_name": "Suppress Parameter Validation: Hive HDFS Scratch Directory", "name": "role_config_suppression_hiveserver2_exec_scratchdir", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Hive Downloaded Resources Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Hive Downloaded Resources Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Hive Downloaded Resources Directory Free Space Monitoring Percentage Thresholds", "name": "hiveserver2_downloaded_resources_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When the average output file size of a job is less than the value of this property, Hive will start an additional map-only job to merge the output files into bigger files. This is only done for map-only jobs if hive.merge.mapfiles is true, for map-reduce jobs if hive.merge.mapredfiles is true, and for Spark jobs if hive.merge.sparkfiles is true.", "display_name": "Small File Average Size Merge Threshold", "name": "hiveserver2_merge_smallfiles_avgsize", "value": "16777216" }, { "desc": "Above this size, queries are converted to fetch tasks.", "display_name": "Fetch Task Query Conversion Threshold", "name": "hiveserver2_fetch_task_conversion_threshold", "value": "268435456" }, { "desc": "This is the amount of extra off-heap memory that can be requested from YARN, per driver. This, together with spark.driver.memory, is the total memory that YARN can use to create JVM for a driver process.", "display_name": "Spark Driver Memory Overhead", "name": "hiveserver2_spark_yarn_driver_memory_overhead", "value": "26" }, { "desc": "Enabled the Calcite-based Cost-Based Optimizer for HiveServer2.", "display_name": "Enable Cost-Based Optimizer for Hive", "name": "hiveserver2_enable_cbo", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hue-service.json0000666000175100017510000011041713245514472024616 0ustar zuulzuul00000000000000[ { "desc": "Create users in Hue when they try to login with their LDAP credentials. For use when using LdapBackend for Hue authentication.", "display_name": "Create LDAP users on login", "name": "create_users_on_login", "value": "true" }, { "desc": "Number of threads used by the Hue web server.", "display_name": "Hue Web Server Threads", "name": "cherrypy_server_threads", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Comma-separated list of regular expressions, which match 'host:port' of requested proxy target.", "display_name": "Whitelist", "name": "whitelist", "value": "(localhost|127\\.0\\.0\\.1):(50030|50070|50060|50075)" }, { "desc": "The health test thresholds of the overall Load Balancer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Load Balancers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Load Balancers falls below the critical threshold.", "display_name": "Healthy Load Balancer Monitoring Thresholds", "name": "hue_load_balancer_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Thrift Usage Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Thrift Usage Validator", "name": "service_config_suppression_hue_hbase_thrift_server_validator", "value": "false" }, { "desc": "The health test thresholds of the overall Kerberos Ticket Renewer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Kerberos Ticket Renewers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Kerberos Ticket Renewers falls below the critical threshold.", "display_name": "Healthy Kerberos Ticket Renewer Monitoring Thresholds", "name": "hue_kt_renewers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hue" }, { "desc": "The distinguished name to use as a search base for finding users and groups. This should be similar to 'dc=hadoop,dc=mycompany,dc=com'.", "display_name": "LDAP Search Base", "name": "base_dn", "value": null }, { "desc": "Whether to suppress the results of the Kerberos Ticket Renewer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Kerberos Ticket Renewer Health", "name": "service_health_suppression_hue_kt_renewers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Password parameter.", "display_name": "Suppress Parameter Validation: Hue Database Password", "name": "service_config_suppression_database_password", "value": "false" }, { "desc": "Whether to suppress the results of the Hue Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hue Server Health", "name": "service_health_suppression_hue_hue_servers_healthy", "value": "false" }, { "desc": "The name of a default group that users will be added to at creation time.", "display_name": "Default User Group", "name": "default_user_group", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Kerberos Ticket Renewer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kerberos Ticket Renewer Count Validator", "name": "service_config_suppression_kt_renewer_count_validator", "value": "false" }, { "desc": "Distinguished name of the user to bind as. Not required if the LDAP server supports anonymous searches. For Active Directory, this would be similar to 'hadoop-admin@mycompany.com'.", "display_name": "LDAP Bind User", "name": "bind_dn", "value": null }, { "desc": "Whether to use StartTLS (as opposed to ldaps) to communicate securely with the LDAP server. This is only effective when the LDAP certificate is specified.", "display_name": "Use StartTLS", "name": "use_start_tls", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Username Pattern parameter.", "display_name": "Suppress Parameter Validation: LDAP Username Pattern", "name": "service_config_suppression_ldap_username_pattern", "value": "false" }, { "desc": "The password of the bind user.", "display_name": "LDAP Bind Password", "name": "bind_password", "value": null }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hue/audit" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default User Group parameter.", "display_name": "Suppress Parameter Validation: Default User Group", "name": "service_config_suppression_default_user_group", "value": "false" }, { "desc": "Type of database used for Hue", "display_name": "Hue Database Type", "name": "database_type", "value": "sqlite3" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the PAM Backend Service Name parameter.", "display_name": "Suppress Parameter Validation: PAM Backend Service Name", "name": "service_config_suppression_pam_auth_service", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Web Interface Role parameter.", "display_name": "Suppress Parameter Validation: HDFS Web Interface Role", "name": "service_config_suppression_hue_webhdfs", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Bind User parameter.", "display_name": "Suppress Parameter Validation: LDAP Bind User", "name": "service_config_suppression_bind_dn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Username parameter.", "display_name": "Suppress Parameter Validation: Hue Database Username", "name": "service_config_suppression_database_user", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Name of host where the Hue database is running. Not necessary for SQLite3.", "display_name": "Hue Database Hostname", "name": "database_host", "value": "localhost" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Temporary Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Temporary Directory", "name": "service_config_suppression_hdfs_tmp_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URL parameter.", "display_name": "Suppress Parameter Validation: LDAP URL", "name": "service_config_suppression_ldap_url", "value": "false" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Name Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Name Attribute", "name": "service_config_suppression_group_name_attr", "value": "false" }, { "desc": "LDAP certificate for authentication over TLS", "display_name": "LDAP Certificate", "name": "ldap_cert", "value": null }, { "desc": "HTTPFS role or Namenode (if webhdfs is enabled) that hue can use to communicate with HDFS.", "display_name": "HDFS Web Interface Role", "name": "hue_webhdfs", "value": null }, { "desc": "Whether to suppress the results of the Load Balancer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Load Balancer Health", "name": "service_health_suppression_hue_load_balancer_healthy", "value": "false" }, { "desc": "Name of the Sqoop service that this Hue service instance depends on", "display_name": "Sqoop Service", "name": "sqoop_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Name of Hue database.", "display_name": "Hue Database Name", "name": "database_name", "value": "hue" }, { "desc": "The attribute of the group object that identifies the members of the group. For Active Directory, this is typically 'member'.", "display_name": "LDAP Group Membership Attribute", "name": "group_member_attr", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Blacklist parameter.", "display_name": "Suppress Parameter Validation: Blacklist", "name": "service_config_suppression_blacklist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Whitelist parameter.", "display_name": "Suppress Parameter Validation: Whitelist", "name": "service_config_suppression_whitelist", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "The base filter for searching for users. For Active Directory, this is typically '(objectClass=user)'.", "display_name": "LDAP User Filter", "name": "user_filter", "value": null }, { "desc": "Name of the Impala service that this Hue service instance depends on", "display_name": "Impala Service", "name": "impala_service", "value": null }, { "desc": "Name of the Oozie service that this Hue service instance depends on", "display_name": "Oozie Service", "name": "oozie_service", "value": null }, { "desc": "Name of the HBase service that this Hue service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Membership Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Membership Attribute", "name": "service_config_suppression_group_member_attr", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Hostname parameter.", "display_name": "Suppress Parameter Validation: Hue Database Hostname", "name": "service_config_suppression_database_host", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini parameter.", "display_name": "Suppress Parameter Validation: Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", "name": "service_config_suppression_hue_service_safety_valve", "value": "false" }, { "desc": "Name of the Solr service that this Hue service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Certificate parameter.", "display_name": "Suppress Parameter Validation: LDAP Certificate", "name": "service_config_suppression_ldap_cert", "value": "false" }, { "desc": "The username to use to log into the Hue database. Not necessary for SQLite3.", "display_name": "Hue Database Username", "name": "database_user", "value": "hue" }, { "desc": "In debug mode, Django displays a detailed traceback when an exception occurs. Debugging information may contain sensitive data. Django remembers every SQL query it executes in debug mode, which will rapidly consume memory.", "display_name": "Enable Django Debug Mode", "name": "django_debug_enable", "value": "false" }, { "desc": "Comma-separated list of regular expressions, which match any prefix of 'host:port/path' of requested proxy target. This does not support matching GET parameters.", "display_name": "Blacklist", "name": "blacklist", "value": "()" }, { "desc": "Search Bind Authentication connects to the LDAP server using credentials provided in the 'bind_dn' and 'bind_password' configurations. If these configurations are not set, then an anonymous search is performed.", "display_name": "Use Search Bind Authentication", "name": "search_bind_authentication", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Username Attribute parameter.", "display_name": "Suppress Parameter Validation: LDAP Username Attribute", "name": "service_config_suppression_user_name_attr", "value": "false" }, { "desc": "Mode of authenticating login credentials. Select desktop.auth.backend.LdapBackend to use LDAP to authenticate login credentials. LDAP requires you to also set the LDAP URL, Active Directory Domain, and optionally LDAP certificate if you are using secure LDAP. Select desktop.auth.backend.PamBackend to use PAM to authenticate login credentials.", "display_name": "Authentication Backend", "name": "auth_backend", "value": "desktop.auth.backend.AllowFirstUserDjangoBackend" }, { "desc": "Whether to suppress configuration warnings produced by the Load Balancer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Load Balancer Count Validator", "name": "service_config_suppression_hue_load_balancer_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Group Filter parameter.", "display_name": "Suppress Parameter Validation: LDAP Group Filter", "name": "service_config_suppression_group_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Active Directory Domain parameter.", "display_name": "Suppress Parameter Validation: Active Directory Domain", "name": "service_config_suppression_nt_domain", "value": "false" }, { "desc": "Enable debug output in HTTP Internal Server Error (status 500) responses. Debugging information may contain sensitive data. If Enable Django Debug Mode is set, this is automatically enabled.", "display_name": "Enable Debugging of Internal Server Error Responses", "name": "http_500_debug_enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Database Directory", "name": "service_config_suppression_database_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "hue_sentry_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hue_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the User Augmentor parameter.", "display_name": "Suppress Parameter Validation: User Augmentor", "name": "service_config_suppression_user_augmentor", "value": "false" }, { "desc": "The username attribute in the LDAP schema. For Active Directory, this is typically 'sAMAccountName'.", "display_name": "LDAP Username Attribute", "name": "user_name_attr", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Search Base parameter.", "display_name": "Suppress Parameter Validation: LDAP Search Base", "name": "service_config_suppression_base_dn", "value": "false" }, { "desc": "Password for Hue database. Not necessary for SQLite3.", "display_name": "Hue Database Password", "name": "database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS HTTPFS Usage Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS HTTPFS Usage Validator", "name": "service_config_suppression_hdfs_httpfs_present_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_hue_sentry_safety_valve", "value": "false" }, { "desc": "Thrift server to use for HBase app.", "display_name": "HBase Thrift Server", "name": "hue_hbase_thrift", "value": null }, { "desc": "Name of the Hive service that this Hue service instance depends on", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Name of the Sentry service that this Hue service instance depends on", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Base filter for searching for groups. For Active Directory, this is typically '(objectClass=group)'.", "display_name": "LDAP Group Filter", "name": "group_filter", "value": null }, { "desc": "Time zone name.", "display_name": "Time Zone", "name": "time_zone", "value": "America/Los_Angeles" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Hue events:\n

\n\n
    \n
  • operation: the Hue operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Database Dump File parameter.", "display_name": "Suppress Parameter Validation: Database Dump File", "name": "service_config_suppression_database_dump_file", "value": "false" }, { "desc": "Whether to suppress the results of the Beeswax Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Beeswax Server Health", "name": "service_health_suppression_hue_beeswax_server_health", "value": "false" }, { "desc": "The health test thresholds of the overall Hue Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Hue Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Hue Servers falls below the critical threshold.", "display_name": "Healthy Hue Server Monitoring Thresholds", "name": "hue_hue_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The group name attribute in the LDAP schema. For Active Directory, this is typically 'cn'.", "display_name": "LDAP Group Name Attribute", "name": "group_name_attr", "value": null }, { "desc": "HDFS directory used for storing temporary files.", "display_name": "HDFS Temporary Directory", "name": "hdfs_tmp_dir", "value": "/tmp" }, { "desc": "Default encoding for site data.", "display_name": "Default Site Encoding", "name": "default_site_encoding", "value": "utf" }, { "desc": "If the database is SQLite3, this is the filename of the database to use, and the directory of this file must be writable by the 'hue' user.", "display_name": "Hue Database Directory", "name": "database_dir", "value": "/var/lib/hue/desktop.db" }, { "desc": "When you enable anonymous usage data collection Hue tracks anonymised pages and application versions in order to gather information about each application's usage levels. The data collected does not include any hostnames or IDs. Data collection option is available on CDH 4.4 and later deployments.", "display_name": "Enable Usage Data Collection", "name": "usage_data_collection_enable", "value": "true" }, { "desc": "Name of the ZooKeeper service that this Hue service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Site Encoding parameter.", "display_name": "Suppress Parameter Validation: Default Site Encoding", "name": "service_config_suppression_default_site_encoding", "value": "false" }, { "desc": "The URL of the LDAP Server; similar to 'ldap://auth.mycompany.com' or 'ldaps://auth.mycompany.com'.", "display_name": "LDAP URL", "name": "ldap_url", "value": null }, { "desc": "The PAM service name to use when authenticating over desktop.auth.backend.PamBackend. This is typically the name of a file under /etc/pam.d/ on the Hue host.", "display_name": "PAM Backend Service Name", "name": "pam_auth_service", "value": "login" }, { "desc": "Whether to suppress configuration warnings produced by the Hue Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hue Server Count Validator", "name": "service_config_suppression_hue_server_count_validator", "value": "false" }, { "desc": "Only applies to Active Directory. The Active Directory Domain will be similar to 'MYCOMPANY.COM'.", "display_name": "Active Directory Domain", "name": "nt_domain", "value": null }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hue" }, { "desc": "Class that defines extra accessor methods for user objects.", "display_name": "User Augmentor", "name": "user_augmentor", "value": "desktop.auth.backend.DefaultUserAugmentor" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hue Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hue_service_env_safety_valve", "value": "false" }, { "desc": "File where the database gets dumped to or loaded from.", "display_name": "Database Dump File", "name": "database_dump_file", "value": "/tmp/hue_database_dump.json" }, { "desc": "LDAP Username Pattern for use with non-Active Directory LDAP implementations. Must contain the special '<username>' string for replacement during authentication.", "display_name": "LDAP Username Pattern", "name": "ldap_username_pattern", "value": null }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hue" }, { "desc": "When computing the overall HUE health, consider Beeswax Server's health", "display_name": "Beeswax Server Role Health Test", "name": "hue_beeswax_server_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP User Filter parameter.", "display_name": "Suppress Parameter Validation: LDAP User Filter", "name": "service_config_suppression_user_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Bind Password parameter.", "display_name": "Suppress Parameter Validation: LDAP Bind Password", "name": "service_config_suppression_bind_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Time Zone parameter.", "display_name": "Suppress Parameter Validation: Time Zone", "name": "service_config_suppression_time_zone", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HUE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Database Name parameter.", "display_name": "Suppress Parameter Validation: Hue Database Name", "name": "service_config_suppression_database_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hue_safety_valve.ini. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", "name": "hue_service_safety_valve", "value": null }, { "desc": "Port on host where the Hue database is running. Not necessary for SQLite3.", "display_name": "Hue Database Port", "name": "database_port", "value": "3306" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/spark-spark_yarn_history_server.json0000666000175100017510000003540213245514472031035 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "History Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SPARK_YARN_HISTORY_SERVER_role_env_safety_valve", "value": null }, { "desc": "The port of the history server WebUI", "display_name": "History Server WebUI Port", "name": "history_server_web_port", "value": "18088" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "How often to poll HDFS for new applications.", "display_name": "HDFS Polling Interval", "name": "history_server_fs_poll_interval", "value": "10" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: History Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_file_descriptor", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Max number of application UIs to keep in the History Server's memory. All applications will still be available, but may take longer to load if they're not in memory.", "display_name": "Retained App Count", "name": "history_server_retained_apps", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The log directory for log files of the role History Server.", "display_name": "History Server Log Directory", "name": "log_dir", "value": "/var/log/spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf parameter.", "display_name": "Suppress Parameter Validation: History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf", "name": "role_config_suppression_spark-history-server.conf_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "spark_yarn_history_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "History Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes.", "display_name": "Java Heap Size of History Server in Bytes", "name": "history_server_max_heapsize", "value": "268435456" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_host_health", "value": "false" }, { "desc": "When computing the overall History Server health, consider the host's health.", "display_name": "History Server Host Health Test", "name": "spark_yarn_history_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_unexpected_exits", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for History Server logs. Typically used by log4j or logback.", "display_name": "History Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: History Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into spark-history-server.conf for this role only.", "display_name": "History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf", "name": "spark-history-server.conf_role_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "role_config_suppression_spark-conf/spark-env.sh_role_safety_valve", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for History Server logs. Typically used by log4j or logback.", "display_name": "History Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Enables the health test that the History Server's process state is consistent with the role configuration", "display_name": "History Server Process Health Test", "name": "spark_yarn_history_server_scm_health_enabled", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the History Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: History Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_spark_yarn_history_server_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into spark-conf/spark-env.sh for this role only.", "display_name": "History Server Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_role_safety_valve", "value": null }, { "desc": "The minimum log level for History Server logs", "display_name": "History Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_spark_on_yarn_spark_yarn_history_server_scm_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/yarn-nodemanager.json0000666000175100017510000013127513245514472025633 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Containers Environment Variable parameter.", "display_name": "Suppress Parameter Validation: Containers Environment Variable", "name": "role_config_suppression_yarn_nodemanager_admin_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NodeManager parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NodeManager", "name": "role_config_suppression_node_manager_java_opts", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed System Users parameter.", "display_name": "Suppress Parameter Validation: Allowed System Users", "name": "role_config_suppression_container_executor_allowed_system_users", "value": "false" }, { "desc": "When computing the overall NodeManager health, consider the host's health.", "display_name": "NodeManager Host Health Test", "name": "nodemanager_host_health_enabled", "value": "true" }, { "desc": "Enables the health test that the NodeManager's process state is consistent with the role configuration", "display_name": "NodeManager Process Health Test", "name": "nodemanager_scm_health_enabled", "value": "true" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_node_manager_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Local Directories parameter.", "display_name": "Suppress Parameter Validation: NodeManager Local Directories", "name": "role_config_suppression_yarn_nodemanager_local_dirs", "value": "false" }, { "desc": "Whether to suppress the results of the ResourceManager Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ResourceManager Connectivity", "name": "role_health_suppression_node_manager_connectivity", "value": "false" }, { "desc": "Number of threads to use for localization fetching.", "display_name": "Localizer Fetch Thread Count", "name": "yarn_nodemanager_localizer_fetch_thread_count", "value": "4" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "nodemanager_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Recovery Directory.", "display_name": "NodeManager Recovery Directory Free Space Monitoring Absolute Thresholds", "name": "nodemanager_recovery_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NodeManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Time in seconds to retain user logs. Only applicable if log aggregation is disabled.", "display_name": "Log Retain Duration", "name": "yarn_nodemanager_log_retain_seconds", "value": "10800" }, { "desc": "Number of threads container manager uses.", "display_name": "Container Manager Thread Count", "name": "yarn_nodemanager_container_manager_thread_count", "value": "20" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Address where the localizer IPC is.", "display_name": "Localizer Cache Cleanup Interval", "name": "yarn_nodemanager_localizer_cache_cleanup_interval_ms", "value": "600000" }, { "desc": "List of users banned from running containers.", "display_name": "Banned System Users", "name": "container_executor_banned_users", "value": "hdfs,yarn,mapred,bin" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Container Log Directories parameter.", "display_name": "Suppress Parameter Validation: NodeManager Container Log Directories", "name": "role_config_suppression_yarn_nodemanager_log_dirs", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Banned System Users parameter.", "display_name": "Suppress Parameter Validation: Banned System Users", "name": "role_config_suppression_container_executor_banned_users", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Local Directories.", "display_name": "NodeManager Local Directories Free Space Monitoring Absolute Thresholds", "name": "nodemanager_local_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "nodemanager_mapred_safety_valve", "value": null }, { "desc": "Amount of physical memory, in MiB, that can be allocated for containers.", "display_name": "Container Memory", "name": "yarn_nodemanager_resource_memory_mb", "value": "8192" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Comma-separated list of arguments which are to be passed to node health script when it is being launched.", "display_name": "Healthchecker Script Arguments", "name": "mapred_healthchecker_script_args", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_node_manager_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_node_manager_unexpected_exits", "value": "false" }, { "desc": "If enabled, adds 'org.apache.hadoop.mapred.ShuffleHandler' to the NodeManager auxiliary services. This is required for MapReduce applications.", "display_name": "Enable Shuffle Auxiliary Service", "name": "mapreduce_aux_service", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Container Log Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Container Log Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Container Log Directories Free Space Monitoring Percentage Thresholds", "name": "nodemanager_log_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Log Directory parameter.", "display_name": "Suppress Parameter Validation: NodeManager Log Directory", "name": "role_config_suppression_node_manager_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The minimum Linux user ID allowed. Used to prevent other super users.", "display_name": "Minimum User ID", "name": "container_executor_min_user_id", "value": "1000" }, { "desc": "The maximum size, in megabytes, per log file for NodeManager logs. Typically used by log4j or logback.", "display_name": "NodeManager Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "List of users explicitly whitelisted to be allowed to run containers. Users with IDs lower than the \"Minimum User Id\" setting may be whitelisted by using this setting.", "display_name": "Allowed System Users", "name": "container_executor_allowed_system_users", "value": "nobody,impala,hive,llama" }, { "desc": "Directory where NodeManager will place its log files.", "display_name": "NodeManager Log Directory", "name": "node_manager_log_dir", "value": "/var/log/hadoop-yarn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Recovery Directory parameter.", "display_name": "Suppress Parameter Validation: NodeManager Recovery Directory", "name": "role_config_suppression_yarn_nodemanager_recovery_dir", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "nodemanager_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "Environment variables that containers may override rather than use NodeManager's default.", "display_name": "Containers Environment Variables Whitelist ", "name": "yarn_nodemanager_env_whitelist", "value": "JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME" }, { "desc": "The address of the NodeManager IPC.", "display_name": "NodeManager IPC Address", "name": "yarn_nodemanager_address", "value": "8041" }, { "desc": "Environment variables that should be forwarded from the NodeManager's environment to the container's.", "display_name": "Containers Environment Variable", "name": "yarn_nodemanager_admin_env", "value": "MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the CGroups Hierarchy parameter.", "display_name": "Suppress Parameter Validation: CGroups Hierarchy", "name": "role_config_suppression_linux_container_executor_cgroups_hierarchy", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_node_manager_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Remote App Log Directory Suffix parameter.", "display_name": "Suppress Parameter Validation: Remote App Log Directory Suffix", "name": "role_config_suppression_yarn_nodemanager_remote_app_log_dir_suffix", "value": "false" }, { "desc": "Number of threads used in cleanup.", "display_name": "Cleanup Thread Count", "name": "yarn_nodemanager_delete_thread_count", "value": "4" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_node_manager_swap_memory_usage", "value": "false" }, { "desc": "Target size of localizer cache in MB, per local directory.", "display_name": "Localizer Cache Target Size", "name": "yarn_nodemanager_localizer_cache_target_size_mb", "value": "10240" }, { "desc": "Absolute path to the script which is periodically run by the node health monitoring service to determine if the node is healthy or not. If the value of this key is empty or the file does not exist in the location configured here, the node health monitoring service is not started.", "display_name": "Healthchecker Script Path", "name": "mapred_healthchecker_script_path", "value": "" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Recovery Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Recovery Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Recovery Directory Free Space Monitoring Percentage Thresholds", "name": "nodemanager_recovery_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_nodemanager_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "nodemanager_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Healthchecker Script Path parameter.", "display_name": "Suppress Parameter Validation: Healthchecker Script Path", "name": "role_config_suppression_mapred_healthchecker_script_path", "value": "false" }, { "desc": "The HTTPS port of the NodeManager web application.", "display_name": "NodeManager Web Application HTTPS Port (TLS/SSL)", "name": "nodemanager_webserver_https_port", "value": "8044" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "nodemanager_web_metric_collection_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "nodemanager_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_node_manager_heap_dump_directory_free_space", "value": "false" }, { "desc": "The local filesystem directory in which the NodeManager stores state when recovery is enabled. Recovery is enabled by default.", "display_name": "NodeManager Recovery Directory", "name": "yarn_nodemanager_recovery_dir", "value": "/var/lib/hadoop-yarn/yarn-nm-recovery" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NodeManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the NodeManager Local Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Local Directories Free Space", "name": "role_health_suppression_nodemanager_local_data_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "HDFS directory where application logs are stored when an application completes.", "display_name": "Remote App Log Directory", "name": "yarn_nodemanager_remote_app_log_dir", "value": "/tmp/logs" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_node_manager_gc_duration", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "nodemanager_gc_duration_window", "value": "5" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NodeManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_nodemanager_role_env_safety_valve", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Path (rooted in the cgroups hierarchy on the machine) where to place YARN-managed cgroups.", "display_name": "CGroups Hierarchy", "name": "linux_container_executor_cgroups_hierarchy", "value": "/hadoop-yarn" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NodeManager", "name": "node_manager_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Number of threads to handle localization requests.", "display_name": "Localizer Client Thread Count", "name": "yarn_nodemanager_localizer_client_thread_count", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Local Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NodeManager Local Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NodeManager Local Directories Free Space Monitoring Percentage Thresholds", "name": "nodemanager_local_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Single User Mode Overrides Validator configuration validator.", "display_name": "Suppress Configuration Validator: Single User Mode Overrides Validator", "name": "role_config_suppression_single_user_mode_override_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_nodemanager_mapred_safety_valve", "value": "false" }, { "desc": "The HTTP Port of the NodeManager web application.", "display_name": "NodeManager Web Application HTTP Port", "name": "nodemanager_webserver_port", "value": "8042" }, { "desc": "Heartbeat interval to ResourceManager", "display_name": "Heartbeat Interval", "name": "yarn_nodemanager_heartbeat_interval_ms", "value": "1000" }, { "desc": "The amount of time to wait for the NodeManager to fully start up and connect to the ResourceManager before enforcing the connectivity check.", "display_name": "NodeManager Connectivity Tolerance at Startup", "name": "nodemanager_connectivity_tolerance_seconds", "value": "180" }, { "desc": "List of directories on the local filesystem where a NodeManager stores intermediate data files.", "display_name": "NodeManager Local Directories", "name": "yarn_nodemanager_local_dirs", "value": null }, { "desc": "Enables the health check that verifies the NodeManager is connected to the ResourceManager", "display_name": "NodeManager Connectivity Health Check", "name": "nodemanager_connectivity_health_enabled", "value": "true" }, { "desc": "Number of seconds after an application finishes before the NodeManager's DeletionService will delete the application's localized file and log directory. To diagnose YARN application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories.", "display_name": "Localized Dir Deletion Delay", "name": "yarn_nodemanager_delete_debug_delay_sec", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Healthchecker Script Arguments parameter.", "display_name": "Suppress Parameter Validation: Healthchecker Script Arguments", "name": "role_config_suppression_mapred_healthchecker_script_args", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for NodeManager logs. Typically used by log4j or logback.", "display_name": "NodeManager Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the NodeManager Container Log Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Container Log Directories Free Space", "name": "role_health_suppression_nodemanager_log_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Remote App Log Directory parameter.", "display_name": "Suppress Parameter Validation: Remote App Log Directory", "name": "role_config_suppression_yarn_nodemanager_remote_app_log_dir", "value": "false" }, { "desc": "List of directories on the local filesystem where a NodeManager stores container log files.", "display_name": "NodeManager Container Log Directories", "name": "yarn_nodemanager_log_dirs", "value": "/var/log/hadoop-yarn/container" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NodeManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "NODEMANAGER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the NodeManager Recovery Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NodeManager Recovery Directory Free Space", "name": "role_health_suppression_nodemanager_recovery_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_node_manager_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Address where the localizer IPC is.", "display_name": "Localizer Port", "name": "yarn_nodemanager_localizer_address", "value": "8040" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Container Executor Group parameter.", "display_name": "Suppress Parameter Validation: Container Executor Group", "name": "role_config_suppression_container_executor_group", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Containers Environment Variables Whitelist parameter.", "display_name": "Suppress Parameter Validation: Containers Environment Variables Whitelist ", "name": "role_config_suppression_yarn_nodemanager_env_whitelist", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The minimum log level for NodeManager logs", "display_name": "NodeManager Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}", "display_name": "Remote App Log Directory Suffix", "name": "yarn_nodemanager_remote_app_log_dir_suffix", "value": "logs" }, { "desc": "The system group that owns the container-executor binary. This does not need to be changed unless the ownership of the binary is explicitly changed.", "display_name": "Container Executor Group", "name": "container_executor_group", "value": "yarn" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_node_manager_host_health", "value": "false" }, { "desc": "Number of virtual CPU cores that can be allocated for containers. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores", "name": "yarn_nodemanager_resource_cpu_vcores", "value": "8" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NodeManager Container Log Directories.", "display_name": "NodeManager Container Log Directories Free Space Monitoring Absolute Thresholds", "name": "nodemanager_log_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum allowed threads for serving shuffle connections. Set to zero to indicate the default of 2 times the number of available processors.", "display_name": "Max Shuffle Threads", "name": "mapreduce_shuffle_max_threads", "value": "80" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NodeManager in Bytes", "name": "node_manager_java_heapsize", "value": "1073741824" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-balancer.json0000666000175100017510000001537713245514472025101 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Balancer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Balancer", "name": "role_config_suppression_balancer_java_opts", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Balancer", "name": "balancer_java_opts", "value": "" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "balancer_config_safety_valve", "value": null }, { "desc": "The policy that should be used to rebalance HDFS storage. The default DataNode policy balances the storage at the DataNode level. This is similar to the balancing policy from prior releases. The BlockPool policy balances the storage at the block pool level as well as at the DataNode level. The BlockPool policy is relevant only to a Federated HDFS service.", "display_name": "Rebalancing Policy", "name": "rebalancing_policy", "value": "DataNode" }, { "desc": "The percentage deviation from average utilization, after which a node will be rebalanced. (for example, '10.0' for 10%)", "display_name": "Rebalancing Threshold", "name": "rebalancer_threshold", "value": "10.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_balancer_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Balancer in Bytes", "name": "balancer_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/sentry-service.json0000666000175100017510000004424013245514472025361 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the Sentry Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Sentry Server Count Validator", "name": "service_config_suppression_sentry_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Password parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Password", "name": "service_config_suppression_sentry_server_database_password", "value": "false" }, { "desc": "Whether to suppress the results of the Sentry Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sentry Server Health", "name": "service_health_suppression_sentry_sentry_servers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Password for Sentry Server database.", "display_name": "Sentry Server Database Password", "name": "sentry_server_database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Sentry events:\n

\n\n
    \n
  • operation: the Sentry operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "List of users allowed to connect to the Sentry Server. These are usually service users such as hive and impala, and the list does not usually need to include end users.", "display_name": "Allowed Connecting Users", "name": "sentry_service_allow_connect", "value": "hive,impala,hue,hdfs" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "sentry" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Sentry events:\n

\n\n
    \n
  • operation: the Sentry operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "User for Sentry Server database.", "display_name": "Sentry Server Database User", "name": "sentry_server_database_user", "value": "sentry" }, { "desc": "Name of the HDFS service that this Sentry service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Type of Sentry Server database.", "display_name": "Sentry Server Database Type", "name": "sentry_server_database_type", "value": "mysql" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Host parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Host", "name": "service_config_suppression_sentry_server_database_host", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sentry Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "SENTRY_service_env_safety_valve", "value": null }, { "desc": "Name of the ZooKeeper service that this Sentry service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Connecting Users parameter.", "display_name": "Suppress Parameter Validation: Allowed Connecting Users", "name": "service_config_suppression_sentry_service_allow_connect", "value": "false" }, { "desc": "Host name of Sentry Server database.", "display_name": "Sentry Server Database Host", "name": "sentry_server_database_host", "value": "localhost" }, { "desc": "The health test thresholds of the overall Sentry Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Sentry Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Sentry Servers falls below the critical threshold.", "display_name": "Healthy Sentry Server Monitoring Thresholds", "name": "sentry_sentry_server_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/sentry/audit" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "sentry" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sentry Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_sentry_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database Name parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database Name", "name": "service_config_suppression_sentry_server_database_name", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "sentry" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "sentry_server_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_sentry_server_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Name of Sentry Server database.", "display_name": "Sentry Server Database Name", "name": "sentry_server_database_name", "value": "sentry" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Admin Groups parameter.", "display_name": "Suppress Parameter Validation: Admin Groups", "name": "service_config_suppression_sentry_service_admin_group", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Port number of Sentry Server database.", "display_name": "Sentry Server Database Port", "name": "sentry_server_database_port", "value": "3306" }, { "desc": "If an end user is in one of these admin groups, that user has administrative privileges on the Sentry Server.", "display_name": "Admin Groups", "name": "sentry_service_admin_group", "value": "hive,impala,hue" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Sentry Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Server Database User parameter.", "display_name": "Suppress Parameter Validation: Sentry Server Database User", "name": "service_config_suppression_sentry_server_database_user", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/impala-catalogserver.json0000666000175100017510000006121713245514472026504 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SSL/TLS Certificate for Catalog Server Webserver parameter.", "display_name": "Suppress Parameter Validation: SSL/TLS Certificate for Catalog Server Webserver", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_catalogserver_swap_memory_usage", "value": "false" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "catalogserver_startup_tolerance", "value": "5" }, { "desc": "Port where Catalog Server debug web server runs.", "display_name": "Catalog Server HTTP Server Port", "name": "catalogserver_webserver_port", "value": "25020" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "catalogserver_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "catalogserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Port where Catalog Server is exported.", "display_name": "Catalog Server Service Port", "name": "catalog_service_port", "value": "26000" }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "Catalog Server Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "If true, loads catalog metadata in the background. If false, metadata is loaded lazily (on access). Only effective in CDH 5 and Impala 1.2.4 and higher.", "display_name": "Load Catalog in Background", "name": "load_catalog_in_background", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_catalogserver_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_catalogserver_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server HBase Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server HBase Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hbase_conf_safety_valve", "value": "false" }, { "desc": "The amount of time to wait for the Catalog Server to fully start up and connect to the StateStore before enforcing the connectivity check.", "display_name": "Catalog Server Connectivity Tolerance at Startup", "name": "catalogserver_connectivity_tolerance", "value": "180" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Timeout for requests to the Hive Metastore Server from Catalog Server. Consider increasing this if you have tables with a lot of metadata and see timeout errors.", "display_name": "Catalog Server Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "3600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogserver_role_env_safety_valve", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Username for Catalog Server web server authentication.", "display_name": "Catalog Server Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Password for Catalog Server web server authentication.", "display_name": "Catalog Server Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Enables the health test that the Impala Catalog Server's process state is consistent with the role configuration", "display_name": "Impala Catalog Server Process Health Test", "name": "catalogserver_scm_health_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml for this role only.", "display_name": "Catalog Server HBase Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hbase_conf_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Catalog Server command line flags. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_cmd_args_safety_valve", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml for this role only.", "display_name": "Catalog Server HDFS Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hdfs_site_conf_safety_valve", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_catalogserver_web_metric_collection", "value": "false" }, { "desc": "Directory where Catalog Server will place its log files.", "display_name": "Catalog Server Log Directory", "name": "log_dir", "value": "/var/log/catalogd" }, { "desc": "Enables the health test that verifies the Catalog Server is connected to the StateStore", "display_name": "Catalog Server Connectivity Health Test", "name": "catalogserver_connectivity_health_enabled", "value": "true" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "catalogserver_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_catalogserver_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_cmd_args_safety_valve", "value": "false" }, { "desc": "Enable/Disable Catalog Server web server. This web server contains useful information about Catalog Server daemon.", "display_name": "Enable Catalog Server Web Server", "name": "catalogd_enable_webserver", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_catalogserver_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "CATALOGSERVER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_catalogserver_memory_rss_health", "value": "false" }, { "desc": "When computing the overall Impala Catalog Server health, consider the host's health.", "display_name": "Impala Catalog Server Host Health Test", "name": "catalogserver_host_health_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Hive Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Hive Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hive_conf_safety_valve", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Catalog Server Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_catalogserver_heap_dump_directory_free_space", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml for this role only.", "display_name": "Catalog Server Hive Advanced Configuration Snippet (Safety Valve)", "name": "catalogd_hive_conf_safety_valve", "value": null }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above.", "display_name": "Catalog Server Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The maximum size, in megabytes, per log file for Impala Catalog Server logs. Typically used by log4j or logback.", "display_name": "Impala Catalog Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Catalog Server HDFS Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Catalog Server HDFS Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_catalogd_hdfs_site_conf_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the StateStore Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: StateStore Connectivity", "name": "role_health_suppression_catalogserver_connectivity", "value": "false" }, { "desc": "Directory where the Catalog Server core dump is placed.", "display_name": "Catalog Server Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/catalogd" }, { "desc": "Local path to the certificate presented by the Catalog Server debug webserver. This file must be in PEM format. If empty, webserver SSL/TLS support is not enabled.", "display_name": "SSL/TLS Certificate for Catalog Server Webserver", "name": "webserver_certificate_file", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The minimum log level for Impala Catalog Server logs", "display_name": "Impala Catalog Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "Catalog Server Maximum Log Files", "name": "max_log_files", "value": "10" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_catalogserver_scm_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-nfsgateway.json0000666000175100017510000006524713245514472025503 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_nfsgateway_role_env_safety_valve", "value": "false" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_nfsgateway_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Log Directory parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Log Directory", "name": "role_config_suppression_nfsgateway_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "nfsgateway_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "NFS clients often reorder writes. As a result, sequential writes can arrive at the NFS Gateway in random order. This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. Please make sure this directory has enough space. For example, if the application uploads 10 files with each having 100MB, it is recommended that this directory have roughly 1GB of space in case write reorder happens (in the worst case) to every file.", "display_name": "Temporary Dump Directory", "name": "dfs_nfs3_dump_dir", "value": "/tmp/.hdfs-nfs" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NFS Gateway in Bytes", "name": "nfsgateway_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "nfsgateway_config_safety_valve", "value": null }, { "desc": "The port number of the system portmap or rpcbind service. This configuration is used by Cloudera Manager to verify if the system portmap or rpcbind service is running before starting NFS Gateway role. Cloudera Manager does not manage the system portmap or rpcbind service.", "display_name": "Portmap (or Rpcbind) Port", "name": "nfs3_portmap_port", "value": "111" }, { "desc": "The NFS Gateway server port.", "display_name": "NFS Gateway Server Port", "name": "nfs3_server_port", "value": "2049" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NFS Gateway", "name": "nfsgateway_java_opts", "value": "" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_nfsgateway_swap_memory_usage", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "When computing the overall NFS Gateway health, consider the host's health.", "display_name": "NFS Gateway Host Health Test", "name": "nfsgateway_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Temporary Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Temporary Dump Directory", "name": "role_config_suppression_dfs_nfs3_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "By default, NFS Gateway exported directories can be mounted by any client. For better access control, update this property with a list of host names and access privileges separated by whitespace characters. Host name format can be a single host, a Java regular expression, or an IPv4 address. The access privilege uses rw to specify readwrite and ro to specify readonly access. If the access privilege is not provided, the default is read-only. Examples of host name format and access privilege: \"192.168.0.0/22 rw\", \"host.*.example.com\", \"host1.test.org ro\".", "display_name": "Allowed Hosts and Privileges", "name": "dfs_nfs_exports_allowed_hosts", "value": "* rw" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The port number of the mount daemon implemented inside the NFS Gateway server role.", "display_name": "NFS Gateway MountD Port", "name": "nfs3_mountd_port", "value": "4242" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NFS Gateway Environment Advanced Configuration Snippet (Safety Valve)", "name": "NFSGATEWAY_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NFS Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_nfsgateway_host_health", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_nfsgateway_log_directory_free_space", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for NFS Gateway logs. Typically used by log4j or logback.", "display_name": "NFS Gateway Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_nfsgateway_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Allowed Hosts and Privileges parameter.", "display_name": "Suppress Parameter Validation: Allowed Hosts and Privileges", "name": "role_config_suppression_dfs_nfs_exports_allowed_hosts", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_nfsgateway_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Temporary Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Temporary Dump Directory Free Space", "name": "role_health_suppression_nfsgateway_dump_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The maximum number of rolled log files to keep for NFS Gateway logs. Typically used by log4j or logback.", "display_name": "NFS Gateway Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory.", "display_name": "Temporary Dump Directory Free Space Monitoring Absolute Thresholds", "name": "nfsgateway_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NFS Gateway parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NFS Gateway", "name": "role_config_suppression_nfsgateway_java_opts", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_nfsgateway_unexpected_exits", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for NFS Gateway logs", "display_name": "NFS Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: NFS Gateway Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_nfsgateway_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Temporary Dump Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Temporary Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Temporary Dump Directory Free Space Monitoring Percentage Thresholds", "name": "nfsgateway_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where NFS Gateway will place its log files.", "display_name": "NFS Gateway Log Directory", "name": "nfsgateway_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "Enables the health test that the NFS Gateway's process state is consistent with the role configuration", "display_name": "NFS Gateway Process Health Test", "name": "nfsgateway_scm_health_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hive-server2-sentry-safety.xml0000666000175100017510000000065513245514472027364 0ustar zuulzuul00000000000000 hive.security.authorization.task.factory org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl hive.server2.session.hook org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook hive.sentry.conf.url file:///{{CMF_CONF_DIR}}/sentry-site.xml sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hive-site.xml0000666000175100017510000000303213245514472024115 0ustar zuulzuul00000000000000 hive.metastore.local false hive.metastore.uris thrift://test-master-001.novalocal:9083 hive.metastore.client.socket.timeout 300 hive.metastore.warehouse.dir /user/hive/warehouse hive.warehouse.subdir.inherit.perms true mapred.reduce.tasks -1 hive.exec.reducers.bytes.per.reducer 1073741824 hive.exec.copyfile.maxsize 33554432 hive.exec.reducers.max 999 hive.metastore.execute.setugi true hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.MemoryTokenStore hive.server2.enable.doAs true fs.hdfs.impl.disable.cache true hive.server2.use.SSL false sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hive-gateway.json0000666000175100017510000001221013245514472024761 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only, a string to be inserted into the client configuration for hive-site.xml.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_client_config_safety_valve", "value": null }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hive" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hive_client_java_opts", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hive-env.sh", "display_name": "Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh", "name": "hive_client_env_safety_valve", "value": null }, { "desc": "Timeout for requests to the Hive Metastore Server. Consider increasing this if you have tables with a lot of metadata and see timeout errors. Used by most Hive Metastore clients such as Hive CLI and HiveServer2, but not by Impala. Impala has a separately configured timeout.", "display_name": "Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "300" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_client_config_safety_valve", "value": "false" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hive_client_java_heapsize", "value": "2147483648" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh parameter.", "display_name": "Suppress Parameter Validation: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hive-env.sh", "name": "role_config_suppression_hive_client_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hive_client_java_opts", "value": "-XX:MaxPermSize=512M -Djava.net.preferIPv4Stack=true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/ks_indexer-service.json0000666000175100017510000004671113245514472026175 0ustar zuulzuul00000000000000[ { "desc": "Name of the HBase service that this Key-Value Store Indexer service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Whether to suppress the results of the Lily HBase Indexer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Lily HBase Indexer Health", "name": "service_health_suppression_ks_indexer_hbase_indexers_healthy", "value": "false" }, { "desc": "The password for the HBase Indexer TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HBase Indexer TLS/SSL Certificate Trust Store Password", "name": "keystore_indexer_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_keystore_indexer_sentry_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_ks_indexer_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Lily HBase Indexer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Lily HBase Indexer Count Validator", "name": "service_config_suppression_hbase_indexer_count_validator", "value": "false" }, { "desc": "Name of the Solr service that this Key-Value Store Indexer service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_keystore_indexer_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Morphlines File parameter.", "display_name": "Suppress Parameter Validation: Morphlines File", "name": "service_config_suppression_morphlines_conf_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Grok Dictionary File parameter.", "display_name": "Suppress Parameter Validation: Grok Dictionary File", "name": "service_config_suppression_grok_dictionary_conf_file", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. Sentry is supported only on CDH 5.4 or later deployments and requires authentication to be turned on for HBase Indexer..", "display_name": "Enable Sentry Authorization using Policy Files", "name": "keystore_indexer_sentry_enabled", "value": "false" }, { "desc": "The health test thresholds of the overall Lily HBase Indexer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Lily HBase Indexers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Lily HBase Indexers falls below the critical threshold.", "display_name": "Healthy Lily HBase Indexer Monitoring Thresholds", "name": "ks_indexer_indexers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Sentry Validator", "name": "service_config_suppression_keystore_indexer_sentry_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Custom Mime-types File parameter.", "display_name": "Suppress Parameter Validation: Custom Mime-types File", "name": "service_config_suppression_custom_mimetypes_file", "value": "false" }, { "desc": "Text that goes into morphlines.conf file used by HBase Indexers. The text goes verbatim into the config file except that $ZK_HOST is replaced by the ZooKeeper quorum of the Solr service.", "display_name": "Morphlines File", "name": "morphlines_conf_file", "value": "SOLR_LOCATOR : {\n # Name of solr collection\n collection : collection\n \n # ZooKeeper ensemble\n zkHost : \"$ZK_HOST\" \n}\n\n\nmorphlines : [\n{\nid : morphline\nimportCommands : [\"org.kitesdk.**\", \"com.ngdata.**\"]\n\ncommands : [ \n {\n extractHBaseCells {\n mappings : [\n {\n inputColumn : \"data:*\"\n outputField : \"data\" \n type : string \n source : value\n }\n ]\n }\n }\n\n\n { logDebug { format : \"output record: {}\", args : [\"@{}\"] } }\n]\n}\n]\n" }, { "desc": "Authentication mechanism used by HBase Indexer.", "display_name": "HBase Indexer Secure Authentication", "name": "hbase_indexer_security_authentication", "value": "simple" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Key-Value Store Indexer Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "keystore_indexer_sentry_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Text that goes verbatim into grok-dictionary.conf file used by HBase Indexers.", "display_name": "Grok Dictionary File", "name": "grok_dictionary_conf_file", "value": "USERNAME [a-zA-Z0-9._-]+\nUSER %{USERNAME}\nINT (?:[+-]?(?:[0-9]+))\nBASE10NUM (?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))\nNUMBER (?:%{BASE10NUM})\nBASE16NUM (?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))\nUUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}\n\n# Networking\nMAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})\nCISCOMAC (?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})\nWINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})\nCOMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})\nIP (?/(?>[\\w_%!$@:.,-]+|\\\\.)*)+\n#UNIXPATH (?/dev/pts/%{NONNEGINT})\nBSDTTY (?>/dev/tty[pq][a-z0-9])\nTTY (?:%{BSDTTY}|%{LINUXTTY})\nWINPATH (?>[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+\nURIPROTO [A-Za-z]+(\\+[A-Za-z+]+)?\nURIHOST %{IPORHOST}(?::%{POSINT:port})?\n# uripath comes loosely from RFC1738, but mostly from what Firefox\n# doesn't turn into %XX\nURIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=#%_\\-]*)+\n#URIPARAM \\?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?\nURIPARAM \\?[A-Za-z0-9$.+!*'|(){},~#%&/=:;_?\\-\\[\\]]*\nURIPATHPARAM %{URIPATH}(?:%{URIPARAM})?\nURI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?\n\n# Months: January, Feb, 3, 03, 12, December\nMONTH \\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b\nMONTHNUM (?:0?[1-9]|1[0-2])\nMONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\n\n# Days: Monday, Tue, Thu, etc...\nDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)\n\n# Years?\nYEAR (?>\\d\\d){1,2}\n# Time: HH:MM:SS\n#TIME \\d{2}:\\d{2}(?::\\d{2}(?:\\.\\d+)?)?\n# I'm still on the fence about using grok to perform the time match,\n# since it's probably slower.\n# TIME %{POSINT<24}:%{POSINT<60}(?::%{POSINT<60}(?:\\.%{POSINT})?)?\nHOUR (?:2[0123]|[01]?[0-9])\nMINUTE (?:[0-5][0-9])\n# '60' is a leap second in most time standards and thus is valid.\nSECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\nTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])\n# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)\nDATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}\nDATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}\nISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))\nISO8601_SECOND (?:%{SECOND}|60)\nTIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?\nDATE %{DATE_US}|%{DATE_EU}\nDATESTAMP %{DATE}[- ]%{TIME}\nTZ (?:[PMCE][SD]T)\nDATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}\nDATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}\n\n# Syslog Dates: Month Day HH:MM:SS\nSYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}\nPROG (?:[\\w._/%-]+)\nSYSLOGPROG %{PROG:program}(?:\\[%{POSINT:pid}\\])?\nSYSLOGHOST %{IPORHOST}\nSYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>\nHTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}\n\n# Shortcuts\nQS %{QUOTEDSTRING}\n\n# Log formats\nSYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:\nCOMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}\n\n# Log Levels\nLOGLEVEL ([T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_keystore_indexer_provider_resource", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Key-Value Store Indexer Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "ks_indexer_env_safety_valve", "value": null }, { "desc": "Text that goes verbatim into custom-mimetypes.xml file used by HBase Indexers.", "display_name": "Custom Mime-types File", "name": "custom_mimetypes_file", "value": "\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop groups or local groups defined in the policy file. When configured with Hadoop groups, Sentry will ask the HDFS Namenode for group mapping for a given user to determine authorization access.", "display_name": "Sentry User to Group Mapping Class", "name": "keystore_indexer_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_keystore_indexer_truststore_file", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HBase Indexer might connect to. This is used when HBase Indexer is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HBase Indexer TLS/SSL Certificate Trust Store File", "name": "keystore_indexer_truststore_file", "value": null }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "keystore_indexer_provider_resource", "value": "/user/hbaseindexer/sentry/sentry-provider.ini" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/zookeeper-service.json0000666000175100017510000003121713245514472026040 0ustar zuulzuul00000000000000[ { "desc": "When enabled, ZooKeeper auto purge feature retains this many most recent snapshots and the corresponding transaction logs in the dataDir and dataLogDir respectively and deletes the rest. Defaults to 5. Minimum value is 3.", "display_name": "Auto Purge Snapshots Retain Count", "name": "autopurgeSnapRetainCount", "value": "5" }, { "desc": "Configures the path of the root znode under which all canary updates are performed", "display_name": "ZooKeeper Canary Root Znode Path", "name": "zookeeper_canary_root_path", "value": "/cloudera_manager_zookeeper_canary" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "zookeeper" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Amount of time, in ticks, to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they are dropped.", "display_name": "Synchronization Limit", "name": "syncLimit", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Server Count Validator", "name": "service_config_suppression_server_count_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Server Health", "name": "service_health_suppression_zookeeper_servers_healthy", "value": "false" }, { "desc": "The time interval in hours for which the purge task has to be triggered. Set to a positive integer (1 and above) to enable the auto purging. Defaults to 24.", "display_name": "Auto Purge Time Interval", "name": "autopurgeInterval", "value": "24" }, { "desc": "Enable Kerberos authentication for ZooKeeper.", "display_name": "Enable Kerberos Authentication", "name": "enableSecurity", "value": "false" }, { "desc": "Whether to suppress the results of the ZooKeeper Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: ZooKeeper Canary", "name": "service_health_suppression_zookeeper_canary_health", "value": "false" }, { "desc": "The number of snapshot files and corresponding transaction logs to keep when running the Cleanup command.", "display_name": "Cleanup Retain Count", "name": "cleanupRetainCount", "value": "5" }, { "desc": "Enables the health check that a client can connect to ZooKeeper and perform basic operations", "display_name": "ZooKeeper Canary Health Check", "name": "zookeeper_canary_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Canary Root Znode Path parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Canary Root Znode Path", "name": "service_config_suppression_zookeeper_canary_root_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_zookeeper_env_safety_valve", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "zookeeper" }, { "desc": "Configures the timeout used by the canary sessions with ZooKeeper servers", "display_name": "ZooKeeper Canary Session Timeout", "name": "zookeeper_canary_session_timeout", "value": "30000" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "The maximum size of the data that can be stored in a znode in bytes.", "display_name": "Jute Max Buffer", "name": "jute_maxbuffer", "value": "4194304" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "zookeeper" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Amount of time, in ticks, to allow followers to connect and sync to a leader. Increase this value as needed, if the amount of data managed by ZooKeeper is large.", "display_name": "Initialization Limit", "name": "initLimit", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "zookeeper_env_safety_valve", "value": null }, { "desc": "Configures the timeout used by the canary for connection establishment with ZooKeeper servers", "display_name": "ZooKeeper Canary Connection Timeout", "name": "zookeeper_canary_connection_timeout", "value": "10000" }, { "desc": "Configures the timeout used by the canary for ZooKeeper operations", "display_name": "ZooKeeper Canary Operation Timeout", "name": "zookeeper_canary_operation_timeout", "value": "30000" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The length of time, in milliseconds, of a single tick, which is the basic time unit used by ZooKeeper. A tick is used to regulate heartbeats and timeouts.", "display_name": "Tick Time", "name": "tickTime", "value": "2000" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "The health test thresholds of the overall Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Servers falls below the critical threshold.", "display_name": "Healthy Server Monitoring Thresholds", "name": "zookeeper_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the ZooKeeper Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: ZooKeeper Server Count Validator", "name": "service_config_suppression_zookeeper_odd_number_of_servers_validator", "value": "false" }, { "desc": "Whether the leader accepts client connections.", "display_name": "Leader Serves", "name": "leaderServes", "value": "yes" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Automatically create data directories at startup, if they do not exist. Enabling this configuration should be used with care as it will suppress any errors in setup of data directories.", "display_name": "Enable auto-creation of data directories", "name": "zookeeper_datadir_autocreate", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/priority-one-confs.json0000666000175100017510000000320513245514472026141 0ustar zuulzuul00000000000000[ "dfs_block_size", "dfs_umaskmode", "dfs_webhdfs_enabled", "dfs_permissions", "dfs_replication", "io_compression_codecs", "io_sort_mb", "dfs_datanode_du_reserved", "dfs_datanode_failed_volumes_tolerated", "dfs_name_dir_restore", "fs_trash_interval", "dfs_safemode_min_datanodes", "dfs_safemode_extension", "dfs_access_time_precision", "yarn_acl_enable", "yarn_admin_acl", "yarn_log_aggregation_enable", "yarn_log_aggregation_retain_seconds", "mapreduce_jobhistory_max_age_ms", "mapreduce_jobhistory_cleaner_interval", "mapreduce_reduce_memory_mb", "mapreduce_reduce_java_opts", "mapreduce_map_memory_mb", "mapreduce_map_java_opts", "yarn_nodemanager_container_manager_thread_count", "yarn_nodemanager_delete_thread_count", "yarn_nodemanager_heartbeat_interval_ms", "yarn_nodemanager_localizer_cache_cleanup_interval_ms", "yarn_nodemanager_localizer_client_thread_count", "yarn_nodemanager_localizer_cache_target_size_mb", "yarn_nodemanager_localizer_fetch_thread_count", "yarn_nodemanager_log_retain_seconds", "yarn_nodemanager_resource_memory_mb", "yarn_resourcemanager_client_thread_count", "yarn_resourcemanager_scheduler_client_thread_count", "yarn_resourcemanager_admin_client_thread_count", "yarn_resourcemanager_amliveliness_monitor_interval_ms", "yarn_am_liveness_monitor_expiry_interval_ms", "yarn_resourcemanager_am_max_retries", "yarn_scheduler_minimum_allocation_mb", "yarn_scheduler_maximum_allocation_mb", "yarn_app_mapreduce_am_command_opts", "yarn_app_mapreduce_am_resource_mb" ] sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hive-service.json0000666000175100017510000013707413245514472025000 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Sentry Validator", "name": "service_config_suppression_hive_sentry_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HiveServer2 Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HiveServer2 Count Validator", "name": "service_config_suppression_hiveserver2_count_validator", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hive" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HiveServer2 might connect to. This is used when HiveServer2 is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HiveServer2 TLS/SSL Certificate Trust Store File", "name": "hiveserver2_truststore_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Name parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Name", "name": "service_config_suppression_hive_metastore_database_name", "value": "false" }, { "desc": "Port number of Hive Metastore database", "display_name": "Hive Metastore Database Port", "name": "hive_metastore_database_port", "value": "3306" }, { "desc": "Smaller than this size, Hive uses a single-threaded copy; larger than this size, Hive uses DistCp.", "display_name": "Hive Copy Large File Size", "name": "hive_exec_copyfile_maxsize", "value": "33554432" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hive/audit" }, { "desc": "Name of Hive Metastore database", "display_name": "Hive Metastore Database Name", "name": "hive_metastore_database_name", "value": "metastore" }, { "desc": "The class to use in Sentry authorization for user to group mapping. Sentry authorization may be configured to use either Hadoop user to group mapping or local groups defined in the policy file. Hadoop user to group mapping may be configured in the Cloudera Manager HDFS service configuration page under the Security section.", "display_name": "Sentry User to Group Mapping Class", "name": "hive_sentry_provider", "value": "org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using the Sentry service instead, add the Sentry service as a dependency to the Hive service. The Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry is supported only on CDH 4.4 or later deployments. Before enabling Sentry, read the requirements and configuration steps in Setting Up Hive Authorization with Sentry.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Bypass Metastore Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Bypass Metastore Validator", "name": "service_config_suppression_hive_bypass_metastore_validator", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the Spark on YARN service that this Hive service instance depends on. If selected and Enable Hive on Spark is set to true, Hive jobs can use the Spark execution engine instead of MapReduce2. Requires that Hive depends on YARN. See Configuring Hive on Spark for more information about Hive on Spark.", "display_name": "Spark On YARN Service", "name": "spark_on_yarn_service", "value": null }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_service_config_safety_valve", "value": null }, { "desc": "Instead of talking to Hive Metastore Server for Metastore information, Hive clients will talk directly to the Metastore database.", "display_name": "Bypass Hive Metastore Server", "name": "hive_bypass_metastore_server", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Host parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Host", "name": "service_config_suppression_hive_metastore_database_host", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_service_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "hive_server2_sentry_safety_valve", "value": null }, { "desc": "Size per reducer. If the input size is 10GiB and this is set to 1GiB, Hive will use 10 reducers.", "display_name": "Hive Bytes Per Reducer", "name": "hive_bytes_per_reducer", "value": "67108864" }, { "desc": "The directory in which Hive on Spark lineage log files are written.", "display_name": "Hive on Spark Lineage Log Directory", "name": "lineage_event_log_dir", "value": "/var/log/hive/lineage" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_hiveserver2_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_hive_server2_sentry_safety_valve", "value": "false" }, { "desc": "Password for Hive Metastore database", "display_name": "Hive Metastore Database Password", "name": "hive_metastore_database_password", "value": "" }, { "desc": "The password for the HiveServer2 TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HiveServer2 TLS/SSL Certificate Trust Store Password", "name": "hiveserver2_truststore_password", "value": null }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Hive events:\n

\n\n
    \n
  • userName: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • operation: the Hive operation being performed.
  • \n
  • databaseName: the databaseName for the operation.
  • \n
  • tableName: the tableName for the operation.
  • \n
\n\n

\nThe default Hive audit event filter discards HDFS directory events generated by\nHive jobs that reference the /tmp directory.\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default Hive audit event filter discards HDFS directory events \",\n \"generated by Hive jobs that reference the /tmp directory.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"operation\", \"match\" : \"QUERY\" },\n { \"name\" : \"objectType\", \"match\" : \"DFS_DIR\"},\n { \"name\" : \"resourcePath\", \"match\" : \"/tmp/hive-(?:.+)?/hive_(?:.+)?/-mr-.*\" }\n ]\n }\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the WebHCat Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: WebHCat Server Count Validator", "name": "service_config_suppression_webhcat_count_validator", "value": "false" }, { "desc": "Directory name where Hive Metastore's database is stored (only for Derby)", "display_name": "Hive Metastore Derby Path", "name": "hive_metastore_derby_path", "value": "/var/lib/hive/cloudera_manager/derby/metastore_db" }, { "desc": "Perform DataNucleus validation of metadata during startup. Note: when enabled, Hive will log DataNucleus warnings even though Hive will function normally.", "display_name": "Hive Metastore Database DataNucleus Metadata Validation", "name": "hive_metastore_database_datanucleus_metadata_validation", "value": "false" }, { "desc": "Name of the Sentry service that this Hive service instance depends on. If selected, Hive uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Max number of reducers to use. If the configuration parameter Hive Reduce Tasks is negative, Hive will limit the number of reducers to the value of this parameter.", "display_name": "Hive Max Reducers", "name": "hive_max_reducers", "value": "1099" }, { "desc": "Type of Hive Metastore database. Note that Derby is not recommended and Cloudera Impala does not support Derby.", "display_name": "Hive Metastore Database Type", "name": "hive_metastore_database_type", "value": "mysql" }, { "desc": "Whether to suppress the results of the HiveServer2 Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HiveServer2 Health", "name": "service_health_suppression_hive_hiveserver2s_healthy", "value": "false" }, { "desc": "The URI of the LDAP server to use if LDAP authentication is enabled. The URI must be prefixed with ldap:// or ldaps://. Usernames and passwords are transmitted in the clear unless an \"ldaps://\" URI is specified. The URI can optionally specify the port; for example, ldaps://ldap_server.example.com:636.", "display_name": "LDAP URI", "name": "hiveserver2_ldap_uri", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hive_service_env_safety_valve", "value": "false" }, { "desc": "Host name of Hive Metastore database", "display_name": "Hive Metastore Database Host", "name": "hive_metastore_database_host", "value": "localhost" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Prevent Metastore operations in the event of schema version incompatibility. Consider setting this to true to reduce probability of schema corruption during Metastore operations. Note that setting this property to true will also set datanucleus.autoCreateSchema property to false and datanucleus.fixedDatastore property to true. Any values set in Cloudera Manager for these properties will be overridden.", "display_name": "Strict Hive Metastore Schema Validation", "name": "hive_metastore_schema_verification", "value": "true" }, { "desc": "This configuration overrides the value set for Hive Proxy User Groups configuration in HDFS service for use by Hive Metastore Server. Specify a comma-delimited list of groups that you want to allow access to Hive Metastore metadata and allow the Hive user to impersonate. A value of '*' allows all groups. The default value of empty inherits the value set for Hive Proxy User Groups configuration in the HDFS service.", "display_name": "Hive Metastore Access Control and Proxy User Groups Override", "name": "hive_proxy_user_groups_list", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Warehouse Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Warehouse Directory", "name": "service_config_suppression_hive_warehouse_directory", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "hive_core_site_safety_valve", "value": null }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "hive.metastore.client.socket.timeout60" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Bypass Sentry Authorization Users parameter.", "display_name": "Suppress Parameter Validation: Bypass Sentry Authorization Users", "name": "service_config_suppression_sentry_metastore_service_users", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_hiveserver2_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Auxiliary JARs Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Auxiliary JARs Directory", "name": "service_config_suppression_hive_aux_jars_path_dir", "value": "false" }, { "desc": "The health test thresholds of the overall WebHCat Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" WebHCat Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" WebHCat Servers falls below the critical threshold.", "display_name": "Healthy WebHCat Server Monitoring Thresholds", "name": "hive_webhcats_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml. Applies to all Hive Replication jobs.", "display_name": "Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_service_replication_config_safety_valve", "value": null }, { "desc": "When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,X. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server looks like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is generally most useful when authenticating against an OpenLDAP server. This parameter is mutually exclusive with LDAP Domain.", "display_name": "LDAP BaseDN", "name": "hiveserver2_ldap_basedn", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Global Policy File parameter.", "display_name": "Suppress Parameter Validation: Sentry Global Policy File", "name": "service_config_suppression_hive_sentry_provider_resource", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Hive warehouse directory is the location in HDFS where Hive's tables are stored. Note that Hive's default value for its warehouse directory is '/user/hive/warehouse'.", "display_name": "Hive Warehouse Directory", "name": "hive_warehouse_directory", "value": "/user/hive/warehouse" }, { "desc": "The password for the HiveServer2 JKS keystore file.", "display_name": "HiveServer2 TLS/SSL Server JKS Keystore File Password", "name": "hiveserver2_keystore_password", "value": null }, { "desc": "Whether to suppress the results of the WebHCat Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: WebHCat Server Health", "name": "service_health_suppression_hive_webhcats_healthy", "value": "false" }, { "desc": "When set, this value is appended to all usernames before authenticating with the LDAP server. For example, if this parameter is set to \"my.domain.com\", and the user authenticating is \"mike\", then \"mike@my.domain.com\" is passed to the LDAP server. If this field is unset, the username is unaltered before being passed to the LDAP server. LDAP Domain is most useful when authenticating against an Active Directory server. This parameter is mutually exclusive with LDAP BaseDN.", "display_name": "LDAP Domain", "name": "hiveserver2_ldap_domain", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Hive Metastore Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Metastore Server Count Validator", "name": "service_config_suppression_hivemetastore_count_validator", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Domain parameter.", "display_name": "Suppress Parameter Validation: LDAP Domain", "name": "service_config_suppression_hiveserver2_ldap_domain", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Derby Path parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Derby Path", "name": "service_config_suppression_hive_metastore_derby_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Proxy Groups Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Proxy Groups Validator", "name": "service_config_suppression_hive_proxy_groups_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Server JKS Keystore File Password", "name": "service_config_suppression_hiveserver2_keystore_password", "value": "false" }, { "desc": "When checked, LDAP-based authentication for users is enabled. This option is incompatible with Kerberos authentication for Hive Server 2 at this time.", "display_name": "Enable LDAP Authentication", "name": "hiveserver2_enable_ldap_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_hive_core_site_safety_valve", "value": "false" }, { "desc": "Default number of reduce tasks per job. Usually set to a prime number close to the number of available hosts. Ignored when mapred.job.tracker is \"local\". Hadoop sets this to 1 by default, while Hive uses -1 as the default. When set to -1, Hive will automatically determine an appropriate number of reducers for each job.", "display_name": "Hive Reduce Tasks", "name": "hive_reduce_tasks", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Name for Sentry Authorization parameter.", "display_name": "Suppress Parameter Validation: Server Name for Sentry Authorization", "name": "service_config_suppression_hive_sentry_server", "value": "false" }, { "desc": "The health test thresholds of the overall Hive Metastore Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Hive Metastore Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Hive Metastore Servers falls below the critical threshold.", "display_name": "Healthy Hive Metastore Server Monitoring Thresholds", "name": "hive_hivemetastores_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "The health test thresholds of the overall HiveServer2 health. The check returns \"Concerning\" health if the percentage of \"Healthy\" HiveServer2s falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" HiveServer2s falls below the critical threshold.", "display_name": "Healthy HiveServer2 Monitoring Thresholds", "name": "hive_hiveserver2s_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Automatically create or upgrade tables in the Hive Metastore database when needed. Consider setting this to false and managing the schema manually.", "display_name": "Auto Create and Upgrade Hive Metastore Database Schema", "name": "hive_metastore_database_auto_create_schema", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userNname, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Hive events:\n

\n\n
    \n
  • operation: the Hive operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • databaseName: the database affected by the operation.
  • \n
  • tableName: the table or view affected by the operation.
  • \n
  • objectType: the type of object affected by the operation.
  • \n
  • resourcePath: the path of the resource affected by the operation.
  • \n
\n\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Enable collection of lineage from the service's roles.", "display_name": "Enable Lineage Collection", "name": "navigator_lineage_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URI parameter.", "display_name": "Suppress Parameter Validation: LDAP URI", "name": "service_config_suppression_hiveserver2_ldap_uri", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Replication Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "service_config_suppression_hive_service_replication_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties parameter.", "display_name": "Suppress Parameter Validation: Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "service_config_suppression_navigator_lineage_client_config_safety_valve", "value": "false" }, { "desc": "Name of the HBase service that this Hive service instance depends on.", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Directory containing auxiliary JARs used by Hive. This should be a directory location and not a classpath containing one or more JARs. This directory must be created and managed manually on hosts that run the Hive Metastore Server, HiveServer2, or the Hive CLI. The directory location is set in the environment as HIVE_AUX_JARS_PATH and will generally override the hive.aux.jars.path property set in XML files, even if hive.aux.jars.path is set in an advanced configuration snippet.", "display_name": "Hive Auxiliary JARs Directory", "name": "hive_aux_jars_path_dir", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database User parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database User", "name": "service_config_suppression_hive_metastore_database_user", "value": "false" }, { "desc": "HDFS path to the global policy file for Sentry authorization. This should be a relative path (and not a full HDFS URL). The global policy file must be in Sentry policy file format.", "display_name": "Sentry Global Policy File", "name": "hive_sentry_provider_resource", "value": "/user/hive/sentry/sentry-provider.ini" }, { "desc": "The server name used when defining privilege rules in Sentry authorization. Sentry uses this name as an alias for the Hive service. It does not correspond to any physical server name.", "display_name": "Server Name for Sentry Authorization", "name": "hive_sentry_server", "value": "server1" }, { "desc": "Whether to suppress the results of the Hive Metastore Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Metastore Server Health", "name": "service_health_suppression_hive_hivemetastores_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HiveServer2 TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HiveServer2 TLS/SSL Server JKS Keystore File Location", "name": "service_config_suppression_hiveserver2_keystore_path", "value": "false" }, { "desc": "Encrypt communication between clients and HiveServer2 using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HiveServer2", "name": "hiveserver2_enable_ssl", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Hive on Spark lineage logs. Typically used by log4j or logback.", "display_name": "Hive on Spark Maximum Lineage Log File Size", "name": "max_lineage_log_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_hiveserver2_ldap_basedn", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Hive service instance depends on.", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "List of users that are allowed to bypass Sentry Authorization in the Hive metastore. These are usually service users that already ensure that all activity has been authorized, such as hive and impala. Only applies when Hive is using Sentry Service.", "display_name": "Bypass Sentry Authorization Users", "name": "sentry_metastore_service_users", "value": "hive,impala,hue,hdfs" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HiveServer2 is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HiveServer2 TLS/SSL Server JKS Keystore File Location", "name": "hiveserver2_keystore_path", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Hive Derby Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Derby Validator", "name": "service_config_suppression_hive_derby_validator", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hive" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hive" }, { "desc": "Let the table directories inherit the permission of the Warehouse or Database directory instead of being created with the permissions derived from dfs umask. This allows Impala to insert into tables created via Hive.", "display_name": "Hive Warehouse Subdirectories Inherit Permissions", "name": "hive_warehouse_subdir_inherit_perms", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Client TLS/SSL In Use With LDAP Authentication Validator configuration validator.", "display_name": "Suppress Configuration Validator: Client TLS/SSL In Use With LDAP Authentication Validator", "name": "service_config_suppression_hive_client_ssl_recommended_with_ldap_auth_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Database Password parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Database Password", "name": "service_config_suppression_hive_metastore_database_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive on Spark Lineage Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hive on Spark Lineage Log Directory", "name": "service_config_suppression_lineage_event_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Access Control and Proxy User Groups Override parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Access Control and Proxy User Groups Override", "name": "service_config_suppression_hive_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "User for Hive Metastore database", "display_name": "Hive Metastore Database User", "name": "hive_metastore_database_user", "value": "hive" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hive Concurrency Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hive Concurrency Configuration Validator", "name": "service_config_suppression_hive_concurrency_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.lineage.client.properties.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "navigator_lineage_client_config_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Hive Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Allows URIs when defining privileges in per-database policy files. Warning: Typically, this configuration should be disabled. Enabling it would allow database policy file owner (which is generally not Hive admin user) to grant load privileges to any directory with read access to Hive admin user, including databases controlled by other database policy files.", "display_name": "Allow URIs in Database Policy File", "name": "sentry_allow_uri_db_policyfile", "value": "false" }, { "desc": "In unsecure mode, setting this property to true will cause the Metastore Server to execute DFS operations using the client's reported user and group permissions. Cloudera Manager will set this for all clients and servers.", "display_name": "Set User and Group Information", "name": "hive_set_ugi", "value": "true" }, { "desc": "Cloudera does not support Hive on Spark in CDH 5.4 or CDH 5.5. Enable Hive to use Spark for execution even though it is not supported. For evaluation purposes only. This configuration only takes effect when Hive is configured with a Spark on YARN Service. See Configuring Hive on Spark for more information about using Hive on Spark.", "display_name": "Enable Hive on Spark (Unsupported)", "name": "enable_hive_on_spark", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "service_config_suppression_hive_service_config_safety_valve", "value": "false" }, { "desc": "MapReduce jobs are run against this service.", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hbase-master.json0000666000175100017510000010330513245514472024750 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"IPC Server handler.*ClosedChannelException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"IPC Server Responder, call.*output error\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Daughter regiondir does not exist: .*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"File.*might still be open.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"File.*might still be open.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Moving table .+ state to enabled but was already enabled\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Received OPENED for region.*but region was in the state.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Log Directory parameter.", "display_name": "Suppress Parameter Validation: Master Log Directory", "name": "role_config_suppression_hbase_master_log_dir", "value": "false" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "master_startup_tolerance", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_master_config_safety_valve", "value": "false" }, { "desc": "Number of pooled threads to handle region closing in the master.", "display_name": "Region Closing Threads", "name": "hbase_master_executor_closeregion_threads", "value": "5" }, { "desc": "When computing the overall Master health, consider the host's health.", "display_name": "Master Host Health Test", "name": "master_host_health_enabled", "value": "true" }, { "desc": "Number of pooled threads to handle the recovery of the region servers in the master.", "display_name": "RegionServer Recovery Threads", "name": "hbase_master_executor_serverops_threads", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where Master will place its log files.", "display_name": "Master Log Directory", "name": "hbase_master_log_dir", "value": "/var/log/hbase" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Time period in seconds to reset long-running metrics (e.g. compactions). This is an HBase specific configuration.", "display_name": "Extended Period", "name": "hbase_metrics_extended_period", "value": "3600" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "master_gc_duration_window", "value": "5" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress the results of the HBase Master Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HBase Master Canary", "name": "role_health_suppression_master_canary_health", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "master_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "A comma-separated list of LogCleanerDelegate(s) that are used in LogsCleaner. WAL/HLog cleaner(s) are called in order, so put the log cleaner that prunes the most log files in the front. To implement your own LogCleanerDelegate, add it to HBase's classpath and add the fully-qualified class name here. You should always add the above default log cleaners in the list, unless you have a special reason not to.", "display_name": "HBase Master Log Cleaner Plugins", "name": "hbase_master_logcleaner_plugins", "value": null }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "master_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_master_log_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "List of org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are loaded by default on the active HMaster process. For any implemented coprocessor methods, the listed classes will be called in order. After implementing your own MasterObserver, just put it in HBase's classpath and add the fully qualified class name here.", "display_name": "HBase Coprocessor Master Classes", "name": "hbase_coprocessor_master_classes", "value": "" }, { "desc": "The host name or IP address of the DNS name server which an HBase Master should use to determine the host name used for communication and display purposes.", "display_name": "HBase Master DNS Name Server", "name": "hbase_master_dns_nameserver", "value": null }, { "desc": "Enables the health test that a client can connect to the HBase Master", "display_name": "HBase Master Canary Health Test", "name": "master_canary_health_enabled", "value": "true" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase Master in Bytes", "name": "hbase_master_java_heapsize", "value": "1073741824" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_master_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "master_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_master_file_descriptor", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "When true, HBase Master will bind to 0.0.0.0. Only available with CDH 4.3 and later.", "display_name": "HBase Master Bind to Wildcard Address", "name": "hbase_master_bind_to_wildcard_address", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_master_scm_health", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The port that the HBase Master binds to.", "display_name": "HBase Master Port", "name": "hbase_master_port", "value": "60000" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_master_gc_duration", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Number of pooled threads to handle region opening in the master.", "display_name": "Region Opening Threads", "name": "hbase_master_executor_openregion_threads", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master Web UI Address parameter.", "display_name": "Suppress Parameter Validation: HBase Master Web UI Address", "name": "role_config_suppression_hbase_master_info_bindaddress", "value": "false" }, { "desc": "The port for the HBase Master web UI. Set to -1 to disable the HBase Master web UI.", "display_name": "HBase Master Web UI Port", "name": "hbase_master_info_port", "value": "60010" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_master_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Master Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "Master Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_master_config_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_master_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase Master", "name": "hbase_master_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "master_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Master Environment Advanced Configuration Snippet (Safety Valve)", "name": "MASTER_role_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The maximum size, in megabytes, per log file for Master logs. Typically used by log4j or logback.", "display_name": "Master Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Maximum time an HLog remains in the .oldlogdir directory until an HBase Master thread deletes it.", "display_name": "Maximum Time to Keep HLogs", "name": "hbase_master_logcleaner_ttl", "value": "60000" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Master Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase Master DNS Network Interface", "name": "role_config_suppression_hbase_master_dns_interface", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_master_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The address for the HBase Master web UI", "display_name": "HBase Master Web UI Address", "name": "hbase_master_info_bindAddress", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master Log Cleaner Plugins parameter.", "display_name": "Suppress Parameter Validation: HBase Master Log Cleaner Plugins", "name": "role_config_suppression_hbase_master_logcleaner_plugins", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_master_web_metric_collection", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Coprocessor Master Classes parameter.", "display_name": "Suppress Parameter Validation: HBase Coprocessor Master Classes", "name": "role_config_suppression_hbase_coprocessor_master_classes", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Master logs. Typically used by log4j or logback.", "display_name": "Master Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Master DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase Master DNS Name Server", "name": "role_config_suppression_hbase_master_dns_nameserver", "value": "false" }, { "desc": "Number of RPC Server instances spun up on HBase Master.", "display_name": "HBase Master Handler Count", "name": "hbase_master_handler_count", "value": "25" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase Master parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase Master", "name": "role_config_suppression_hbase_master_java_opts", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The name of the DNS network interface from which an HBase Master should report its IP address.", "display_name": "HBase Master DNS Network Interface", "name": "hbase_master_dns_interface", "value": null }, { "desc": "The minimum log level for Master logs", "display_name": "Master Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Master Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Master Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_master_role_env_safety_valve", "value": "false" }, { "desc": "Enables the health test that the Master's process state is consistent with the role configuration", "display_name": "Master Process Health Test", "name": "master_scm_health_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/kms-service.json0000666000175100017510000000750313245514472024630 0ustar zuulzuul00000000000000[ { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "kms" }, { "desc": "Authentication type for the KMS. Can either be \"simple\" or \"kerberos\".", "display_name": "Authentication Type", "name": "hadoop_kms_authentication_type", "value": "simple" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "kms" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has all of the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file-descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Java KeyStore KMS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "KMS_service_env_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/ks_indexer-hbase_indexer.json0000666000175100017510000005334013245514472027331 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Kerberos principal used by the Lily HBase Indexer roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "hbase" }, { "desc": "The group that the HBase Indexer process should run as.", "display_name": "System Group", "name": "hbase_indexer_process_groupname", "value": "hbase" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Lily HBase Indexer", "name": "hbase_indexer_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Indexer Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Indexer Log Directory", "name": "role_config_suppression_hbase_indexer_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "HTTP port used by HBase Indexer.", "display_name": "HBase Indexer HTTP Port", "name": "hbase_indexer_http_port", "value": "11060" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbase_indexer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_indexer_swap_memory_usage", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_indexer_host_health", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The user that the HBase Indexer process should run as.", "display_name": "System User", "name": "hbase_indexer_process_username", "value": "hbase" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_hbase_indexer_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Lily HBase Indexer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Lily HBase Indexer", "name": "role_config_suppression_hbase_indexer_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_indexer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_indexer_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_indexer_unexpected_exits", "value": "false" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only. A string to be inserted into hbase-indexer-site.xml for this role only.", "display_name": "Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml", "name": "hbase_indexer_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Lily HBase Indexer Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASE_INDEXER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_hbase_indexer_process_username", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Lily HBase Indexer logs. Typically used by log4j or logback.", "display_name": "Lily HBase Indexer Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where HBase Indexer will place its log files.", "display_name": "HBase Indexer Log Directory", "name": "hbase_indexer_log_dir", "value": "/var/log/hbase-solr" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_indexer_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Advanced Configuration Snippet (Safety Valve) for hbase-indexer-site.xml", "name": "role_config_suppression_hbase_indexer_config_safety_valve", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "When computing the overall Lily HBase Indexer health, consider the host's health.", "display_name": "Lily HBase Indexer Host Health Test", "name": "hbase_indexer_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Lily HBase Indexer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbase_indexer_role_env_safety_valve", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_indexer_log_directory_free_space", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for Lily HBase Indexer logs. Typically used by log4j or logback.", "display_name": "Lily HBase Indexer Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Enables the health test that the Lily HBase Indexer's process state is consistent with the role configuration", "display_name": "Lily HBase Indexer Process Health Test", "name": "hbase_indexer_scm_health_enabled", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for Lily HBase Indexer logs", "display_name": "Lily HBase Indexer Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Lily HBase Indexer in Bytes", "name": "hbase_indexer_java_heapsize", "value": "1073741824" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hbase-hbaserestserver.json0000666000175100017510000007126613245514472026676 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase REST Server", "name": "hbase_restserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_hbase_restserver_keystore_file", "value": "false" }, { "desc": "Maximum size of the HBase REST Server thread pool. The server can process this number of concurrent requests. Setting this too high can lead to out of memory errors.", "display_name": "HBase REST Server Maximum Threads", "name": "hbase_restserver_threads_max", "value": "100" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hbaserestserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The host name or IP address of the DNS name server which an HBase REST Server should use to determine the host name used for communication and display purposes.", "display_name": "HBase REST Server DNS Name Server", "name": "hbase_restserver_dns_nameserver", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Log Directory", "name": "role_config_suppression_hbase_restserver_log_dir", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HBase REST Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HBASERESTSERVER_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_restserver_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_hbase_restserver_keystore_password", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hbaserestserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "HBase REST Server Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_restserver_config_safety_valve", "value": null }, { "desc": "The password for the HBase REST Server JKS keystore file.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore File Password", "name": "hbase_restserver_keystore_password", "value": null }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hbase_rest_server_scm_health", "value": "false" }, { "desc": "Directory where HBase REST Server will place its log files.", "display_name": "HBase REST Server Log Directory", "name": "hbase_restserver_log_dir", "value": "/var/log/hbase" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when HBase REST Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore File Location", "name": "hbase_restserver_keystore_file", "value": null }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Encrypt communication between clients and HBase REST Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for HBase REST Server", "name": "hbase_restserver_ssl_enable", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hbase_rest_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase REST Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase REST Server", "name": "role_config_suppression_hbase_restserver_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server DNS Name Server parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server DNS Name Server", "name": "role_config_suppression_hbase_restserver_dns_nameserver", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hbase_rest_server_file_descriptor", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HBase REST Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Minimum size of the HBase REST Server thread pool. The server will maintain at least this number of threads in the pool at all times. The thread pool can grow up to the maximum size set by hbase.rest.threads.max.", "display_name": "HBase REST Server Minimum Threads", "name": "hbase_restserver_threads_min", "value": "2" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hbase_rest_server_swap_memory_usage", "value": "false" }, { "desc": "When false, all HTTP methods are permitted (GET/PUT/POST/DELETE). When true, only GET is permitted.", "display_name": "Enable HBase REST Server Read Only Mode", "name": "hbase_restserver_readonly", "value": "false" }, { "desc": "The port that HBase REST Server Web UI binds to.", "display_name": "HBase REST Server Web UI Port", "name": "hbase_restserver_info_port", "value": "8085" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When computing the overall HBase REST Server health, consider the host's health.", "display_name": "HBase REST Server Host Health Test", "name": "hbaserestserver_host_health_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The password that protects the private key contained in the JKS keystore used when HBase REST Server is acting as a TLS/SSL server.", "display_name": "HBase REST Server TLS/SSL Server JKS Keystore Key Password", "name": "hbase_restserver_keystore_keypassword", "value": null }, { "desc": "The maximum size, in megabytes, per log file for HBase REST Server logs. Typically used by log4j or logback.", "display_name": "HBase REST Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Enables the health test that the HBase REST Server's process state is consistent with the role configuration", "display_name": "HBase REST Server Process Health Test", "name": "hbaserestserver_scm_health_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "If true, HBase REST Server Web UI will bind to a wildcard address (0.0.0.0). Otherwise it will bind to a host name. Only available in CDH 4.3 and later.", "display_name": "HBase REST Server Web UI Bind to Wildcard Address", "name": "hbase_restserver_info_bind_to_wildcard", "value": "true" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "HBase REST Server will bind to this address.", "display_name": "HBase REST Server Host Address", "name": "hbase_restserver_host", "value": "0.0.0.0" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hbase_rest_server_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server Host Address parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server Host Address", "name": "role_config_suppression_hbase_restserver_host", "value": "false" }, { "desc": "The name of the DNS network interface from which an HBase REST Server should report its IP address.", "display_name": "HBase REST Server DNS Network Interface", "name": "hbase_restserver_dns_interface", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server DNS Network Interface", "name": "role_config_suppression_hbase_restserver_dns_interface", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HBase REST Server logs. Typically used by log4j or logback.", "display_name": "HBase REST Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The port that HBase REST Server binds to.", "display_name": "HBase REST Server Port", "name": "hbase_restserver_port", "value": "20550" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hbase_rest_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase REST Server TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: HBase REST Server TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_hbase_restserver_keystore_keypassword", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hbase_rest_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase REST Server in Bytes", "name": "hbase_restserver_java_heapsize", "value": "1073741824" }, { "desc": "The minimum log level for HBase REST Server logs", "display_name": "HBase REST Server Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/yarn-resourcemanager.json0000666000175100017510000012604213245514472026531 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "resourcemanager_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For applications that request containers on particular racks, the minimum time in milliseconds the Fair Scheduler waits before accepting a placement on another rack. Requires Fair Scheduler continuous scheduling to be enabled. If continuous scheduling is disabled, yarn.scheduler.fair.locality.threshold.rack should be used instead.", "display_name": "Fair Scheduler Rack Locality Delay", "name": "yarn_scheduler_fair_locality_delay_rack_ms", "value": "4000" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "An XML string that will be inserted verbatim into the Fair Scheduler allocations file. For CDH5, overrides the configuration set using the Pools configuration UI. For CDH4, this is the only way to configure the Fair Scheduler for YARN.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "resourcemanager_fair_scheduler_configuration", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "The expiry interval to wait until a NodeManager is considered dead.", "display_name": "NodeManager Monitor Expiry", "name": "yarn_nm_liveness_monitor_expiry_interval_ms", "value": "600000" }, { "desc": "The HTTP port of the ResourceManager web application.", "display_name": "ResourceManager Web Application HTTP Port", "name": "resourcemanager_webserver_port", "value": "8088" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The address of the applications manager interface in the ResourceManager.", "display_name": "ResourceManager Address", "name": "yarn_resourcemanager_address", "value": "8032" }, { "desc": "Enables the health test that the ResourceManager's process state is consistent with the role configuration", "display_name": "ResourceManager Process Health Test", "name": "resourcemanager_scm_health_enabled", "value": "true" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "resourcemanager_web_metric_collection_enabled", "value": "true" }, { "desc": "The periodic interval that the ResourceManager will check whether ApplicationMasters is still alive.", "display_name": "ApplicationMaster Monitor Interval", "name": "yarn_resourcemanager_amliveliness_monitor_interval_ms", "value": "1000" }, { "desc": "Number of threads used to handle the ResourceManager admin interface.", "display_name": "Admin Client Thread Count", "name": "yarn_resourcemanager_admin_client_thread_count", "value": "1" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "If enabled, the ResourceManager binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind ResourceManager to Wildcard Address", "name": "yarn_rm_bind_wildcard", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_fair_scheduler_configuration", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_resource_manager_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The address of the admin interface in the ResourceManager.", "display_name": "Administration Address", "name": "yarn_resourcemanager_admin_address", "value": "8033" }, { "desc": "The periodic interval that the ResourceManager will check whether NodeManagers are still alive.", "display_name": "NodeManager Monitor Interval", "name": "yarn_resourcemanager_nm_liveness_monitor_interval_ms", "value": "1000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_resource_manager_swap_memory_usage", "value": "false" }, { "desc": "The maximum number of application attempts. This is a global setting for all ApplicationMasters. Each ApplicationMaster can specify its individual maximum through the API, but if the individual maximum is more than the global maximum, the ResourceManager overrides it.", "display_name": "ApplicationMaster Maximum Attempts", "name": "yarn_resourcemanager_am_max_retries", "value": "2" }, { "desc": "For advanced use only. A string to be inserted into nodes_exclude.txt for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt", "name": "rm_hosts_exclude_safety_valve", "value": null }, { "desc": "Enable continuous scheduling in the Fair Scheduler. When enabled, scheduling decisions are decoupled from NodeManager heartbeats, leading to faster resource allocations.", "display_name": "Enable Fair Scheduler Continuous Scheduling", "name": "yarn_scheduler_fair_continuous_scheduling_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for ResourceManager", "name": "resource_manager_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_resource_manager_web_metric_collection", "value": "false" }, { "desc": "The periodic interval that the ResourceManager will check whether containers are still alive.", "display_name": "Container Monitor Interval", "name": "yarn_resourcemanager_container_liveness_monitor_interval_ms", "value": "600000" }, { "desc": "The maximum size, in megabytes, per log file for ResourceManager logs. Typically used by log4j or logback.", "display_name": "ResourceManager Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "resourcemanager_mapred_safety_valve", "value": null }, { "desc": "For applications that request containers on particular racks, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another rack. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. If not set, this means don't pass up any scheduling opportunities. Requires Fair Scheduler continuous scheduling to be disabled. If continuous scheduling is enabled, yarn.scheduler.fair.locality-delay-rack-ms should be used instead.", "display_name": "Fair Scheduler Rack Locality Threshold", "name": "resourcemanager_fair_scheduler_locality_threshold_rack", "value": null }, { "desc": "The class to use as the resource scheduler. FairScheduler is only supported in CDH 4.2.1 and later.", "display_name": "Scheduler Class", "name": "yarn_resourcemanager_scheduler_class", "value": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When enabled, any applications that were running on the cluster when the ResourceManager died will be recovered when the ResourceManager next starts. Note: If RM-HA is enabled, then this configuration is always enabled.", "display_name": "Enable ResourceManager Recovery", "name": "yarn_resourcemanager_recovery_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Maximum Attempts parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Maximum Attempts", "name": "role_config_suppression_yarn_resourcemanager_am_max_retries", "value": "false" }, { "desc": "The expiry interval to wait until an ApplicationMaster is considered dead.", "display_name": "ApplicationMaster Monitor Expiry", "name": "yarn_am_liveness_monitor_expiry_interval_ms", "value": "600000" }, { "desc": "Enter an XML string that represents the Capacity Scheduler configuration.", "display_name": "Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)", "name": "resourcemanager_capacity_scheduler_configuration", "value": "\n\n \n yarn.scheduler.capacity.root.queues\n default\n \n \n yarn.scheduler.capacity.root.capacity\n 100\n \n \n yarn.scheduler.capacity.root.default.capacity\n 100\n \n\n" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The largest amount of physical memory, in MiB, that can be requested for a container.", "display_name": "Container Memory Maximum", "name": "yarn_scheduler_maximum_allocation_mb", "value": "65536" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Log Directory parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Log Directory", "name": "role_config_suppression_resource_manager_log_dir", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When enabled, if a pool's minimum share is not met for some period of time, the Fair Scheduler preempts applications in other pools. Preemption guarantees that production applications are not starved while also allowing the cluster to be used for experimental and research applications. To minimize wasted computation, the Fair Scheduler preempts the most recently launched applications.", "display_name": "Fair Scheduler Preemption", "name": "resourcemanager_fair_scheduler_preemption", "value": "false" }, { "desc": "For applications that request containers on particular nodes, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another node. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. If not set, this means don't pass up any scheduling opportunities. Requires Fair Scheduler continuous scheduling to be disabled. If continuous scheduling is enabled, yarn.scheduler.fair.locality-delay-node-ms should be used instead.", "display_name": "Fair Scheduler Node Locality Threshold", "name": "resourcemanager_fair_scheduler_locality_threshold_node", "value": null }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "For applications that request containers on particular nodes, the minimum time in milliseconds the Fair Scheduler waits before accepting a placement on another node. Requires Fair Scheduler continuous scheduling to be enabled. If continuous scheduling is disabled, yarn.scheduler.fair.locality.threshold.node should be used instead.", "display_name": "Fair Scheduler Node Locality Delay", "name": "yarn_scheduler_fair_locality_delay_node_ms", "value": "2000" }, { "desc": "The minimum log level for ResourceManager logs", "display_name": "ResourceManager Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "resourcemanager_startup_tolerance_minutes", "value": "5" }, { "desc": "When enabled, ResourceManager has proxy user privileges.", "display_name": "Enable ResourceManger Proxy User Privileges", "name": "yarn_resourcemanager_proxy_user_privileges_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_resourcemanager_config_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into nodes_allow.txt for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt", "name": "rm_hosts_allow_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_resource_manager_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_resource_manager_log_directory_free_space", "value": "false" }, { "desc": "The maximum number of completed applications that the ResourceManager keeps.", "display_name": "Max Completed Applications", "name": "yarn_resourcemanager_max_completed_applications", "value": "10000" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "resourcemanager_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_resourcemanager_capacity_scheduler_configuration", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_allow.txt", "name": "role_config_suppression_rm_hosts_allow_safety_valve", "value": "false" }, { "desc": "The address of the scheduler interface in the ResourceManager.", "display_name": "Scheduler Address", "name": "yarn_resourcemanager_scheduler_address", "value": "8030" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "resourcemanager_gc_duration_window", "value": "5" }, { "desc": "The timeout for the ResourceManager session with ZooKeeper. The session expires if the ZooKeeper ensemble does not hear from the ResourceManager within the specified timeout period (no heartbeat). Session expiration is managed by the ZooKeeper ensemble, not by the ResourceManager.", "display_name": "ZooKeeper Session Timeout", "name": "yarn_resourcemanager_zk_timeout_ms", "value": "60000" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_resource_manager_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "resourcemanager_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_resource_manager_gc_duration", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The smallest number of virtual CPU cores that can be requested for a container. If using the Capacity or FIFO scheduler (or any scheduler, prior to CDH 5), virtual core requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores Minimum", "name": "yarn_scheduler_minimum_allocation_vcores", "value": "1" }, { "desc": "If using the Fair Scheduler, memory requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 5.", "display_name": "Container Memory Increment", "name": "yarn_scheduler_increment_allocation_mb", "value": "512" }, { "desc": "When computing the overall ResourceManager health, consider the host's health.", "display_name": "ResourceManager Host Health Test", "name": "resourcemanager_host_health_enabled", "value": "true" }, { "desc": "When enabled, the Fair Scheduler will assign shares to individual apps based on their size, rather than providing an equal share to all apps regardless of size.", "display_name": "Fair Scheduler Size-Based Weight", "name": "resourcemanager_fair_scheduler_size_based_weight", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The address of the resource tracker interface in the ResourceManager.", "display_name": "Resource Tracker Address", "name": "yarn_resourcemanager_resource_tracker_address", "value": "8031" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_resource_manager_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for nodes_exclude.txt", "name": "role_config_suppression_rm_hosts_exclude_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "ResourceManager Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Enables multiple Fair Scheduler container assignments in one heartbeat, which improves cluster throughput when there are many small tasks to run.", "display_name": "Fair Scheduler Assign Multiple Tasks", "name": "resourcemanager_fair_scheduler_assign_multiple", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "resourcemanager_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: ResourceManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_resourcemanager_mapred_safety_valve", "value": "false" }, { "desc": "Directory where ResourceManager will place its log files.", "display_name": "ResourceManager Log Directory", "name": "resource_manager_log_dir", "value": "/var/log/hadoop-yarn" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Number of threads to handle resource tracker calls.", "display_name": "Resource Tracker Thread Count", "name": "yarn_resourcemanager_resource_tracker_client_thread_count", "value": "50" }, { "desc": "If using the Fair Scheduler, virtual core requests will be rounded up to the nearest multiple of this number. This parameter has no effect prior to CDH 5.", "display_name": "Container Virtual CPU Cores Increment", "name": "yarn_scheduler_increment_allocation_vcores", "value": "1" }, { "desc": "The HTTPS port of the ResourceManager web application.", "display_name": "ResourceManager Web Application HTTPS Port (TLS/SSL)", "name": "resourcemanager_webserver_https_port", "value": "8090" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of ResourceManager in Bytes", "name": "resource_manager_java_heapsize", "value": "1073741824" }, { "desc": "The number of threads used to handle applications manager requests.", "display_name": "Client Thread Count", "name": "yarn_resourcemanager_client_thread_count", "value": "50" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_resource_manager_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for ResourceManager logs. Typically used by log4j or logback.", "display_name": "ResourceManager Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "ResourceManager Environment Advanced Configuration Snippet (Safety Valve)", "name": "RESOURCEMANAGER_role_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for ResourceManager parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for ResourceManager", "name": "role_config_suppression_resource_manager_java_opts", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The number of threads used to handle requests through the scheduler interface.", "display_name": "Scheduler Thread Count", "name": "yarn_resourcemanager_scheduler_client_thread_count", "value": "50" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "When set to true, the Fair Scheduler uses the username as the default pool name, in the event that a pool name is not specified. When set to false, all applications are run in a shared pool, called default.", "display_name": "Fair Scheduler User As Default Queue", "name": "resourcemanager_fair_scheduler_user_as_default_queue", "value": "true" }, { "desc": "The largest number of virtual CPU cores that can be requested for a container. This parameter has no effect prior to CDH 4.4.", "display_name": "Container Virtual CPU Cores Maximum", "name": "yarn_scheduler_maximum_allocation_vcores", "value": "32" }, { "desc": "The smallest amount of physical memory, in MiB, that can be requested for a container. If using the Capacity or FIFO scheduler (or any scheduler, prior to CDH 5), memory requests will be rounded up to the nearest multiple of this number.", "display_name": "Container Memory Minimum", "name": "yarn_scheduler_minimum_allocation_mb", "value": "1024" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hue-hue_load_balancer.json0000666000175100017510000003767013245514472026576 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hue_load_balancer_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Load Balancer Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Load Balancer Log Directory", "name": "role_config_suppression_hue_load_balancer_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hue_load_balancer_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Path to TLS/SSL Private Key File parameter.", "display_name": "Suppress Parameter Validation: Path to TLS/SSL Private Key File", "name": "role_config_suppression_ssl_certificate_key", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hue_load_balancer_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hue_load_balancer_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf parameter.", "display_name": "Suppress Parameter Validation: Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf", "name": "role_config_suppression_hue_load_balancer_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Path to TLS/SSL Certificate File parameter.", "display_name": "Suppress Parameter Validation: Path to TLS/SSL Certificate File", "name": "role_config_suppression_ssl_certificate", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Load Balancer Environment Advanced Configuration Snippet (Safety Valve)", "name": "HUE_LOAD_BALANCER_role_env_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into httpd.conf for this role only. This can only add options to the configuration, and cannot override previously defined options.", "display_name": "Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf", "name": "hue_load_balancer_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hue_load_balancer_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Port to use to connect to the Hue through the Load Balancer.", "display_name": "Hue Load Balancer Port", "name": "listen", "value": "8889" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hue_load_balancer_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Load Balancer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Load Balancer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hue_load_balancer_role_env_safety_valve", "value": "false" }, { "desc": "Path to the TLS/SSL private key file on the host running the Hue Load Balancer, used to encrypt traffic between the browser and the Load Balancer. This file must be in PEM format, and must be readable by the Hue system user. The Hue Load Balancer only supports a key without a passphrase.", "display_name": "Path to TLS/SSL Private Key File", "name": "ssl_certificate_key", "value": null }, { "desc": "Directory where Hue Load Balancer will place its log files.", "display_name": "Hue Load Balancer Log Directory", "name": "hue_load_balancer_log_dir", "value": "/var/log/hue-httpd" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When computing the overall Load Balancer health, consider the host's health.", "display_name": "Load Balancer Host Health Test", "name": "hue_load_balancer_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hue_load_balancer_unexpected_exits", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Path to the TLS/SSL certificate file on the host running the Hue Load Balancer, used to encrypt traffic between the browser and the Load Balancer. This file must be in PEM format, and must be readable by the Hue system user.", "display_name": "Path to TLS/SSL Certificate File", "name": "ssl_certificate", "value": null }, { "desc": "Enables the health test that the Load Balancer's process state is consistent with the role configuration", "display_name": "Load Balancer Process Health Test", "name": "hue_load_balancer_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hue_load_balancer_swap_memory_usage", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-datanode.json0000666000175100017510000012636013245514472025104 0ustar zuulzuul00000000000000[ { "desc": "Specifies the maximum number of threads to use for transferring data in and out of the DataNode.", "display_name": "Maximum Number of Transfer Threads", "name": "dfs_datanode_max_xcievers", "value": "4096" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Transceiver Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Transceiver Usage", "name": "role_health_suppression_data_node_transceivers_usage", "value": "false" }, { "desc": "In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This behavior will always be disabled for workloads that read only short sections of a block (e.g HBase random-IO workloads). This property is supported in CDH3u3 or later deployments.", "display_name": "Enable purging cache after reads", "name": "dfs_datanode_drop_cache_behind_reads", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether DataNodes should use DataNode hostnames when connecting to DataNodes for data transfer. This property is supported in CDH3u4 or later deployments.", "display_name": "Use DataNode Hostname", "name": "dfs_datanode_use_datanode_hostname", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "datanode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Enables the health test that the DataNode's process state is consistent with the role configuration", "display_name": "DataNode Process Health Test", "name": "datanode_scm_health_enabled", "value": "true" }, { "desc": "In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spare usage for more cacheable data. This property is supported in CDH3u3 or later deployments.", "display_name": "Enable purging cache after writes", "name": "dfs_datanode_drop_cache_behind_writes", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of DataNode in Bytes", "name": "datanode_java_heapsize", "value": "1073741824" }, { "desc": "Directory where DataNode will place its log files.", "display_name": "DataNode Log Directory", "name": "datanode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_data_node_web_metric_collection", "value": "false" }, { "desc": "While reading block files, the DataNode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader's position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the DataNode will attempt to read ahead. A value of 0 disables this feature. This property is supported in CDH3u3 or later deployments.", "display_name": "Number of read ahead bytes", "name": "dfs_datanode_readahead_bytes", "value": "4194304" }, { "desc": "Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls how much DataNode volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis.", "display_name": "Available Space Policy Balanced Threshold", "name": "dfs_datanode_available_space_balanced_threshold", "value": "10737418240" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "datanode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a DataNode Data Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "DataNode Data Directory Free Space Monitoring Percentage Thresholds", "name": "datanode_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "datanode_pause_duration_window", "value": "5" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's DataNode Data Directory.", "display_name": "DataNode Data Directory Free Space Monitoring Absolute Thresholds", "name": "datanode_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Permissions for the directories on the local file system where the DataNode stores its blocks. The permissions must be octal. 755 and 700 are typical values.", "display_name": "DataNode Data Directory Permissions", "name": "dfs_datanode_data_dir_perm", "value": "700" }, { "desc": "Maximum amount of bandwidth that each DataNode can use for balancing. Specified in bytes per second.", "display_name": "DataNode Balancing Bandwidth", "name": "dfs_balance_bandwidthPerSec", "value": "10485760" }, { "desc": "The health test thresholds of transceivers usage in a DataNode. Specified as a percentage of the total configured number of transceivers.", "display_name": "DataNode Transceivers Usage Thresholds", "name": "datanode_transceivers_usage_thresholds", "value": "{\"critical\":\"95.0\",\"warning\":\"75.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory parameter.", "display_name": "Suppress Parameter Validation: DataNode Data Directory", "name": "role_config_suppression_dfs_data_dir_list", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_data_node_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 5, \"content\":\"Datanode registration failed\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Got a command from standby NN - ignoring command:.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Failed Volumes Tolerated Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Failed Volumes Tolerated Validator", "name": "role_config_suppression_datanode_failed_volumes_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Data Directory Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Data Directory Status", "name": "role_health_suppression_data_node_volume_failures", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the DataNode Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: DataNode Data Directory Free Space", "name": "role_health_suppression_datanode_data_directories_free_space", "value": "false" }, { "desc": "When computing the overall DataNode health, consider the host's health.", "display_name": "DataNode Host Health Test", "name": "datanode_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for DataNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for DataNode", "name": "role_config_suppression_datanode_java_opts", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for DataNode logs. Typically used by log4j or logback.", "display_name": "DataNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for DataNode", "name": "datanode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DateNode Plugins parameter.", "display_name": "Suppress Parameter Validation: DateNode Plugins", "name": "role_config_suppression_dfs_datanode_plugins_list", "value": "false" }, { "desc": "Whether to suppress the results of the NameNode Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Connectivity", "name": "role_health_suppression_data_node_ha_connectivity", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Port for the DataNode HTTP web UI. Combined with the DataNode's hostname to build its HTTP address.", "display_name": "DataNode HTTP Web UI Port", "name": "dfs_datanode_http_port", "value": "50075" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of free space in a DataNode. Specified as a percentage of the capacity on the DataNode.", "display_name": "DataNode Free Space Monitoring Thresholds", "name": "datanode_free_space_thresholds", "value": "{\"critical\":\"10.0\",\"warning\":\"20.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Port for DataNode's XCeiver Protocol. Combined with the DataNode's hostname to build its address.", "display_name": "DataNode Transceiver Port", "name": "dfs_datanode_port", "value": "50010" }, { "desc": "The health test thresholds of the number of blocks on a DataNode", "display_name": "DataNode Block Count Thresholds", "name": "datanode_block_count_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"500000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The health test thresholds of failed volumes in a DataNode.", "display_name": "DataNode Volume Failures Thresholds", "name": "datanode_volume_failures_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_data_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "The minimum log level for DataNode logs", "display_name": "DataNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_data_node_scm_health", "value": "false" }, { "desc": "DataNode Policy for picking which volume should get a new block. The Available Space policy is only available starting with CDH 4.3.", "display_name": "DataNode Volume Choosing Policy", "name": "dfs_datanode_volume_choosing_policy", "value": "org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Data Directory Permissions parameter.", "display_name": "Suppress Parameter Validation: DataNode Data Directory Permissions", "name": "role_config_suppression_dfs_datanode_data_dir_perm", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "datanode_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: DataNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: DataNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_datanode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_data_node_host_health", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Reserved Space Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Reserved Space Validator", "name": "role_config_suppression_datanode_reserved_space_validator", "value": "false" }, { "desc": "Reserved space in bytes per volume for non Distributed File System (DFS) use.", "display_name": "Reserved Space for Non DFS Use", "name": "dfs_datanode_du_reserved", "value": "10737418240" }, { "desc": "The maximum amount of memory a DataNode may use to cache data blocks in memory. Setting it to zero will disable caching.", "display_name": "Maximum Memory Used for Caching", "name": "dfs_datanode_max_locked_memory", "value": "4294967296" }, { "desc": "Enables the health test that verifies the DataNode is connected to the NameNode", "display_name": "DataNode Connectivity Health Test", "name": "datanode_connectivity_health_enabled", "value": "true" }, { "desc": "Minimum number of running threads for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Min Threadcount", "name": "dfs_thrift_threads_min", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_datanode_config_safety_valve", "value": "false" }, { "desc": "The base port where the secure DataNode web UI listens. Combined with the DataNode's hostname to build its secure web UI address.", "display_name": "Secure DataNode Web UI Port (TLS/SSL)", "name": "dfs_datanode_https_port", "value": "50475" }, { "desc": "The amount of time to wait for the DataNode to fully start up and connect to the NameNode before enforcing the connectivity check.", "display_name": "DataNode Connectivity Tolerance at Startup", "name": "datanode_connectivity_tolerance", "value": "180" }, { "desc": "Maximum number of running threads for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Max Threadcount", "name": "dfs_thrift_threads_max", "value": "20" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Free Space", "name": "role_health_suppression_data_node_free_space_remaining", "value": "false" }, { "desc": "If this configuration is enabled, the DataNode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. This property is supported in CDH3u3 or later deployments.", "display_name": "Enable immediate enqueuing of data to disk after writes", "name": "dfs_datanode_sync_behind_writes", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_data_node_swap_memory_usage", "value": "false" }, { "desc": "Comma-separated list of DataNode plug-ins to be activated. If one plug-in cannot be loaded, all the plug-ins are ignored.", "display_name": "DateNode Plugins", "name": "dfs_datanode_plugins_list", "value": "" }, { "desc": "Comma-delimited list of directories on the local file system where the DataNode stores HDFS block data. Typical values are /data/N/dfs/dn for N = 1, 2, 3.... These directories should be mounted using the noatime option, and the disks should be configured using JBOD. RAID is not recommended. Warning: Be very careful when modifying this property. Removing or changing entries can result in data loss. To hot swap drives in CDH 5.4 and higher, override the value of this property for the specific DataNode role instance that has the drive to be hot-swapped; do not modify the property value in the role group. See Configuring Hot Swap for DataNodes for more information.", "display_name": "DataNode Data Directory", "name": "dfs_data_dir_list", "value": null }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_data_node_pause_duration", "value": "false" }, { "desc": "The number of volumes that are allowed to fail before a DataNode stops offering service. By default, any volume failure will cause a DataNode to shutdown.", "display_name": "DataNode Failed Volumes Tolerated", "name": "dfs_datanode_failed_volumes_tolerated", "value": "0" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "DataNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "If enabled, the DataNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind DataNode to Wildcard Address", "name": "dfs_datanode_bind_wildcard", "value": "false" }, { "desc": "The number of server threads for the DataNode.", "display_name": "Handler Count", "name": "dfs_datanode_handler_count", "value": "3" }, { "desc": "The maximum number of rolled log files to keep for DataNode logs. Typically used by log4j or logback.", "display_name": "DataNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "DataNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "DATANODE_role_env_safety_valve", "value": null }, { "desc": "Timeout in seconds for the Hue Thrift server running on each DataNode", "display_name": "Hue Thrift Server Timeout", "name": "dfs_thrift_timeout", "value": "60" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_data_node_log_directory_free_space", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Block Count heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Block Count", "name": "role_health_suppression_data_node_block_count", "value": "false" }, { "desc": "Port for the various DataNode Protocols. Combined with the DataNode's hostname to build its IPC port address.", "display_name": "DataNode Protocol Port", "name": "dfs_datanode_ipc_port", "value": "50020" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: DataNode Log Directory", "name": "role_config_suppression_datanode_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of DataNode in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of DataNode in Bytes", "name": "role_config_suppression_datanode_java_heapsize", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "datanode_config_safety_valve", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Only used when the DataNode Volume Choosing Policy is set to Available Space. Controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations.", "display_name": "Available Space Policy Balanced Preference", "name": "dfs_datanode_available_space_balanced_preference", "value": "0.75" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_data_node_unexpected_exits", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "datanode_web_metric_collection_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/impala-service.json0000666000175100017510000012446713245514472025312 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Impala Llama ApplicationMaster Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Llama ApplicationMaster Health", "name": "service_health_suppression_impala_llamas_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Pattern parameter.", "display_name": "Suppress Parameter Validation: LDAP Pattern", "name": "service_config_suppression_ldap_bind_pattern", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Bypass Hive Metastore Validator configuration validator.", "display_name": "Suppress Configuration Validator: Bypass Hive Metastore Validator", "name": "service_config_suppression_impala_bypass_hms_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "The user that this Impala's processes should run as (except Llama, which has its own user).", "display_name": "Impala System User (except Llama)", "name": "process_username", "value": "impala" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for Impala events:\n

\n\n
    \n
  • userName: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • operation: the Impala operation being performed.
  • \n
  • databaseName: the databaseName for the operation.
  • \n
  • tableName: the tableName for the operation.
  • \n
\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": null }, { "desc": "When set, this value is appended to all usernames before authenticating with the LDAP server. For example, if this parameter is set to \"my.domain.com\", and the user authenticating to the Impala daemon is \"mike\", then \"mike@my.domain.com\" is passed to the LDAP server. If this field is unset, the username remains unaltered before being passed to the LDAP server. This parameter is mutually exclusive with LDAP BaseDN and LDAP Pattern.", "display_name": "LDAP Domain", "name": "ldap_domain", "value": null }, { "desc": "Name of the HDFS service that this Impala service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Maximum amount of time (in milliseconds) that a request waits to be admitted before timing out. Must be a positive integer.", "display_name": "Admission Control Queue Timeout", "name": "admission_control_queue_timeout", "value": "60000" }, { "desc": "Use Dynamic Resource Pools to configure Impala admission control or RM for this Impala service. These features are only supported in Impala 1.3 or higher deployments.", "display_name": "Enable Dynamic Resource Pools", "name": "admission_control_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Local Path Access User Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Local Path Access User Validator", "name": "service_config_suppression_local_path_access_user_validator", "value": "false" }, { "desc": "Use Sentry to enable role-based, fine-grained authorization. This configuration enables Sentry using policy files. To enable Sentry using Sentry service instead, add Sentry service as a dependency to Impala service. Sentry service provides concurrent and secure access to authorization policy metadata and is the recommended option for enabling Sentry. Sentry is supported only on Impala 1.1 or later deployments.", "display_name": "Enable Sentry Authorization using Policy Files", "name": "sentry_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml parameter.", "display_name": "Suppress Parameter Validation: Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "service_config_suppression_impalad_sentry_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Impala Daemon command-line flags. Applies to all roles in this service. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "impala_cmd_args_safety_valve", "value": null }, { "desc": "Timeout in milliseconds for all HBase RPCs made by Impala. Overrides configuration in HBase service.", "display_name": "HBase RPC Timeout", "name": "hbase_rpc_timeout", "value": "3000" }, { "desc": "Use debug build of Impala binaries when starting roles. Useful when performing diagnostic activities to get more information in the stacktrace or core dump.", "display_name": "Use Debug Build", "name": "use_debug_build", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Proxy User Configuration parameter.", "display_name": "Suppress Parameter Validation: Proxy User Configuration", "name": "service_config_suppression_impala_authorized_proxy_user_config", "value": "false" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Local path to the private key that matches the certificate specified in the Certificate for Clients. This file must be in PEM format, and is required if the SSL/TLS Certificate for Clients is supplied.", "display_name": "SSL/TLS Private Key for Clients", "name": "ssl_private_key", "value": null }, { "desc": "Whether to suppress the results of the Impala Catalog Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Catalog Server Health", "name": "service_health_suppression_impala_catalogserver_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the LDAP Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: LDAP Configuration Validator", "name": "service_config_suppression_impala_ldap_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala System Group (except Llama) parameter.", "display_name": "Suppress Parameter Validation: Impala System Group (except Llama)", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress the results of the Assignment Locality heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Assignment Locality", "name": "service_health_suppression_impala_assignment_locality", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Daemon Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Daemon Count Validator", "name": "service_config_suppression_impalad_count_validator", "value": "false" }, { "desc": "Specifies the set of authorized proxy users (users who can impersonate other users during authorization) and whom they are allowed to impersonate. Input is a semicolon-separated list of key=value pairs of authorized proxy users to the user(s) they can impersonate. These users are specified as a comma separated list of short usernames, or '*' to indicate all users. For example: joe=alice,bob;hue=*;admin=*. Only valid when Sentry is enabled.", "display_name": "Proxy User Configuration", "name": "impala_authorized_proxy_user_config", "value": "hue=*" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Catalog Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Catalog Server Count Validator", "name": "service_config_suppression_catalogserver_count_validator", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Configures the maximum number of concurrently running queries for admission control when using a single pool. -1 indicates no limit and 0 indicates all incoming requests will be rejected. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Max Running Queries", "name": "admission_control_single_pool_max_requests", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_impala_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Impala StateStore Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala StateStore Health", "name": "service_health_suppression_impala_statestore_health", "value": "false" }, { "desc": "Name of YARN service to use for resource management integration between Impala and YARN. This service dependency and the existence of a Llama role is required for using said integration.", "display_name": "YARN Service for Resource Management", "name": "yarn_service", "value": null }, { "desc": "Name of the Sentry service that this Impala service instance depends on. If selected, Impala uses this Sentry service to look up authorization privileges. Before enabling Sentry, read the requirements and configuration steps in Setting Up The Sentry Service.", "display_name": "Sentry Service", "name": "sentry_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties parameter.", "display_name": "Suppress Parameter Validation: Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "service_config_suppression_navigator_lineage_client_config_safety_valve", "value": "false" }, { "desc": "Time in seconds before Impala Daemon or Catalog Server times out with the StateStore.", "display_name": "StateStoreSubscriber Timeout", "name": "statestore_subscriber_timeout", "value": "30" }, { "desc": "Configures the max memory of running queries for admission control when using a single pool. -1 or 0 indicates no limit. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Mem Limit", "name": "admission_control_single_pool_mem_limit", "value": "-1" }, { "desc": "Maximum number of HBase client retries for Impala. Used as a maximum for all operations such as fetching of the root region from the root RegionServer, getting a cell's value, and starting a row update. Overrides configuration in the HBase service.", "display_name": "Maximum HBase Client Retries", "name": "hbase_client_retries_number", "value": "3" }, { "desc": "Controls the aggregate metrics generated for Impala queries. The structure is a JSON list of the attributes to aggregate and the entities to aggregate to. For example, if the attributeName is 'hdfs_bytes_read' and the aggregationTargets is ['USER'] then the Service Monitor will create the metric 'impala_query_hdfs_bytes_read_rate' and, every ten minutes, will record the total hdfs bytes read for each user across all their Impala queries. By default it will also record the number of queries issues ('num_impala_queries_rate') for both users and pool. For a full list of the supported attributes see the Impala search page. Note that the valid aggregation targets are USER, YARN_POOL, and IMPALA (the service), and that these aggregate metrics can be viewed on both the reports and charts search pages.", "display_name": "Impala Query Aggregates", "name": "impala_query_aggregates", "value": "[\n {\n \"attributeName\": \"hdfs_bytes_read\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"hdfs_bytes_written\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"thread_cpu_time\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"bytes_streamed\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n {\n \"attributeName\": \"cm_cpu_milliseconds\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"IMPALA\", \"CLUSTER\"]\n },\n { \"attributeName\": \"query_duration\",\n \"aggregationTargets\": [\"USER\", \"YARN_POOL\", \"IMPALA\", \"CLUSTER\"]\n }\n]\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Query Aggregates parameter.", "display_name": "Suppress Parameter Validation: Impala Query Aggregates", "name": "service_config_suppression_impala_query_aggregates", "value": "false" }, { "desc": "The health test thresholds of the overall Impala Llama ApplicationMaster health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Impala Llama ApplicationMasters falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Impala Llama ApplicationMasters falls below the critical threshold.", "display_name": "Healthy Impala Llama ApplicationMaster Monitoring Thresholds", "name": "impala_llamas_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala System User (except Llama) parameter.", "display_name": "Suppress Parameter Validation: Impala System User (except Llama)", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Daemon Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Health", "name": "service_health_suppression_impala_impalads_healthy", "value": "false" }, { "desc": "Local path to the X509 certificate that will identify the Impala daemon to clients during SSL/TLS connections. This file must be in PEM format.", "display_name": "SSL/TLS Certificate for Clients", "name": "ssl_server_certificate", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SSL/TLS CA Certificate parameter.", "display_name": "Suppress Parameter Validation: SSL/TLS CA Certificate", "name": "service_config_suppression_ssl_client_ca_certificate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SSL/TLS Private Key for Clients parameter.", "display_name": "Suppress Parameter Validation: SSL/TLS Private Key for Clients", "name": "service_config_suppression_ssl_private_key", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "A list specifying the rules to run to determine which Fair Scheduler configuration to use. Typically edited using the Rules configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Configuration Rules", "name": "impala_schedule_rules", "value": "[]" }, { "desc": "When checked, LDAP-based authentication for users is enabled. Usernames and passwords are transmitted in the clear unless encryption is turned on. To encrypt the network traffic from the Impala daemon to the LDAP server, use either an ldaps:// URI or select 'Enable LDAP TLS'. To encrypt network traffic from clients to the Impala daemon, specify 'Enable TLS/SSL for Impala Client Services'.", "display_name": "Enable LDAP Authentication", "name": "enable_ldap_auth", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala StateStore Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala StateStore Count Validator", "name": "service_config_suppression_statestore_count_validator", "value": "false" }, { "desc": "Use Impala Admission Control to throttle Impala requests. Unless 'Enable Dynamic Resource Pools' is enabled, Impala uses a single, default pool that is configured using the Single Pool configurations below. These features are only supported in Impala 1.3 or higher deployments.", "display_name": "Enable Impala Admission Control", "name": "all_admission_control_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Domain parameter.", "display_name": "Suppress Parameter Validation: LDAP Domain", "name": "service_config_suppression_ldap_domain", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP BaseDN parameter.", "display_name": "Suppress Parameter Validation: LDAP BaseDN", "name": "service_config_suppression_ldap_basedn", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into sentry-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "Impala Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml", "name": "impalad_sentry_safety_valve", "value": null }, { "desc": "Configures the maximum number of queued queries for admission control when using a single pool. -1 or 0 disables queuing, i.e. incoming requests are rejected if they can not be executed immediately. Ignored when Dynamic Resource Pools for Admission Control is enabled.", "display_name": "Single Pool Max Queued Queries", "name": "admission_control_single_pool_max_queued", "value": "200" }, { "desc": "The health test thresholds of the overall Impala Daemon health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Impala Daemons falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Impala Daemons falls below the critical threshold.", "display_name": "Healthy Impala Daemon Monitoring Thresholds", "name": "impala_impalads_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the Enable HDFS Block Metadata API Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Enable HDFS Block Metadata API Configuration Validator", "name": "service_config_suppression_impala_hdfs_dfs_datanode_hdfs_blocks_metadata_enabled_set_validator", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Impala Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "impala_service_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_impala_cmd_args_safety_valve", "value": "false" }, { "desc": "Number of minutes between reestablishing our ticket with the Kerberos server.", "display_name": "Kerberos Re-init Interval", "name": "kerberos_reinit_interval", "value": "60" }, { "desc": "If true, attempts to establish a TLS (Transport Layer Security) connection with the LDAP server. Only supported in Impala 1.4 or CDH 5.1 or higher. Not required when using an LDAP URI with prefix ldaps://, because that already specifies TLS.", "display_name": "Enable LDAP TLS", "name": "enable_ldap_tls", "value": "false" }, { "desc": "When set, this parameter allows arbitrary mapping from usernames into a Distinguished Name (DN). The string specified must have a placeholder named \"#UID\" inside it, and that #UID is replaced with the username. For example, you could mimic the behavior of LDAP BaseDN by specifying \"uid=#UID,ou=People,dc=cloudera,dc=com\". When the username of \"mike\" comes in, it replaces the #UID and the result is \"uid=mike,ou=People,dc=cloudera,dc=com\". This option should be used when more control over the DN is needed. This parameter is mutually exclusive with LDAP Domain and LDAP BaseDN.", "display_name": "LDAP Pattern", "name": "ldap_bind_pattern", "value": null }, { "desc": "The URI of the LDAP server to use if LDAP is enabled. The URI must be prefixed with ldap:// or ldaps://. The URI can optionally specify the port, for example: ldap://ldap_server.example.com:389. ldaps:// is only supported in Impala 1.4 or CDH 5.1 or higher, and usually requires that you specify a port.", "display_name": "LDAP URI", "name": "impala_ldap_uri", "value": null }, { "desc": "The health test thresholds for the assignment locality health test. Specified as a percentage of total assignments.", "display_name": "Assignment Locality Ratio Thresholds", "name": "impala_assignment_locality_thresholds", "value": "{\"critical\":\"5.0\",\"warning\":\"80.0\"}" }, { "desc": "Controls which queries a non-admin user can see in the queries list view", "display_name": "Non-Admin Users Query List Visibility Settings", "name": "user_query_list_settings", "value": "ALL" }, { "desc": "The location on disk of the certificate, in PEM format, used to confirm the authenticity of SSL/TLS servers that the Impala daemons might connect to.", "display_name": "SSL/TLS CA Certificate", "name": "ssl_client_ca_certificate", "value": null }, { "desc": "Used to generate a core dump to get more information about an Impala crash. Unless otherwise configured systemwide using /proc/sys/kernel/core_pattern, the dump is generated in the 'current directory' of the Impala process (usually a subdirectory of the /var/run/cloudera-scm-agent/process directory). The core file can be very large.", "display_name": "Enable Core Dump", "name": "enable_core_dump", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "true" }, { "desc": "Password protecting the SSL/TLS Private Key for Clients. Leave blank if there is no password.", "display_name": "SSL/TLS Private Key Password for Clients", "name": "ssl_private_key_password", "value": null }, { "desc": "The minimum number of assignments that must occur during the test time period before the threshold values will be checked. Until this number of assignments have been observed in the test time period the health test will be disabled.", "display_name": "Assignment Locality Minimum Assignments", "name": "impala_assignment_locality_minimum", "value": "10" }, { "desc": "Enable collection of lineage from the service's roles.", "display_name": "Enable Lineage Collection", "name": "navigator_lineage_enabled", "value": "true" }, { "desc": "JSON representation of all the configurations that the Fair Scheduler can take on across all schedules. Typically edited using the Pools configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler Allocations", "name": "impala_scheduled_allocations", "value": "{\"defaultMinSharePreemptionTimeout\":null,\"defaultQueueSchedulingPolicy\":null,\"fairSharePreemptionTimeout\":null,\"queueMaxAMShareDefault\":null,\"queueMaxAppsDefault\":null,\"queuePlacementRules\":null,\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"minSharePreemptionTimeout\":null,\"name\":\"root\",\"queues\":[{\"aclAdministerApps\":null,\"aclSubmitApps\":null,\"minSharePreemptionTimeout\":null,\"name\":\"default\",\"queues\":[],\"schedulablePropertiesList\":[{\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null}],\"schedulablePropertiesList\":[{\"impalaMaxMemory\":null,\"impalaMaxQueuedQueries\":null,\"impalaMaxRunningQueries\":null,\"maxAMShare\":null,\"maxResources\":null,\"maxRunningApps\":null,\"minResources\":null,\"scheduleName\":\"default\",\"weight\":null}],\"schedulingPolicy\":null}],\"userMaxAppsDefault\":null,\"users\":[]}" }, { "desc": "When computing the overall IMPALA health, consider Impala Catalog Server's health", "display_name": "Impala Catalog Server Role Health Test", "name": "impala_catalogserver_health_enabled", "value": "true" }, { "desc": "Name of the HBase service that this Impala service instance depends on", "display_name": "HBase Service", "name": "hbase_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Permissions Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Permissions Validator", "name": "service_config_suppression_short_circuit_reads_data_directory_permissions_validator", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • userName: treats the field value as a userName, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare Impala events:\n

\n\n
    \n
  • operation: the Impala operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • databaseName: the database affected by the operation.
  • \n
  • tableName: the table affected by the operation.
  • \n
  • objectType: the type of object affected by the operation.
  • \n
  • privilege: the privilege associated with the operation.
  • \n
\n\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": null }, { "desc": "Name of the Hive service that this Impala service instance depends on", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Impala Llama ApplicationMaster Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Llama ApplicationMaster Count Validator", "name": "service_config_suppression_llama_count_validator", "value": "false" }, { "desc": "When set, this parameter is used to convert the username into the LDAP Distinguished Name (DN), so that the resulting DN looks like uid=username,X. For example, if this parameter is set to \"ou=People,dc=cloudera,dc=com\", and the username passed in is \"mike\", the resulting authentication passed to the LDAP server looks like \"uid=mike,ou=People,dc=cloudera,dc=com\". This parameter is frequently useful when authenticating against an OpenLDAP server. This parameter is mutually exclusive with LDAP Domain and LDAP Pattern.", "display_name": "LDAP BaseDN", "name": "ldap_baseDN", "value": null }, { "desc": "Name of the ZooKeeper service to use for leader election and fencing when Llama is configured for high availability. This service dependency is required when more than one Llama role is present.", "display_name": "ZooKeeper Service for Llama HA", "name": "zookeeper_service", "value": null }, { "desc": "The time period over which to compute the assignment locality ratio. Specified in minutes.", "display_name": "Assignment Locality Monitoring Period", "name": "impala_assignment_locality_window", "value": "15" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "impala" }, { "desc": "The group that this Impala's processes should run as (except Llama, which has its own group).", "display_name": "Impala System Group (except Llama)", "name": "process_groupname", "value": "impala" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SSL/TLS Certificate for Clients parameter.", "display_name": "Suppress Parameter Validation: SSL/TLS Certificate for Clients", "name": "service_config_suppression_ssl_server_certificate", "value": "false" }, { "desc": "Controls which queries admin users can see in the queries list view", "display_name": "Admin Users Query List Visibility Settings", "name": "admin_query_list_settings", "value": "ALL" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Encrypt communication between clients (like ODBC, JDBC, and the Impala shell) and the Impala daemon using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Impala Client Services", "name": "client_services_ssl_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SSL/TLS Private Key Password for Clients parameter.", "display_name": "Suppress Parameter Validation: SSL/TLS Private Key Password for Clients", "name": "service_config_suppression_ssl_private_key_password", "value": "false" }, { "desc": "When computing the overall IMPALA health, consider Impala StateStore's health", "display_name": "Impala StateStore Role Health Test", "name": "impala_statestore_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP URI parameter.", "display_name": "Suppress Parameter Validation: LDAP URI", "name": "service_config_suppression_impala_ldap_uri", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Sentry Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Sentry Validator", "name": "service_config_suppression_impala_sentry_validator", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.lineage.client.properties.", "display_name": "Impala Client Advanced Configuration Snippet (Safety Valve) for navigator.lineage.client.properties", "name": "navigator_lineage_client_config_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hive-hivemetastore.json0000666000175100017510000006022013245514472026203 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Hive Metastore Server", "name": "hive_metastore_java_opts", "value": "-XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Hive Metastore Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Hive Metastore Server", "name": "role_config_suppression_hive_metastore_java_opts", "value": "false" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "hivemetastore_pause_duration_window", "value": "5" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hivemetastore_host_health", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hivemetastore_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Maximum number of worker threads in the Hive Metastore Server's thread pool", "display_name": "Max Hive Metastore Server Threads", "name": "hive_metastore_max_threads", "value": "100000" }, { "desc": "Enables the health test that checks that basic Hive Metastore operations succeed", "display_name": "Hive Metastore Canary Health Test", "name": "metastore_canary_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hivemetastore_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hive_metrics_sample_file_location", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Minimum number of worker threads in the Hive Metastore Server's thread pool", "display_name": "Min Hive Metastore Server Threads", "name": "hive_metastore_min_threads", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_metastore_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hivemetastore_swap_memory_usage", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hivemetastore_file_descriptor", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hivemetastore_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Log Directory", "name": "role_config_suppression_hive_log_dir", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Controls whether the Hive metrics subsystem is enabled for the role.", "display_name": "Enable Metrics Subsystem", "name": "hive_metrics_enabled", "value": "true" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "When computing the overall Hive Metastore Server health, consider the host's health.", "display_name": "Hive Metastore Server Host Health Test", "name": "hivemetastore_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "hivemetastore_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Hive Metastore Server in Bytes", "name": "hive_metastore_java_heapsize", "value": "8589934592" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hivemetastore_heap_dump_directory_free_space", "value": "false" }, { "desc": "Port on which Hive Metastore Server will listen for connections.", "display_name": "Hive Metastore Server Port", "name": "hive_metastore_port", "value": "9083" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_hivemetastore_pause_duration", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for Hive Metastore Server logs. Typically used by log4j or logback.", "display_name": "Hive Metastore Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_metastore_config_safety_valve", "value": null }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Hive Metastore Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Hive Metastore Canary", "name": "role_health_suppression_hivemetastore_canary_health", "value": "false" }, { "desc": "Enables the health test that the Hive Metastore Server's process state is consistent with the role configuration", "display_name": "Hive Metastore Server Process Health Test", "name": "hivemetastore_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hivemetastore_unexpected_exits", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Logging Frequency", "name": "hive_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_metastore_config_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The maximum number of rolled log files to keep for Hive Metastore Server logs. Typically used by log4j or logback.", "display_name": "Hive Metastore Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hive/metrics-hivemetastore/metrics.log. The setting only has an effect if \"Enable Metrics Subsystem\" is set to true.", "display_name": "Metrics Sample File Location", "name": "hive_metrics_sample_file_location", "value": null }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Directory where Hive Metastore Server will place its log files.", "display_name": "Hive Metastore Server Log Directory", "name": "hive_log_dir", "value": "/var/log/hive" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The delegation token store implementation class. Use DBTokenStore for Highly Available Metastore Configuration.", "display_name": "Hive Metastore Delegation Token Store", "name": "hive_metastore_delegation_token_store", "value": "org.apache.hadoop.hive.thrift.MemoryTokenStore" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for Hive Metastore Server logs", "display_name": "Hive Metastore Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Hive Metastore Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_metastore_env_safety_valve", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hbase-service.json0000666000175100017510000013423213245514472025120 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "Timeout (in ms) for the distributed log splitting manager to receive response from a worker.", "display_name": "SplitLog Manager Timeout", "name": "hbase_service_splitlog_manager_timeout", "value": "120000" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for HBase events:\n

\n
    \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • username: the user performing the action.
  • \n
  • tableName: the table affected by the operation.
  • \n
  • family: the column family affected by the operation.
  • \n
  • qualifier: the qualifier the operation.
  • \n
  • action: the action being performed.
  • \n
\n\n

\nThe default HBase audit event filter discards events that affect the internal\n-ROOT-, .META. and _acl_ tables.\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default HBase audit event filter discards events that affect the \",\n \"internal -ROOT-, .META. and _acl_ tables.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"tableName\", \"match\" : \"(?:-ROOT-|.META.|_acl_|hbase:meta|hbase:acl)\" }\n ]\n }\n ]\n}\n" }, { "desc": "Enables the canary that checks HBase region availability by scanning a row from every region.", "display_name": "HBase Region Health Canary", "name": "hbase_region_health_canary_enabled", "value": "true" }, { "desc": "Name of the HDFS service that this HBase service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Period of time, in milliseconds, to pause between connection retries to ZooKeeper. Used together with ${zookeeper.retries} in an exponential backoff fashion when making queries to ZooKeeper.", "display_name": "ZooKeeper Connection Retry Pause Duration", "name": "zookeeper_pause", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "When computing the overall HBase cluster health, consider the active HBase Master's health.", "display_name": "Active Master Health Test", "name": "hbase_master_health_enabled", "value": "true" }, { "desc": "Size of the threadpool used for hedged reads in hdfs clients. If a read from a block is slow, a parallel 'hedged' read will be started against a different block replica. The first one to return with a result is used while the other one is cancelled. This 'hedged' read feature helps rein in the outliers. A value of zero disables the feature.", "display_name": "HDFS Hedged Read Threadpool Size", "name": "hbase_server_dfs_client_hedged_read_threadpool_size", "value": "0" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hbase/audit" }, { "desc": "Use this to enable Http server usage on thrift, which is mainly needed for \"doAs\" functionality.", "display_name": "Enable HBase Thrift Http Server", "name": "hbase_thriftserver_http", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase User to Impersonate parameter.", "display_name": "Suppress Parameter Validation: HBase User to Impersonate", "name": "service_config_suppression_hbase_user_to_impersonate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the AWS S3 Secret Access Key for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: AWS S3 Secret Access Key for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_secret_access_key", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Timeout for all HBase RPCs in milliseconds.", "display_name": "RPC Timeout", "name": "hbase_rpc_timeout", "value": "60000" }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted web UIs.", "display_name": "TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "hbase_ssl_server_safety_valve", "value": null }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HBase Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hbase_service_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "service_config_suppression_hbase_service_config_safety_valve", "value": "false" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "The number of times to retry connections to ZooKeeper. Used for reading and writing root region location. Used together with ${zookeeper.pause} in an exponential backoff fashion when making queries to ZooKeeper.", "display_name": "ZooKeeper Connection Retries", "name": "zookeeper_retries", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Timeout for graceful shutdown of this HBase service. Once this timeout is reached, any remaining running roles are abruptly shutdown. A value of 0 means no timeout.", "display_name": "Graceful Shutdown Timeout", "name": "hbase_graceful_stop_timeout", "value": "180" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_hbase_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The health test thresholds of the overall RegionServer health. The check returns \"Concerning\" health if the percentage of \"Healthy\" RegionServers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" RegionServers falls below the critical threshold.", "display_name": "Healthy RegionServer Monitoring Thresholds", "name": "hbase_regionservers_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress the results of the RegionServer Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: RegionServer Health", "name": "service_health_suppression_hbase_region_servers_healthy", "value": "false" }, { "desc": "An alert is published if the HBase region health canary detects at least this many unhealthy regions. This setting takes precedence over the hbase_canary_alert_unhealthy_region_percent_threshold config.", "display_name": "HBase Canary Unhealthy Region Count Alert Threshold", "name": "hbase_canary_alert_unhealthy_region_count_threshold", "value": null }, { "desc": "Set to true to cause the hosting server (Master or RegionServer) to abort if a coprocessor throws a Throwable object that is not IOException or a subclass of IOException. Setting it to true might be useful in development environments where one wants to terminate the server as soon as possible to simplify coprocessor failure analysis.", "display_name": "HBase Coprocessor Abort on Error", "name": "hbase_coprocessor_abort_on_error", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Write buffer size in bytes. A larger buffer requires more memory on both the client and the server because the server instantiates the passed write buffer to process it but reduces the number of remote procedure calls (RPC). To estimate the amount of server memory used, multiply the value of 'hbase.client.write.buffer' by the value of 'hbase.regionserver.handler.count'.", "display_name": "HBase Client Write Buffer", "name": "hbase_client_write_buffer", "value": "2097152" }, { "desc": "The user the management services impersonate when connecting to HBase. If no value is specified, the HBase superuser is used.", "display_name": "HBase User to Impersonate", "name": "hbase_user_to_impersonate", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Amazon S3 Path for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: Amazon S3 Path for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_path", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "hbase" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Scheduler Pool for Remote Snapshots in AWS S3 parameter.", "display_name": "Suppress Parameter Validation: Scheduler Pool for Remote Snapshots in AWS S3", "name": "service_config_suppression_hbase_snapshot_s3_scheduler_pool", "value": "false" }, { "desc": "Allow indexing of tables in HBase by Lily HBase Indexer. Note: Replication must be enabled for indexing to work.", "display_name": "Enable Indexing", "name": "hbase_enable_indexing", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Superusers parameter.", "display_name": "Suppress Parameter Validation: HBase Superusers", "name": "service_config_suppression_hbase_superuser", "value": "false" }, { "desc": "Password for the server keystore file used for encrypted web UIs.", "display_name": "TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Ratio of Lily HBase Indexers used by each HBase RegionServer while doing replication.", "display_name": "Replication Source Ratio", "name": "hbase_replication_source_ratio", "value": "1.0" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "hbase_core_site_safety_valve", "value": null }, { "desc": "The timeout before injecting a snapshot timeout error when waiting for a snapshot completion.", "display_name": "HBase Master Snapshot Waiting Timeout", "name": "hbase_snapshot_master_timeoutMillis", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Authentication And Authorization Validation configuration validator.", "display_name": "Suppress Configuration Validator: HBase Authentication And Authorization Validation", "name": "service_config_suppression_hbase_authentication_and_authorization_validator", "value": "false" }, { "desc": "The maximum amount of time the Hbase RegionServer waits for a snapshot to complete.", "display_name": "HBase RegionServer Snapshot Timeout", "name": "hbase_snapshot_region_timeout", "value": "60000" }, { "desc": "If this is set to \"kerberos\", HBase REST Server will authenticate its clients. HBase Proxy User Hosts and Groups should be configured to allow specific users to access HBase through REST Server.", "display_name": "HBase REST Authentication", "name": "hbase_restserver_security_authentication", "value": "simple" }, { "desc": "Specifies the combined maximum allowed size of a KeyValue instance. This option configures an upper boundary for a single entry saved in a storage file. This option prevents a region from splitting if the data is too large. Set this option to a fraction of the maximum region size. To disable this check, use a value of zero or less.", "display_name": "Maximum Size of HBase Client KeyValue", "name": "hbase_client_keyvalue_maxsize", "value": "10485760" }, { "desc": "Enable snapshots. Disabling snapshots requires deletion of all snapshots before restarting the HBase master; the HBase master will not start if snapshots are disabled and snapshots exist.", "display_name": "Enable Snapshots", "name": "hbase_snapshot_enabled", "value": "true" }, { "desc": "Maximum number of client retries. Used as a maximum for all operations such as fetching of the root region from the root RegionServer, getting a cell's value, and starting a row update.", "display_name": "Maximum HBase Client Retries", "name": "hbase_client_retries_number", "value": "35" }, { "desc": "Set to true to use HBase Secure RPC Engine for remote procedure calls (RPC). This is only effective in simple authentication mode. Does not provide authentication for RPC calls, but provides user information in the audit logs. Changing this setting requires a restart of this and all dependent services and redeployment of client configurations, along with a restart of the Service Monitor management role.", "display_name": "HBase Secure RPC Engine", "name": "hbase_secure_rpc_engine", "value": "false" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "zookeeper.recovery.retry0zookeeper.recovery.retry.intervalmill3000hbase.zookeeper.recoverable.waittime1000zookeeper.session.timeout30000hbase.rpc.timeout10000hbase.client.retries.number1hbase.client.rpc.maxattempts1hbase.client.operation.timeout10000" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Amazon S3 Access Key ID for Remote Snapshots parameter.", "display_name": "Suppress Parameter Validation: Amazon S3 Access Key ID for Remote Snapshots", "name": "service_config_suppression_hbase_snapshot_s3_access_key_id", "value": "false" }, { "desc": "Whether to suppress the results of the HBase Master Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HBase Master Health", "name": "service_health_suppression_hbase_master_health", "value": "false" }, { "desc": "Enable HBase authorization.", "display_name": "HBase Secure Authorization", "name": "hbase_security_authorization", "value": "false" }, { "desc": "Period of time, in milliseconds, to pause between searches for work. Used as a sleep interval by service threads such as a META scanner and log roller.", "display_name": "HBase Server Thread Wake Frequency", "name": "hbase_server_thread_wakefrequency", "value": "10000" }, { "desc": "Whether to suppress configuration warnings produced by the HBase Thrift Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase Thrift Server Count Validator", "name": "service_config_suppression_hbasethriftserver_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Root Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Root Directory", "name": "service_config_suppression_hdfs_rootdir", "value": "false" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the RegionServer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: RegionServer Count Validator", "name": "service_config_suppression_regionserver_count_validator", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode Rootserver parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode Rootserver", "name": "service_config_suppression_zookeeper_znode_rootserver", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Znode Parent parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Znode Parent", "name": "service_config_suppression_zookeeper_znode_parent", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Region Health Canary Exclude Tables parameter.", "display_name": "Suppress Parameter Validation: HBase Region Health Canary Exclude Tables", "name": "service_config_suppression_hbase_region_health_canary_exclude_tables", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HBase Proxy User Groups", "name": "service_config_suppression_hbase_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Service Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_hbase_core_site_safety_valve", "value": "false" }, { "desc": "The root znode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file paths are configured with a relative path, so they will all go under this directory unless changed.", "display_name": "ZooKeeper Znode Parent", "name": "zookeeper_znode_parent", "value": "/hbase" }, { "desc": "Access key ID required to access Amazon S3 to store remote snapshots.", "display_name": "Amazon S3 Access Key ID for Remote Snapshots", "name": "hbase_snapshot_s3_access_key_id", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "AWS secret access key required to access S3 to store remote snapshots.", "display_name": "AWS S3 Secret Access Key for Remote Snapshots", "name": "hbase_snapshot_s3_secret_access_key", "value": null }, { "desc": "Number of rows to fetch when calling next on a scanner if it is not served from memory. Higher caching values enable faster scanners but require more memory and some calls of next may take longer when the cache is empty.", "display_name": "HBase Client Scanner Caching", "name": "hbase_client_scanner_caching", "value": "100" }, { "desc": "Tables to exclude in the HBase Region Health Canary which will scan a row from every region.", "display_name": "HBase Region Health Canary Exclude Tables", "name": "hbase_region_health_canary_exclude_tables", "value": "" }, { "desc": "The maximum amount of time the HBase master waits for a snapshot to complete.", "display_name": "HBase Master Snapshot Timeout", "name": "hbase_snapshot_master_timeout_millis", "value": "60000" }, { "desc": "Allow HBase tables to be replicated.", "display_name": "Enable Replication", "name": "hbase_enable_replication", "value": "false" }, { "desc": "Start a process to periodically check that RegionServer is alive when RegionServer is started. Note: This canary is different from the Cloudera Service Monitoring canary and is provided by the HBase service itself.", "display_name": "Enable HBase Canary", "name": "hbase_regionserver_enable_canary", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "true" }, { "desc": "Path to ZooKeeper Node holding root region location. This is written by the HBase Master and read by clients and RegionServers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, the root location is stored at /hbase/root-region-server.", "display_name": "ZooKeeper Znode Rootserver", "name": "zookeeper_znode_rootserver", "value": "root-region-server" }, { "desc": "When computing the overall HBase cluster health, consider the health of the backup HBase Masters.", "display_name": "Backup Masters Health Test", "name": "hbase_backup_masters_health_enabled", "value": "true" }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted web UIs.", "display_name": "TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "An alert is published if the HBase region health canary runs slowly.", "display_name": "HBase Region Health Canary Slow Run Alert Enabled", "name": "hbase_region_health_canary_slow_run_alert_enabled", "value": "true" }, { "desc": "Name of the scheduler pool to use for MR jobs created during export/import of remote snapshots in AWS S3.", "display_name": "Scheduler Pool for Remote Snapshots in AWS S3", "name": "hbase_snapshot_s3_scheduler_pool", "value": null }, { "desc": "Configure the type of encrypted communication to be used with RPC.", "display_name": "HBase Transport Security", "name": "hbase_rpc_protection", "value": "authentication" }, { "desc": "An alert is published if the HBase region health canary detects at least this percentage of total regions are unhealthy. This threshold is used if the explicit count is not set via the hbase_canary_alert_unhealthy_region_count_threshold config.", "display_name": "HBase Canary Unhealthy Region Percentage Alert Threshold", "name": "hbase_canary_alert_unhealthy_region_percent_threshold", "value": "0.1" }, { "desc": "Duration to wait before starting up a 'hedged' read.", "display_name": "HDFS Hedged Read Delay Threshold", "name": "hbase_server_dfs_client_hedged_read_threshold_millis", "value": "500" }, { "desc": "Enable TLS/SSL encryption for HBase web UIs.", "display_name": "Web UI TLS/SSL Encryption Enabled", "name": "hbase_hadoop_ssl_enabled", "value": "false" }, { "desc": "List of users or groups, who are allowed full privileges, regardless of stored ACLs, across the cluster. Only used when HBase security is enabled.", "display_name": "HBase Superusers", "name": "hbase_superuser", "value": "" }, { "desc": "Use this to allow proxy users on thrift gateway, which is mainly needed for \"doAs\" functionality.", "display_name": "Enable HBase Thrift Proxy Users", "name": "hbase_thriftserver_support_proxyuser", "value": "false" }, { "desc": "Maximum number of hlog entries to replicate in one go. If this is large, and a consumer takes a while to process the events, the HBase RPC call will time out.", "display_name": "Replication Batch Size", "name": "hbase_replication_source_nb_capacity", "value": "1000" }, { "desc": "Whether to suppress configuration warnings produced by the Master Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Master Count Validator", "name": "service_config_suppression_master_count_validator", "value": "false" }, { "desc": "ZooKeeper session timeout in milliseconds. HBase passes this to the ZooKeeper quorum as the suggested maximum time for a session. See http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions The client sends a requested timeout, the server responds with the timeout that it can give the client.", "display_name": "ZooKeeper Session Timeout", "name": "zookeeper_session_timeout", "value": "60000" }, { "desc": "Enable HBase row-level authorization.", "display_name": "HBase Row-Level Authorization", "name": "hbase_row_level_authorization", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HBase user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'. Note: This property is used only if HBase REST/Thrift Server Authentication is enabled.", "display_name": "HBase Proxy User Hosts", "name": "hbase_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HBase Proxy User Hosts", "name": "service_config_suppression_hbase_proxy_user_hosts_list", "value": "false" }, { "desc": "Amazon S3 path where remote snapshots should be stored.", "display_name": "Amazon S3 Path for Remote Snapshots", "name": "hbase_snapshot_s3_path", "value": null }, { "desc": "Comma-delimited list of groups that you want to allow the HBase user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'. Note: This property is used only if HBase REST/Thrift Server Authentication is enabled.", "display_name": "HBase Proxy User Groups", "name": "hbase_proxy_user_groups_list", "value": "*" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • username: treats the field value as a user name, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can used to compare HBase events:\n

\n\n
    \n
  • operation: the HBase operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • tableName: the name of the table affected by the operation.
  • \n
  • family: the column family affected by the operation.
  • \n
  • qualifier: the qualifier of the operation.
  • \n
\n\n

\nThe default event tracker for HBase services defines equality by comparing the\nusername, operation, table name, family, and qualifier of the events.\n

\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": "{\n \"comment\" : [\n \"The default event tracker for HBase services defines equality by \",\n \"comparing the username, operation, table name, family, and qualifier of \",\n \"the events.\"\n ],\n \"timeToLive\" : 60000,\n \"fields\" : [\n { \"type\": \"value\", \"name\" : \"tableName\" },\n { \"type\": \"value\", \"name\" : \"family\" },\n { \"type\": \"value\", \"name\" : \"qualifier\" },\n { \"type\": \"value\", \"name\" : \"operation\" },\n { \"type\": \"username\", \"name\" : \"username\" }\n ]\n}\n" }, { "desc": "The tolerance window that will be used in HBase service tests that depend on detection of the active HBase Master.", "display_name": "HBase Active Master Detection Window", "name": "hbase_active_master_detecton_window", "value": "3" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_service_config_safety_valve", "value": null }, { "desc": "Choose the authentication mechanism used by HBase.", "display_name": "HBase Secure Authentication", "name": "hbase_security_authentication", "value": "simple" }, { "desc": "A general client pause time value. Used mostly as a time period to wait before retrying operations such as a failed get or region lookup.", "display_name": "HBase Client Pause", "name": "hbase_client_pause", "value": "100" }, { "desc": "Name of the ZooKeeper service that this HBase service instance depends on.", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "If this is set, HBase Thrift Server authenticates its clients. HBase Proxy User Hosts and Groups should be configured to allow specific users to access HBase through Thrift Server.", "display_name": "HBase Thrift Authentication", "name": "hbase_thriftserver_security_authentication", "value": "none" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hbase" }, { "desc": "Whether to suppress configuration warnings produced by the HBase REST Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase REST Server Count Validator", "name": "service_config_suppression_hbaserestserver_count_validator", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "hbase" }, { "desc": "The HDFS directory shared by HBase RegionServers.", "display_name": "HDFS Root Directory", "name": "hdfs_rootdir", "value": "/hbase" }, { "desc": "Whether to suppress configuration warnings produced by the ZooKeeper Max Session Timeout Validator configuration validator.", "display_name": "Suppress Configuration Validator: ZooKeeper Max Session Timeout Validator", "name": "service_config_suppression_zookeeper_max_session_timeout_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "HBase Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hbase_service_env_safety_valve", "value": null }, { "desc": "Whether the step to reload regions back onto the original RegionServers should be skipped during rolling restart. This can be used to increase the speed of rolling restart or upgrade operations, but can result in regions being moved multiple times, decreasing performance for clients during rolling restart.", "display_name": "Skip Region Reload During Rolling Restart", "name": "hbase_skip_reload_during_rr", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HBASE Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-gateway.json0000666000175100017510000001402413245514472024757 0ustar zuulzuul00000000000000[ { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for hdfs-site.xml.", "display_name": "HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "hdfs_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether HDFS clients will use the legacy block reader.", "display_name": "Use Legacy Blockreader", "name": "dfs_client_use_legacy_blockreader", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Trash Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Trash Enabled Validator", "name": "role_config_suppression_hdfs_trash_disabled_validator", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hdfs_client_java_heapsize", "value": "268435456" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_hdfs_client_config_safety_valve", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hadoop" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hbase_client_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Move deleted files to the trash so that they can be recovered if necessary. This client side configuration takes effect only if the HDFS service-wide trash is disabled (NameNode Filesystem Trash Interval set to 0) and is ignored otherwise. The trash is not automatically emptied when enabled with this configuration.", "display_name": "Use Trash", "name": "dfs_client_use_trash", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_client_read_shortcircuit", "value": "false" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "role_config_suppression_hdfs_client_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh", "display_name": "HDFS Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "hdfs_client_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hbase_client_java_opts", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/cdh_config.sh0000777000175100017510000000010113245514472024112 0ustar zuulzuul00000000000000#!/bin/bash tox -evenv -- python $(dirname $0)/cdh_config.py $* sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/zookeeper-server.json0000666000175100017510000011531513245514472025710 0ustar zuulzuul00000000000000[ { "desc": "The port to monitor for inter-server communication", "display_name": "Quorum Port", "name": "quorumPort", "value": "3181" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "zookeeper_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Log Directory parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Log Directory", "name": "role_config_suppression_zk_server_log_dir", "value": "false" }, { "desc": "Specifies the name of the user that has read-only privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Name of User with Read-Only access to the JMX Agent", "name": "jmx_passwd_file_readonly_user", "value": "monitorRole" }, { "desc": "Enables the health test that the Server's process state is consistent with the role configuration", "display_name": "Server Process Health Test", "name": "zookeeper_server_scm_health_enabled", "value": "true" }, { "desc": "The address (IPv4, IPv6, or hostname) to monitor for client connections. This is the address that clients attempt to connect to. This setting is optional, because by default, ZooKeeper binds in such a way that any connection to the client port for any address/interface/NIC on the server will be accepted.", "display_name": "Client Port Address", "name": "clientPortAddress", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Specifies the password of the user that has read-only privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Password of User with Read-Only Access to the JMX agent", "name": "jmx_passwd_file_readonly_user_password", "value": "MONITOR" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the Quorum Membership heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Quorum Membership", "name": "role_health_suppression_zookeeper_server_quorum_membership", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Port Address parameter.", "display_name": "Suppress Parameter Validation: Client Port Address", "name": "role_config_suppression_clientportaddress", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health check thresholds of the weighted average size of the ZooKeeper Server connection count over a recent period. See ZooKeeper Server Connection Count Monitoring Period.", "display_name": "ZooKeeper Server Connection Count Thresholds", "name": "zookeeper_server_connection_count_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into zoo.cfg for this role only.", "display_name": "Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg", "name": "zookeeper_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Transaction Log Directory.", "display_name": "Transaction Log Directory Free Space Monitoring Absolute Thresholds", "name": "zookeeper_server_data_log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The port to monitor for client connections. This is the port that clients attempt to connect to.", "display_name": "Client Port", "name": "clientPort", "value": "2181" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg parameter.", "display_name": "Suppress Parameter Validation: Server Advanced Configuration Snippet (Safety Valve) for zoo.cfg", "name": "role_config_suppression_zookeeper_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Password of User with Read-Only Access to the JMX agent parameter.", "display_name": "Suppress Parameter Validation: Password of User with Read-Only Access to the JMX agent", "name": "role_config_suppression_jmx_passwd_file_readonly_user_password", "value": "false" }, { "desc": "The disk location that ZooKeeper will use to store its database snapshots.", "display_name": "Data Directory", "name": "dataDir", "value": "/var/lib/zookeeper" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_zookeeper_server_gc_duration", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_zookeeper_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Unique identifier for each ZooKeeper server, typically starts at 1", "display_name": "ZooKeeper Server ID", "name": "serverId", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_zookeeper_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_zookeeper_server_file_descriptor", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "The disk location that ZooKeeper will use to store its transaction logs.", "display_name": "Transaction Log Directory", "name": "dataLogDir", "value": "/var/lib/zookeeper" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Data Directory.", "display_name": "Data Directory Free Space Monitoring Absolute Thresholds", "name": "zookeeper_server_data_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The maximum size, in megabytes, per log file for Server logs. Typically used by log4j or logback.", "display_name": "Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Password of user with read-write access to the JMX agent parameter.", "display_name": "Suppress Parameter Validation: Password of user with read-write access to the JMX agent", "name": "role_config_suppression_jmx_passwd_file_readwrite_user_password", "value": "false" }, { "desc": "The minimum session timeout, in milliseconds, that the ZooKeeper Server will allow the client to negotiate", "display_name": "Minimum Session Timeout", "name": "minSessionTimeout", "value": "4000" }, { "desc": "The percentage thresholds of the ratio of the maximum request latency to the maximum client-negotiable session timeout since the server was started.", "display_name": "Maximum Latency Monitoring Thresholds", "name": "zookeeper_server_max_latency_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"75.0\"}" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Enables authentication when interacting with the JMX agent on the ZooKeeper server.", "display_name": "Enable Authenticated Communication with the JMX Agent", "name": "enable_jmx_authentication", "value": "false" }, { "desc": "Enables the quorum membership check for this ZooKeeper Server.", "display_name": "Enable the Quorum Membership Check", "name": "zookeeper_server_quorum_membership_enabled", "value": "true" }, { "desc": "The port used by the ZooKeeper Server's RMI server to handle JMX RMI requests. This is added as \"-Dcom.sun.management.jmxremote.rmi.port=\" to the ZooKeeper Server's JVM command line. This has an effect only in Oracle JDK 7u4 and higher. If the setting is left blank, the JMX Remote Port value is used. If set to 0 or -1, this setting is ignored. When this setting is not in effect, the JVM uses a random port for the RMI server.", "display_name": "JMX RMI Server Port", "name": "server_jmx_rmi_port", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_zookeeper_server_host_health", "value": "false" }, { "desc": "Directory where ZooKeeper will place its log files.", "display_name": "ZooKeeper Log Directory", "name": "zk_server_log_dir", "value": "/var/log/zookeeper" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Transaction Log Directory parameter.", "display_name": "Suppress Parameter Validation: Transaction Log Directory", "name": "role_config_suppression_datalogdir", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The minimum log level for Server logs", "display_name": "Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SERVER_role_env_safety_valve", "value": null }, { "desc": "The period to review when computing the moving average of the outstanding requests queue size. Specified in minutes.", "display_name": "ZooKeeper Server Outstanding Requests Monitoring Period", "name": "zookeeper_server_outstanding_requests_window", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Name of User with Read-Write Access to the JMX Agent parameter.", "display_name": "Suppress Parameter Validation: Name of User with Read-Write Access to the JMX Agent", "name": "role_config_suppression_jmx_passwd_file_readwrite_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Zookeeper Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Zookeeper Server", "name": "role_config_suppression_zk_server_java_opts", "value": "false" }, { "desc": "The port used by the ZooKeeper Server's RMI registry. This is required to enable JMX access through RMI which is required for Cloudera Manager ZooKeeper monitoring. This is added as \"-Dcom.sun.management.jmxremote.port\" to the ZooKeeper Server's JVM command line.", "display_name": "JMX Remote Port", "name": "server_jmx_agent_port", "value": "9010" }, { "desc": "The tolerance window that will be used in the detection of a ZooKeeper server's membership in a quorum. Specified in minutes.", "display_name": "Quorum Membership Detection Window", "name": "zookeeper_server_quorum_membership_detection_window", "value": "3" }, { "desc": "The health check thresholds of the weighted average size of the ZooKeeper Server outstanding requests queue over a recent period. See ZooKeeper Server Outstanding Requests Monitoring Period.", "display_name": "ZooKeeper Server Outstanding Requests Thresholds", "name": "zookeeper_server_outstanding_requests_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_server_role_env_safety_valve", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Specifies the name of the user that has read-write privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Name of User with Read-Write Access to the JMX Agent", "name": "jmx_passwd_file_readwrite_user", "value": "controlRole" }, { "desc": "The port to monitor for leadership election", "display_name": "Election Port", "name": "electionPort", "value": "4181" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The maximum number of concurrent connections (at the socket level) that a single client, identified by the IP address, may make to a single member of the ZooKeeper ensemble. This setting is used to prevent certain classes of DoS attacks, including file descriptor exhaustion. To remove the limit on concurrent connections, set this value to 0.", "display_name": "Maximum Client Connections", "name": "maxClientCnxns", "value": "60" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Zookeeper Server", "name": "zk_server_java_opts", "value": "" }, { "desc": "Whether to suppress the results of the Maximum Request Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Maximum Request Latency", "name": "role_health_suppression_zookeeper_server_max_latency", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_zookeeper_server_unexpected_exits", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "zookeeper_server_gc_duration_window", "value": "5" }, { "desc": "Specifies the password of the user that has read-write privileges when using password file based authentication for JMX access. JMX authentication must be enabled for this setting to take effect.", "display_name": "Password of user with read-write access to the JMX agent", "name": "jmx_passwd_file_readwrite_user_password", "value": "CONTROL" }, { "desc": "Enables the JMX agent on the ZooKeeper server. Turning this off on any of the ZooKeeper servers that are part of a service will prevent Cloudera Manager from being able to monitor that server and may affect the monitoring provided on the entire service.", "display_name": "Enable JMX Agent", "name": "enable_jmx_agent", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_zookeeper_server_scm_health", "value": "false" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Transaction Log Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Transaction Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Transaction Log Directory Free Space Monitoring Percentage Thresholds", "name": "zookeeper_server_data_log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Data Directory parameter.", "display_name": "Suppress Parameter Validation: Data Directory", "name": "role_config_suppression_datadir", "value": "false" }, { "desc": "Whether to suppress the results of the Outstanding Requests heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Outstanding Requests", "name": "role_health_suppression_zookeeper_server_outstanding_requests", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Name of User with Read-Only access to the JMX Agent parameter.", "display_name": "Suppress Parameter Validation: Name of User with Read-Only access to the JMX Agent", "name": "role_config_suppression_jmx_passwd_file_readonly_user", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "zookeeper_server_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of ZooKeeper Server in Bytes", "name": "zookeeper_server_java_heapsize", "value": "1073741824" }, { "desc": "The maximum number of rolled log files to keep for Server logs. Typically used by log4j or logback.", "display_name": "Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_zookeeper_server_log_directory_free_space", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Data Directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Data Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Data Directory Free Space Monitoring Percentage Thresholds", "name": "zookeeper_server_data_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The period to review when computing the moving average of the connection count. Specified in minutes.", "display_name": "ZooKeeper Server Connection Count Monitoring Period", "name": "zookeeper_server_connection_count_window", "value": "3" }, { "desc": "When computing the overall Server health, consider the host's health.", "display_name": "Server Host Health Test", "name": "zookeeper_server_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The maximum session timeout, in milliseconds, that the ZooKeeper Server will allow the client to negotiate", "display_name": "Maximum Session Timeout", "name": "maxSessionTimeout", "value": "40000" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Transaction Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Transaction Log Directory Free Space", "name": "role_health_suppression_zookeeper_server_data_log_directory_free_space", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the Data Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Data Directory Free Space", "name": "role_health_suppression_zookeeper_server_data_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Connection Count heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Connection Count", "name": "role_health_suppression_zookeeper_server_connection_count", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/spark-gateway.json0000666000175100017510000002026613245514472025160 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Data Serializer parameter.", "display_name": "Suppress Parameter Validation: Spark Data Serializer", "name": "role_config_suppression_spark_data_serializer", "value": "false" }, { "desc": "Python library paths to add to PySpark applications.", "display_name": "Extra Python Path", "name": "spark_python_path", "value": "" }, { "desc": "When dynamic allocation is enabled, time after which idle executors will be stopped.", "display_name": "Executor Idle Timeout", "name": "spark_dynamic_allocation_idle_timeout", "value": "60" }, { "desc": "Enable Usage of External Shuffle Service. The External Shuffle Service is not robust to NodeManager restarts and so is not recommended for production use.", "display_name": "Enable Shuffle Service", "name": "spark_shuffle_service_enabled", "value": "true" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/spark" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "51" }, { "desc": "Enable dynamic allocation of executors in Spark applications.", "display_name": "Enable Dynamic Allocation", "name": "spark_dynamic_allocation_enabled", "value": "true" }, { "desc": "Which deploy mode to use by default. Can be overridden by users when launching applications.", "display_name": "Default Application Deploy Mode", "name": "spark_deploy_mode", "value": "client" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-env.sh.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "When dynamic allocation is enabled, timeout before requesting new executors when there are backlogged tasks.", "display_name": "Scheduler Backlog Timeout", "name": "spark_dynamic_allocation_scheduler_backlog_timeout", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Extra Python Path parameter.", "display_name": "Suppress Parameter Validation: Extra Python Path", "name": "role_config_suppression_spark_python_path", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/spark-defaults.conf.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", "name": "spark-conf/spark-defaults.conf_client_config_safety_valve", "value": null }, { "desc": "When dynamic allocation is enabled, timeout before requesting new executors after the initial backlog timeout has already expired. By default this is the same value as the initial backlog timeout.", "display_name": "Sustained Scheduler Backlog Timeout", "name": "spark_dynamic_allocation_sustained_scheduler_backlog_timeout", "value": null }, { "desc": "When dynamic allocation is enabled, maximum number of executors to allocate. By default, Spark relies on YARN to control the maximum number of executors for the application.", "display_name": "Maximum Executor Count", "name": "spark_dynamic_allocation_max_executors", "value": null }, { "desc": "When dynamic allocation is enabled, time after which idle executors with cached RDDs blocks will be stopped. By default, they're never stopped. This configuration is only available starting in CDH 5.5.", "display_name": "Caching Executor Idle Timeout", "name": "spark_dynamic_allocation_cached_idle_timeout", "value": null }, { "desc": "Write Spark application history logs to HDFS.", "display_name": "Enable History", "name": "spark_history_enabled", "value": "true" }, { "desc": "When dynamic allocation is enabled, minimum number of executors to keep alive while the application is running.", "display_name": "Minimum Executor Count", "name": "spark_dynamic_allocation_min_executors", "value": "0" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", "name": "role_config_suppression_spark-conf/spark-defaults.conf_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/log4j.properties parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/log4j.properties", "name": "role_config_suppression_spark-conf/log4j.properties_client_config_safety_valve", "value": "false" }, { "desc": "Name of class implementing org.apache.spark.serializer.Serializer to use in Spark applications.", "display_name": "Spark Data Serializer", "name": "spark_data_serializer", "value": "org.apache.spark.serializer.KryoSerializer" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for spark-conf/log4j.properties.", "display_name": "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/log4j.properties", "name": "spark-conf/log4j.properties_client_config_safety_valve", "value": null }, { "desc": "When dynamic allocation is enabled, number of executors to allocate when the application starts. By default, this is the same value as the minimum number of executors.", "display_name": "Initial Executor Count", "name": "spark_dynamic_allocation_initial_executors", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "role_config_suppression_spark-conf/spark-env.sh_client_config_safety_valve", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-service.json0000666000175100017510000026005713245514472024767 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Groups", "name": "service_config_suppression_hdfs_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml parameter.", "display_name": "Suppress Parameter Validation: Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "service_config_suppression_core_site_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties parameter.", "display_name": "Suppress Parameter Validation: HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "service_config_suppression_navigator_client_config_safety_valve", "value": "false" }, { "desc": "List of fencing methods to use for service fencing. shell(./cloudera_manager_agent_fencer.py) is a fencing mechanism designed to use the Cloudera Manager agent. The sshfence method uses SSH. If using custom fencers (that may communicate with shared store, power units, or network switches), use the shell to invoke them.", "display_name": "HDFS High Availability Fencing Methods", "name": "dfs_ha_fencing_methods", "value": "shell(./cloudera_manager_agent_fencer.py)" }, { "desc": "Maximum number of rolled-over audit logs to retain. The logs are not deleted if they contain audit events that have not yet been propagated to the Audit Server.", "display_name": "Number of Audit Logs to Retain", "name": "navigator_client_max_num_audit_log", "value": "10" }, { "desc": "Maximum size of audit log file in MB before it is rolled over.", "display_name": "Maximum Audit Log File Size", "name": "navigator_audit_log_max_file_size", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Groups", "name": "service_config_suppression_yarn_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the FailoverProxyProvider Class parameter.", "display_name": "Suppress Parameter Validation: FailoverProxyProvider Class", "name": "service_config_suppression_dfs_ha_proxy_provider", "value": "false" }, { "desc": "If false, permission checking is turned off for files in HDFS.", "display_name": "Check HDFS Permissions", "name": "dfs_permissions", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_service_env_safety_valve", "value": "false" }, { "desc": "The user that this service's processes should run as (except the HttpFS server, which has its own user)", "display_name": "System User", "name": "process_username", "value": "hdfs" }, { "desc": "Comma-delimited list of groups that you want to allow the Hue user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hue Proxy User Groups", "name": "hue_proxy_user_groups_list", "value": "*" }, { "desc": "The password for the TLS/SSL keystore.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Keystore Password", "name": "hadoop_group_mapping_ldap_keystore_passwd", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Groups", "name": "service_config_suppression_flume_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Failover Controller Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Failover Controller Count Validator", "name": "service_config_suppression_failovercontroller_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hue Proxy User Hosts", "name": "hue_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Groups", "name": "service_config_suppression_oozie_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Admin Users parameter.", "display_name": "Suppress Parameter Validation: Authorized Admin Users", "name": "service_config_suppression_hadoop_authorized_admin_users", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_hdfs_replication_env_safety_valve", "value": "false" }, { "desc": "The service monitor will use this directory to create files to test if the hdfs service is healthy. The directory and files are created with permissions specified by 'HDFS Health Canary Directory Permissions'", "display_name": "HDFS Health Canary Directory", "name": "firehose_hdfs_canary_directory", "value": "/tmp/.cloudera_health_monitoring_canary_files" }, { "desc": "Path to the directory where audit logs will be written. The directory will be created if it doesn't exist.", "display_name": "Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/hadoop-hdfs/audit" }, { "desc": "Class for user to group mapping (get groups for a given user).", "display_name": "Hadoop User Group Mapping Implementation", "name": "hadoop_security_group_mapping", "value": "org.apache.hadoop.security.ShellBasedUnixGroupsMapping" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop HTTP Authentication Cookie Domain parameter.", "display_name": "Suppress Parameter Validation: Hadoop HTTP Authentication Cookie Domain", "name": "service_config_suppression_hadoop_http_auth_cookie_domain", "value": "false" }, { "desc": "Comma-delimited list of groups that you want to allow the YARN user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "YARN Proxy User Groups", "name": "yarn_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Count Validator", "name": "service_config_suppression_namenode_count_validator", "value": "false" }, { "desc": "Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Oozie Proxy User Groups", "name": "oozie_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Groups", "name": "service_config_suppression_http_proxy_user_groups_list", "value": "false" }, { "desc": "Comma-separated list of users authorized to used Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Users", "name": "hadoop_authorized_users", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the Hadoop TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hadoop TLS/SSL Validator", "name": "service_config_suppression_hadoop_ssl_validator", "value": "false" }, { "desc": "Enable HDFS short-circuit read. This allows a client colocated with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality.", "display_name": "Enable HDFS Short-Circuit Read", "name": "dfs_datanode_read_shortcircuit", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Tracker parameter.", "display_name": "Suppress Parameter Validation: Audit Event Tracker", "name": "service_config_suppression_navigator_event_tracker", "value": "false" }, { "desc": "The distinguished name of the user to bind as when connecting to the LDAP server. This may be left blank if the LDAP server supports anonymous binds.", "display_name": "Hadoop User Group Mapping LDAP Bind User", "name": "hadoop_group_mapping_ldap_bind_user", "value": "" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the Balancer Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Balancer Count Validator", "name": "service_config_suppression_balancer_count_validator", "value": "false" }, { "desc": "Password for the TLS/SSL client truststore. Defines a cluster-wide default that can be overridden by individual services.", "display_name": "Cluster-Wide Default TLS/SSL Client Truststore Password", "name": "ssl_client_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Bind User parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Bind User", "name": "service_config_suppression_hadoop_group_mapping_ldap_bind_user", "value": "false" }, { "desc": "The password of the bind user.", "display_name": "Hadoop User Group Mapping LDAP Bind User Password", "name": "hadoop_group_mapping_ldap_bind_passwd", "value": "" }, { "desc": "Action to take when the audit event queue is full. Drop the event or shutdown the affected process.", "display_name": "Audit Queue Policy", "name": "navigator_audit_queue_policy", "value": "DROP" }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Mountpoints Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Mountpoints Validator", "name": "service_config_suppression_nameservice_mountpoints_validator", "value": "false" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping Search Base parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping Search Base", "name": "service_config_suppression_hadoop_group_mapping_ldap_base", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Short-Circuit Read Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Short-Circuit Read Enabled Validator", "name": "service_config_suppression_short_circuit_read_validator", "value": "false" }, { "desc": "Enable/Disable the Log and Query Redaction Policy for this cluster.", "display_name": "Enable Log and Query Redaction", "name": "redaction_policy_enabled", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml. Applies to all roles and client configurations in this HDFS service as well as all its dependent services. Any configs added here will be overridden by their default values in HDFS (which can be found in hdfs-default.xml).", "display_name": "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "core_site_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Authentication And Authorization Validation configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Authentication And Authorization Validation", "name": "service_config_suppression_hdfs_authentication_and_authorization_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Proxy User Groups", "name": "service_config_suppression_smon_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the DataNode Local Path Access Users parameter.", "display_name": "Suppress Parameter Validation: DataNode Local Path Access Users", "name": "service_config_suppression_dfs_block_local_path_access_user", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The health test thresholds of the overall DataNode health. The check returns \"Concerning\" health if the percentage of \"Healthy\" DataNodes falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" DataNodes falls below the critical threshold.", "display_name": "Healthy DataNode Monitoring Thresholds", "name": "hdfs_datanodes_healthy_thresholds", "value": "{\"critical\":\"90.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Private Keys for SSH Fencing Strategy parameter.", "display_name": "Suppress Parameter Validation: Private Keys for SSH Fencing Strategy", "name": "service_config_suppression_dfs_ha_fencing_ssh_private_key_files", "value": "false" }, { "desc": "Enable WebHDFS interface", "display_name": "Enable WebHDFS", "name": "dfs_webhdfs_enabled", "value": "true" }, { "desc": "The short name of the Hue Kerberos principal. Normally, you do not need to specify this configuration. Cloudera Manager auto-configures this property so that Hue and Cloudera Manamgent Service work properly.", "display_name": "Hue's Kerberos Principal Short Name", "name": "hue_kerberos_principal_shortname", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Search Filter parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Search Filter", "name": "service_config_suppression_hadoop_group_mapping_ldap_group_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue's Kerberos Principal Short Name parameter.", "display_name": "Suppress Parameter Validation: Hue's Kerberos Principal Short Name", "name": "service_config_suppression_hue_kerberos_principal_shortname", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Enter a FailoverProxyProvider implementation to configure two URIs to connect to during fail-over. The first configured address is tried first, and on a fail-over event the other address is tried.", "display_name": "FailoverProxyProvider Class", "name": "dfs_ha_proxy_provider", "value": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" }, { "desc": "The domain to use for the HTTP cookie that stores the authentication token. In order for authentiation to work correctly across all Hadoop nodes' web-consoles the domain must be correctly set. Important: when using IP addresses, browsers ignore cookies with domain settings. For this setting to work properly all nodes in the cluster must be configured to generate URLs with hostname.domain names on it.", "display_name": "Hadoop HTTP Authentication Cookie Domain", "name": "hadoop_http_auth_cookie_domain", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Location", "name": "service_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "The search base for the LDAP connection. This is a distinguished name, and will typically be the root of the LDAP directory.", "display_name": "Hadoop User Group Mapping Search Base", "name": "hadoop_group_mapping_ldap_base", "value": "" }, { "desc": "Comma-delimited list of hosts where you want to allow the HDFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HDFS Proxy User Hosts", "name": "hdfs_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the NFS Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: NFS Gateway Count Validator", "name": "service_config_suppression_nfsgateway_count_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Corrupt Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Corrupt Blocks", "name": "service_health_suppression_hdfs_blocks_with_corrupt_replicas", "value": "false" }, { "desc": "

\nEvent filters are defined in a JSON object like the following:\n

\n\n
\n{\n  \"defaultAction\" : (\"accept\", \"discard\"),\n  \"rules\" : [\n    {\n      \"action\" : (\"accept\", \"discard\"),\n      \"fields\" : [\n        {\n          \"name\" : \"fieldName\",\n          \"match\" : \"regex\"\n        }\n      ]\n    }\n  ]\n}\n
\n\n

\nA filter has a default action and a list of rules, in order of precedence.\nEach rule defines an action, and a list of fields to match against the\naudit event.\n

\n\n

\nA rule is \"accepted\" if all the listed field entries match the audit\nevent. At that point, the action declared by the rule is taken.\n

\n\n

\nIf no rules match the event, the default action is taken. Actions\ndefault to \"accept\" if not defined in the JSON object.\n

\n\n

\nThe following is the list of fields that can be filtered for HDFS events:\n

\n\n
    \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • command: the HDFS operation being performed.
  • \n
  • src: the source path for the operation.
  • \n
  • dest: the destination path for the operation.
  • \n
  • permissions: the permissions associated with the operation.
  • \n
\n\n

\nThe default HDFS audit event filter discards events generated by the internal\nCloudera and Hadoop users (cloudera-scm, dr.who, hbase, hive, impala, mapred,\nsolr, and spark), 'ls' actions performed by the hdfs user, operations in the\nCloudera Hive canary directory, and events that affect files in the /tmp\ndirectory.\"\n

\n", "display_name": "Audit Event Filter", "name": "navigator_audit_event_filter", "value": "{\n \"comment\" : [\n \"The default HDFS audit event filter discards events generated by the internal \",\n \"Cloudera and Hadoop users (cloudera-scm, dr.who, hbase, hive, impala, \",\n \"mapred, solr, and spark), 'ls' actions performed by the hdfs user, \",\n \"operations in the Cloudera Hive canary directory, and events that affect \",\n \"files in the /tmp directory.\"\n ],\n \"defaultAction\" : \"accept\",\n \"rules\" : [\n {\n \"action\" : \"discard\",\n \"fields\" : [\n {\n \"name\" : \"username\",\n \"match\" : \"(?:cloudera-scm|dr.who|hbase|hive|impala|mapred|solr|spark)(?:/.+)?\"\n }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n {\n \"name\" : \"username\",\n \"match\" : \"(?:hdfs)(?:/.+)?\"\n },\n {\n \"name\" : \"operation\",\n \"match\" : \"(?:listStatus|listCachePools|listCacheDirectives|getfileinfo)\"\n }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/user/hue/\\\\.cloudera_manager_hive_metastore_canary(?:/.*)?\" }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/user/hue/\\\\.Trash/Current/user/hue/\\\\.cloudera_manager_hive_metastore_canary(?:/.*)?\" }\n ]\n },\n {\n \"action\" : \"discard\",\n \"fields\" : [\n { \"name\" : \"src\", \"match\" : \"/tmp(?:/.*)?\" }\n ]\n }\n ]\n}\n" }, { "desc": "Comma-separated list of groups authorized to used Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Groups", "name": "hadoop_authorized_groups", "value": "" }, { "desc": "If AES/CTR/NoPadding is chosen for the Data Transfer Encryption Algorithm, this specifies the length (in bits) of the AES key. When this parameter is changed, a full, non-rolling restart of the cluster must be performed.", "display_name": "Data Transfer Cipher Suite Key Strength", "name": "dfs_encrypt_data_transfer_cipher_keybits", "value": "256" }, { "desc": "Password that protects the private key contained in the server keystore used for encrypted shuffle and encrypted web UIs. Applies to all configurations of daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": null }, { "desc": "Comma-delimited list of groups that you want to allow the Hive user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hive Proxy User Groups", "name": "hive_proxy_user_groups_list", "value": "*" }, { "desc": "Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Oozie Proxy User Hosts", "name": "oozie_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress the results of the Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Free Space", "name": "service_health_suppression_hdfs_free_space_remaining", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the Cloudera Service Monitor user to impersonate other users. The default '*' allows all hosts. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'.", "display_name": "Service Monitor Proxy User Hosts", "name": "smon_proxy_user_hosts_list", "value": "*" }, { "desc": "A list of path prefixes that define the HDFS regions where authorization is enforced using Sentry information. Only relevant when Sentry Synchronization is enabled.", "display_name": "Sentry Synchronization Path Prefixes", "name": "hdfs_sentry_sync_path_prefixes", "value": "/user/hive/warehouse" }, { "desc": "Comma-delimited list of groups that you want to allow the mapred user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Mapred Proxy User Groups", "name": "mapred_proxy_user_groups_list", "value": "*" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_service_env_safety_valve", "value": null }, { "desc": "For paths where authorization is enforced by Sentry Synchronization, file permissions will use this parameter as the group. This group should normally include the hive and impala users.", "display_name": "Sentry Authorization Provider Group", "name": "sentry_authorization_provider_hdfs_group", "value": "hive" }, { "desc": "Additional mapping rules that will be inserted before rules generated from the list of trusted realms and before the default rule. After changing this value and restarting the service, any services depending on this one must be restarted as well. The hadoop.security.auth_to_local property is configured using this information.", "display_name": "Additional Rules to Map Kerberos Principals to Short Names", "name": "extra_auth_to_local_rules", "value": null }, { "desc": "Password for the server keystore file used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Password", "name": "ssl_server_keystore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "service_config_suppression_hadoop_policy_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HDFS Encryption Validator configuration validator.", "display_name": "Suppress Configuration Validator: HDFS Encryption Validator", "name": "service_config_suppression_hdfs_encryption_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "service_config_suppression_hdfs_service_config_safety_valve", "value": "false" }, { "desc": "Enables Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Note: This is effective only if Kerberos is enabled for the HDFS service.", "display_name": "Enable Kerberos Authentication for HTTP Web-Consoles", "name": "hadoop_secure_web_ui", "value": "false" }, { "desc": "For advanced use only. Key-value pairs (one on each line) to be inserted into the HDFS replication configuration for hadoop-env.sh.", "display_name": "HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "hdfs_replication_haoop_env_sh_safety_valve", "value": null }, { "desc": "Comma-delimited list of groups to allow the HDFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "HDFS Proxy User Groups", "name": "hdfs_proxy_user_groups_list", "value": "*" }, { "desc": "Quality of protection for secured RPC connections between NameNode and HDFS clients. For effective RPC protection, enable Kerberos authentication.", "display_name": "Hadoop RPC Protection", "name": "hadoop_rpc_protection", "value": "authentication" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Hosts", "name": "service_config_suppression_hue_proxy_user_hosts_list", "value": "false" }, { "desc": "Path to the TLS/SSL client truststore file. Defines a cluster-wide default that can be overridden by individual services. This truststore must be in JKS format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. The contents of the truststore can be modified without restarting any roles. By default, changes to its contents are picked up within ten seconds. If not set, the default Java truststore is used to verify certificates.", "display_name": "Cluster-Wide Default TLS/SSL Client Truststore Location", "name": "ssl_client_truststore_location", "value": null }, { "desc": "Comma-separated list of users authorized to perform admin operations on Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Admin Users", "name": "hadoop_authorized_admin_users", "value": "*" }, { "desc": "The health check thresholds of the number of missing blocks. Specified as a percentage of the total number of blocks.", "display_name": "Missing Block Monitoring Thresholds", "name": "hdfs_missing_blocks_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The amount of time after NameNode(s) start that the lack of an active NameNode will be tolerated. This is intended to allow either the auto-failover daemon to make a NameNode active, or a specifically issued failover command to take effect. This is an advanced option that does not often need to be changed.", "display_name": "NameNode Activation Startup Tolerance", "name": "hdfs_namenode_activation_startup_tolerance", "value": "180" }, { "desc": "

Note: Do not edit this property in the classic layout. Switch to the new layout to use preconfigured redaction rules and test your rules inline.

Use this property to define a list of rules to be followed for redacting sensitive information from log files and query strings. Click + to add a new redaction rule. You can choose one of the preconfigured rules or add a custom rule. When specifying a custom rule, the Search field should contain a regular expression that will be matched against the data. If a match is found, it is replaced by the contents of the Replace field.

Trigger is an optional field. It can be used to specify a simple string to be searched in the data. If the string is found, the redactor attempts to find a match for the Search regex. If no trigger is specified, redaction occurs by matching the Search regular expression. Use the Trigger field to enhance performance: simple string matching is faster than regular expression matching.

Test your rules by entering sample text into the Test Redaction Rules text box and clicking Test Redaction. If no rules match, the text you entered is returned unchanged.

", "display_name": "Log and Query Redaction Policy", "name": "redaction_policy", "value": null }, { "desc": "Comma-delimited list of hosts that you want to allow the YARN user to impersonate. The default '*' allows all hosts. To disable entirely, use a string that does not correspond to a host name, such as '_no_host'.", "display_name": "YARN Proxy User Hosts", "name": "yarn_proxy_user_hosts_list", "value": "*" }, { "desc": "Comma-delimited list of groups to allow the HttpFS user to impersonate. The default '*' allows all groups. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "HttpFS Proxy User Groups", "name": "httpfs_proxy_user_groups_list", "value": "*" }, { "desc": "Allows the flume user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Flume Proxy User Groups", "name": "flume_proxy_user_groups_list", "value": "*" }, { "desc": "Comma-delimited list of hosts where you want to allow the mapred user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Mapred Proxy User Hosts", "name": "mapred_proxy_user_hosts_list", "value": "*" }, { "desc": "For advanced use only, a list of configuration properties that will be used by the Service Monitor instead of the current client configuration for the service.", "display_name": "Service Monitor Client Config Overrides", "name": "smon_client_config_overrides", "value": "dfs.socket.timeout3000dfs.datanode.socket.write.timeout3000ipc.client.connect.max.retries1fs.permissions.umask-mode000" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the DataNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: DataNode Count Validator", "name": "service_config_suppression_datanode_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Enable TLS/SSL encryption for HDFS, MapReduce, and YARN web UIs, as well as encrypted shuffle for MapReduce and YARN.", "display_name": "Hadoop TLS/SSL Enabled", "name": "hdfs_hadoop_ssl_enabled", "value": "false" }, { "desc": "List of Kerberos realms that Hadoop services should trust. If empty, defaults to the default_realm property configured in the krb5.conf file. After changing this value and restarting the service, all services depending on this service must also be restarted. Adds mapping rules for each domain to the hadoop.security.auth_to_local property in core-site.xml.", "display_name": "Trusted Kerberos Realms", "name": "trusted_realms", "value": "" }, { "desc": "Whether to suppress the results of the Failover Controllers Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Failover Controllers Health", "name": "service_health_suppression_hdfs_failover_controllers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Proxy User Hosts", "name": "service_config_suppression_smon_proxy_user_hosts_list", "value": "false" }, { "desc": "Typically, HDFS clients and servers communicate by opening sockets via an IP address. In certain networking configurations, it is preferable to open sockets after doing a DNS lookup on the hostname. Enable this property to open sockets after doing a DNS lookup on the hostname. This property is supported in CDH3u4 or later deployments.", "display_name": "Use DataNode Hostname", "name": "dfs_client_use_datanode_hostname", "value": "false" }, { "desc": "Enables the health check that verifies that the failover controllers associated with this service are healthy and running.", "display_name": "Failover Controllers Healthy", "name": "failover_controllers_healthy_enabled", "value": "true" }, { "desc": "The attribute of the group object that identifies the users that are members of the group. The default will usually be appropriate for any LDAP installation.", "display_name": "Hadoop User Group Mapping LDAP Group Membership Attribute", "name": "hadoop_group_mapping_ldap_member_attr", "value": "member" }, { "desc": "Comma separated list of users allowed to do short circuit read. A short circuit read allows a client co-located with the data to read HDFS file blocks directly from HDFS. If empty, will default to the DataNode process' user.", "display_name": "DataNode Local Path Access Users", "name": "dfs_block_local_path_access_user", "value": null }, { "desc": "The timeout, in milliseconds, to use with the Cloudera Manager agent-based fencer.", "display_name": "Timeout for Cloudera Manager Fencing Strategy", "name": "dfs_ha_fencing_cloudera_manager_timeout_millis", "value": "10000" }, { "desc": "Enable collection of audit events from the service's roles.", "display_name": "Enable Audit Collection", "name": "navigator_audit_enabled", "value": "true" }, { "desc": "Maximum bandwidth used for image transfer in bytes per second. This can help keep normal NameNode operations responsive during checkpointing. A default value of 0 indicates that throttling is disabled.", "display_name": "FsImage Transfer Bandwidth", "name": "dfs_image_transfer_bandwidthPerSec", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Groups", "name": "service_config_suppression_mapred_proxy_user_groups_list", "value": "false" }, { "desc": "The user the management services impersonates when connecting to HDFS. If no value is specified, the HDFS superuser is used.", "display_name": "HDFS User to Impersonate", "name": "hdfs_user_to_impersonate", "value": null }, { "desc": "File path to the SSL keystore containing the TLS/SSL certificate required by the LDAP server.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Keystore", "name": "hadoop_group_mapping_ldap_keystore", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Synchronization Path Prefixes parameter.", "display_name": "Suppress Parameter Validation: Sentry Synchronization Path Prefixes", "name": "service_config_suppression_hdfs_sentry_sync_path_prefixes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Event Filter parameter.", "display_name": "Suppress Parameter Validation: Audit Event Filter", "name": "service_config_suppression_navigator_audit_event_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-Wide Default TLS/SSL Client Truststore Password parameter.", "display_name": "Suppress Parameter Validation: Cluster-Wide Default TLS/SSL Client Truststore Password", "name": "service_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "The name of the group of superusers.", "display_name": "Superuser Group", "name": "dfs_permissions_supergroup", "value": "supergroup" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP URL parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP URL", "name": "service_config_suppression_hadoop_group_mapping_ldap_url", "value": "false" }, { "desc": "Allows the Cloudera Service Monitor user to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. This property is used only if Service Monitor is using a different Kerberos principal than the Hue service. To disable entirely, use a string that does not correspond to a group name, such as '_no_group_'.", "display_name": "Service Monitor Proxy User Groups", "name": "smon_proxy_user_groups_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Groups parameter.", "display_name": "Suppress Parameter Validation: Authorized Groups", "name": "service_config_suppression_hadoop_authorized_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Hosts", "name": "service_config_suppression_httpfs_proxy_user_hosts_list", "value": "false" }, { "desc": "The minimal block replication.", "display_name": "Minimal Block Replication", "name": "dfs_replication_min", "value": "1" }, { "desc": "The health check thresholds of free space in HDFS. Specified as a percentage of total HDFS capacity.", "display_name": "HDFS Free Space Monitoring Thresholds", "name": "hdfs_free_space_thresholds", "value": "{\"critical\":\"10.0\",\"warning\":\"20.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Hadoop Group Name parameter.", "display_name": "Suppress Parameter Validation: Shared Hadoop Group Name", "name": "service_config_suppression_hdfs_hadoop_group_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore File Password", "name": "service_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Single User Mode Overrides Validator configuration validator.", "display_name": "Suppress Configuration Validator: Single User Mode Overrides Validator", "name": "service_config_suppression_single_user_mode_override_validator", "value": "false" }, { "desc": "The maximal block replication.", "display_name": "Maximal Block Replication", "name": "dfs_replication_max", "value": "512" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Client Config Overrides parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Client Config Overrides", "name": "service_config_suppression_smon_client_config_overrides", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'. This is used by WebHCat.", "display_name": "HTTP Proxy User Hosts", "name": "HTTP_proxy_user_hosts_list", "value": "*" }, { "desc": "The service monitor will use these permissions to create the directory and files to test if the hdfs service is healthy. Permissions are specified using the 10-character unix-symbolic format e.g. '-rwxr-xr-x'.", "display_name": "HDFS Health Canary Directory Permissions", "name": "firehose_hdfs_canary_directory_permissions", "value": "-rwxrwxrwx" }, { "desc": "Enable authorization", "display_name": "Hadoop Secure Authorization", "name": "hadoop_security_authorization", "value": "false" }, { "desc": "The Key Management Server used by HDFS. This must be set to use encryption for data at rest.", "display_name": "KMS Service", "name": "kms_service", "value": null }, { "desc": "The attribute of the group object that identifies the group name. The default will usually be appropriate for all LDAP systems.", "display_name": "Hadoop User Group Mapping LDAP Group Name Attribute", "name": "hadoop_group_mapping_ldap_group_name_attr", "value": "cn" }, { "desc": "Enables DataNode support for the experimental DistributedFileSystem.getFileVBlockStorageLocations API. Applicable to CDH 4.1 and onwards.", "display_name": "Enable HDFS Block Metadata API", "name": "dfs_datanode_hdfs_blocks_metadata_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: HttpFS Proxy User Groups", "name": "service_config_suppression_httpfs_proxy_user_groups_list", "value": "false" }, { "desc": "The tolerance window that will be used in HDFS service tests that depend on detection of the active NameNode.", "display_name": "Active NameNode Detection Window", "name": "hdfs_active_namenode_detecton_window", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Flume Proxy User Hosts", "name": "service_config_suppression_flume_proxy_user_hosts_list", "value": "false" }, { "desc": "Default block replication. The number of replications to make when the file is created. The default value is used if a replication number is not specified.", "display_name": "Replication Factor", "name": "dfs_replication", "value": "3" }, { "desc": "Comma-delimited list of groups that you want to allow the HTTP user to impersonate. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'. This is used by WebHCat.", "display_name": "HTTP Proxy User Groups", "name": "HTTP_proxy_user_groups_list", "value": "*" }, { "desc": "The name of the system group shared by all the core Hadoop services.", "display_name": "Shared Hadoop Group Name", "name": "hdfs_hadoop_group_name", "value": "hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP User Search Filter parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP User Search Filter", "name": "service_config_suppression_hadoop_group_mapping_ldap_user_filter", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS High Availability Fencing Methods parameter.", "display_name": "Suppress Parameter Validation: HDFS High Availability Fencing Methods", "name": "service_config_suppression_dfs_ha_fencing_methods", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Hosts", "name": "service_config_suppression_hive_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Superuser Group parameter.", "display_name": "Suppress Parameter Validation: Superuser Group", "name": "service_config_suppression_dfs_permissions_supergroup", "value": "false" }, { "desc": "ACLs (Access Control Lists) enhance the existing HDFS permission model to support controlling file access for arbitrary combinations of users and groups instead of a single owner, single group, and all other users. When ACLs are disabled, the NameNode rejects all attempts to set an ACL.", "display_name": "Enable Access Control Lists", "name": "dfs_namenode_acls_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codecs parameter.", "display_name": "Suppress Parameter Validation: Compression Codecs", "name": "service_config_suppression_io_compression_codecs", "value": "false" }, { "desc": "An additional filter to use when searching for LDAP users. The default will usually be appropriate for Active Directory installations. If connecting to a generic LDAP server, ''sAMAccountName'' will likely be replaced with ''uid''. {0} is a special string used to denote where the username fits into the filter.", "display_name": "Hadoop User Group Mapping LDAP User Search Filter", "name": "hadoop_group_mapping_ldap_user_filter", "value": "(&(objectClass=user)(sAMAccountName={0}))" }, { "desc": "Path to the keystore file containing the server certificate and private key used for encrypted shuffle and encrypted web UIs. Applies to configurations of all daemon roles of this service.", "display_name": "Hadoop TLS/SSL Server Keystore File Location", "name": "ssl_server_keystore_location", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the JournalNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: JournalNode Count Validator", "name": "service_config_suppression_journalnode_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HDFS Proxy User Hosts", "name": "service_config_suppression_hdfs_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Health Canary Directory Permissions parameter.", "display_name": "Suppress Parameter Validation: HDFS Health Canary Directory Permissions", "name": "service_config_suppression_firehose_hdfs_canary_directory_permissions", "value": "false" }, { "desc": "

\nConfigures the rules for event tracking and coalescing. This feature is\nused to define equivalency between different audit events. When\nevents match, according to a set of configurable parameters, only one\nentry in the audit list is generated for all the matching events.\n

\n\n

\nTracking works by keeping a reference to events when they first appear,\nand comparing other incoming events against the \"tracked\" events according\nto the rules defined here.\n

\n\n

Event trackers are defined in a JSON object like the following:

\n\n
\n{\n  \"timeToLive\" : [integer],\n  \"fields\" : [\n    {\n      \"type\" : [string],\n      \"name\" : [string]\n    }\n  ]\n}\n
\n\n

\nWhere:\n

\n\n
    \n
  • timeToLive: maximum amount of time an event will be tracked, in\n milliseconds. Must be provided. This defines how long, since it's\n first seen, an event will be tracked. A value of 0 disables tracking.
  • \n\n
  • fields: list of fields to compare when matching events against\n tracked events.
  • \n
\n\n

\nEach field has an evaluator type associated with it. The evaluator defines\nhow the field data is to be compared. The following evaluators are\navailable:\n

\n\n
    \n
  • value: uses the field value for comparison.
  • \n\n
  • username: treats the field value as a user name, and ignores any\n host-specific data. This is useful for environment using Kerberos,\n so that only the principal name and realm are compared.
  • \n
\n\n

\nThe following is the list of fields that can be used to compare HDFS events:\n

\n\n
    \n
  • operation: the HDFS operation being performed.
  • \n
  • username: the user performing the action.
  • \n
  • ipAddress: the IP from where the request originated.
  • \n
  • allowed: whether the operation was allowed or denied.
  • \n
  • src: the source path for the operation.
  • \n
  • dest: the destination path for the operation.
  • \n
  • permissions: the permissions associated with the operation.
  • \n
\n\n

\nThe default event tracker for HDFS services defines equality by comparing the\nusername, operation, and source path of the events.\n

\n", "display_name": "Audit Event Tracker", "name": "navigator_event_tracker", "value": "{\n \"comment\" : [\n \"The default event tracker for HDFS services defines equality by \",\n \"comparing the username, operation, and source path of the events.\"\n ],\n \"timeToLive\" : 60000,\n \"fields\" : [\n { \"type\": \"value\", \"name\" : \"src\" },\n { \"type\": \"value\", \"name\" : \"operation\" },\n { \"type\": \"username\", \"name\" : \"username\" }\n ]\n}\n" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Health Canary Directory parameter.", "display_name": "Suppress Parameter Validation: HDFS Health Canary Directory", "name": "service_config_suppression_firehose_hdfs_canary_directory", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the environment of HDFS replication jobs.", "display_name": "HDFS Replication Advanced Configuration Snippet (Safety Valve)", "name": "hdfs_replication_env_safety_valve", "value": null }, { "desc": "The default block size in bytes for new HDFS files. Note that this value is also used as the HBase Region Server HLog block size.", "display_name": "HDFS Block Size", "name": "dfs_block_size", "value": "134217728" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hue Proxy User Groups", "name": "service_config_suppression_hue_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP TLS/SSL Keystore Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP TLS/SSL Keystore Password", "name": "service_config_suppression_hadoop_group_mapping_ldap_keystore_passwd", "value": "false" }, { "desc": "Path on the DataNode's local file system to a UNIX domain socket used for communication between the DataNode and local HDFS clients. This socket is used for Short Circuit Reads. Only the HDFS System User and \"root\" should have write access to the parent directory and all of its ancestors. This property is supported in CDH 4.2 or later deployments.", "display_name": "UNIX Domain Socket path", "name": "dfs_domain_socket_path", "value": "/var/run/hdfs-sockets/dn" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User's Home Directory parameter.", "display_name": "Suppress Parameter Validation: System User's Home Directory", "name": "service_config_suppression_hdfs_user_home_dir", "value": "false" }, { "desc": "Algorithm to encrypt data transfer between DataNodes and clients, and among DataNodes. If 3des or rc4 are chosen, the entire communication is encrypted with that algorithm. In CDH 5.4 and higher, if AES/CTR/NoPadding is chosen, 3des is used for the initial key exchange, and then AES/CTR/NoPadding is used for the transfer of data. This is the most secure option, and is recommended for clusters running CDH 5.4 or higher. It also requires that the \"openssl-devel\" package be installed on all machines in the cluster. When this parameter is changed, a full, nonrolling restart of the cluster must be performed.", "display_name": "Data Transfer Encryption Algorithm", "name": "dfs_encrypt_data_transfer_algorithm", "value": "rc4" }, { "desc": "The health check thresholds of the number of under-replicated blocks. Specified as a percentage of the total number of blocks.", "display_name": "Under-replicated Block Monitoring Thresholds", "name": "hdfs_under_replicated_blocks_thresholds", "value": "{\"critical\":\"40.0\",\"warning\":\"10.0\"}" }, { "desc": "Whether to suppress the results of the DataNode Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: DataNode Health", "name": "service_health_suppression_hdfs_data_nodes_healthy", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "hdfs_service_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: YARN Proxy User Hosts", "name": "service_config_suppression_yarn_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress the results of the HDFS Canary heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HDFS Canary", "name": "service_health_suppression_hdfs_canary_health", "value": "false" }, { "desc": "Enable encryption of data transfer between DataNodes and clients, and among DataNodes. For effective data transfer protection, enable Kerberos authentication and choose Privacy Quality of RPC Protection.", "display_name": "Enable Data Transfer Encryption", "name": "dfs_encrypt_data_transfer", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "service_config_suppression_hdfs_ssl_client_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the HttpFS Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: HttpFS Count Validator", "name": "service_config_suppression_httpfs_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the SecondaryNameNode Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: SecondaryNameNode Count Validator", "name": "service_config_suppression_secondarynamenode_count_validator", "value": "false" }, { "desc": "Timeout in milliseconds for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations(). This value is only emitted for Impala.", "display_name": "HDFS File Block Storage Location Timeout", "name": "dfs_client_file_block_storage_locations_timeout", "value": "10000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sentry Authorization Provider Group parameter.", "display_name": "Suppress Parameter Validation: Sentry Authorization Provider Group", "name": "service_config_suppression_sentry_authorization_provider_hdfs_group", "value": "false" }, { "desc": "Comma-separated list of compression codecs that can be used in job or map compression.", "display_name": "Compression Codecs", "name": "io_compression_codecs", "value": "org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Membership Attribute parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Membership Attribute", "name": "service_config_suppression_hadoop_group_mapping_ldap_member_attr", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Heap Size Validator", "name": "service_config_suppression_nameservice_namenodes_heap_size_validator", "value": "false" }, { "desc": "Choose the authentication mechanism used by Hadoop", "display_name": "Hadoop Secure Authentication", "name": "hadoop_security_authentication", "value": "simple" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Replication Factor", "name": "service_config_suppression_dfs_replication", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into hadoop-policy.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for hadoop-policy.xml", "name": "hadoop_policy_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Log and Query Redaction Policy parameter.", "display_name": "Suppress Parameter Validation: Log and Query Redaction Policy", "name": "service_config_suppression_redaction_policy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Bind User Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Bind User Password", "name": "service_config_suppression_hadoop_group_mapping_ldap_bind_passwd", "value": "false" }, { "desc": "When computing the overall HDFS cluster health, consider the active NameNode's health", "display_name": "Active NameNode Role Health Check", "name": "hdfs_namenode_health_enabled", "value": "true" }, { "desc": "For advanced use only, a string to be inserted into ssl-server.xml. Applies to configurations of all roles in this service except client configuration.", "display_name": "HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "hdfs_ssl_server_safety_valve", "value": null }, { "desc": "The home directory of the system user on the local filesystem. This setting must reflect the system's configured value - only changing it here will not change the actual home directory.", "display_name": "System User's Home Directory", "name": "hdfs_user_home_dir", "value": "/var/lib/hadoop-hdfs" }, { "desc": "Whether to suppress the results of the Missing Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Missing Blocks", "name": "service_health_suppression_hdfs_missing_blocks", "value": "false" }, { "desc": "SASL protection mode for secured connections to the DataNodes when reading or writing data.", "display_name": "DataNode Data Transfer Protection", "name": "dfs_data_transfer_protection", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Users parameter.", "display_name": "Suppress Parameter Validation: Authorized Users", "name": "service_config_suppression_hadoop_authorized_users", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the flume user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Flume Proxy User Hosts", "name": "flume_proxy_user_hosts_list", "value": "*" }, { "desc": "The URL for the LDAP server to use for resolving user groups when using LdapGroupsMapping.", "display_name": "Hadoop User Group Mapping LDAP URL", "name": "hadoop_group_mapping_ldap_url", "value": "" }, { "desc": "SSH connection timeout, in milliseconds, to use with the built-in sshfence fencer.", "display_name": "Timeout for SSH Fencing Strategy", "name": "dfs_ha_fencing_ssh_connect_timeout", "value": "30000" }, { "desc": "Enable automatic synchronization of HDFS ACLs with Sentry privileges. HDFS Access Control Lists and Check HDFS Permissions must be enabled when this feature is enabled. Use Sentry Synchronization Path Prefixes to define the HDFS regions where authorization is enforced using Sentry information. For more information, see Synchronizing HDFS ACLs and Sentry Authorization.", "display_name": "Enable Sentry Synchronization", "name": "hdfs_sentry_sync_enable", "value": "false" }, { "desc": "Comma-delimited list of hosts where you allow the HttpFS user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HttpFS Proxy User Hosts", "name": "httpfs_proxy_user_hosts_list", "value": "*" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Name of the ZooKeeper service that this HDFS service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Authorized Admin Groups parameter.", "display_name": "Suppress Parameter Validation: Authorized Admin Groups", "name": "service_config_suppression_hadoop_authorized_admin_groups", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mapred Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Mapred Proxy User Hosts", "name": "service_config_suppression_mapred_proxy_user_hosts_list", "value": "false" }, { "desc": "The amount of time to wait for HDFS filesystem image transfer from NameNode to complete.", "display_name": "FsImage Transfer Timeout", "name": "dfs_image_transfer_timeout", "value": "60000" }, { "desc": "Whether to suppress configuration warnings produced by the Auto Failover Validator configuration validator.", "display_name": "Suppress Configuration Validator: Auto Failover Validator", "name": "service_config_suppression_auto_failover_validator", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "hdfs" }, { "desc": "Whether to suppress configuration warnings produced by the Nameservice Checkpoint Configuration Validator configuration validator.", "display_name": "Suppress Configuration Validator: Nameservice Checkpoint Configuration Validator", "name": "service_config_suppression_nameservice_checkpoint_configuration_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: Oozie Proxy User Hosts", "name": "service_config_suppression_oozie_proxy_user_hosts_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Proxy User Hosts parameter.", "display_name": "Suppress Parameter Validation: HTTP Proxy User Hosts", "name": "service_config_suppression_http_proxy_user_hosts_list", "value": "false" }, { "desc": "The group that this service's processes should run as (except the HttpFS server, which has its own group)", "display_name": "System Group", "name": "process_groupname", "value": "hdfs" }, { "desc": "Comma-delimited list of hosts where you want to allow the Hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hive Proxy User Hosts", "name": "hive_proxy_user_hosts_list", "value": "*" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Trusted Kerberos Realms parameter.", "display_name": "Suppress Parameter Validation: Trusted Kerberos Realms", "name": "service_config_suppression_trusted_realms", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the NFS High Availability Validator configuration validator.", "display_name": "Suppress Configuration Validator: NFS High Availability Validator", "name": "service_config_suppression_nfs_ha_validator", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "Whether to suppress the results of the NameNode Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Health", "name": "service_health_suppression_hdfs_ha_namenode_health", "value": "false" }, { "desc": "Default umask for file and directory creation, specified in an octal value (with a leading 0)", "display_name": "Default Umask", "name": "dfs_umaskmode", "value": "022" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the UNIX Domain Socket path parameter.", "display_name": "Suppress Parameter Validation: UNIX Domain Socket path", "name": "service_config_suppression_dfs_domain_socket_path", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS User to Impersonate parameter.", "display_name": "Suppress Parameter Validation: HDFS User to Impersonate", "name": "service_config_suppression_hdfs_user_to_impersonate", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hive Proxy User Groups parameter.", "display_name": "Suppress Parameter Validation: Hive Proxy User Groups", "name": "service_config_suppression_hive_proxy_user_groups_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Audit Log Directory", "name": "service_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "Enables the health check that a client can create, read, write, and delete files", "display_name": "HDFS Canary Health Check", "name": "hdfs_canary_health_enabled", "value": "true" }, { "desc": "The health check thresholds of the number of blocks that have at least one corrupt replica. Specified as a percentage of the total number of blocks.", "display_name": "Blocks With Corrupt Replicas Monitoring Thresholds", "name": "hdfs_blocks_with_corrupt_replicas_thresholds", "value": "{\"critical\":\"1.0\",\"warning\":\"0.5\"}" }, { "desc": "Comma-separated list of groups authorized to perform admin operations on Hadoop. This is emitted only if authorization is enabled.", "display_name": "Authorized Admin Groups", "name": "hadoop_authorized_admin_groups", "value": "" }, { "desc": "An additional filter to use when searching for groups.", "display_name": "Hadoop User Group Mapping LDAP Group Search Filter", "name": "hadoop_group_mapping_ldap_group_filter", "value": "(objectClass=group)" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Rules to Map Kerberos Principals to Short Names parameter.", "display_name": "Suppress Parameter Validation: Additional Rules to Map Kerberos Principals to Short Names", "name": "service_config_suppression_extra_auth_to_local_rules", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Cluster-Wide Default TLS/SSL Client Truststore Location parameter.", "display_name": "Suppress Parameter Validation: Cluster-Wide Default TLS/SSL Client Truststore Location", "name": "service_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml parameter.", "display_name": "Suppress Parameter Validation: HDFS Service Advanced Configuration Snippet (Safety Valve) for ssl-server.xml", "name": "service_config_suppression_hdfs_ssl_server_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Under-Replicated Blocks heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Under-Replicated Blocks", "name": "service_health_suppression_hdfs_under_replicated_blocks", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for navigator.client.properties.", "display_name": "HDFS Client Advanced Configuration Snippet (Safety Valve) for navigator.client.properties", "name": "navigator_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP Group Name Attribute parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP Group Name Attribute", "name": "service_config_suppression_hadoop_group_mapping_ldap_group_name_attr", "value": "false" }, { "desc": "When computing the overall HDFS cluster health, consider the health of the standby NameNode.", "display_name": "Standby NameNode Health Check", "name": "hdfs_standby_namenodes_health_enabled", "value": "true" }, { "desc": "The SSH private key files to use with the built-in sshfence fencer. These are to be accessible to the hdfs user on the machines running the NameNodes.", "display_name": "Private Keys for SSH Fencing Strategy", "name": "dfs_ha_fencing_ssh_private_key_files", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "service_config_suppression_hdfs_replication_haoop_env_sh_safety_valve", "value": "false" }, { "desc": "Whether or not to use TLS/SSL when connecting to the LDAP server.", "display_name": "Hadoop User Group Mapping LDAP TLS/SSL Enabled", "name": "hadoop_group_mapping_ldap_use_ssl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop TLS/SSL Server Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Hadoop TLS/SSL Server Keystore Key Password", "name": "service_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop User Group Mapping LDAP TLS/SSL Keystore parameter.", "display_name": "Suppress Parameter Validation: Hadoop User Group Mapping LDAP TLS/SSL Keystore", "name": "service_config_suppression_hadoop_group_mapping_ldap_keystore", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into ssl-client.xml. Applies cluster-wide, but can be overridden by individual services.", "display_name": "HDFS Advanced Configuration Snippet (Safety Valve) for ssl-client.xml", "name": "hdfs_ssl_client_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Redaction Policy Validator configuration validator.", "display_name": "Suppress Configuration Validator: Redaction Policy Validator", "name": "service_config_suppression_redaction_policy_validator", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hbase-gateway.json0000666000175100017510000001504713245514472025123 0ustar zuulzuul00000000000000[ { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hbase" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for hbase-site.xml.", "display_name": "HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_client_config_safety_valve", "value": "false" }, { "desc": "If a multiget operation is performed with 'Consistency.TIMELINE', the read RPC is sent to the primary region server first. After this timeout, a parallel RPC for secondary region replicas is also sent if the primary does not respond. After this, the result is returned from whichever RPC is finished first. If the response returns from the primary region replica, that the data is the most recent. Result.isStale() API has been added to inspect the staleness. If the result is from a secondary region, Result.isStale() is set to true.", "display_name": "HBase Client Multiget Timeout For Secondary Region Replicas", "name": "hbase_client_primaryCallTimeout_multiget", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "Whether to enable interruption of RPC threads at the client. The default value of true enables primary RegionServers to access data from other regions' secondary replicas.", "display_name": "Enable Client RPC Threads Interruption", "name": "hbase_ipc_client_allowsInterrupt", "value": "true" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hbase-env.sh", "display_name": "HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh", "name": "hbase_client_env_safety_valve", "value": null }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_hbase_client_java_opts", "value": "false" }, { "desc": "If a get operation is performed with 'Consistency.TIMELINE', the read RPC is sent to the primary region server first. After this timeout, parallel RPC for secondary region replicas is also sent if the primary does not respond. After this, the result is returned from whichever RPC is finished first. If the response returns from the primary region replica, that the data is the most recent. Result.isStale() API has been added to inspect the staleness. If the result is from a secondary region, Result.isStale() is set to true.", "display_name": "HBase Client Get Timeout For Secondary Region Replicas", "name": "hbase_client_primaryCallTimeout_get", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh parameter.", "display_name": "Suppress Parameter Validation: HBase Client Environment Advanced Configuration Snippet (Safety Valve) for hbase-env.sh", "name": "role_config_suppression_hbase_client_env_safety_valve", "value": "false" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "hbase_client_java_heapsize", "value": "268435456" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "hbase_client_java_opts", "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/sqoop-sqoop_server.json0000666000175100017510000005733713245514472026300 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where Sqoop 2 Server will place its log files.", "display_name": "Sqoop 2 Server Log Directory", "name": "sqoop_log_dir", "value": "/var/log/sqoop2" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Password for Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Password", "name": "sqoop_repository_database_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_sqoop_server_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties", "name": "role_config_suppression_sqoop_config_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Admin port of Sqoop 2 Server. (Note: This configuration only applies from CDH 4.3 onwards.)", "display_name": "Sqoop 2 Admin Port", "name": "sqoop_admin_port", "value": "8005" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum number of clients allowed to connect to the Sqoop 2 Server.", "display_name": "Maximum Client Connections", "name": "max_client_connections", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "For advanced use only. A string to be inserted into sqoop.properties for this role only.", "display_name": "Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties", "name": "sqoop_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_sqoop_server_unexpected_exits", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "HTTP port of Sqoop 2 Server. (Note: This configuration only applies from CDH 4.3 onwards.)", "display_name": "Sqoop 2 HTTP Port", "name": "sqoop_http_port", "value": "12000" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_sqoop_server_file_descriptor", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Sqoop 2 Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "SQOOP_SERVER_role_env_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "sqoop_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_sqoop_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_sqoop_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database User parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database User", "name": "role_config_suppression_sqoop_repository_database_user", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_sqoop_server_scm_health", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Sqoop 2 Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Type of Sqoop repository database.", "display_name": "Sqoop Repository Database Type", "name": "sqoop_repository_database_type", "value": "derby" }, { "desc": "User for Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database User", "name": "sqoop_repository_database_user", "value": "sa" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Password parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Password", "name": "role_config_suppression_sqoop_repository_database_password", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Name of Sqoop repository database. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Name", "name": "sqoop_repository_database_name", "value": "sqoop" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Log Directory", "name": "role_config_suppression_sqoop_log_dir", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Sqoop 2 Server logs. Typically used by log4j or logback.", "display_name": "Sqoop 2 Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Sqoop 2 Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Sqoop 2 Server", "name": "role_config_suppression_sqoop_java_opts", "value": "false" }, { "desc": "Directory where the Sqoop 2 Server places its metastore data. This is used only when Sqoop Repository Database Type is Derby.", "display_name": "Sqoop 2 Server Metastore Directory", "name": "sqoop_metastore_data_dir", "value": "/var/lib/sqoop2" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Server Metastore Directory parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Server Metastore Directory", "name": "role_config_suppression_sqoop_metastore_data_dir", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Sqoop 2 Server", "name": "sqoop_java_opts", "value": "" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Host parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Host", "name": "role_config_suppression_sqoop_repository_database_host", "value": "false" }, { "desc": "When computing the overall Sqoop 2 Server health, consider the host's health.", "display_name": "Sqoop 2 Server Host Health Test", "name": "sqoop_server_host_health_enabled", "value": "true" }, { "desc": "The maximum number of rolled log files to keep for Sqoop 2 Server logs. Typically used by log4j or logback.", "display_name": "Sqoop 2 Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Enables the health test that the Sqoop 2 Server's process state is consistent with the role configuration", "display_name": "Sqoop 2 Server Process Health Test", "name": "sqoop_server_scm_health_enabled", "value": "true" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Sqoop 2 Server in Bytes", "name": "sqoop_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_sqoop_server_log_directory_free_space", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Host name of Sqoop repository database. If the port is non-default for your database type, use host:port notation. Does not apply if you are using Derby as the database type.", "display_name": "Sqoop Repository Database Host", "name": "sqoop_repository_database_host", "value": "localhost" }, { "desc": "The minimum log level for Sqoop 2 Server logs", "display_name": "Sqoop 2 Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop Repository Database Name parameter.", "display_name": "Suppress Parameter Validation: Sqoop Repository Database Name", "name": "role_config_suppression_sqoop_repository_database_name", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_sqoop_server_host_health", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/kms-kms.json0000666000175100017510000004506113245514472023763 0ustar zuulzuul00000000000000[ { "desc": "Allows the oozie superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Oozie Proxy User Groups", "name": "hadoop_kms_proxyuser_oozie_groups", "value": "*" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kms_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Allows the mapred superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Mapred Proxy User Groups", "name": "hadoop_kms_proxyuser_mapred_groups", "value": "*" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Comma-delimited list of hosts where you want to allow the mapred user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Mapred Proxy User Hosts", "name": "hadoop_kms_proxyuser_mapred_hosts", "value": "*" }, { "desc": "Comma-delimited list of hosts where you want to allow the httpfs user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HttpFS Proxy User Hosts", "name": "hadoop_kms_proxyuser_httpfs_hosts", "value": "*" }, { "desc": "Maximum number of threads used to handle KMS requests.", "display_name": "KMS Max Threads", "name": "kms_max_threads", "value": "250" }, { "desc": "Comma-delimited list of hosts where you want to allow the oozie user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Oozie Proxy User Hosts", "name": "hadoop_kms_proxyuser_oozie_hosts", "value": "*" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Comma-delimited list of hosts where you want to allow the yarn user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "YARN Proxy User Hosts", "name": "hadoop_kms_proxyuser_yarn_hosts", "value": "*" }, { "desc": "Enables the health test that the Key Management Server's process state is consistent with the role configuration", "display_name": "Key Management Server Process Health Test", "name": "kms_scm_health_enabled", "value": "true" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "The password for the Key Management Server JKS keystore file.", "display_name": "Key Management Server TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Allows the hive superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hive Proxy User Groups", "name": "hadoop_kms_proxyuser_hive_groups", "value": "*" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Comma-delimited list of hosts where you want to allow the HTTP user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HTTP Proxy User Hosts", "name": "hadoop_kms_proxyuser_HTTP_hosts", "value": "*" }, { "desc": "Comma-delimited list of hosts where you want to allow the hdfs user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "HDFS Proxy User Hosts", "name": "hadoop_kms_proxyuser_hdfs_hosts", "value": "*" }, { "desc": "Allows the hdfs superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HDFS Proxy User Groups", "name": "hadoop_kms_proxyuser_hdfs_groups", "value": "*" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "For advanced use only, a string to be inserted into kms-acls.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml", "name": "kms-acls.xml_role_safety_valve", "value": null }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Comma-delimited list of hosts where you want to allow the hue user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hue Proxy User Hosts", "name": "hadoop_kms_proxyuser_hue_hosts", "value": "*" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Encrypt communication between clients and Key Management Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Key Management Server", "name": "ssl_enabled", "value": "false" }, { "desc": "The log directory for log files of the role Key Management Server.", "display_name": "Key Management Server Log Directory", "name": "log_dir", "value": "/var/log/hadoop-kms" }, { "desc": "Comma-delimited list of hosts where you want to allow the hive user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Hive Proxy User Hosts", "name": "hadoop_kms_proxyuser_hive_hosts", "value": "*" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Key Management Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Key Management Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "KMS_role_env_safety_valve", "value": null }, { "desc": "Port used by clients to interact with the KMS.", "display_name": "KMS HTTP Port", "name": "kms_http_port", "value": "16000" }, { "desc": "Allows the yarn superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "YARN Proxy User Groups", "name": "hadoop_kms_proxyuser_yarn_groups", "value": "*" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Allows the hue superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Hue Proxy User Groups", "name": "hadoop_kms_proxyuser_hue_groups", "value": "*" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Comma-delimited list of hosts where you want to allow the flume user to impersonate other users. The default '*' allows all hosts. To disable entirely, use a string that doesn't correspond to a host name, such as '_no_host'.", "display_name": "Flume Proxy User Hosts", "name": "hadoop_kms_proxyuser_flume_hosts", "value": "*" }, { "desc": "The maximum size, in megabytes, per log file for Key Management Server logs. Typically used by log4j or logback.", "display_name": "Key Management Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "The password for the JavaKeyStoreProvider keystore file.", "display_name": "JavaKeyStoreProvider Password", "name": "hadoop_security_keystore_javaKeyStoreProvider_password", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "When computing the overall Key Management Server health, consider the host's health.", "display_name": "Key Management Server Host Health Test", "name": "kms_host_health_enabled", "value": "true" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has all of the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file-descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "A comma-separated list of users (no spaces) for whom to disallow access to key material. These users can still fetch key metadata and create encrypted encryption keys, but are unable to do any other KMS operations. Typically, HDFS superusers will be specified here.", "display_name": "KMS Blacklist Users", "name": "kms_blacklist_users", "value": "" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Key Management Server is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Key Management Server TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "Allows the httpfs superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HttpFS Proxy User Groups", "name": "hadoop_kms_proxyuser_httpfs_groups", "value": "*" }, { "desc": "Directory where configuration and binaries are staged before starting KMS. Does not normally need to be modified.", "display_name": "KMS Staging Directory", "name": "kms_staging_dir", "value": "/var/lib/hadoop-kms" }, { "desc": "For advanced use only, a string to be inserted into core-site.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml", "name": "core-site.xml_role_safety_valve", "value": null }, { "desc": "The maximum number of rolled log files to keep for Key Management Server logs. Typically used by log4j or logback.", "display_name": "Key Management Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Allows the flume superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "Flume Proxy User Groups", "name": "hadoop_kms_proxyuser_flume_groups", "value": "*" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Allows the HTTP superuser to impersonate any members of a comma-delimited list of groups. The default '*' allows all groups. To disable entirely, use a string that doesn't correspond to a group name, such as '_no_group_'.", "display_name": "HTTP Proxy User Groups", "name": "hadoop_kms_proxyuser_HTTP_groups", "value": "*" }, { "desc": "Directory of the keystore file kms.keystore used by JavaKeyStoreProvider that backs the KMS.", "display_name": "JavaKeyStoreProvider Directory", "name": "hadoop_security_key_provider_dir", "value": "/var/lib/kms" }, { "desc": "For advanced use only, a string to be inserted into kms-site.xml for this role only.", "display_name": "Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml", "name": "kms-site.xml_role_safety_valve", "value": null }, { "desc": "The minimum log level for Key Management Server logs", "display_name": "Key Management Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Maximum heap size of the KMS.", "display_name": "KMS Heap Size", "name": "kms_heap_size", "value": "1073741824" }, { "desc": "Port used to access the KMS' embedded Tomcat admin console.", "display_name": "KMS Admin Port", "name": "kms_admin_port", "value": "16001" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hbase-regionserver.json0000666000175100017510000016502613245514472026177 0ustar zuulzuul00000000000000[ { "desc": "For advanced use only. A string to be inserted into hbase-site.xml for this role only.", "display_name": "RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "hbase_regionserver_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the HBase RegionServer Multiple HDFS WAL with Replication Validator configuration validator.", "display_name": "Suppress Configuration Validator: HBase RegionServer Multiple HDFS WAL with Replication Validator", "name": "role_config_suppression_hbase_regionserver_multiwal_replication_validator", "value": "false" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Maximum number of HStoreFiles to compact per minor compaction.", "display_name": "Maximum Number of HStoreFiles Compaction", "name": "hbase_hstore_compaction_max", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: RegionServer Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_regionserver_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase HRegion Major Compaction parameter.", "display_name": "Suppress Parameter Validation: HBase HRegion Major Compaction", "name": "role_config_suppression_hbase_hregion_majorcompaction", "value": "false" }, { "desc": "The address for the HBase RegionServer web UI", "display_name": "HBase RegionServer Web UI Address", "name": "hbase_regionserver_info_bindAddress", "value": null }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Number of threads to use while loading and unloading regions to or from a RegionServer. Can be used to increase the speed of decommissioning or rolling restart operations.", "display_name": "Region Mover Threads", "name": "hbase_regionserver_regionmover_thread_count", "value": "1" }, { "desc": "The health test thresholds of the average size of the HBase RegionServer flush queue over a recent period. See HBase RegionServer Flush Queue Monitoring Period.", "display_name": "HBase RegionServer Flush Queue Monitoring Thresholds", "name": "regionserver_flush_queue_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "The amount of time to wait for the HBase Region Server to fully start up and connect to the HBase Master before enforcing the connectivity check.", "display_name": "HBase Region Server Connectivity Tolerance at Startup", "name": "regionserver_connectivity_tolerance", "value": "180" }, { "desc": "Whether to suppress the results of the Store File Index Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Store File Index Size", "name": "role_health_suppression_region_server_store_file_idx_size", "value": "false" }, { "desc": "The hashing algorithm for use in HashFunction. Two values are supported: 'murmur' (for MurmurHash) and 'jenkins' (for JenkinsHash).", "display_name": "HBase Hash Type", "name": "hbase_hash_type", "value": "murmur" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer DNS Network Interface parameter.", "display_name": "Suppress Parameter Validation: RegionServer DNS Network Interface", "name": "role_config_suppression_hbase_regionserver_dns_interface", "value": "false" }, { "desc": "The port for the HBase RegionServer web UI. Set to -1 to disable RegionServer web UI.", "display_name": "HBase RegionServer Web UI port", "name": "hbase_regionserver_info_port", "value": "60030" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The period over which to compute the moving average of the compaction queue size.", "display_name": "HBase RegionServer Compaction Queue Monitoring Period", "name": "regionserver_compaction_queue_window", "value": "5" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HBase RegionServer", "name": "hbase_regionserver_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "RegionServer Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The health test thresholds of the latency that the RegionServer sees for HDFS read operations", "display_name": "HBase RegionServer HDFS Read Latency Thresholds", "name": "regionserver_read_latency_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"50.0\"}" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds of the size used by the HStoreFile index. Specified as a percentage of the total heap size.", "display_name": "Percentage of Heap Used by HStoreFile Index", "name": "regionserver_store_file_idx_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "Maximum number of Write-Ahead Log (WAL) files. This value multiplied by HDFS Block Size (dfs.blocksize) is the size of the WAL that will need to be replayed when a server crashes. This value is inversely proportional to the frequency of flushes to disk.", "display_name": "Maximum number of Write-Ahead Log (WAL) files", "name": "hbase_regionserver_maxlogs", "value": "32" }, { "desc": "List of coprocessors that are loaded by default on all tables. For any override coprocessor method, these classes will be called in order. After implementing your own coprocessor, just put it in HBase's classpath and add the fully qualified class name here. A coprocessor can also be loaded on demand by setting HTableDescriptor.", "display_name": "HBase Coprocessor Region Classes", "name": "hbase_coprocessor_region_classes", "value": "" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "regionserver_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_region_server_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_region_server_scm_health", "value": "false" }, { "desc": "Blocks writes if the size of the memstore increases to the value of 'hbase.hregion.block.memstore' multiplied by the value of 'hbase.hregion.flush.size' bytes. This setting is useful for preventing runaway memstore during spikes in update traffic. Without an upper-bound, memstore fills such that when it flushes, the resultant process of flushing files take a long time to compact or split, or worse, an \"out of memory\" error occurs.", "display_name": "HBase Memstore Block Multiplier", "name": "hbase_hregion_memstore_block_multiplier", "value": "2" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Sync the HLog to HDFS after this interval, in milliseconds, if it has not accumulated the number of HLog Entries specified to trigger a sync.", "display_name": "Synch Interval of HLog Entries", "name": "hbase_regionserver_optionallogflushinterval", "value": "1000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \"IPC Server handler.*took.*appending an edit to hlog.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"ABORTING region server serverName.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"DFSOutputStream ResponseProcessor exception.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Error Recovery for block blk.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Failed init\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Problem renewing lease for DFSClient.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"remote error telling master we are up\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\", \"content\": \"Session.*for server.*closing socket connection and attempting reconnect.\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting .*compaction on region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_REGION_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^completed compaction on region (.+) after (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_REGION_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting compaction on (.+) in region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"COLUMN_FAMILY\", \"group1\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^completed compaction: regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Completed compaction: Request = regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^aborted compaction: regionName=(.+), storeName=(.+), fileCount=(.+), fileSize=(.+), priority=(.+), time=(.+); duration=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_COMPACTION_COLUMN_FAMILY_ABORTED\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\", \"group1\": \"COLUMN_FAMILY\", \"group2\": \"FILE_COUNT\", \"group3\": \"FILE_SIZE\", \"group4\": \"PRIORITY\", \"group5\": \"REQUEST_TIME_NANOS\", \"group6\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Finished memstore flush of .+ for region (.+) in (.+), sequenceid=(.+), compaction requested=(.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DURATION\", \"group2\": \"SEQUENCE_ID\", \"group3\": \"COMPACTION_REQUESTED\" },\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Flush of region (.+) due to global heap pressure$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_DUE_TO_HEAP_PRESSURE\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"WARN\", \"content\": \"^Region (.+) has too many store files; delaying flush up to 90000ms$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_FLUSH_DELAYED_TOO_MANY_STORE_FILES\", \"attribute:SEVERITY\": \"CRITICAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Starting split of region (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_STARTED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Running rollback/cleanup of failed split of (.+);.+$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_ABORTED\", \"attribute:SEVERITY\": \"IMPORTANT\", \"group0\": \"REGION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Region split, hbase:meta updated, and report to master. Parent=(.+), new regions: (.+,.*,.+), (.+,.*,.+). Split took (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DAUGHTER_REGIONS\", \"group2\": \"DAUGHTER_REGIONS\", \"group3\": \"DURATION\"},\n {\"alert\": false, \"rate\": -1, \"threshold\":\"INFO\", \"content\": \"^Region split, META updated, and report to master. Parent=(.+), new regions: (.+,.*,.+), (.+,.*,.+). Split took (.+)$\", \"attribute:CATEGORY\": \"HBASE\", \"attribute:EVENTCODE\": \"EV_HBASE_SPLIT_COMPLETED\", \"attribute:SEVERITY\": \"INFORMATIONAL\", \"group0\": \"REGION\", \"group1\": \"DAUGHTER_REGIONS\", \"group2\": \"DAUGHTER_REGIONS\", \"group3\": \"DURATION\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}" }, { "desc": "Whether to suppress the results of the Cluster Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Cluster Connectivity", "name": "role_health_suppression_region_server_master_connectivity", "value": "false" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_region_server_gc_duration", "value": "false" }, { "desc": "Number of threads for completing small compactions.", "display_name": "RegionServer Small Compactions Thread Count", "name": "hbase_regionserver_thread_compaction_small", "value": "1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds of the total size of RegionServer's memstores. Specified as a percentage of the configured upper limit. See Maximum Size of All Memstores in RegionServer.", "display_name": "HBase RegionServer Memstore Size Thresholds", "name": "regionserver_memstore_size_thresholds", "value": "{\"critical\":\"100.0\",\"warning\":\"95.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Secure Bulk Load Directory parameter.", "display_name": "Suppress Parameter Validation: HBase Secure Bulk Load Directory", "name": "role_config_suppression_hbase_bulkload_staging_dir", "value": "false" }, { "desc": "Percentage of maximum heap (-Xmx setting) to allocate to block cache used by HFile/StoreFile. To disable, set this value to 0 .", "display_name": "HFile Block Cache Size", "name": "hfile_block_cache_size", "value": "0.4" }, { "desc": "The maximum size, in megabytes, per log file for RegionServer logs. Typically used by log4j or logback.", "display_name": "RegionServer Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "If the memstores in a region are this size or larger when closing, run a pre-flush process to clear out memstores before putting up the region closed flag and taking the region offline. On close, a flush process is run under the close flag up to empty memory. During this time, the region is offline and no writes are taken. If the memstore content is large, the flush process could take a long time to complete. The pre-flush process cleans out the bulk of the memstore before putting up the close flag and taking the region offline, so that the flush process that runs under the close flag has little to do.", "display_name": "HBase Memstore Pre-close Flush Size", "name": "hbase_hregion_preclose_flush_size", "value": "5242880" }, { "desc": "If this number of HStoreFiles in any one HStore is exceeded, then a compaction is run to rewrite all HStoreFiles files as one HStoreFile. (One HStoreFile is written per flush of memstore.) You can delay compaction by specifying a larger number, but the compaction will take longer when it does run. During a compaction, updates cannot be flushed to disk. Long compactions require memory sufficient to carry the logging of all updates across the duration of the compaction. If too large, clients timeout during compaction.", "display_name": "HStore Compaction Threshold", "name": "hbase_hstore_compactionThreshold", "value": "3" }, { "desc": "The health test thresholds for the latency of HDFS write operations that the RegionServer detects", "display_name": "HBase RegionServer HDFS Sync Latency Thresholds", "name": "regionserver_sync_latency_thresholds", "value": "{\"critical\":\"5000.0\",\"warning\":\"500.0\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_region_server_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Codecs parameter.", "display_name": "Suppress Parameter Validation: RegionServer Codecs", "name": "role_config_suppression_hbase_regionserver_codecs", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The maximum size byte array that should come from the MSLAB, in bytes.", "display_name": "Maximum Byte Array from MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_max_allocation", "value": "262144" }, { "desc": "The period over which to compute the moving average of the HDFS sync latency of the HBase RegionServer.", "display_name": "HBase RegionServer HDFS Sync Latency Monitoring Period", "name": "regionserver_sync_latency_window", "value": "5" }, { "desc": "Maximum size of all memstores in a RegionServer before new updates are blocked and flushes are forced.", "display_name": "Maximum Size of All Memstores in RegionServer", "name": "hbase_regionserver_global_memstore_upperLimit", "value": "0.4" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_region_server_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where RegionServer will place its log files.", "display_name": "RegionServer Log Directory", "name": "hbase_regionserver_log_dir", "value": "/var/log/hbase" }, { "desc": "The period of time that an HRegion will block updates after reaching the HStoreFile limit that is specified by 'hbase.hstore.blockingStoreFiles'. After this time has elapsed, the HRegion will stop blocking updates even if a compaction has not been completed.", "display_name": "HStore Blocking Wait Time", "name": "hbase_hstore_blockingWaitTime", "value": "90000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Flush Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Flush Queue Size", "name": "role_health_suppression_region_server_flush_queue", "value": "false" }, { "desc": "Timeout for Canary to perform its checks.", "display_name": "Canary Timeout", "name": "hbase_regionserver_canary_timeout", "value": "15" }, { "desc": "Interval, in milliseconds, between messages from the RegionServer to the HBase Master. Use a high value such as 3000 for clusters that have more than 10 hosts.", "display_name": "HBase RegionServer Message Interval", "name": "hbase_regionserver_msginterval", "value": "3000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase Coprocessor Region Classes parameter.", "display_name": "Suppress Parameter Validation: HBase Coprocessor Region Classes", "name": "role_config_suppression_hbase_coprocessor_region_classes", "value": "false" }, { "desc": "The lease period, in milliseconds, for the HBase RegionServer. Clients must report in within this period or else they are considered dead.", "display_name": "HBase RegionServer Lease Period", "name": "hbase_regionserver_lease_period", "value": "60000" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer DNS Nameserver parameter.", "display_name": "Suppress Parameter Validation: RegionServer DNS Nameserver", "name": "role_config_suppression_hbase_regionserver_dns_nameserver", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "regionserver_web_metric_collection_enabled", "value": "true" }, { "desc": "The name of the DNS Network Interface from which a RegionServer should report its IP address.", "display_name": "RegionServer DNS Network Interface", "name": "hbase_regionserver_dns_interface", "value": null }, { "desc": "The number of reservation blocks that are used to prevent unstable RegionServers caused by an OOME.", "display_name": "RegionServer Reservation Blocks", "name": "hbase_regionserver_nbreservationblocks", "value": "4" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HLog Reader Implementation parameter.", "display_name": "Suppress Parameter Validation: HLog Reader Implementation", "name": "role_config_suppression_hbase_regionserver_hlog_reader_impl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HBase RegionServer parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HBase RegionServer", "name": "role_config_suppression_hbase_regionserver_java_opts", "value": "false" }, { "desc": "The implementation that should be used by the RegionServer for the write-ahead-log.", "display_name": "WAL Provider", "name": "hbase_regionserver_wal_provider", "value": "multiwal" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "regionserver_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The size of the chunks allocated by MSLAB, in bytes.", "display_name": "Chunk Size Allocated by MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_chunksize", "value": "2097152" }, { "desc": "Whether to suppress the results of the Memstore Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Memstore Size", "name": "role_health_suppression_region_server_memstore_size", "value": "false" }, { "desc": "The HLog file reader implementation.", "display_name": "HLog Reader Implementation", "name": "hbase_regionserver_hlog_reader_impl", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml parameter.", "display_name": "Suppress Parameter Validation: RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml", "name": "role_config_suppression_hbase_regionserver_config_safety_valve", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: RegionServer Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The period over which to compute the moving average of the HDFS read latency of the HBase RegionServer.", "display_name": "HBase RegionServer HDFS Read Latency Monitoring Period", "name": "regionserver_read_latency_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of HBase RegionServer in Bytes parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of HBase RegionServer in Bytes", "name": "role_config_suppression_hbase_regionserver_java_heapsize", "value": "false" }, { "desc": "The directory for HBase secure bulk Load.", "display_name": "HBase Secure Bulk Load Directory", "name": "hbase_bulkload_staging_dir", "value": "/tmp/hbase-staging" }, { "desc": "An interface that is assignable to HRegionInterface. Used in HBase Client for opening a proxy to a remote HBase RegionServer.", "display_name": "HBase RegionServer Interface Class", "name": "hbase_regionserver_class", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase RegionServer Web UI Address parameter.", "display_name": "Suppress Parameter Validation: HBase RegionServer Web UI Address", "name": "role_config_suppression_hbase_regionserver_info_bindaddress", "value": "false" }, { "desc": "When true, HBase RegionServer will bind to 0.0.0.0. Only available in CDH 4.3 and later.", "display_name": "HBase RegionServer Bind to Wildcard Address", "name": "hbase_regionserver_bind_to_wildcard_address", "value": "true" }, { "desc": "The time, in milliseconds, between 'major' compactions of all HStoreFiles in a region. To disable automated major compactions, set this value to 0.", "display_name": "HBase HRegion Major Compaction", "name": "hbase_hregion_majorcompaction", "value": "604800000" }, { "desc": "The host name or IP address of the DNS name server which a RegionServer should use to determine the host name used by the HBase Master for communication and display purposes.", "display_name": "RegionServer DNS Nameserver", "name": "hbase_regionserver_dns_nameserver", "value": null }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Enable MemStore-Local Allocation Buffer (MSLAB) Allocation Scheme. Note: This feature is experimental in CDH3.", "display_name": "Enable MSLAB Allocation Scheme", "name": "hbase_hregion_memstore_mslab_enabled", "value": "true" }, { "desc": "Enables the health test that the RegionServer is connected to the Master", "display_name": "HBase RegionServer to Master Connectivity Test", "name": "regionserver_master_connectivity_enabled", "value": "true" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": "32768" }, { "desc": "Limit for the number of regions after which no more region splitting should take place. This is not a hard limit for the number of regions but acts as a guideline for the RegionServer to stop splitting after a certain limit.", "display_name": "HBase Region Split Limit", "name": "hbase_regionserver_regionSplitLimit", "value": "2147483647" }, { "desc": "When using Multiple HDFS WAL as the WALProvider, sets how many write-ahead-logs each region server should run. Will result in this number of HDFS pipelines. Writes for a given Region only go to a single pipeline, spreading total RegionServer load.", "display_name": "Per-RegionServer Number of WAL Pipelines", "name": "hbase_regionserver_wal_pipelines", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HLog Writer Implementation parameter.", "display_name": "Suppress Parameter Validation: HLog Writer Implementation", "name": "role_config_suppression_hbase_regionserver_hlog_writer_impl", "value": "false" }, { "desc": "Enables the health test that the RegionServer's process state is consistent with the role configuration", "display_name": "RegionServer Process Health Test", "name": "regionserver_scm_health_enabled", "value": "true" }, { "desc": "Memstore will be flushed to disk if size of the memstore exceeds this value in number of bytes. This value is checked by a thread that runs the frequency specified by hbase.server.thread.wakefrequency.", "display_name": "HBase Memstore Flush Size", "name": "hbase_hregion_memstore_flush_size", "value": "134217728" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "When memstores are being forced to flush to make room in memory, keep flushing until this amount is reached. If this amount is equal to 'hbase.regionserver.global.memstore.upperLimit', then minimum possible flushing will occur when updates are blocked due to memstore limiting.", "display_name": "Low Watermark for Memstore Flush", "name": "hbase_regionserver_global_memstore_lowerLimit", "value": "0.38" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the RegionServer Log Directory parameter.", "display_name": "Suppress Parameter Validation: RegionServer Log Directory", "name": "role_config_suppression_hbase_regionserver_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the RegionServer File Descriptor Limit Validator configuration validator.", "display_name": "Suppress Configuration Validator: RegionServer File Descriptor Limit Validator", "name": "role_config_suppression_regionserver_fd_limit_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_region_server_host_health", "value": "false" }, { "desc": "Whether asynchronous WAL replication to the secondary region replicas is enabled.\u2028 If enabled, a replication peer named 'region_replica_replication' is created\u2028 that tails the logs and replicates the mutations to region replicas for tables that\u2028 have region replication > 1. Disabling this replication also\u2028 requires disabling the replication peer using shell or the ReplicationAdmin Java class.\u2028 Replication to secondary region replicas works over standard intercluster replication. If disabled explicitly, enable replication by setting 'hbase.replication' to true for this feature to work.\u2028", "display_name": "Enable Replication To Secondary Region Replicas", "name": "hbase_region_replica_replication_enabled", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "regionserver_gc_duration_window", "value": "5" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "RegionServer Environment Advanced Configuration Snippet (Safety Valve)", "name": "REGIONSERVER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_region_server_swap_memory_usage", "value": "false" }, { "desc": "Time period in seconds to reset long-running metrics (e.g. compactions). This is an HBase specific configuration.", "display_name": "Extended Period", "name": "hbase_metrics_extended_period", "value": "3600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Comma-separated list of codecs that the RegionServer requires to start. Use this setting to make sure that all RegionServers joining a cluster are installed with a particular set of codecs.", "display_name": "RegionServer Codecs", "name": "hbase_regionserver_codecs", "value": "" }, { "desc": "Number of handlers for processing priority requests in a RegionServer.", "display_name": "HBase RegionServer Meta-Handler Count", "name": "hbase_regionserver_metahandler_count", "value": "10" }, { "desc": "The maximum number of rolled log files to keep for RegionServer logs. Typically used by log4j or logback.", "display_name": "RegionServer Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_region_server_file_descriptor", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the Compaction Queue Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Compaction Queue Size", "name": "role_health_suppression_region_server_compaction_queue", "value": "false" }, { "desc": "Period, in milliseconds, at which to roll the commit log.", "display_name": "HBase RegionServer Log Roll Period", "name": "hbase_regionserver_logroll_period", "value": "3600000" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_region_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Jitter outer bound for major compactions.\u2028 On each RegionServer, the hbase.region.majorcompaction\u2028 interval is multiplied by a random fraction that is inside the bounds of this\u2028 maximum. This + or - product is added to when the next\u2028 major compaction is to run. Major compaction\u2028 should not occur on every RegionServer at the same time. The\u2028 smaller this number, the closer together the compactions.", "display_name": "HBase HRegion Major Compaction Jitter", "name": "hbase_hregion_majorcompaction_jitter", "value": "0.5" }, { "desc": "The period over which to compute the moving average of the flush queue size.", "display_name": "HBase RegionServer Flush Queue Monitoring Period", "name": "regionserver_flush_queue_window", "value": "5" }, { "desc": "Maximum HStoreFile size. If any one of a column families' HStoreFiles has grown to exceed this value, the hosting HRegion is split in two.", "display_name": "HBase Maximum File Size", "name": "hbase_hregion_max_filesize", "value": "10737418240" }, { "desc": "The health test thresholds of the weighted average size of the HBase RegionServer compaction queue over a recent period. See HBase RegionServer Compaction Queue Monitoring Period.", "display_name": "HBase RegionServer Compaction Queue Monitoring Thresholds", "name": "regionserver_compaction_queue_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10.0\"}" }, { "desc": "Read threadpool size used by the RegionServer HBase IPC Server.", "display_name": "RegionServer IPC Read Threadpool Size", "name": "hbase_ipc_server_read_threadpool_size", "value": "10" }, { "desc": "If there are more than this number of HStoreFiles in any one HStore, then updates are blocked for this HRegion until a compaction is completed, or until the value specified for 'hbase.hstore.blockingWaitTime' has been exceeded.", "display_name": "HStore Blocking Store Files", "name": "hbase_hstore_blockingStoreFiles", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "regionserver_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "The HLog file writer implementation.", "display_name": "HLog Writer Implementation", "name": "hbase_regionserver_hlog_writer_impl", "value": null }, { "desc": "Duration between consecutive checks done by the Canary.", "display_name": "Canary Interval", "name": "hbase_regionserver_canary_interval", "value": "6" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The minimum log level for RegionServer logs", "display_name": "RegionServer Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The port that an HBase RegionServer binds to.", "display_name": "HBase RegionServer Port", "name": "hbase_regionserver_port", "value": "60020" }, { "desc": "Number of RPC Server instances spun up on RegionServers.", "display_name": "HBase RegionServer Handler Count", "name": "hbase_regionserver_handler_count", "value": "30" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HBase RegionServer in Bytes", "name": "hbase_regionserver_java_heapsize", "value": "4294967296" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HBase RegionServer Interface Class parameter.", "display_name": "Suppress Parameter Validation: HBase RegionServer Interface Class", "name": "role_config_suppression_hbase_regionserver_class", "value": "false" }, { "desc": "When computing the overall RegionServer health, consider the host's health.", "display_name": "RegionServer Host Health Test", "name": "regionserver_host_health_enabled", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/impala-llama.json0000666000175100017510000007306613245514472024736 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_llama_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether Llama should cache allocated resources on release.", "display_name": "Enable Resource Caching", "name": "llama_am_cache_enabled", "value": "true" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Kerberos principal used by the Impala Llama ApplicationMaster roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "llama" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_llama_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama System Group parameter.", "display_name": "Suppress Parameter Validation: Llama System Group", "name": "role_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_llama_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_llama_host_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "ACL for Impala ApplicationMaster clients. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. \"alice,bob users,wheel\". A special value of \"*\" means all users are allowed. These take effect only if security is enabled.", "display_name": "Client ACLs", "name": "llama_am_server_thrift_client_acl", "value": "*" }, { "desc": "For advanced use only. A string to be inserted into llama-site.xml for this role only.", "display_name": "Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml", "name": "llama_config_valve", "value": null }, { "desc": "When computing the overall Impala Llama ApplicationMaster health, consider the host's health.", "display_name": "Impala Llama ApplicationMaster Host Health Test", "name": "llama_host_health_enabled", "value": "true" }, { "desc": "Port on which the Llama ApplicationMaster listens to HTTP requests.", "display_name": "Llama HTTP Port", "name": "llama_http_port", "value": "15001" }, { "desc": "Maximum number of retries for a client notification. After the maximum number of client notification retries has been reached without success the client is considered lost and all its reservations are released. A successful client notification resets the retries count.", "display_name": "Maximum Client Notification Retries", "name": "llama_am_server_thrift_client_notifier_max_retries", "value": "5" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama Log Directory parameter.", "display_name": "Suppress Parameter Validation: Llama Log Directory", "name": "role_config_suppression_llama_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Advanced Configuration Snippet (Safety Valve) for llama-site.xml", "name": "role_config_suppression_llama_config_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "Enter an XML string that will be inserted verbatim into the Fair Scheduler allocations file. Overrides the configuration set using the Pools configuration UI. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "llama_fair_scheduler_safety_valve", "value": null }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Client notification retry interval, in milliseconds.", "display_name": "Client Notification Retry Interval", "name": "llama_am_server_thrift_client_notifier_retry_interval_ms", "value": "5000" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Enables the health test that the Impala Llama ApplicationMaster's process state is consistent with the role configuration", "display_name": "Impala Llama ApplicationMaster Process Health Test", "name": "llama_scm_health_enabled", "value": "true" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The group that the Llama processes should run as.", "display_name": "Llama System Group", "name": "process_groupname", "value": "llama" }, { "desc": "Timeout policy for resources being cached.", "display_name": "Resource Caching Idle Timeout", "name": "llama_am_cache_eviction_timeout_policy_idle_timeout_ms", "value": "30000" }, { "desc": "Port on which the Llama ApplicationMaster listens to administrative requests on its administrative Thrift interface.", "display_name": "Llama Thrift Admin Port", "name": "llama_am_server_thrift_admin_address", "value": "15002" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Impala Llama ApplicationMaster in Bytes", "name": "llama_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Administrative Interface ACLs parameter.", "display_name": "Suppress Parameter Validation: Administrative Interface ACLs", "name": "role_config_suppression_llama_am_server_thrift_admin_acl", "value": "false" }, { "desc": "Maximum amount of time the backed off reservations will be in 'backed off' state. The actual amount time is a random value between the minimum and the maximum.", "display_name": "Anti-Deadlock Maximum Delay", "name": "llama_am_gang_anti_deadlock_max_delay_ms", "value": "30000" }, { "desc": "Queues Llama ApplicationMaster should connect to at start up.", "display_name": "Core Queues", "name": "llama_am_core_queues", "value": "" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Maximum number of threads used by the Llama ApplicationMaster auxiliary service uses for serving client requests.", "display_name": "Thrift Server Maximum Threads", "name": "llama_am_server_thrift_server_max_threads", "value": "50" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve)", "name": "LLAMA_role_env_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Interval of time without any new allocation that will trigger the Impala ApplicationMaster anti-deadlock logic.", "display_name": "Anti-Deadlock No Allocation Limit Interval", "name": "llama_am_gang_anti_deadlock_no_allocation_limit_ms", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Time in milliseconds after which Llama will discard its AM for a queue that has been empty of reservations. Does not apply to queues specified with the Core Queues property.", "display_name": "Queue Expiration Age", "name": "llama_am_queue_expire_ms", "value": "300000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client ACLs parameter.", "display_name": "Suppress Parameter Validation: Client ACLs", "name": "role_config_suppression_llama_am_server_thrift_client_acl", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Llama ApplicationMaster Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_llama_role_env_safety_valve", "value": "false" }, { "desc": "Minimum number of threads used by the Llama ApplicationMaster auxiliary service uses for serving client requests.", "display_name": "Thrift Server Minimum Threads", "name": "llama_am_server_thrift_server_min_threads", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "llama_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Core Queues parameter.", "display_name": "Suppress Parameter Validation: Core Queues", "name": "role_config_suppression_llama_am_core_queues", "value": "false" }, { "desc": "If enabled, the Impala Llama ApplicationMaster binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind Impala Llama ApplicationMaster to Wildcard Address", "name": "llama_bind_wildcard", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The maximum size, in megabytes, per log file for Impala Llama ApplicationMaster logs. Typically used by log4j or logback.", "display_name": "Impala Llama ApplicationMaster Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Minimum amount of time the backed off reservations will be in 'backed off' state. The actual amount time is a random value between the minimum and the maximum.", "display_name": "Anti-Deadlock Minimum Delay", "name": "llama_am_gang_anti_deadlock_min_delay_ms", "value": "10000" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Percentage of resources that will be backed off by the Impala ApplicationMaster anti-deadlock logic. Random reservations will be backed off until the percentage of backed off resources reaches this percentage.", "display_name": "Anti-Deadlock Backoff Percentage", "name": "llama_am_gang_anti_deadlock_backoff_percent", "value": "30" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Llama Server", "name": "llama_java_opts", "value": "" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Llama ApplicationMaster heartbeat interval, in milliseconds. On each heartbeat the ApplicationMaster submits new reservations to YARN ResourceManager and gets updates from it.", "display_name": "AM Heartbeat Interval", "name": "llama_am_server_thrift_client_notifier_heartbeat_ms", "value": "5000" }, { "desc": "The user that the Llama process should run as.", "display_name": "Llama System User", "name": "process_username", "value": "llama" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Port on which the Llama ApplicationMaster serves its Thrift interface.", "display_name": "Llama Thrift Port", "name": "llama_port", "value": "15000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Fair Scheduler XML Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Fair Scheduler XML Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_llama_fair_scheduler_safety_valve", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Impala Llama ApplicationMaster Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_llama_unexpected_exits", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_llama_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Llama System User parameter.", "display_name": "Suppress Parameter Validation: Llama System User", "name": "role_config_suppression_process_username", "value": "false" }, { "desc": "Whether to break resource requests into smaller requests of standard size before the resource cache. The sizes are taken from Yarn settings Container Memory Increment and Container Virtual CPU Cores Increment.", "display_name": "Enable Resource Cache Normalization", "name": "llama_am_resource_normalizing_enabled", "value": "true" }, { "desc": "The maximum number of rolled log files to keep for Impala Llama ApplicationMaster logs. Typically used by log4j or logback.", "display_name": "Impala Llama ApplicationMaster Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Socket time, in milliseconds, used Llama ApplicationMaster auxiliary service for all its server and client Thrift connections.", "display_name": "Thrift Transport Timeout", "name": "llama_am_server_thrift_transport_timeout_ms", "value": "60000" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_llama_swap_memory_usage", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "ACL for Impala ApplicationMaster admins. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. \"alice,bob users,wheel\". A special value of \"*\" means all users are allowed. These take effect only if security is enabled.", "display_name": "Administrative Interface ACLs", "name": "llama_am_server_thrift_admin_acl", "value": "*" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Llama Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Llama Server", "name": "role_config_suppression_llama_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "Directory where Llama will place its log files.", "display_name": "Llama Log Directory", "name": "llama_log_dir", "value": "/var/log/impala-llama" }, { "desc": "The minimum log level for Impala Llama ApplicationMaster logs", "display_name": "Impala Llama ApplicationMaster Logging Threshold", "name": "log_threshold", "value": "INFO" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/yarn-gateway.json0000666000175100017510000006457213245514472025021 0ustar zuulzuul00000000000000[ { "desc": "For MapReduce job outputs that are compressed, specify the compression codec to use. Will be part of generated client configuration.", "display_name": "Compression Codec of MapReduce Job Output", "name": "mapred_output_compression_codec", "value": "org.apache.hadoop.io.compress.DefaultCodec" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for mapred-site.xml.", "display_name": "MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "mapreduce_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Reduce Task Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: Reduce Task Java Opts Base", "name": "role_config_suppression_mapreduce_reduce_java_opts", "value": "false" }, { "desc": "Whether map tasks should attempt to use the optimized native implementation of the map-side output collector. This can improve performance of many jobs that are shuffle-intensive. Experimental in CDH 5.2.", "display_name": "Enable Optimized Map-side Output Collector", "name": "mapreduce_enable_native_map_output_collector", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Map Task Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Map Task Maximum Heap Size Validator", "name": "role_config_suppression_mapreduce_map_java_opts_max_heap_mapreduce_map_memory_mb_validator", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/hadoop" }, { "desc": "Whether to suppress configuration warnings produced by the ApplicationMaster Java Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: ApplicationMaster Java Maximum Heap Size Validator", "name": "role_config_suppression_yarn_app_mapreduce_am_max_heap_yarn_app_mapreduce_am_resource_mb_validator", "value": "false" }, { "desc": "The number of streams to merge at the same time while sorting files. That is, the number of sort heads to use during the merge sort on the reducer side. This determines the number of open file handles. Merging more files in parallel reduces merge sort iterations and improves run time by eliminating disk I/O. Note that merging more files in parallel uses more memory. If 'io.sort.factor' is set too high or the maximum JVM heap is set too low, excessive garbage collection will occur. The Hadoop default is 10, but Cloudera recommends a higher value. Will be part of generated client configuration.", "display_name": "I/O Sort Factor", "name": "io_sort_factor", "value": "64" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "92" }, { "desc": "Location to store the job history files of running jobs. This is a path on the host where the JobTracker is running.", "display_name": "Running Job History Location", "name": "hadoop_job_history_dir", "value": "/var/log/hadoop-mapreduce/history" }, { "desc": "The number of virtual CPU cores allocated for each map task of a job. This parameter has no effect prior to CDH 4.4.", "display_name": "Map Task CPU Virtual Cores", "name": "mapreduce_map_cpu_vcores", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codec of MapReduce Job Output parameter.", "display_name": "Suppress Parameter Validation: Compression Codec of MapReduce Job Output", "name": "role_config_suppression_mapred_output_compression_codec", "value": "false" }, { "desc": "If enabled, multiple instances of some reduce tasks may be executed in parallel.", "display_name": "Reduce Tasks Speculative Execution", "name": "mapred_reduce_tasks_speculative_execution", "value": "false" }, { "desc": "Limit on the number of counters allowed per job.", "display_name": "Job Counters Limit", "name": "mapreduce_job_counters_limit", "value": "120" }, { "desc": "The application framework to run jobs with. If not set, jobs will be run with the local job runner.", "display_name": "Application Framework", "name": "mapreduce_framework_name", "value": "yarn" }, { "desc": "Whether to suppress configuration warnings produced by the Job Submit Replication Validator configuration validator.", "display_name": "Suppress Configuration Validator: Job Submit Replication Validator", "name": "role_config_suppression_mapreduce_replication_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Reduce Task Maximum Heap Size Validator configuration validator.", "display_name": "Suppress Configuration Validator: Reduce Task Maximum Heap Size Validator", "name": "role_config_suppression_mapreduce_reduce_java_opts_max_heap_mapreduce_reduce_memory_mb_validator", "value": "false" }, { "desc": "Base sleep time between failover attempts. Used only if RM HA is enabled.", "display_name": "Client Failover Sleep Base Time", "name": "client_failover_sleep_base", "value": "100" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Running Job History Location parameter.", "display_name": "Suppress Parameter Validation: Running Job History Location", "name": "role_config_suppression_hadoop_job_history_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into the client configuration for yarn-site.xml.", "display_name": "YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "yarn_client_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Size of buffer for read and write operations of SequenceFiles.", "display_name": "SequenceFile I/O Buffer Size", "name": "io_file_buffer_size", "value": "65536" }, { "desc": "Fraction of the number of map tasks in the job which should be completed before reduce tasks are scheduled for the job.", "display_name": "Number of Map Tasks to Complete Before Reduce Tasks", "name": "mapred_reduce_slowstart_completed_maps", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Maximum Number of Attempts for MapReduce Jobs parameter.", "display_name": "Suppress Parameter Validation: Maximum Number of Attempts for MapReduce Jobs", "name": "role_config_suppression_mapreduce_am_max_attempts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "The maximum heap size, in bytes, of the Java MapReduce ApplicationMaster. This number will be formatted and concatenated with 'ApplicationMaster Java Opts Base' to pass to Hadoop.", "display_name": "ApplicationMaster Java Maximum Heap Size", "name": "yarn_app_mapreduce_am_max_heap", "value": "825955249" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Compression Codec of MapReduce Map Output parameter.", "display_name": "Suppress Parameter Validation: Compression Codec of MapReduce Map Output", "name": "role_config_suppression_mapred_map_output_compression_codec", "value": "false" }, { "desc": "The replication level for submitted job files.", "display_name": "Mapreduce Submit Replication", "name": "mapred_submit_replication", "value": "10" }, { "desc": "The total amount of memory buffer, in megabytes, to use while sorting files. Note that this memory comes out of the user JVM heap size (meaning total user JVM heap - this amount of memory = total user usable heap space. Note that Cloudera's default differs from Hadoop's default; Cloudera uses a bigger buffer by default because modern machines often have more RAM. The smallest value across all TaskTrackers will be part of generated client configuration.", "display_name": "I/O Sort Memory Buffer (MiB)", "name": "io_sort_mb", "value": "256" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_yarn_client_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "The ratio of heap size to container size for both map and reduce tasks. The heap should be smaller than the container size to allow for some overhead of the JVM.", "display_name": "Heap to Container Size Ratio", "name": "mapreduce_job_heap_memory_mb_ratio", "value": "0.8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Client Java Configuration Options parameter.", "display_name": "Suppress Parameter Validation: Client Java Configuration Options", "name": "role_config_suppression_mapreduce_client_java_opts", "value": "false" }, { "desc": "The default number of parallel transfers run by reduce during the copy (shuffle) phase. This number should be between sqrt(nodes*number_of_map_slots_per_node) and nodes*number_of_map_slots_per_node/2. Will be part of generated client configuration.", "display_name": "Default Number of Parallel Transfers During Shuffle", "name": "mapred_reduce_parallel_copies", "value": "10" }, { "desc": "The amount of physical memory, in MiB, allocated for each reduce task of a job. This parameter has no effect prior to CDH 4.4.", "display_name": "Reduce Task Memory", "name": "mapreduce_reduce_memory_mb", "value": "0" }, { "desc": "The maximum permissible size of the split metainfo file. The JobTracker won't attempt to read split metainfo files bigger than the configured value. No limits if set to -1.", "display_name": "JobTracker MetaInfo Maxsize", "name": "mapreduce_jobtracker_split_metainfo_maxsize", "value": "10000000" }, { "desc": "Compression level for the codec used to compress MapReduce outputs. Default compression is a balance between speed and compression ratio.", "display_name": "Compression Level of Codecs", "name": "zlib_compress_level", "value": "DEFAULT_COMPRESSION" }, { "desc": "Whether to enable ubertask optimization, which runs \"sufficiently small\" jobs sequentially within a single JVM. \"Small\" is defined by the mapreduce.job.ubertask.maxmaps, mapreduce.job.ubertask.maxreduces, and mapreduce.job.ubertask.maxbytes settings.", "display_name": "Enable Ubertask Optimization", "name": "mapreduce_job_ubertask_enabled", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Gateway Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Environment parameter.", "display_name": "Suppress Parameter Validation: MR Application Environment", "name": "role_config_suppression_mapreduce_admin_user_env", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Environment parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Environment", "name": "role_config_suppression_yarn_app_mapreduce_am_admin_user_env", "value": "false" }, { "desc": "These are Java command line arguments. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Client Java Configuration Options", "name": "mapreduce_client_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MR Application Classpath parameter.", "display_name": "Suppress Parameter Validation: MR Application Classpath", "name": "role_config_suppression_mapreduce_application_classpath", "value": "false" }, { "desc": "The physical memory requirement, in MiB, for the ApplicationMaster.", "display_name": "ApplicationMaster Memory", "name": "yarn_app_mapreduce_am_resource_mb", "value": "1024" }, { "desc": "For MapReduce map outputs that are compressed, specify the compression codec to use. Will be part of generated client configuration.", "display_name": "Compression Codec of MapReduce Map Output", "name": "mapred_map_output_compression_codec", "value": "org.apache.hadoop.io.compress.SnappyCodec" }, { "desc": "The maximum Java heap size, in bytes, of the reduce processes. This number will be formatted and concatenated with 'Reduce Task Java Opts Base' to pass to Hadoop.", "display_name": "Reduce Task Maximum Heap Size", "name": "mapreduce_reduce_java_opts_max_heap", "value": "0" }, { "desc": "Maximum size in bytes for the Java process heap memory. Passed to Java -Xmx.", "display_name": "Client Java Heap Size in Bytes", "name": "mapreduce_client_java_heapsize", "value": "825955249" }, { "desc": "The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string.", "display_name": "MapReduce Task Timeout", "name": "mapred_task_timeout", "value": "600000" }, { "desc": "The virtual CPU cores requirement, for the ApplicationMaster. This parameter has no effect prior to CDH 4.4.", "display_name": "ApplicationMaster Virtual CPU Cores", "name": "yarn_app_mapreduce_am_resource_cpu_vcores", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Map Task Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: Map Task Java Opts Base", "name": "role_config_suppression_mapreduce_map_java_opts", "value": "false" }, { "desc": "The amount of physical memory, in MiB, allocated for each map task of a job.", "display_name": "Map Task Memory", "name": "mapreduce_map_memory_mb", "value": "0" }, { "desc": "Threshold for number of reduces, beyond which a job is considered too big for ubertask optimization. Note: As of CDH 5, MR2 does not support more than one reduce in an ubertask. (Zero is valid.)", "display_name": "Ubertask Maximum Reduces", "name": "mapreduce_job_ubertask_maxreduces", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_mapreduce_client_config_safety_valve", "value": "false" }, { "desc": "Classpaths to include for MapReduce applications. During evaluation, the string '{version}' in the value of this parameter will be replaced by the actual MapReduce version.", "display_name": "MR Application Classpath", "name": "mapreduce_application_classpath", "value": "$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH" }, { "desc": "Java command line arguments passed to the MapReduce ApplicationMaster.", "display_name": "ApplicationMaster Java Opts Base", "name": "yarn_app_mapreduce_am_command_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Maximum sleep time between failover attempts. Used only if RM HA is enabled.", "display_name": "Client Failover Sleep Max Time", "name": "client_failover_sleep_max", "value": "2000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh parameter.", "display_name": "Suppress Parameter Validation: Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "role_config_suppression_mapreduce_client_env_safety_valve", "value": "false" }, { "desc": "Compress the output of MapReduce jobs. Will be part of generated client configuration.", "display_name": "Compress MapReduce Job Output", "name": "mapred_output_compress", "value": "false" }, { "desc": "For MapReduce job outputs that are compressed as SequenceFiles, you can select one of these compression type options: NONE, RECORD or BLOCK. Cloudera recommends BLOCK. Will be part of generated client configuration.", "display_name": "Compression Type of MapReduce Job Output", "name": "mapred_output_compression_type", "value": "BLOCK" }, { "desc": "Java opts for the map processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp pass a value of: \"-verbose:gc -Xloggc:/tmp/@taskid@.gc\". The configuration variable 'Map Task Memory' can be used to control the maximum memory of the map processes.", "display_name": "Map Task Java Opts Base", "name": "mapreduce_map_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ApplicationMaster Java Opts Base parameter.", "display_name": "Suppress Parameter Validation: ApplicationMaster Java Opts Base", "name": "role_config_suppression_yarn_app_mapreduce_am_command_opts", "value": "false" }, { "desc": "Additional execution environment entries for map and reduce task processes.", "display_name": "MR Application Environment", "name": "mapreduce_admin_user_env", "value": "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH" }, { "desc": "The soft limit in either the buffer or record collection buffers. When this limit is reached, a thread will begin to spill the contents to disk in the background. Note that this does not imply any chunking of data to the spill. A value less than 0.5 is not recommended. The syntax is in decimal units; the default is 80% and is formatted 0.8. Will be part of generated client configuration.", "display_name": "I/O Sort Spill Percent", "name": "io_sort_spill_percent", "value": "0.8" }, { "desc": "The default number of reduce tasks per job. Will be part of generated client configuration.", "display_name": "Default Number of Reduce Tasks per Job", "name": "mapred_reduce_tasks", "value": "1" }, { "desc": "Maximum allowed connections for the shuffle. Set to 0 (zero) to indicate no limit on the number of connections.", "display_name": "Max Shuffle Connections", "name": "mapreduce_shuffle_max_connections", "value": "80" }, { "desc": "The maximum Java heap size, in bytes, of the map processes. This number will be formatted and concatenated with 'Map Task Java Opts Base' to pass to Hadoop.", "display_name": "Map Task Maximum Heap Size", "name": "mapreduce_map_java_opts_max_heap", "value": "0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the I/O Sort Factor parameter.", "display_name": "Suppress Parameter Validation: I/O Sort Factor", "name": "role_config_suppression_io_sort_factor", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into the client configuration for hadoop-env.sh", "display_name": "Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh", "name": "mapreduce_client_env_safety_valve", "value": null }, { "desc": "The number of virtual CPU cores for each reduce task of a job.", "display_name": "Reduce Task CPU Virtual Cores", "name": "mapreduce_reduce_cpu_vcores", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the Max Attempts Validator configuration validator.", "display_name": "Suppress Configuration Validator: Max Attempts Validator", "name": "role_config_suppression_yarn_mapreduce_max_attempts_validator", "value": "false" }, { "desc": "Environment variables for the MapReduce ApplicationMaster. These settings can be overridden in the ApplicationMaster User Environment (yarn.app.mapreduce.am.env).", "display_name": "ApplicationMaster Environment", "name": "yarn_app_mapreduce_am_admin_user_env", "value": "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH" }, { "desc": "Java opts for the reduce processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp pass a value of: \"-verbose:gc -Xloggc:/tmp/@taskid@.gc\". The configuration variable 'Reduce Task Memory' can be used to control the maximum memory of the reduce processes.", "display_name": "Reduce Task Java Opts Base", "name": "mapreduce_reduce_java_opts", "value": "-Djava.net.preferIPv4Stack=true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "If enabled, uses compression on the map outputs before they are sent across the network. Will be part of generated client configuration.", "display_name": "Use Compression on Map Outputs", "name": "mapred_compress_map_output", "value": "true" }, { "desc": "The maximum number of application attempts for MapReduce jobs. The value of this parameter overrides ApplicationMaster Maximum Attempts for MapReduce jobs.", "display_name": "Maximum Number of Attempts for MapReduce Jobs", "name": "mapreduce_am_max_attempts", "value": "2" }, { "desc": "The minimum log level for Gateway logs", "display_name": "Gateway Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Threshold for number of maps, beyond which a job is considered too big for ubertask optimization.", "display_name": "Ubertask Maximum Maps", "name": "mapreduce_job_ubertask_maxmaps", "value": "9" }, { "desc": "If enabled, multiple instances of some map tasks may be executed in parallel.", "display_name": "Map Tasks Speculative Execution", "name": "mapred_map_tasks_speculative_execution", "value": "false" }, { "desc": "Threshold for number of input bytes, beyond which a job is considered too big for ubertask optimization. If no value is specified, dfs.block.size is used as a default.", "display_name": "Ubertask Maximum Job Size", "name": "mapreduce_job_ubertask_maxbytes", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hue-hue_server.json0000666000175100017510000005323413245514472025330 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Top Banner Custom HTML parameter.", "display_name": "Suppress Parameter Validation: Top Banner Custom HTML", "name": "role_config_suppression_banner_html", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Hue TLS/SSL Validator configuration validator.", "display_name": "Suppress Configuration Validator: Hue TLS/SSL Validator", "name": "role_config_suppression_hue_ssl_validator", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Path to TLS/SSL Private Key parameter.", "display_name": "Suppress Parameter Validation: Path to TLS/SSL Private Key", "name": "role_config_suppression_ssl_private_key", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "hue_server_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "An optional, custom one-line HTML code to display as a banner on top of all Hue Server web pages. Useful in displaying cluster identity of the Hue Server.", "display_name": "Top Banner Custom HTML", "name": "banner_html", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Private Key Password parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Private Key Password", "name": "role_config_suppression_ssl_private_key_password", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_hue_server_unexpected_exits", "value": "false" }, { "desc": "Enable HTTPS for the Hue web server.", "display_name": "Enable Hue Server HTTPS", "name": "ssl_enable", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Directory where Hue Server will place its log files.", "display_name": "Hue Server Log Directory", "name": "hue_server_log_dir", "value": "/var/log/hue" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_hue_server_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hue Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hue_server_role_env_safety_valve", "value": "false" }, { "desc": "Timeout in seconds for Thrift calls to HiveServer2 and Impala.", "display_name": "HiveServer2 and Impala Thrift Connection Timeout", "name": "hs2_conn_timeout", "value": "120" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Path to TLS/SSL Certificate parameter.", "display_name": "Suppress Parameter Validation: Path to TLS/SSL Certificate", "name": "role_config_suppression_ssl_certificate", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_hue_server_web_metric_collection", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: Hue Server Log Directory", "name": "role_config_suppression_hue_server_log_dir", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_hue_server_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_hue_server_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_hue_server_heap_dump_directory_free_space", "value": "false" }, { "desc": "Location on HDFS where the jobsub examples and templates are stored.", "display_name": "Jobsub Examples and Templates Directory", "name": "hue_server_remote_data_dir", "value": "/user/hue/jobsub" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_hue_server_file_descriptor", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_hue_server_host_health", "value": "false" }, { "desc": "When computing the overall Hue Server health, consider the host's health.", "display_name": "Hue Server Host Health Test", "name": "hue_server_host_health_enabled", "value": "true" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Port to use to connect to the Hue server.", "display_name": "Hue HTTP Port", "name": "hue_http_port", "value": "8888" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Jobsub Examples and Templates Directory parameter.", "display_name": "Suppress Parameter Validation: Jobsub Examples and Templates Directory", "name": "role_config_suppression_hue_server_remote_data_dir", "value": "false" }, { "desc": "The frequency at which the metrics are logged to the sample file.", "display_name": "Metrics Sample File Logging Frequency", "name": "hue_metrics_sample_logging_frequency", "value": "30000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Path to the TLS/SSL certificate on the host running the Hue web server. This file must be in PEM format, and must be readable by the Hue system user.", "display_name": "Path to TLS/SSL Certificate", "name": "ssl_certificate", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Secret Key parameter.", "display_name": "Suppress Parameter Validation: Secret Key", "name": "role_config_suppression_secret_key", "value": "false" }, { "desc": "The full path to a file with a sample of metrics exposed by the role. The sample is updated at the frequency configured by Metrics Sample File Logging Frequency. By default, the sample file is logged to a directory under the role log directory, e.g., /var/log/hue/metrics-hue_server/metrics.log.", "display_name": "Metrics Sample File Location", "name": "hue_metrics_sample_file_location", "value": null }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Hue Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "HUE_SERVER_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini parameter.", "display_name": "Suppress Parameter Validation: Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini", "name": "role_config_suppression_hue_server_hue_safety_valve", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "hue_server_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Metrics Sample File Location parameter.", "display_name": "Suppress Parameter Validation: Metrics Sample File Location", "name": "role_config_suppression_hue_metrics_sample_file_location", "value": "false" }, { "desc": "Path to the TLS/SSL truststore on the host running the Hue web server. This file must be readable by the Hue system user and in PEM format. The truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. This file is created by concatenating together, in PEM format, all of the appropriate certificates. If not set, the system's default OpenSSL trusted certificate authority list is used. This option replaces the former REQUESTS_CA_BUNDLE environment variable.", "display_name": "TLS/SSL Truststore", "name": "ssl_cacerts", "value": null }, { "desc": "Path to the TLS/SSL private key on the host running the Hue web server. This file must be in PEM format, and must be readable by the Hue system user.", "display_name": "Path to TLS/SSL Private Key", "name": "ssl_private_key", "value": null }, { "desc": "For advanced use only. A string to be inserted into hue_safety_valve_server.ini for this role only.", "display_name": "Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini", "name": "hue_server_hue_safety_valve", "value": null }, { "desc": "The password protecting the TLS/SSL Private Key. Leave blank if there is no password.", "display_name": "TLS/SSL Private Key Password", "name": "ssl_private_key_password", "value": null }, { "desc": "If enabled, the Hue server binds to the wildcard address (\"0.0.0.0\") for its ports.", "display_name": "Bind Hue Server to Wildcard Address", "name": "hue_server_bind_wildcard", "value": "false" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "hue_server_web_metric_collection_enabled", "value": "true" }, { "desc": "Enables the health test that the Hue Server's process state is consistent with the role configuration", "display_name": "Hue Server Process Health Test", "name": "hue_server_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the TLS/SSL Truststore parameter.", "display_name": "Suppress Parameter Validation: TLS/SSL Truststore", "name": "role_config_suppression_ssl_cacerts", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Random string used for secure hashing in the session store.", "display_name": "Secret Key", "name": "secret_key", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hive-metastore-sentry-safety.xml0000666000175100017510000000135213245514472027772 0ustar zuulzuul00000000000000 hive.metastore.client.impl org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient Sets custom Hive metastore client which Sentry uses to filter out metadata. hive.metastore.pre.event.listeners org.apache.sentry.binding.metastore.MetastoreAuthzBinding list of comma separated listeners for metastore events. hive.metastore.event.listeners org.apache.sentry.binding.metastore.SentryMetastorePostEventListener list of comma separated listeners for metastore, post events. sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hive-webhcat.json0000666000175100017510000005134213245514472024746 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_webhcat_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_webhcat_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_webhcat_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_webhcat_scm_health", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Port on which WebHCat Server will listen for connections.", "display_name": "WebHCat Server Port", "name": "hive_webhcat_address_port", "value": "50111" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hive_webhcat_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of WebHCat Server in Bytes", "name": "hive_webhcat_java_heapsize", "value": "268435456" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Log Directory", "name": "role_config_suppression_hcatalog_log_dir", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hive-site.xml for this role only.", "display_name": "WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "hive_webhcat_hive_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "webhcat_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "WebHCat Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "WebHCat Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "hive_webhcat_env_safety_valve", "value": null }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for WebHCat Server", "name": "hive_webhcat_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for WebHCat Server logs. Typically used by log4j or logback.", "display_name": "WebHCat Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Enables the health test that the WebHCat Server's process state is consistent with the role configuration", "display_name": "WebHCat Server Process Health Test", "name": "webhcat_scm_health_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_webhcat_swap_memory_usage", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into webhcat-site.xml for this role only.", "display_name": "WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml", "name": "hive_webhcat_config_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml", "name": "role_config_suppression_hive_webhcat_hive_config_safety_valve", "value": "false" }, { "desc": "When computing the overall WebHCat Server health, consider the host's health.", "display_name": "WebHCat Server Host Health Test", "name": "webhcat_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_webhcat_host_health", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_webhcat_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for WebHCat Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for WebHCat Server", "name": "role_config_suppression_hive_webhcat_java_opts", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for WebHCat Server logs. Typically used by log4j or logback.", "display_name": "WebHCat Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml parameter.", "display_name": "Suppress Parameter Validation: WebHCat Server Advanced Configuration Snippet (Safety Valve) for webhcat-site.xml", "name": "role_config_suppression_hive_webhcat_config_safety_valve", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The minimum log level for WebHCat Server logs", "display_name": "WebHCat Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Directory where WebHCat Server will place its log files.", "display_name": "WebHCat Server Log Directory", "name": "hcatalog_log_dir", "value": "/var/log/hcatalog" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/mapred-site.xml0000666000175100017510000001050113245514472024431 0ustar zuulzuul00000000000000 mapreduce.job.split.metainfo.maxsize 10000000 mapreduce.job.counters.max 120 mapreduce.output.fileoutputformat.compress false mapreduce.output.fileoutputformat.compress.type BLOCK mapreduce.output.fileoutputformat.compress.codec org.apache.hadoop.io.compress.DefaultCodec mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.compress true zlib.compress.level DEFAULT_COMPRESSION mapreduce.task.io.sort.factor 64 mapreduce.map.sort.spill.percent 0.8 mapreduce.reduce.shuffle.parallelcopies 10 mapreduce.task.timeout 600000 mapreduce.client.submit.file.replication 10 mapreduce.job.reduces 1 mapreduce.task.io.sort.mb 256 mapreduce.map.speculative false mapreduce.reduce.speculative false mapreduce.job.reduce.slowstart.completedmaps 0.8 mapreduce.jobhistory.address test-master-001.novalocal:10020 mapreduce.jobhistory.webapp.address test-master-001.novalocal:19888 mapreduce.jobhistory.webapp.https.address test-master-001.novalocal:19890 mapreduce.jobhistory.admin.address test-master-001.novalocal:10033 mapreduce.framework.name yarn yarn.app.mapreduce.am.staging-dir /user yarn.app.mapreduce.am.resource.mb 1024 yarn.app.mapreduce.am.resource.cpu-vcores 1 mapreduce.job.ubertask.enable false yarn.app.mapreduce.am.command-opts -Djava.net.preferIPv4Stack=true -Xmx825955249 mapreduce.map.java.opts -Djava.net.preferIPv4Stack=true -Xmx825955249 mapreduce.reduce.java.opts -Djava.net.preferIPv4Stack=true -Xmx825955249 yarn.app.mapreduce.am.admin.user.env LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH mapreduce.map.memory.mb 1024 mapreduce.map.cpu.vcores 1 mapreduce.reduce.memory.mb 1024 mapreduce.reduce.cpu.vcores 1 mapreduce.application.classpath $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH mapreduce.admin.user.env LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH mapreduce.shuffle.max.connections 80 sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/sqoop-service.json0000666000175100017510000001606713245514472025204 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_sqoop_env_safety_valve", "value": "false" }, { "desc": "When computing the overall SQOOP health, consider Sqoop 2 Server's health", "display_name": "Sqoop 2 Server Role Health Test", "name": "sqoop_sqoop_server_health_enabled", "value": "true" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "sqoop" }, { "desc": "Whether to suppress the results of the Sqoop 2 Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Sqoop 2 Server Health", "name": "service_health_suppression_sqoop_sqoop_server_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Sqoop 2 Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Sqoop 2 Server Count Validator", "name": "service_config_suppression_sqoop_server_count_validator", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Sqoop 2 Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "sqoop_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "sqoop2" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "sqoop2" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "MapReduce jobs are run against this service.", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/solr-gateway.json0000666000175100017510000000261613245514472025016 0ustar zuulzuul00000000000000[ { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Deploy Directory parameter.", "display_name": "Suppress Parameter Validation: Deploy Directory", "name": "role_config_suppression_client_config_root_dir", "value": "false" }, { "desc": "The directory where the client configs will be deployed", "display_name": "Deploy Directory", "name": "client_config_root_dir", "value": "/etc/solr" }, { "desc": "The priority level that the client configuration will have in the Alternatives system on the hosts. Higher priority levels will cause Alternatives to prefer this configuration over any others.", "display_name": "Alternatives Priority", "name": "client_config_priority", "value": "90" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/impala-impalad.json0000666000175100017510000011572713245514472025260 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Impala Daemon Scratch Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Scratch Directories Free Space", "name": "role_health_suppression_impalad_scratch_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Lineage Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Lineage Enabled Validator", "name": "role_config_suppression_impala_lineage_enabled_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemons Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Impala Daemons Load Balancer", "name": "role_config_suppression_impalad_load_balancer", "value": "false" }, { "desc": "Encrypt and verify the integrity of all data spilled to disk as part of a query. This feature is only supported for Impala 2.0 and higher and CDH 5.2 and higher (which includes Impala 2.0).", "display_name": "Disk Spill Encryption", "name": "disk_spill_encryption", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_cmd_args_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Impala Audit Enabled Validator configuration validator.", "display_name": "Suppress Configuration Validator: Impala Audit Enabled Validator", "name": "role_config_suppression_impala_audit_enabled_validator", "value": "false" }, { "desc": "Directory where an Impala Daemon core dump is placed.", "display_name": "Impala Daemon Core Dump Directory", "name": "core_dump_dir", "value": "/var/log/impalad" }, { "desc": "Port on which HiveServer2 client requests are served by Impala Daemons.", "display_name": "Impala Daemon HiveServer2 Port", "name": "hs2_port", "value": "21050" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_fair_scheduler_safety_valve", "value": "false" }, { "desc": "Maximum number of seconds that Impala attempts to register or re-register with Llama. If registration is unsuccessful, Impala cancels the action with an error, which could result in an impalad startup failure or a cancelled query. A setting of -1 seconds means try indefinitely.", "display_name": "Llama Registration Timeout Seconds", "name": "llama_registration_timeout_secs", "value": "30" }, { "desc": "When computing the overall Impala Daemon health, consider the host's health.", "display_name": "Impala Daemon Host Health Test", "name": "impalad_host_health_enabled", "value": "true" }, { "desc": "The directory in which Impala daemon audit event log files are written. If \"Impala Audit Event Generation\" property is enabled, Impala will generate its audit logs in this directory.", "display_name": "Impala Daemon Audit Log Directory", "name": "audit_event_log_dir", "value": "/var/log/impalad/audit" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "A list of key-value pairs of additional query options to pass to the Impala Daemon command line, separated by ','.", "display_name": "Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve)", "name": "default_query_options", "value": "" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_impalad_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Web Server User Password parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Web Server User Password", "name": "role_config_suppression_webserver_htpassword_password", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Enable or disable the Impala Daemon web server. This web server contains useful information about Impala Daemon.", "display_name": "Enable Impala Daemon Web Server", "name": "impalad_enable_webserver", "value": "true" }, { "desc": "An XML snippet to append to llama-site.xml for Impala Daemons. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve)", "name": "impala_llama_site_conf_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "impalad_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)", "name": "IMPALAD_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Web Server Username parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Web Server Username", "name": "role_config_suppression_webserver_htpassword_user", "value": "false" }, { "desc": "Enables the health test that verifies the Impala Daemon is connected to the StateStore.", "display_name": "Impala Daemon Connectivity Health Test", "name": "impalad_connectivity_health_enabled", "value": "true" }, { "desc": "The amount of time at Impala Daemon startup allowed for the Impala Daemon to start accepting new queries for processing.", "display_name": "Impala Daemon Ready Status Startup Tolerance", "name": "impalad_ready_status_check_startup_tolerance", "value": "180" }, { "desc": "The maximum size (in queries) of the Impala Daemon audit event log file before a new one is created.", "display_name": "Impala Daemon Maximum Audit Log File Size", "name": "max_audit_event_log_file_size", "value": "5000" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_impalad_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Local UDF Library Dir parameter.", "display_name": "Suppress Parameter Validation: Local UDF Library Dir", "name": "role_config_suppression_local_library_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the LDAP Server CA Certificate parameter.", "display_name": "Suppress Parameter Validation: LDAP Server CA Certificate", "name": "role_config_suppression_impalad_ldap_ca_certificate", "value": "false" }, { "desc": "The directory in which Impala daemon lineage log files are written. If \"Impala Lineage Generation\" property is enabled, Impala generates its lineage logs in this directory.", "display_name": "Impala Daemon Lineage Log Directory", "name": "lineage_event_log_dir", "value": "/var/log/impalad/lineage" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Llama Site Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_llama_site_conf_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SSL/TLS Certificate for Impala Daemon Webserver parameter.", "display_name": "Suppress Parameter Validation: SSL/TLS Certificate for Impala Daemon Webserver", "name": "role_config_suppression_webserver_certificate_file", "value": "false" }, { "desc": "User-defined function (UDF) libraries are copied from HDFS into this local directory.", "display_name": "Local UDF Library Dir", "name": "local_library_dir", "value": "/var/lib/impala/udfs" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Audit Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Audit Log Directory", "name": "role_config_suppression_audit_event_log_dir", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "impalad_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_impalad_unexpected_exits", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for Impala Daemon logs. Typically used by log4j or logback.", "display_name": "Impala Daemon Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Username for Impala Daemon webserver authentication.", "display_name": "Impala Daemon Web Server Username", "name": "webserver_htpassword_user", "value": null }, { "desc": "Password for Impala Daemon webserver authentication.", "display_name": "Impala Daemon Web Server User Password", "name": "webserver_htpassword_password", "value": null }, { "desc": "Directory where Impala Daemon will place its log files.", "display_name": "Impala Daemon Log Directory", "name": "log_dir", "value": "/var/log/impalad" }, { "desc": "An XML string to use verbatim for the contents of fair-scheduler.xml for Impala Daemons. This configuration only has effect on Impala versions 1.3 or greater.", "display_name": "Impala Daemon Fair Scheduler Advanced Configuration Snippet (Safety Valve)", "name": "impalad_fair_scheduler_safety_valve", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be added (verbatim) to Impala Daemon command-line flags. Key names should begin with a hyphen(-). For example: -log_filename=foo.log", "display_name": "Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)", "name": "impalad_cmd_args_safety_valve", "value": null }, { "desc": "Number of seconds to wait between attempts during Llama registration.", "display_name": "Llama Registration Wait Seconds", "name": "llama_registration_wait_secs", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Query Options Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_default_query_options", "value": "false" }, { "desc": "Port on which ImpalaBackendService is exported.", "display_name": "Impala Daemon Backend Port", "name": "be_port", "value": "22000" }, { "desc": "For advanced use only, a string to be inserted into hbase-site.xml for this role only.", "display_name": "Impala Daemon HBase Advanced Configuration Snippet (Safety Valve)", "name": "impalad_hbase_conf_safety_valve", "value": null }, { "desc": "For advanced use only, a string to be inserted into hive-site.xml for this role only.", "display_name": "Impala Daemon Hive Advanced Configuration Snippet (Safety Valve)", "name": "impala_hive_conf_safety_valve", "value": null }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds on the resident size of the process.", "display_name": "Resident Set Size Thresholds", "name": "process_resident_set_size_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon HBase Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon HBase Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_hbase_conf_safety_valve", "value": "false" }, { "desc": "Verbose logging level for the GLog logger. These messages are always logged at 'INFO' log level, so this setting has no effect if Logging Threshold is set to 'WARN' or above. ", "display_name": "Impala Daemon Verbose Log Level", "name": "log_verbose_level", "value": "1" }, { "desc": "Whether to suppress the results of the Resident Set Size heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Resident Set Size", "name": "role_health_suppression_impalad_memory_rss_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_hdfs_site_conf_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_impalad_swap_memory_usage", "value": "false" }, { "desc": "The timeout used by the Cloudera Manager Agent's query monitor when communicating with the Impala Daemon web server, specified in seconds.", "display_name": "Query Monitoring Timeout", "name": "executing_queries_timeout_seconds", "value": "5.0" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_impalad_heap_dump_directory_free_space", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Hive Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Hive Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impala_hive_conf_safety_valve", "value": "false" }, { "desc": "Enables audit event generation by Impala daemons. The audit log file will be placed in the directory specified by 'Impala Daemon Audit Log Directory' parameter.", "display_name": "Enable Impala Audit Event Generation", "name": "enable_audit_event_log", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Scratch Directories parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Scratch Directories", "name": "role_config_suppression_scratch_dirs", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_impalad_scm_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_impalad_file_descriptor", "value": "false" }, { "desc": "Whether to suppress the results of the Impala Daemon Ready Check heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Impala Daemon Ready Check", "name": "role_health_suppression_impalad_ready_status", "value": "false" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"any\"}" }, { "desc": "Enables the health check that determines if the Impala daemon is ready to process queries.", "display_name": "Impala Daemon Ready Status Health Check", "name": "impalad_ready_status_check_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Core Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Core Dump Directory", "name": "role_config_suppression_core_dump_dir", "value": "false" }, { "desc": "The polling period of the Impala query monitor in the Cloudera Manager Agent, specified in seconds. If set to zero, query monitoring is disabled.", "display_name": "Query Monitoring Period", "name": "query_monitoring_period_seconds", "value": "1.0" }, { "desc": "Port on which Beeswax client requests are served by Impala Daemons.", "display_name": "Impala Daemon Beeswax Port", "name": "beeswax_port", "value": "21000" }, { "desc": "Local path to the certificate presented by the Impala daemon debug webserver. This file must be in .pem format. If empty, webserver SSL/TLS support is not enabled.", "display_name": "SSL/TLS Certificate for Impala Daemon Webserver", "name": "webserver_certificate_file", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_impalad_role_env_safety_valve", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Impala Daemon Scratch Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Impala Daemon Scratch Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Impala Daemon Scratch Directories Free Space Monitoring Percentage Thresholds", "name": "impalad_scratch_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Maximum number of query results a client may request to be cached on a per-query basis to support restarting fetches. This option guards against unreasonably large result caches requested by clients. Requests exceeding this maximum will be rejected.", "display_name": "Result Cache Maximum Size", "name": "impalad_result_cache_max_size", "value": "100000" }, { "desc": "Memory limit in bytes for Impala Daemon, enforced by the daemon itself. If reached, queries running on the Impala Daemon may be killed. Leave it blank to let Impala pick its own limit. Use a value of -1 B to specify no limit.", "display_name": "Impala Daemon Memory Limit", "name": "impalad_memory_limit", "value": null }, { "desc": "Abort Impala startup if there are improper configs or running on unsupported hardware.", "display_name": "Abort on Config Error", "name": "abort_on_config_error", "value": "true" }, { "desc": "Directories where Impala Daemon will write data such as spilling information to disk to free up memory. This can potentially be large amounts of data.", "display_name": "Impala Daemon Scratch Directories", "name": "scratch_dirs", "value": null }, { "desc": "Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; 1 means buffer WARNING only, ...)", "display_name": "Impala Daemon Log Buffer Level", "name": "logbuflevel", "value": "0" }, { "desc": "The amount of time to wait for the Impala Daemon to fully start up and connect to the StateStore before enforcing the connectivity check.", "display_name": "Impala Daemon Connectivity Tolerance at Startup", "name": "impalad_connectivity_tolerance", "value": "180" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_impalad_host_health", "value": "false" }, { "desc": "The maximum size (in entries) of the Impala daemon lineage log file before a new one is created.", "display_name": "Impala Daemon Maximum Lineage Log File Size", "name": "max_lineage_log_file_size", "value": "5000" }, { "desc": "The location on disk of the certificate, in .pem format, used to confirm the authenticity of the LDAP server certificate. This is the Certificate Authority (CA) certificate, and it was used to sign the LDAP server certificate. If not set, Impala by default trusts all certificates supplied by the LDAP server, which means that an attacker could potentially intercept otherwise encrypted usernames and passwords.", "display_name": "LDAP Server CA Certificate", "name": "impalad_ldap_ca_certificate", "value": null }, { "desc": "Timeout for requests to the Hive Metastore Server from Impala. Consider increasing this if you have tables with a lot of metadata and see timeout errors.", "display_name": "Impala Daemon Hive Metastore Connection Timeout", "name": "hive_metastore_timeout", "value": "3600" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Port where StateStoreSubscriberService is running.", "display_name": "StateStoreSubscriber Service Port", "name": "state_store_subscriber_port", "value": "23000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Impala Daemon Lineage Log Directory parameter.", "display_name": "Suppress Parameter Validation: Impala Daemon Lineage Log Directory", "name": "role_config_suppression_lineage_event_log_dir", "value": "false" }, { "desc": "Address of the load balancer used for Impala daemons. Should be specified in host:port format. If this is specified and Kerberos is enabled, Cloudera Manager adds a principal for 'impala/<load_balancer_host>@<realm>' to the keytab for all Impala daemons.", "display_name": "Impala Daemons Load Balancer", "name": "impalad_load_balancer", "value": null }, { "desc": "Maximum number of times a request to reserve, expand, or release resources is attempted until the request is cancelled.", "display_name": "Llama Maximum Request Attempts", "name": "llama_max_request_attempts", "value": "5" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The minimum log level for Impala Daemon logs", "display_name": "Impala Daemon Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Port where Impala debug web server runs.", "display_name": "Impala Daemon HTTP Server Port", "name": "impalad_webserver_port", "value": "25000" }, { "desc": "Enables the health test that the Impala Daemon's process state is consistent with the role configuration", "display_name": "Impala Daemon Process Health Test", "name": "impalad_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the StateStore Connectivity heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: StateStore Connectivity", "name": "role_health_suppression_impalad_connectivity", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Enables lineage generation by Impala daemons. The lineage log file is placed in the directory specified by the 'Impala Daemon Lineage Log Directory' parameter.", "display_name": "Enable Impala Lineage Generation", "name": "enable_lineage_log", "value": "true" }, { "desc": "Port where Llama notification callback should be started", "display_name": "Llama Callback Port", "name": "llama_callback_port", "value": "28000" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's Impala Daemon Scratch Directories.", "display_name": "Impala Daemon Scratch Directories Free Space Monitoring Absolute Thresholds", "name": "impalad_scratch_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "For advanced use only, a string to be inserted into hdfs-site.xml for this role only.", "display_name": "Impala Daemon HDFS Advanced Configuration Snippet (Safety Valve)", "name": "impala_hdfs_site_conf_safety_valve", "value": null }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "impalad_web_metric_collection_enabled", "value": "true" }, { "desc": "The number of log files that are kept for each severity level before all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files are retained and log rotation is effectively disabled.", "display_name": "Impala Maximum Log Files", "name": "max_log_files", "value": "10" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-namenode.json0000666000175100017510000015076013245514472025114 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress the results of the Checkpoint Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Checkpoint Status", "name": "role_health_suppression_name_node_ha_checkpoint_age", "value": "false" }, { "desc": "Name of the journal located on each JournalNode filesystem.", "display_name": "Quorum-based Storage Journal name", "name": "dfs_namenode_quorum_journal_name", "value": null }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Indicate whether or not to avoid reading from stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Stale DataNodes are moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.", "display_name": "Avoid Reading Stale DataNode", "name": "dfs_namenode_avoid_read_stale_datanode", "value": "false" }, { "desc": "The base port where the DFS NameNode web UI listens. If the port number is 0, then the server starts on a free port. Combined with the NameNode's hostname to build its HTTP address.", "display_name": "NameNode Web UI Port", "name": "dfs_http_port", "value": "50070" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for NameNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for NameNode", "name": "role_config_suppression_namenode_java_opts", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_name_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "namenode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "The health check thresholds of the NameNode's RPC latency.", "display_name": "NameNode RPC Latency Thresholds", "name": "namenode_rpc_latency_thresholds", "value": "{\"critical\":\"5000.0\",\"warning\":\"1000.0\"}" }, { "desc": "The period to review when computing the moving average of extra time the pause monitor spent paused.", "display_name": "Pause Duration Monitoring Period", "name": "namenode_pause_duration_window", "value": "5" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_name_node_log_directory_free_space", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress the results of the RPC Latency heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: RPC Latency", "name": "role_health_suppression_name_node_rpc_latency", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_name_node_file_descriptor", "value": "false" }, { "desc": "The maximum number of outgoing replication threads a node can have at one time. This limit is waived for the highest priority replications. Configure dfs.namenode.replication.max-streams-hard-limit to set the absolute limit, including the highest-priority replications.", "display_name": "Maximum Number of Replication Threads on a DataNode", "name": "dfs_namenode_replication_max_streams", "value": "20" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "NameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "When the ratio of number stale DataNodes to total DataNodes marked is greater than this ratio, permit writing to stale nodes to prevent causing hotspots.", "display_name": "Write Stale DataNode Ratio", "name": "dfs_namenode_write_stale_datanode_ratio", "value": "0.5" }, { "desc": "The health test thresholds of the number of transactions since the last HDFS namespace checkpoint. Specified as a percentage of the configured checkpointing transaction limit.", "display_name": "Filesystem Checkpoint Transactions Monitoring Thresholds", "name": "namenode_checkpoint_transactions_thresholds", "value": "{\"critical\":\"400.0\",\"warning\":\"200.0\"}" }, { "desc": "The health test thresholds for the weighted average extra time the pause monitor spent paused. Specified as a percentage of elapsed wall clock time.", "display_name": "Pause Duration Thresholds", "name": "namenode_pause_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_name_node_swap_memory_usage", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress the results of the NameNode Data Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: NameNode Data Directories Free Space", "name": "role_health_suppression_name_node_data_directories_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Filesystem Trash Interval On Validator configuration validator.", "display_name": "Suppress Configuration Validator: Filesystem Trash Interval On Validator", "name": "role_config_suppression_fs_trash_interval_minimum_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_namenode_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Safe Mode Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Safe Mode Status", "name": "role_health_suppression_name_node_safe_mode", "value": "false" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_name_node_web_metric_collection", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Plugins parameter.", "display_name": "Suppress Parameter Validation: NameNode Plugins", "name": "role_config_suppression_dfs_namenode_plugins_list", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Nameservice parameter.", "display_name": "Suppress Parameter Validation: NameNode Nameservice", "name": "role_config_suppression_dfs_federation_namenode_nameservice", "value": "false" }, { "desc": "Enables the health test of the rolling metadata upgrade status of the NameNode. This covers rolling metadata upgrades. Nonrolling metadata upgrades are covered in a separate health test.", "display_name": "HDFS Rolling Metadata Upgrade Status Health Test", "name": "namenode_rolling_upgrade_status_enabled", "value": "true" }, { "desc": "Mount points that are mapped to this NameNode's nameservice.", "display_name": "Mount Points", "name": "nameservice_mountpoints", "value": "/" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "namenode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"},\n {\"alert\": false, \"rate\": 1, \"threshold\":\"INFO\", \"content\":\"Triggering checkpoint.*\"}\n ]\n}\n" }, { "desc": "Enables the health test that the NameNode is not in safemode", "display_name": "NameNode Safemode Health Test", "name": "namenode_safe_mode_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Edits Directories parameter.", "display_name": "Suppress Parameter Validation: NameNode Edits Directories", "name": "role_config_suppression_dfs_namenode_edits_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt", "name": "role_config_suppression_namenode_hosts_exclude_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Service Handler Count Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Service Handler Count Minimum Validator", "name": "role_config_suppression_dfs_namenode_service_handler_count_minimum_validator", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of NameNode in Bytes", "name": "namenode_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress the results of the JournalNode Sync Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: JournalNode Sync Status", "name": "role_health_suppression_name_node_journal_node_sync_status", "value": "false" }, { "desc": "The maximum size, in megabytes, per log file for NameNode logs. Typically used by log4j or logback.", "display_name": "NameNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Determines where on the local file system the NameNode should store the name table (fsimage). For redundancy, enter a comma-delimited list of directories to replicate the name table in all of the directories. Typical values are /data/N/dfs/nn where N=1..3.", "display_name": "NameNode Data Directories", "name": "dfs_name_dir_list", "value": null }, { "desc": "Whether to suppress the results of the Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Upgrade Status", "name": "role_health_suppression_name_node_upgrade_status", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Shared Edits Directory parameter.", "display_name": "Suppress Parameter Validation: Shared Edits Directory", "name": "role_config_suppression_dfs_namenode_shared_edits_dir", "value": "false" }, { "desc": "Default time interval for marking a DataNode as \"stale\". If the NameNode has not received heartbeat messages from a DataNode for more than this time interval, the DataNode is marked and treated as \"stale\" by default.", "display_name": "Stale DataNode Time Interval", "name": "dfs_namenode_stale_datanode_interval", "value": "30000" }, { "desc": "Whether to suppress the results of the Pause Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Pause Duration", "name": "role_health_suppression_name_node_pause_duration", "value": "false" }, { "desc": "The port where the NameNode runs the HDFS protocol. Combined with the NameNode's hostname to build its address.", "display_name": "NameNode Port", "name": "namenode_port", "value": "8020" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "The period to review when computing the moving average of the NameNode's RPC latency.", "display_name": "NameNode RPC Latency Monitoring Window", "name": "namenode_rpc_latency_window", "value": "5" }, { "desc": "Optional port for the service-rpc address which can be used by HDFS daemons instead of sharing the RPC address used by the clients.", "display_name": "NameNode Service RPC Port", "name": "dfs_namenode_servicerpc_address", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Quorum-based Storage Journal name parameter.", "display_name": "Suppress Parameter Validation: Quorum-based Storage Journal name", "name": "role_config_suppression_dfs_namenode_quorum_journal_name", "value": "false" }, { "desc": "The health check thresholds for the number of out-of-sync JournalNodes for this NameNode.", "display_name": "NameNode Out-Of-Sync JournalNodes Thresholds", "name": "namenode_out_of_sync_journal_nodes_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The health test thresholds of the age of the HDFS namespace checkpoint. Specified as a percentage of the configured checkpoint interval.", "display_name": "Filesystem Checkpoint Age Monitoring Thresholds", "name": "namenode_checkpoint_age_thresholds", "value": "{\"critical\":\"400.0\",\"warning\":\"200.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "Enables the health test that the NameNode's process state is consistent with the role configuration", "display_name": "NameNode Process Health Test", "name": "namenode_scm_health_enabled", "value": "true" }, { "desc": "The absolute maximum number of outgoing replication threads a given node can have at one time. The regular limit (dfs.namenode.replication.max-streams) is waived for highest-priority block replications. Highest replication priority is for blocks that are at a very high risk of loss if the disk or server on which they remain fails. These are usually blocks with only one copy, or blocks with zero live copies but a copy in a node being decommissioned. dfs.namenode.replication.max-streams-hard-limit provides a limit on the total number of outgoing replication threads, including threads of all priorities.", "display_name": "Hard Limit on the Number of Replication Threads on a Datanode", "name": "dfs_namenode_replication_max_streams_hard_limit", "value": "40" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "NameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "NAMENODE_role_env_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Directory where NameNode will place its log files.", "display_name": "NameNode Log Directory", "name": "namenode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Enables the health test of the metadata upgrade status of the NameNode. This covers nonrolling metadata upgrades. Rolling metadata upgrades are covered in a separate health test.", "display_name": "HDFS Metadata Upgrade Status Health Test", "name": "namenode_upgrade_status_enabled", "value": "true" }, { "desc": "Nameservice of this NameNode. The Nameservice represents the interface to this NameNode and its High Availability partner. The Nameservice also represents the namespace associated with a federated NameNode.", "display_name": "NameNode Nameservice", "name": "dfs_federation_namenode_nameservice", "value": null }, { "desc": "The minimum log level for NameNode logs", "display_name": "NameNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.replication.min. Enter a value less than or equal to 0 to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safemode permanent.", "display_name": "Safemode Threshold Percentage", "name": "dfs_safemode_threshold_pct", "value": "0.999" }, { "desc": "Directories on the local file system to store the NameNode edits. If not set, the edits are stored in the NameNode's Data Directories. The value of this configuration is automatically generated to be the Quorum-based Storage URI if there are JournalNodes and this NameNode is not Highly Available.", "display_name": "NameNode Edits Directories", "name": "dfs_namenode_edits_dir", "value": null }, { "desc": "If set to false and if one of the replicas of the NameNode storage fails, such as temporarily failure of NFS, this directory is not used until the NameNode restarts. If enabled, failed storage is re-checked on every checkpoint and, if it becomes valid, the NameNode will try to restore the edits and fsimage.", "display_name": "Restore NameNode Directories at Checkpoint Time", "name": "dfs_name_dir_restore", "value": "false" }, { "desc": "The amount of time allowed after this role is started that failures of health checks that rely on communication with this role will be tolerated.", "display_name": "Health Check Startup Tolerance", "name": "namenode_startup_tolerance", "value": "5" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for NameNode", "name": "namenode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether to suppress configuration warnings produced by the Java Heap Size of NameNode in Bytes Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: Java Heap Size of NameNode in Bytes Minimum Validator", "name": "role_config_suppression_namenode_java_heapsize_minimum_validator", "value": "false" }, { "desc": "Enable Automatic Failover to maintain High Availability. Requires a ZooKeeper service and a High Availability NameNode partner.", "display_name": "Enable Automatic Failover", "name": "autofailover_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a NameNode Data Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "NameNode Data Directories Free Space Monitoring Percentage Thresholds", "name": "namenode_data_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "For advanced use only. A string to be inserted into dfs_hosts_exclude.txt for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_exclude.txt", "name": "namenode_hosts_exclude_safety_valve", "value": null }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's NameNode Data Directories.", "display_name": "NameNode Data Directories Free Space Monitoring Absolute Thresholds", "name": "namenode_data_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_namenode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Rolling Upgrade Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Rolling Upgrade Status", "name": "role_health_suppression_name_node_rolling_upgrade_status", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: NameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "This determines the percentage amount of block invalidations (deletes) to do over a single DataNode heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DataNode.", "display_name": "Invalidate Work Percentage Per Iteration", "name": "dfs_namenode_invalidate_work_pct_per_iteration", "value": "0.32" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "namenode_web_metric_collection_enabled", "value": "true" }, { "desc": "Timeout when writing edits to a JournalNode. This only applies when NameNode high availability is enabled.", "display_name": "JournalNode Write Transactions Timeout", "name": "dfs_qjournal_write_txns_timeout_ms", "value": "20000" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The time between two periodic file system checkpoints.", "display_name": "Filesystem Checkpoint Period", "name": "fs_checkpoint_period", "value": "3600" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "The access time for HDFS file is precise upto this value. Setting the value of 0 disables access times for HDFS. When using the NFS Gateway role, make sure this property is enabled.", "display_name": "Access Time Precision", "name": "dfs_access_time_precision", "value": "3600000" }, { "desc": "Minimum number of running threads for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Min Threadcount", "name": "dfs_thrift_threads_min", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Topology Script File Name parameter.", "display_name": "Suppress Parameter Validation: Topology Script File Name", "name": "role_config_suppression_topology_script_file_name", "value": "false" }, { "desc": "Maximum number of running threads for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Max Threadcount", "name": "dfs_thrift_threads_max", "value": "20" }, { "desc": "Whether to suppress configuration warnings produced by the NameNode Handler Count Minimum Validator configuration validator.", "display_name": "Suppress Configuration Validator: NameNode Handler Count Minimum Validator", "name": "role_config_suppression_dfs_namenode_handler_count_minimum_validator", "value": "false" }, { "desc": "Full path to a custom topology script on the host file system. The topology script is used to determine the rack location of nodes. If left blank, a topology script will be provided that uses your hosts' rack information, visible in the \"Hosts\" page.", "display_name": "Topology Script File Name", "name": "topology_script_file_name", "value": null }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The minimum log level for NameNode block state change log messages. Setting this to WARN or higher greatly reduces the amount of log output related to block state changes.", "display_name": "NameNode Block State Change Logging Threshold", "name": "namenode_blockstatechange_log_threshold", "value": "INFO" }, { "desc": "Indicate whether or not to avoid writing to stale DataNodes for which heartbeat messages have not been received by the NameNode for more than Stale DataNode Time Interval. Writes avoid using stale DataNodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of DataNodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads.", "display_name": "Avoid Writing Stale DataNode", "name": "dfs_namenode_avoid_write_stale_datanode", "value": "false" }, { "desc": "The health test thresholds of failed status directories in a NameNode.", "display_name": "NameNode Directory Failures Thresholds", "name": "namenode_directory_failures_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_name_node_unexpected_exits", "value": "false" }, { "desc": "Number of minutes between trash checkpoints. Also controls the number of minutes after which a trash checkpoint directory is deleted. To disable the trash feature, enter 0.", "display_name": "Filesystem Trash Interval", "name": "fs_trash_interval", "value": "1440" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "namenode_config_safety_valve", "value": null }, { "desc": "This determines the total amount of block transfers to begin in parallel at a DataNode for replication, when such a command list is being sent over a DataNode heartbeat by the NameNode. The actual number is obtained by multiplying this value by the total number of live nodes in the cluster. The result number is the number of blocks to transfer immediately, per DataNode heartbeat.", "display_name": "Replication Work Multiplier Per Iteration", "name": "dfs_namenode_replication_work_multiplier_per_iteration", "value": "10" }, { "desc": "The maximum number of rolled log files to keep for NameNode logs. Typically used by log4j or logback.", "display_name": "NameNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Determines extension of safemode in milliseconds after the threshold level is reached.", "display_name": "Safemode Extension", "name": "dfs_safemode_extension", "value": "30000" }, { "desc": "Timeout in seconds for the Hue Thrift server running on the NameNode", "display_name": "Hue Thrift Server Timeout", "name": "dfs_thrift_timeout", "value": "60" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The number of server threads for the NameNode used for service calls. Only used when NameNode Service RPC Port is configured.", "display_name": "NameNode Service Handler Count", "name": "dfs_namenode_service_handler_count", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt parameter.", "display_name": "Suppress Parameter Validation: NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt", "name": "role_config_suppression_namenode_hosts_allow_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Data Directories parameter.", "display_name": "Suppress Parameter Validation: NameNode Data Directories", "name": "role_config_suppression_dfs_name_dir_list", "value": "false" }, { "desc": "The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired.", "display_name": "Filesystem Checkpoint Transaction Threshold", "name": "fs_checkpoint_txns", "value": "1000000" }, { "desc": "When computing the overall NameNode health, consider the host's health.", "display_name": "NameNode Host Health Test", "name": "namenode_host_health_enabled", "value": "true" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_name_node_scm_health", "value": "false" }, { "desc": "The base port where the secure NameNode web UI listens.", "display_name": "Secure NameNode Web UI Port (TLS/SSL)", "name": "dfs_https_port", "value": "50470" }, { "desc": "For advanced use only. A string to be inserted into dfs_hosts_allow.txt for this role only.", "display_name": "NameNode Advanced Configuration Snippet (Safety Valve) for dfs_hosts_allow.txt", "name": "namenode_hosts_allow_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Mount Points parameter.", "display_name": "Suppress Parameter Validation: Mount Points", "name": "role_config_suppression_nameservice_mountpoints", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "If enabled, the NameNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind NameNode to Wildcard Address", "name": "namenode_bind_wildcard", "value": "false" }, { "desc": "Comma-separated list of NameNode plug-ins to be activated. If one plug-in cannot be loaded, all the plug-ins are ignored.", "display_name": "NameNode Plugins", "name": "dfs_namenode_plugins_list", "value": "" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_name_node_host_health", "value": "false" }, { "desc": "Directory on a shared storage device, such as a Quorum-based Storage URI or a local directory that is an NFS mount from a NAS, to store the NameNode edits. The value of this configuration is automatically generated to be the Quourm Journal URI if there are JournalNodes and this NameNode is Highly Available.", "display_name": "Shared Edits Directory", "name": "dfs_namenode_shared_edits_dir", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The number of server threads for the NameNode.", "display_name": "NameNode Handler Count", "name": "dfs_namenode_handler_count", "value": "30" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the NameNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: NameNode Log Directory", "name": "role_config_suppression_namenode_log_dir", "value": "false" }, { "desc": "Whether to suppress the results of the Name Directory Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Name Directory Status", "name": "role_health_suppression_name_node_directory_failures", "value": "false" }, { "desc": "Specifies the number of DataNodes that must be live before the name node exits safemode. Enter a value less than or equal to 0 to take the number of live DataNodes into account when deciding whether to remain in safemode during startup. Values greater than the number of DataNodes in the cluster will make safemode permanent.", "display_name": "Safemode Minimum DataNodes", "name": "dfs_safemode_min_datanodes", "value": "0" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/spark-service.json0000666000175100017510000002251113245514472025152 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Spark Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_spark_on_yarn_service_env_safety_valve", "value": "false" }, { "desc": "Enable whether the Spark communication protocols do authentication using a shared secret.", "display_name": "Spark Authentication", "name": "spark_authenticate", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark History Location (HDFS) parameter.", "display_name": "Suppress Parameter Validation: Spark History Location (HDFS)", "name": "service_config_suppression_spark_history_log_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into spark-conf/spark-env.sh. Applies to configurations of all roles in this service except client configuration.", "display_name": "Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "spark-conf/spark-env.sh_service_safety_valve", "value": null }, { "desc": "The location of Spark application history logs in HDFS. Changing this value will not move existing logs to the new location.", "display_name": "Spark History Location (HDFS)", "name": "spark_history_log_dir", "value": "/user/spark/applicationHistory" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "spark" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark JAR Location (HDFS) parameter.", "display_name": "Suppress Parameter Validation: Spark JAR Location (HDFS)", "name": "service_config_suppression_spark_jar_hdfs_path", "value": "false" }, { "desc": "The location of the Spark JAR in HDFS. If left blank, Cloudera Manager will use the Spark JAR installed on the cluster nodes.", "display_name": "Spark JAR Location (HDFS)", "name": "spark_jar_hdfs_path", "value": "" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "spark" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Spark Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "SPARK_ON_YARN_service_env_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the History Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: History Server Count Validator", "name": "service_config_suppression_spark_yarn_history_server_count_validator", "value": "false" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the Gateway Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Gateway Count Validator", "name": "service_config_suppression_gateway_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Name of the YARN (MR2 Included) service that this Spark service instance depends on", "display_name": "YARN (MR2 Included) Service", "name": "yarn_service", "value": null }, { "desc": "The port the Spark Shuffle Service listens for fetch requests.", "display_name": "Spark Shuffle Service Port", "name": "spark_shuffle_service_port", "value": "7337" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh parameter.", "display_name": "Suppress Parameter Validation: Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh", "name": "service_config_suppression_spark-conf/spark-env.sh_service_safety_valve", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-secondarynamenode.json0000666000175100017510000007566113245514472027032 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.IOException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.net.SocketClosedException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.io.EOFException\"},\n {\"alert\": false, \"rate\": 0, \"exceptiontype\": \"java.nio.channels.CancelledKeyException\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Unknown job [^ ]+ being deleted.*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"Error executing shell command .+ No such process.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\".*attempt to override final parameter.+\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\":\"[^ ]+ is a deprecated filesystem name. Use.*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"},\n {\"alert\": false, \"rate\": 1, \"threshold\":\"INFO\", \"content\":\"Triggering checkpoint.*\"}\n ]\n}\n" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_secondary_name_node_web_metric_collection", "value": "false" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for Secondary NameNode", "name": "secondarynamenode_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories.", "display_name": "HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds", "name": "secondarynamenode_checkpoint_directories_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of Secondary NameNode in Bytes", "name": "secondary_namenode_java_heapsize", "value": "1073741824" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Nameservice parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Nameservice", "name": "role_config_suppression_dfs_secondarynamenode_nameservice", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_secondary_name_node_scm_health", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "secondarynamenode_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "role_config_suppression_secondarynamenode_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_secondary_name_node_heap_dump_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Log Directory parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Log Directory", "name": "role_config_suppression_secondarynamenode_log_dir", "value": "false" }, { "desc": "Enables the health test that the SecondaryNameNode's process state is consistent with the role configuration", "display_name": "SecondaryNameNode Process Health Test", "name": "secondarynamenode_scm_health_enabled", "value": "true" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "The number of transactions after which the NameNode or SecondaryNameNode will create a checkpoint of the namespace, regardless of whether the checkpoint period has expired.", "display_name": "Filesystem Checkpoint Transaction Threshold", "name": "fs_checkpoint_txns", "value": "1000000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HDFS Checkpoint Directories parameter.", "display_name": "Suppress Parameter Validation: HDFS Checkpoint Directories", "name": "role_config_suppression_fs_checkpoint_dir_list", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "secondarynamenode_web_metric_collection_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into hdfs-site.xml for this role only.", "display_name": "SecondaryNameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", "name": "secondarynamenode_config_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for Secondary NameNode parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for Secondary NameNode", "name": "role_config_suppression_secondarynamenode_java_opts", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_secondary_name_node_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_secondary_name_node_gc_duration", "value": "false" }, { "desc": "Advanced Configuration Snippet (Safety Valve) for Hadoop Metrics2. Properties will be inserted into hadoop-metrics2.properties.", "display_name": "Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "hadoop_metrics2_safety_valve", "value": null }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "SECONDARYNAMENODE_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_secondary_name_node_unexpected_exits", "value": "false" }, { "desc": "If enabled, the SecondaryNameNode binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind SecondaryNameNode to Wildcard Address", "name": "secondary_namenode_bind_wildcard", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "SecondaryNameNode Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The time between two periodic file system checkpoints.", "display_name": "Filesystem Checkpoint Period", "name": "fs_checkpoint_period", "value": "3600" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: SecondaryNameNode Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_secondarynamenode_role_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress the results of the HDFS Checkpoint Directories Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: HDFS Checkpoint Directories Free Space", "name": "role_health_suppression_secondary_name_node_checkpoint_directories_free_space", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_secondary_name_node_host_health", "value": "false" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "secondarynamenode_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's HDFS Checkpoint Directories. Specified as a percentage of the capacity on that filesystem. This setting is not used if a HDFS Checkpoint Directories Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "HDFS Checkpoint Directories Free Space Monitoring Percentage Thresholds", "name": "secondarynamenode_checkpoint_directories_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "When computing the overall SecondaryNameNode health, consider the host's health.", "display_name": "SecondaryNameNode Host Health Test", "name": "secondarynamenode_host_health_enabled", "value": "true" }, { "desc": "The maximum size, in megabytes, per log file for SecondaryNameNode logs. Typically used by log4j or logback.", "display_name": "SecondaryNameNode Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_secondary_name_node_file_descriptor", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "secondarynamenode_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Hadoop Metrics2 Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_hadoop_metrics2_safety_valve", "value": "false" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "secondarynamenode_gc_duration_window", "value": "5" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_secondary_name_node_log_directory_free_space", "value": "false" }, { "desc": "Determines where on the local file system the DFS SecondaryNameNode should store the temporary images to merge. For redundancy, enter a comma-delimited list of directories to replicate the image in all of the directories. Typical values are /data/N/dfs/snn for N = 1, 2, 3...", "display_name": "HDFS Checkpoint Directories", "name": "fs_checkpoint_dir_list", "value": null }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "The maximum number of rolled log files to keep for SecondaryNameNode logs. Typically used by log4j or logback.", "display_name": "SecondaryNameNode Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Nameservice of this SecondaryNameNode", "display_name": "SecondaryNameNode Nameservice", "name": "dfs_secondarynamenode_nameservice", "value": null }, { "desc": "The minimum log level for SecondaryNameNode logs", "display_name": "SecondaryNameNode Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The base port where the secure SecondaryNameNode web UI listens.", "display_name": "Secure SecondaryNameNode Web UI Port (TLS/SSL)", "name": "dfs_secondary_https_port", "value": "50495" }, { "desc": "Directory where SecondaryNameNode will place its log files.", "display_name": "SecondaryNameNode Log Directory", "name": "secondarynamenode_log_dir", "value": "/var/log/hadoop-hdfs" }, { "desc": "The SecondaryNameNode HTTP port. If the port is 0, then the server starts on a free port. Combined with the SecondaryNameNode's hostname to build its HTTP address.", "display_name": "SecondaryNameNode Web UI Port", "name": "dfs_secondary_http_port", "value": "50090" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/yarn-jobhistory.json0000666000175100017510000007374013245514472025551 0ustar zuulzuul00000000000000[ { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of JobHistory Server in Bytes", "name": "mr2_jobhistory_java_heapsize", "value": "1073741824" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "

This file contains the rules which govern how log messages are turned into events by the custom log4j appender that this role loads. It is in JSON format, and is composed of a list of rules. Every log message is evaluated against each of these rules in turn to decide whether or not to send an event for that message.

Each rule has some or all of the following fields:

  • alert - whether or not events generated from this rule should be promoted to alerts. A value of \"true\" will cause alerts to be generated. If not specified, the default is \"false\".
  • rate (mandatory) - the maximum number of log messages matching this rule that may be sent as events every minute. If more than rate matching log messages are received in a single minute, the extra messages are ignored. If rate is less than 0, the number of messages per minute is unlimited.
  • periodminutes - the number of minutes during which the publisher will only publish rate events or fewer. If not specified, the default is one minute
  • threshold - apply this rule only to messages with this log4j severity level or above. An example is \"WARN\" for warning level messages or higher.
  • content - match only those messages whose contents match this regular expression.
  • exceptiontype - match only those messages which are part of an exception message. The exception type must match this regular expression.

Example:{\"alert\": false, \"rate\": 10, \"exceptiontype\": \"java.lang.StringIndexOutOfBoundsException\"}

This rule will send events to Cloudera Manager for every StringIndexOutOfBoundsException, up to a maximum of 10 every minute.

", "display_name": "Rules to Extract Events from Log Files", "name": "log_event_whitelist", "value": "{\n \"version\": \"0\",\n \"rules\": [\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"FATAL\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Instead, use .*\"},\n {\"alert\": false, \"rate\": 0, \"threshold\":\"WARN\", \"content\": \".* is deprecated. Use .* instead\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 2, \"exceptiontype\": \".*\"},\n {\"alert\": false, \"rate\": 1, \"periodminutes\": 1, \"threshold\":\"WARN\"}\n ]\n}\n" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for JobHistory Server", "name": "mr2_jobhistory_java_opts", "value": "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Kerberos principal used by the JobHistory Server roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "mapred" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_jobhistory_log_directory_free_space", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Enables the health test that the JobHistory Server's process state is consistent with the role configuration", "display_name": "JobHistory Server Process Health Test", "name": "jobhistory_scm_health_enabled", "value": "true" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Whether to suppress the results of the GC Duration heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: GC Duration", "name": "role_health_suppression_jobhistory_gc_duration", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_jobhistory_scm_health", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_jobhistory_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_history_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_jobhistory_host_health", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Directory where JobHistory Server will place its log files.", "display_name": "JobHistory Server Log Directory", "name": "mr2_jobhistory_log_dir", "value": "/var/log/hadoop-mapreduce" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_jobhistory_swap_memory_usage", "value": "false" }, { "desc": "Time interval for history cleaner to check for files to delete. Files are only deleted if they are older than mapreduce.jobhistory.max-age-ms.", "display_name": "Job History Files Cleaner Interval", "name": "mapreduce_jobhistory_cleaner_interval", "value": "86400000" }, { "desc": "The period to review when computing the moving average of garbage collection time.", "display_name": "Garbage Collection Duration Monitoring Period", "name": "jobhistory_gc_duration_window", "value": "5" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "When computing the overall JobHistory Server health, consider the host's health.", "display_name": "JobHistory Server Host Health Test", "name": "jobhistory_host_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "role_config_suppression_jobhistory_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Rules to Extract Events from Log Files parameter.", "display_name": "Suppress Parameter Validation: Rules to Extract Events from Log Files", "name": "role_config_suppression_log_event_whitelist", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the MapReduce ApplicationMaster Staging Root Directory parameter.", "display_name": "Suppress Parameter Validation: MapReduce ApplicationMaster Staging Root Directory", "name": "role_config_suppression_yarn_app_mapreduce_am_staging_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "JobHistory Server Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Log Directory parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Log Directory", "name": "role_config_suppression_mr2_jobhistory_log_dir", "value": "false" }, { "desc": "The port of the MapReduce JobHistory Server administrative interface. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Server Admin Interface Port", "name": "mapreduce_jobhistory_admin_address", "value": "10033" }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_jobhistory_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_jobhistory_file_descriptor", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into yarn-site.xml for this role only.", "display_name": "JobHistory Server Advanced Configuration Snippet (Safety Valve) for yarn-site.xml", "name": "jobhistory_config_safety_valve", "value": null }, { "desc": "Job history files older than this time duration will deleted when the history cleaner runs.", "display_name": "Job History Files Maximum Age", "name": "mapreduce_jobhistory_max_age_ms", "value": "604800000" }, { "desc": "The group that the JobHistory Server process should run as.", "display_name": "System Group", "name": "history_process_groupname", "value": "hadoop" }, { "desc": "The port of the MapReduce JobHistory Server. Together with the hostname of the JobHistory role, forms the address.", "display_name": "MapReduce JobHistory Server Port", "name": "mapreduce_jobhistory_address", "value": "10020" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_history_process_username", "value": "false" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "jobhistory_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "The HTTP port of the MapReduce JobHistory Server web application. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Web Application HTTP Port", "name": "mapreduce_jobhistory_webapp_address", "value": "19888" }, { "desc": "The root HDFS directory of the staging area for users' MR2 jobs; for example /user. The staging directories are always named after the user.", "display_name": "MapReduce ApplicationMaster Staging Root Directory", "name": "yarn_app_mapreduce_am_staging_dir", "value": "/user" }, { "desc": "Enables the health test that the Cloudera Manager Agent can successfully contact and gather metrics from the web server.", "display_name": "Web Metric Collection", "name": "jobhistory_web_metric_collection_enabled", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "The minimum log level for JobHistory Server logs", "display_name": "JobHistory Server Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The maximum size, in megabytes, per log file for JobHistory Server logs. Typically used by log4j or logback.", "display_name": "JobHistory Server Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "For advanced use only. A string to be inserted into mapred-site.xml for this role only.", "display_name": "JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "jobhistory_mapred_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_jobhistory_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "JobHistory Server Environment Advanced Configuration Snippet (Safety Valve)", "name": "JOBHISTORY_role_env_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for JobHistory Server parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for JobHistory Server", "name": "role_config_suppression_mr2_jobhistory_java_opts", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for JobHistory Server logs. Typically used by log4j or logback.", "display_name": "JobHistory Server Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "If enabled, the JobHistory Server binds to the wildcard address (\"0.0.0.0\") on all of its ports.", "display_name": "Bind JobHistory Server to Wildcard Address", "name": "yarn_jobhistory_bind_wildcard", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "The user that the JobHistory Server process should run as.", "display_name": "System User", "name": "history_process_username", "value": "mapred" }, { "desc": "The HTTPS port of the MapReduce JobHistory Server web application. Together with the host name of the JobHistory role forms the address.", "display_name": "MapReduce JobHistory Web Application HTTPS Port (TLS/SSL)", "name": "mapreduce_jobhistory_webapp_https_address", "value": "19890" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml parameter.", "display_name": "Suppress Parameter Validation: JobHistory Server Advanced Configuration Snippet (Safety Valve) for mapred-site.xml", "name": "role_config_suppression_jobhistory_mapred_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The health test thresholds for the weighted average time spent in Java garbage collection. Specified as a percentage of elapsed wall clock time.", "display_name": "Garbage Collection Duration Thresholds", "name": "jobhistory_gc_duration_thresholds", "value": "{\"critical\":\"60.0\",\"warning\":\"30.0\"}" }, { "desc": "The health test thresholds on the duration of the metrics request to the web server.", "display_name": "Web Metric Collection Duration", "name": "jobhistory_web_metric_collection_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"10000.0\"}" }, { "desc": "Whether to suppress the results of the Web Server Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Web Server Status", "name": "role_health_suppression_jobhistory_web_metric_collection", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/flume-service.json0000666000175100017510000002445313245514472025151 0ustar zuulzuul00000000000000[ { "desc": "Name of the Hbase service that this Flume service instance depends on", "display_name": "Hbase Service", "name": "hbase_service", "value": null }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Flume might connect to. This is used when Flume is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Flume TLS/SSL Certificate Trust Store File", "name": "flume_truststore_file", "value": null }, { "desc": "Whether to suppress the results of the Agent Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Agent Health", "name": "service_health_suppression_flume_agents_healthy", "value": "false" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "flume" }, { "desc": "Name of the HDFS service that this Flume service instance depends on", "display_name": "HDFS Service", "name": "hdfs_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Agent Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Agent Count Validator", "name": "service_config_suppression_agent_count_validator", "value": "false" }, { "desc": "Name of the Solr service that this Flume service instance depends on", "display_name": "Solr Service", "name": "solr_service", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Flume Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "flume_env_safety_valve", "value": null }, { "desc": "Sets the maximum number of Flume components that will be returned under Flume Metric Details. Increasing this value will negatively impact the interactive performance of the Flume Metrics Details page.", "display_name": "Maximum displayed Flume metrics components", "name": "flume_context_groups_request_limit", "value": "1000" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "flume" }, { "desc": "The health test thresholds of the overall Agent health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Agents falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Agents falls below the critical threshold.", "display_name": "Healthy Agent Monitoring Thresholds", "name": "flume_agents_healthy_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"95.0\"}" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "flume" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The frequency in which the log4j event publication appender will retry sending undelivered log events to the Event server, in seconds", "display_name": "Log Event Retry Frequency", "name": "log_event_retry_frequency", "value": "30" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "When set, each role identifies important log events and forwards them to Cloudera Manager.", "display_name": "Enable Log Event Capture", "name": "catch_events", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Flume Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_flume_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Flume TLS/SSL Certificate Trust Store Password", "name": "service_config_suppression_flume_truststore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "The password for the Flume TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Flume TLS/SSL Certificate Trust Store Password", "name": "flume_truststore_password", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Flume TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Flume TLS/SSL Certificate Trust Store File", "name": "service_config_suppression_flume_truststore_file", "value": "false" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/kafka-service.json0000666000175100017510000005267413245514472025124 0ustar zuulzuul00000000000000[ { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Controlled Shutdown Maximum Attempts parameter.", "display_name": "Suppress Parameter Validation: Controlled Shutdown Maximum Attempts", "name": "service_config_suppression_controlled.shutdown.max.retries", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_kafka_service_env_safety_valve", "value": "false" }, { "desc": "The default number of partitions for automatically created topics.", "display_name": "Default Number of Partitions", "name": "num.partitions", "value": "1" }, { "desc": "The amount of time to retain delete messages for log compacted topics. Once a consumer has seen an original message you need to ensure it also sees the delete message. If you removed the delete message too quickly, this might not happen. As a result there is a configurable delete retention time.", "display_name": "Log Compaction Delete Record Retention Time", "name": "log.cleaner.delete.retention.ms", "value": "604800000" }, { "desc": "Enables auto creation of topics on the server. If this is set to true, then attempts to produce, consume, or fetch metadata for a non-existent topic automatically create the topic with the default replication factor and number of partitions.", "display_name": "Topic Auto Creation", "name": "auto.create.topics.enable", "value": "true" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Number of threads used to replicate messages from leaders. Increasing this value increases the degree of I/O parallelism in the follower broker.", "display_name": "Number of Replica Fetchers", "name": "num.replica.fetchers", "value": "1" }, { "desc": "Enables Kafka monitoring.", "display_name": "Enable Kafka Monitoring (Note: Requires Kafka-1.3.0 parcel or higher)", "name": "monitoring.enabled", "value": "true" }, { "desc": "If automatic leader rebalancing is enabled, the controller tries to balance leadership for partitions among the brokers by periodically returning leadership for each partition to the preferred replica, if it is available.", "display_name": "Enable Automatic Leader Rebalancing", "name": "auto.leader.rebalance.enable", "value": "true" }, { "desc": "Number of unsuccessful controlled shutdown attempts before executing an unclean shutdown. For example, the default value of 3 means that the system will attempt a controlled shutdown 3 times before executing an unclean shutdown.", "display_name": "Controlled Shutdown Maximum Attempts", "name": "controlled.shutdown.max.retries", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "The number of partitions for the offset commit topic. Since changing this after deployment is currently unsupported, we recommend using a higher setting for production (for example, 100-200).", "display_name": "Offset Commit Topic Number of Partitions", "name": "offsets.topic.num.partitions", "value": "50" }, { "desc": "If a follower has not sent any fetch requests, nor has it consumed up to the leader's log end offset during this time, the leader removes the follower from the ISR set.", "display_name": "Allowed Replica Time Lag", "name": "replica.lag.time.max.ms", "value": "10000" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "kafka" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Replica Maximum Fetch Size parameter.", "display_name": "Suppress Parameter Validation: Replica Maximum Fetch Size", "name": "service_config_suppression_replica.fetch.max.bytes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "The replication factor for the offset commit topic. A higher setting is recommended in order to ensure higher availability (for example, 3 or 4) . If the offsets topic is created when there are fewer brokers than the replication factor, then the offsets topic is created with fewer replicas.", "display_name": "Offset Commit Topic Replication Factor", "name": "offsets.topic.replication.factor", "value": "3" }, { "desc": "Whether to suppress configuration warnings produced by the Kafka Broker Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kafka Broker Count Validator", "name": "service_config_suppression_kafka_broker_count_validator", "value": "false" }, { "desc": "Controls how frequently the log cleaner will attempt to clean the log. This ratio bounds the maximum space wasted in the log by duplicates. For example, at 0.5 at most 50% of the log could be duplicates. A higher ratio will mean fewer, more efficient cleanings but will mean more wasted space in the log.", "display_name": "Log Cleaner Clean Ratio", "name": "log.cleaner.min.cleanable.ratio", "value": "0.5" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "kafka" }, { "desc": "Enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so might result in data loss.", "display_name": "Enable Unclean Leader Election", "name": "unclean.leader.election.enable", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Default Replication Factor", "name": "service_config_suppression_default.replication.factor", "value": "false" }, { "desc": "Enables controlled shutdown of the broker. If enabled, the broker moves all leaders on it to other brokers before shutting itself down. This reduces the unavailability window during shutdown.", "display_name": "Enable Controlled Shutdown", "name": "controlled.shutdown.enable", "value": "true" }, { "desc": "The frequency with which to check for leader imbalance.", "display_name": "Leader Imbalance Check Interval", "name": "leader.imbalance.check.interval.seconds", "value": "300" }, { "desc": "The maximum number of bytes to fetch for each partition in fetch requests replicas send to the leader. This value should be larger than message.max.bytes.", "display_name": "Replica Maximum Fetch Size", "name": "replica.fetch.max.bytes", "value": "1048576" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Session Timeout parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Session Timeout", "name": "service_config_suppression_zookeeper.session.timeout.ms", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of Replica Fetchers parameter.", "display_name": "Suppress Parameter Validation: Number of Replica Fetchers", "name": "service_config_suppression_num.replica.fetchers", "value": "false" }, { "desc": "Any producer distinguished by clientId will get throttled if it produces more bytes than this value per-second. Only respected by Kafka 2.0 or later.", "display_name": "Default Producer Quota", "name": "quota.producer.default", "value": null }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Kafka Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_service_env_safety_valve", "value": null }, { "desc": "The maximum size of a message that the server can receive. It is important that this property be in sync with the maximum fetch size the consumers use, or else an unruly producer could publish messages too large for consumers to consume.", "display_name": "Maximum Message Size", "name": "message.max.bytes", "value": "1000000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Leader Imbalance Allowed Per Broker parameter.", "display_name": "Suppress Parameter Validation: Leader Imbalance Allowed Per Broker", "name": "service_config_suppression_leader.imbalance.per.broker.percentage", "value": "false" }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Enables the log cleaner to compact topics with cleanup.policy=compact on this cluster.", "display_name": "Enable Log Compaction", "name": "log.cleaner.enable", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Minimum Number of Replicas in ISR parameter.", "display_name": "Suppress Parameter Validation: Minimum Number of Replicas in ISR", "name": "service_config_suppression_min.insync.replicas", "value": "false" }, { "desc": "ZNode in ZooKeeper that should be used as a root for this Kafka cluster.", "display_name": "ZooKeeper Root", "name": "zookeeper.chroot", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the Kafka MirrorMaker Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Kafka MirrorMaker Count Validator", "name": "service_config_suppression_kafka_mirror_maker_count_validator", "value": "false" }, { "desc": "The number of messages written to a log partition before triggering an fsync on the log. Setting this lower syncs data to disk more often, but has a major impact on performance. We recommend use of replication for durability rather than depending on single-server fsync; however, this setting can be used to be extra certain. If used in conjunction with log.flush.interval.ms, the log is flushed when either criteria is met.", "display_name": "Log Flush Message Interval", "name": "log.flush.interval.messages", "value": null }, { "desc": "The number of background threads to use for log cleaning.", "display_name": "Number of Log Cleaner Threads", "name": "log.cleaner.threads", "value": "1" }, { "desc": "Enable Kerberos authentication for this KAFKA service.", "display_name": "Enable Kerberos Authentication", "name": "kerberos.auth.enable", "value": "false" }, { "desc": "List of metric reporter class names. HTTP reporter is included by default.", "display_name": "List of Metric Reporters", "name": "kafka.metrics.reporters", "value": "nl.techop.kafka.KafkaHttpMetricsReporter" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Root parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Root", "name": "service_config_suppression_zookeeper.chroot", "value": "false" }, { "desc": "The frequency, in ms, with which the log flusher checks whether any log is eligible to be flushed to disk.", "display_name": "Log Flush Scheduler Interval", "name": "log.flush.scheduler.interval.ms", "value": null }, { "desc": "The minimum number of replicas in the in-sync replica needed to satisfy a produce request where required.acks=-1 (that is, all).", "display_name": "Minimum Number of Replicas in ISR", "name": "min.insync.replicas", "value": "1" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Offset Commit Topic Replication Factor parameter.", "display_name": "Suppress Parameter Validation: Offset Commit Topic Replication Factor", "name": "service_config_suppression_offsets.topic.replication.factor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Leader Imbalance Check Interval parameter.", "display_name": "Suppress Parameter Validation: Leader Imbalance Check Interval", "name": "service_config_suppression_leader.imbalance.check.interval.seconds", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "If the server fails to send a heartbeat to ZooKeeper within this period of time, it is considered dead. If set too low, ZooKeeper might falsely consider a server dead; if set too high, ZooKeeper might take too long to recognize a dead server.", "display_name": "ZooKeeper Session Timeout", "name": "zookeeper.session.timeout.ms", "value": "6000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Maximum Message Size parameter.", "display_name": "Suppress Parameter Validation: Maximum Message Size", "name": "service_config_suppression_message.max.bytes", "value": "false" }, { "desc": "Enables topic deletion using admin tools. When delete topic is disabled, deleting topics through the admin tools has no effect.", "display_name": "Enable Delete Topic", "name": "delete.topic.enable", "value": "true" }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "If a replica falls more than this number of messages behind the leader, the leader removes the follower from the ISR and treats it as dead. This property is deprecated in Kafka 1.4.0; higher versions use only replica.lag.time.max.ms.", "display_name": "Allowed Replica Message Lag", "name": "replica.lag.max.messages", "value": "4000" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Default Number of Partitions parameter.", "display_name": "Suppress Parameter Validation: Default Number of Partitions", "name": "service_config_suppression_num.partitions", "value": "false" }, { "desc": "Name of the ZooKeeper service that this Kafka service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "The total memory used for log deduplication across all cleaner threads. This memory is statically allocated and will not cause GC problems.", "display_name": "Log Cleaner Deduplication Buffer Size", "name": "log.cleaner.dedupe.buffer.size", "value": "134217728" }, { "desc": "Any consumer distinguished by clientId/consumer group will get throttled if it fetches more bytes than this value per-second. Only respected by Kafka 2.0 or later.", "display_name": "Default Consumer Quota", "name": "quota.consumer.default", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Offset Commit Topic Number of Partitions parameter.", "display_name": "Suppress Parameter Validation: Offset Commit Topic Number of Partitions", "name": "service_config_suppression_offsets.topic.num.partitions", "value": "false" }, { "desc": "The default replication factor for automatically created topics.", "display_name": "Default Replication Factor", "name": "default.replication.factor", "value": "1" }, { "desc": "The maximum time between fsync calls on the log. If used in conjuction with log.flush.interval.messages, the log is flushed when either criteria is met.", "display_name": "Log Flush Time Interval", "name": "log.flush.interval.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the List of Metric Reporters parameter.", "display_name": "Suppress Parameter Validation: List of Metric Reporters", "name": "service_config_suppression_kafka.metrics.reporters", "value": "false" }, { "desc": "The percentage of leader imbalance allowed per broker. The controller rebalances leadership if this ratio goes above the configured value per broker.", "display_name": "Leader Imbalance Allowed Per Broker", "name": "leader.imbalance.per.broker.percentage", "value": "10" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/hdfs-httpfs.json0000666000175100017510000006323713245514472024640 0ustar zuulzuul00000000000000[ { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Log Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Log Directory Free Space Monitoring Percentage Thresholds", "name": "log_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Kerberos principal used by the HttpFS roles.", "display_name": "Role-Specific Kerberos Principal", "name": "kerberos_role_princ_name", "value": "httpfs" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_httpfs_scm_health", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "httpfs_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that HttpFS might connect to. This is used when HttpFS is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "HttpFS TLS/SSL Certificate Trust Store File", "name": "httpfs_https_truststore_file", "value": null }, { "desc": "Whether to suppress the results of the Heap Dump Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Heap Dump Directory Free Space", "name": "role_health_suppression_httpfs_heap_dump_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Keystore File parameter.", "display_name": "Suppress Parameter Validation: HttpFS Keystore File", "name": "role_config_suppression_httpfs_https_keystore_file", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory. Specified as a percentage of the capacity on that filesystem. This setting is not used if a Heap Dump Directory Free Space Monitoring Absolute Thresholds setting is configured.", "display_name": "Heap Dump Directory Free Space Monitoring Percentage Thresholds", "name": "heap_dump_directory_free_space_percentage_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"never\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Log Directory parameter.", "display_name": "Suppress Parameter Validation: HttpFS Log Directory", "name": "role_config_suppression_httpfs_log_dir", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "role_config_suppression_httpfs_process_username", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's heap dump directory.", "display_name": "Heap Dump Directory Free Space Monitoring Absolute Thresholds", "name": "heap_dump_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_httpfs_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Heap Dump Directory parameter.", "display_name": "Suppress Parameter Validation: Heap Dump Directory", "name": "role_config_suppression_oom_heap_dump_dir", "value": "false" }, { "desc": "The method used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles that have an HTTP server endpoint exposing the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.", "display_name": "Stacks Collection Method", "name": "stacks_collection_method", "value": "jstack" }, { "desc": "When computing the overall HttpFS health, consider the host's health.", "display_name": "HttpFS Host Health Test", "name": "httpfs_host_health_enabled", "value": "true" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "The group that the HttpFS server process should run as.", "display_name": "System Group", "name": "httpfs_process_groupname", "value": "httpfs" }, { "desc": "Whether or not periodic stacks collection is enabled.", "display_name": "Stacks Collection Enabled", "name": "stacks_collection_enabled", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role-Specific Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Role-Specific Kerberos Principal", "name": "role_config_suppression_kerberos_role_princ_name", "value": "false" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HttpFS Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "Enables the health test that the HttpFS's process state is consistent with the role configuration", "display_name": "HttpFS Process Health Test", "name": "httpfs_scm_health_enabled", "value": "true" }, { "desc": "Password of the keystore used by the HttpFS role for TLS/SSL.", "display_name": "HttpFS Keystore Password", "name": "httpfs_https_keystore_password", "value": null }, { "desc": "The password for the HttpFS TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "HttpFS TLS/SSL Certificate Trust Store Password", "name": "httpfs_https_truststore_password", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "role_config_suppression_httpfs_process_groupname", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_httpfs_https_truststore_file", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Signature Secret parameter.", "display_name": "Suppress Parameter Validation: Signature Secret", "name": "role_config_suppression_hdfs_httpfs_signature_secret", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml parameter.", "display_name": "Suppress Parameter Validation: HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml", "name": "role_config_suppression_httpfs_config_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Configuration Options for HttpFS parameter.", "display_name": "Suppress Parameter Validation: Java Configuration Options for HttpFS", "name": "role_config_suppression_httpfs_java_opts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Keystore Password parameter.", "display_name": "Suppress Parameter Validation: HttpFS Keystore Password", "name": "role_config_suppression_httpfs_https_keystore_password", "value": "false" }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_httpfs_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: HttpFS Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_httpfs_role_env_safety_valve", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "The amount of stacks data that is retained. After the retention limit is reached, the oldest data is deleted.", "display_name": "Stacks Collection Data Retention", "name": "stacks_collection_data_retention", "value": "104857600" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "Directory where HttpFS will place its log files.", "display_name": "HttpFS Log Directory", "name": "httpfs_log_dir", "value": "/var/log/hadoop-httpfs" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "HttpFS Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "The health test thresholds for monitoring of free space on the filesystem that contains this role's log directory.", "display_name": "Log Directory Free Space Monitoring Absolute Thresholds", "name": "log_directory_free_space_absolute_thresholds", "value": "{\"critical\":\"5.36870912E9\",\"warning\":\"1.073741824E10\"}" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "For advanced use only. A string to be inserted into httpfs-site.xml for this role only.", "display_name": "HttpFS Advanced Configuration Snippet (Safety Valve) for httpfs-site.xml", "name": "httpfs_config_safety_valve", "value": null }, { "desc": "The maximum size, in megabytes, per log file for HttpFS logs. Typically used by log4j or logback.", "display_name": "HttpFS Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "When set, a SIGKILL signal is sent to the role process when java.lang.OutOfMemoryError is thrown.", "display_name": "Kill When Out of Memory", "name": "oom_sigkill_enabled", "value": "true" }, { "desc": "These arguments will be passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags would be passed here.", "display_name": "Java Configuration Options for HttpFS", "name": "httpfs_java_opts", "value": "" }, { "desc": "When set, generates heap dump file when java.lang.OutOfMemoryError is thrown.", "display_name": "Dump Heap When Out of Memory", "name": "oom_heap_dump_enabled", "value": "false" }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "HttpFS Environment Advanced Configuration Snippet (Safety Valve)", "name": "HTTPFS_role_env_safety_valve", "value": null }, { "desc": "The secret to use for signing client authentication tokens.", "display_name": "Signature Secret", "name": "hdfs_httpfs_signature_secret", "value": "hadoop httpfs secret" }, { "desc": "Address of the load balancer used for HttpFS roles. Should be specified in host:port format. Note: Changing this property will regenerate Kerberos keytabs for all HttpFS roles.", "display_name": "HttpFS Load Balancer", "name": "httpfs_load_balancer", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Location of the keystore file used by the HttpFS role for TLS/SSL.", "display_name": "HttpFS Keystore File", "name": "httpfs_https_keystore_file", "value": "/var/run/hadoop-httpfs/.keystore" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Stacks Collection Directory parameter.", "display_name": "Suppress Parameter Validation: Stacks Collection Directory", "name": "role_config_suppression_stacks_collection_directory", "value": "false" }, { "desc": "The directory in which stacks logs are placed. If not set, stacks are logged into a stacks subdirectory of the role's log directory.", "display_name": "Stacks Collection Directory", "name": "stacks_collection_directory", "value": null }, { "desc": "Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.", "display_name": "Java Heap Size of HttpFS in Bytes", "name": "httpfs_java_heapsize", "value": "268435456" }, { "desc": "The port for the administration interface.", "display_name": "Administration Port", "name": "hdfs_httpfs_admin_port", "value": "14001" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_httpfs_file_descriptor", "value": "false" }, { "desc": "The maximum number of rolled log files to keep for HttpFS logs. Typically used by log4j or logback.", "display_name": "HttpFS Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS Load Balancer parameter.", "display_name": "Suppress Parameter Validation: HttpFS Load Balancer", "name": "role_config_suppression_httpfs_load_balancer", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "The port where the REST interface to HDFS is available. The REST interface is served over HTTPS if TLS/SSL is enabled for HttpFS, or over HTTP otherwise.", "display_name": "REST Port", "name": "hdfs_httpfs_http_port", "value": "14000" }, { "desc": "Path to directory where heap dumps are generated when java.lang.OutOfMemoryError error is thrown. This directory is automatically created if it does not exist. If this directory already exists, role user must have write access to this directory. If this directory is shared among multiple roles, it should have 1777 permissions. The heap dump files are created with 600 permissions and are owned by the role user. The amount of free space in this directory should be greater than the maximum Java Process heap size configured for this role.", "display_name": "Heap Dump Directory", "name": "oom_heap_dump_dir", "value": "/tmp" }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_httpfs_host_health", "value": "false" }, { "desc": "The frequency with which stacks are collected.", "display_name": "Stacks Collection Frequency", "name": "stacks_collection_frequency", "value": "5.0" }, { "desc": "Whether to suppress the results of the Log Directory Free Space heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Log Directory Free Space", "name": "role_health_suppression_httpfs_log_directory_free_space", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HttpFS TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: HttpFS TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_httpfs_https_truststore_password", "value": "false" }, { "desc": "The minimum log level for HttpFS logs", "display_name": "HttpFS Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "Use TLS/SSL for HttpFS.", "display_name": "Use TLS/SSL", "name": "httpfs_use_ssl", "value": "false" }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "The user that the HttpFS server process should run as.", "display_name": "System User", "name": "httpfs_process_username", "value": "httpfs" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/oozie-service.json0000666000175100017510000003204613245514472025163 0ustar zuulzuul00000000000000[ { "desc": "Namespace used by this Oozie service in ZooKeeper when High Availability is enabled.", "display_name": "ZooKeeper Namespace", "name": "oozie_zookeeper_namespace", "value": "oozie" }, { "desc": "The user that this service's processes should run as.", "display_name": "System User", "name": "process_username", "value": "oozie" }, { "desc": "The timeout in seconds used for the Oozie Upload ShareLib command. When the value is zero, there is no timeout for the command.", "display_name": "Oozie Upload ShareLib Command Timeout", "name": "oozie_upload_sharelib_cmd_timeout", "value": "270" }, { "desc": "URL of the JMS Broker used by the Oozie service in JMS integration is enabled.", "display_name": "JMS Broker", "name": "oozie_jms_broker", "value": "tcp://localhost:61616" }, { "desc": "Encrypt communication between clients and Oozie using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Oozie", "name": "oozie_use_ssl", "value": "false" }, { "desc": "List of event listeners used by the Oozie service. Listeners needed for JMS or SLA integration are automatically emitted if they are enabled.", "display_name": "Oozie Event Listeners", "name": "oozie_event_listeners", "value": "" }, { "desc": "Name of the ZooKeeper service that this Oozie service instance depends on", "display_name": "ZooKeeper Service", "name": "zookeeper_service", "value": null }, { "desc": "Whether to suppress the results of the Oozie Server Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Oozie Server Health", "name": "service_health_suppression_oozie_oozie_servers_healthy", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Service Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Oozie Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_oozie_env_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the JMS Broker parameter.", "display_name": "Suppress Parameter Validation: JMS Broker", "name": "service_config_suppression_oozie_jms_broker", "value": "false" }, { "desc": "Whether to configure Oozie properties needed for SLA integration", "display_name": "Enable SLA Integration", "name": "oozie_use_sla", "value": "false" }, { "desc": "Coordinator Job Lookup trigger command is scheduled at this interval (in seconds).", "display_name": "Coordinator Job Lookup Interval", "name": "oozie_service_coord_lookup_interval", "value": "300" }, { "desc": "Kerberos principal short name used by all roles of this service.", "display_name": "Kerberos Principal", "name": "kerberos_princ_name", "value": "oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ShareLib Root Directory parameter.", "display_name": "Suppress Parameter Validation: ShareLib Root Directory", "name": "service_config_suppression_oozie_sharelib_rootdir", "value": "false" }, { "desc": "Root of the directory in HDFS where the Oozie ShareLibs are stored. The libraries are stored in the share/lib subdirectory under the specified root directory.", "display_name": "ShareLib Root Directory", "name": "oozie_sharelib_rootdir", "value": "/user/oozie" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Event Listeners parameter.", "display_name": "Suppress Parameter Validation: Oozie Event Listeners", "name": "service_config_suppression_oozie_event_listeners", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Load Balancer parameter.", "display_name": "Suppress Parameter Validation: Oozie Load Balancer", "name": "service_config_suppression_oozie_load_balancer", "value": "false" }, { "desc": "The group that this service's processes should run as.", "display_name": "System Group", "name": "process_groupname", "value": "oozie" }, { "desc": "Whether to configure Oozie properties needed for JMS integration", "display_name": "Enable JMS Integration", "name": "oozie_use_jms", "value": "false" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration.", "display_name": "Oozie Service Environment Advanced Configuration Snippet (Safety Valve)", "name": "oozie_env_safety_valve", "value": null }, { "desc": "A list of credential class mappings for CredentialsProvider.", "display_name": "Oozie Credential Classes", "name": "oozie_credential_classes", "value": "hcat=org.apache.oozie.action.hadoop.HCatCredentials,hbase=org.apache.oozie.action.hadoop.HbaseCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Service Level Health Alerts", "name": "enable_alerts", "value": "true" }, { "desc": "Address of the load balancer used if Oozie HA is enabled. Should be specified in host:port format.", "display_name": "Oozie Load Balancer", "name": "oozie_load_balancer", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Triggers parameter.", "display_name": "Suppress Parameter Validation: Service Triggers", "name": "service_config_suppression_service_triggers", "value": "false" }, { "desc": "Name of the Spark on Yarn service that this Oozie service instance depends on", "display_name": "Spark on Yarn Service", "name": "spark_on_yarn_service", "value": null }, { "desc": "

The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad\",\n  \"streamThreshold\": 10, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Service Triggers", "name": "service_triggers", "value": "[]" }, { "desc": "The health test thresholds of the overall Oozie Server health. The check returns \"Concerning\" health if the percentage of \"Healthy\" Oozie Servers falls below the warning threshold. The check is unhealthy if the total percentage of \"Healthy\" and \"Concerning\" Oozie Servers falls below the critical threshold.", "display_name": "Healthy Oozie Server Monitoring Thresholds", "name": "oozie_servers_healthy_thresholds", "value": "{\"critical\":\"51.0\",\"warning\":\"99.0\"}" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Oozie Credential Classes parameter.", "display_name": "Suppress Parameter Validation: Oozie Credential Classes", "name": "service_config_suppression_oozie_credential_classes", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System User parameter.", "display_name": "Suppress Parameter Validation: System User", "name": "service_config_suppression_process_username", "value": "false" }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kerberos Principal parameter.", "display_name": "Suppress Parameter Validation: Kerberos Principal", "name": "service_config_suppression_kerberos_princ_name", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the ZooKeeper Namespace parameter.", "display_name": "Suppress Parameter Validation: ZooKeeper Namespace", "name": "service_config_suppression_oozie_zookeeper_namespace", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the System Group parameter.", "display_name": "Suppress Parameter Validation: System Group", "name": "service_config_suppression_process_groupname", "value": "false" }, { "desc": "Name of the Hive service that this Oozie service instance depends on. This is used to configure Oozie HCat integration.", "display_name": "Hive Service", "name": "hive_service", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the Oozie Server Count Validator configuration validator.", "display_name": "Suppress Configuration Validator: Oozie Server Count Validator", "name": "service_config_suppression_oozie_server_count_validator", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "service_config_suppression_smon_derived_configs_safety_valve", "value": "false" }, { "desc": "Service to run MapReduce jobs against", "display_name": "MapReduce Service", "name": "mapreduce_yarn_service", "value": null }, { "desc": "For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones.", "display_name": "Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve)", "name": "smon_derived_configs_safety_valve", "value": null }, { "desc": "Use ACLs on Znode while a secure ZooKeeper is used for Oozie High Availability. Note: This config is not emitted if ZooKeeper is not secure.", "display_name": "Use ACLs on Znode", "name": "oozie_zk_secure", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/v5_5_0/resources/kafka-kafka_broker.json0000666000175100017510000006744113245514472026103 0ustar zuulzuul00000000000000[ { "desc": "

The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed.

Each trigger has the following fields:

  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger is not evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here can lead to inconsistencies.

For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file descriptors opened:

[{\"triggerName\": \"sample-trigger\",\n  \"triggerExpression\": \"IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad\",\n  \"streamThreshold\": 0, \"enabled\": \"true\"}]

See the trigger rules documentation for more details on how to write triggers using tsquery.

The JSON format is evolving and may change and, as a result, backward compatibility is not guaranteed between releases.

", "display_name": "Role Triggers", "name": "role_triggers", "value": "[]" }, { "desc": "The port to give out to producers, consumers, and other brokers to use in establishing connections. This only needs to be set if this port is different from the port the server should bind to.", "display_name": "Advertised Port", "name": "advertised.port", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Additional Broker Java Options parameter.", "display_name": "Suppress Parameter Validation: Additional Broker Java Options", "name": "role_config_suppression_broker_java_opts", "value": "false" }, { "desc": "If set, this is the hostname given out to producers, consumers, and other brokers to use in establishing connections. Never set this property at the group level; it should always be overriden on instance level.", "display_name": "Advertised Host", "name": "advertised.host.name", "value": null }, { "desc": "The period to review when computing unexpected exits.", "display_name": "Unexpected Exits Monitoring Period", "name": "unexpected_exits_window", "value": "5" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Role Triggers parameter.", "display_name": "Suppress Parameter Validation: Role Triggers", "name": "role_config_suppression_role_triggers", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the HTTP Metric Report Host parameter.", "display_name": "Suppress Parameter Validation: HTTP Metric Report Host", "name": "role_config_suppression_kafka.http.metrics.host", "value": "false" }, { "desc": "Whether to suppress the results of the Process Status heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Process Status", "name": "role_health_suppression_kafka_kafka_broker_scm_health", "value": "false" }, { "desc": "Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager.", "display_name": "Cgroup I/O Weight", "name": "rm_io_weight", "value": "500" }, { "desc": "When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold", "display_name": "Enable Health Alerts for this Role", "name": "enable_alerts", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore File Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore File Password", "name": "role_config_suppression_ssl_server_keystore_password", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Certificate Trust Store Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Certificate Trust Store Password", "name": "role_config_suppression_ssl_client_truststore_password", "value": "false" }, { "desc": "The location on disk of the trust store, in .jks format, used to confirm the authenticity of TLS/SSL servers that Kafka Broker might connect to. This is used when Kafka Broker is the client in a TLS/SSL connection. This trust store must contain the certificate(s) used to sign the service(s) connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.", "display_name": "Kafka Broker TLS/SSL Certificate Trust Store File", "name": "ssl_client_truststore_location", "value": "" }, { "desc": "Port the HTTP metric reporter listens on.", "display_name": "HTTP Metric Report Port", "name": "kafka.http.metrics.port", "value": "24042" }, { "desc": "Kafka broker port.", "display_name": "TCP Port", "name": "port", "value": "9092" }, { "desc": "The log for a topic partition is stored as a directory of segment files. This setting controls the size to which a segment file can grow before a new segment is rolled over in the log. This value should be larger than message.max.bytes.", "display_name": "Segment File Size", "name": "log.segment.bytes", "value": "1073741824" }, { "desc": "For advanced use only. A string to be inserted into kafka-monitoring.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties", "name": "kafka-monitoring.properties_role_safety_valve", "value": null }, { "desc": "The frequency, in milliseconds, that the log cleaner checks whether any log segment is eligible for deletion, per retention policies.", "display_name": "Data Retention Check Interval", "name": "log.retention.check.interval.ms", "value": "300000" }, { "desc": "The maximum number of rolled log files to keep for Kafka Broker logs. Typically used by log4j or logback.", "display_name": "Kafka Broker Maximum Log File Backups", "name": "max_log_backup_index", "value": "10" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Advertised Host parameter.", "display_name": "Suppress Parameter Validation: Advertised Host", "name": "role_config_suppression_advertised.host.name", "value": "false" }, { "desc": "The minimum log level for Kafka Broker logs", "display_name": "Kafka Broker Logging Threshold", "name": "log_threshold", "value": "INFO" }, { "desc": "The health test thresholds on the swap memory usage of the process.", "display_name": "Process Swap Memory Thresholds", "name": "process_swap_memory_thresholds", "value": "{\"critical\":\"never\",\"warning\":\"any\"}" }, { "desc": "When computing the overall Kafka Broker health, consider the host's health.", "display_name": "Kafka Broker Host Health Test", "name": "kafka_broker_host_health_enabled", "value": "true" }, { "desc": "Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in megabytes. Kafka does not generally require setting large heap sizes. It is better to let the file system cache utilize the available memory.", "display_name": "Java Heap Size of Broker", "name": "broker_max_heap_size", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the CDH Version Validator configuration validator.", "display_name": "Suppress Configuration Validator: CDH Version Validator", "name": "role_config_suppression_cdh_version_validator", "value": "false" }, { "desc": "Whether to suppress the results of the Swap Memory Usage heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Swap Memory Usage", "name": "role_health_suppression_kafka_kafka_broker_swap_memory_usage", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Log Directory parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Log Directory", "name": "role_config_suppression_log_dir", "value": "false" }, { "desc": "The number of I/O threads that the server uses for executing requests. You should have at least as many threads as you have disks.", "display_name": "Number of I/O Threads", "name": "num.io.threads", "value": "8" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore File Location parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore File Location", "name": "role_config_suppression_ssl_server_keystore_location", "value": "false" }, { "desc": "Protocol to be used for inter-broker communication.", "display_name": "Inter Broker Protocol", "name": "security.inter.broker.protocol", "value": "PLAINTEXT" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties", "name": "role_config_suppression_ssl.properties_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role.", "display_name": "Unexpected Exits Thresholds", "name": "unexpected_exits_thresholds", "value": "{\"critical\":\"any\",\"warning\":\"never\"}" }, { "desc": "For advanced use only. A string to be inserted into kafka.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties", "name": "kafka.properties_role_safety_valve", "value": null }, { "desc": "When set, Cloudera Manager will send alerts when this entity's configuration changes.", "display_name": "Enable Configuration Change Alerts", "name": "enable_config_alerts", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Server JKS Keystore Key Password parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Server JKS Keystore Key Password", "name": "role_config_suppression_ssl_server_keystore_keypassword", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out (in hours). Secondary to the log.retention.ms property. The special value of -1 is interpreted as unlimited. This property is deprecated in Kafka 1.4.0. Use log.retention.ms.", "display_name": "Data Retention Hours", "name": "log.retention.hours", "value": "168" }, { "desc": "Kafka broker secure port.", "display_name": "TLS/SSL Port", "name": "ssl_port", "value": "9093" }, { "desc": "The amount of data to retain in the log for each topic-partition. This is the limit per partition: multiply by the number of partitions to get the total data retained for the topic. The special value of -1 is interpreted as unlimited. If both log.retention.ms and log.retention.bytes are set, a segment is deleted when either limit is exceeded.", "display_name": "Data Retention Size", "name": "log.retention.bytes", "value": "-1" }, { "desc": "When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure.", "display_name": "Automatically Restart Process", "name": "process_auto_restart", "value": "false" }, { "desc": "Host the HTTP metric reporter binds to.", "display_name": "HTTP Metric Report Host", "name": "kafka.http.metrics.host", "value": "0.0.0.0" }, { "desc": "The maximum size, in megabytes, per log file for Kafka Broker logs. Typically used by log4j or logback.", "display_name": "Kafka Broker Max Log Size", "name": "max_log_size", "value": "200" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties", "name": "role_config_suppression_kafka-monitoring.properties_role_safety_valve", "value": "false" }, { "desc": "The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit.", "display_name": "File Descriptor Monitoring Thresholds", "name": "kafka_broker_fd_thresholds", "value": "{\"critical\":\"70.0\",\"warning\":\"50.0\"}" }, { "desc": "For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration.", "display_name": "Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)", "name": "KAFKA_BROKER_role_env_safety_valve", "value": null }, { "desc": "Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Soft Limit", "name": "rm_memory_soft_limit", "value": "-1" }, { "desc": "Port for JMX.", "display_name": "JMX Port", "name": "jmx_port", "value": "9393" }, { "desc": "The log directory for log files of the role Kafka Broker.", "display_name": "Kafka Broker Log Directory", "name": "log_dir", "value": "/var/log/kafka" }, { "desc": "The maximum time before a new log segment is rolled out. This property is used in Cloudera Kafka 1.4.0 and later in place of log.roll.hours.", "display_name": "Data Log Roll Time", "name": "log.roll.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties", "name": "role_config_suppression_kafka.properties_role_safety_valve", "value": "false" }, { "desc": "The password that protects the private key contained in the JKS keystore used when Kafka Broker is acting as a TLS/SSL server.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore Key Password", "name": "ssl_server_keystore_keypassword", "value": "" }, { "desc": "For advanced use only, a string to be inserted into log4j.properties for this role only.", "display_name": "Kafka Broker Logging Advanced Configuration Snippet (Safety Valve)", "name": "log4j_safety_valve", "value": null }, { "desc": "Whether to suppress the results of the Unexpected Exits heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Unexpected Exits", "name": "role_health_suppression_kafka_kafka_broker_unexpected_exits", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Data Directories parameter.", "display_name": "Suppress Parameter Validation: Data Directories", "name": "role_config_suppression_log.dirs", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out. If both log.retention.ms and log.retention.bytes are set, a segment is deleted when either limit is exceeded. The special value of -1 is interpreted as unlimited. This property is used in Kafka 1.4.0 and later in place of log.retention.hours.", "display_name": "Data Retention Time", "name": "log.retention.ms", "value": null }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker TLS/SSL Certificate Trust Store File parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker TLS/SSL Certificate Trust Store File", "name": "role_config_suppression_ssl_client_truststore_location", "value": "false" }, { "desc": "The maximum time before a new log segment is rolled out (in hours). This property is deprecated in Cloudera Kafka 1.4.0; use log.roll.ms.", "display_name": "Data Log Roll Hours", "name": "log.roll.hours", "value": "168" }, { "desc": "These arguments are passed as part of the Java command line. Commonly, garbage collection flags or extra debugging flags are passed here.", "display_name": "Additional Broker Java Options", "name": "broker_java_opts", "value": "-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" }, { "desc": "The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Kafka Broker is acting as a TLS/SSL server. The keystore must be in JKS format.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore File Location", "name": "ssl_server_keystore_location", "value": "" }, { "desc": "If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value.", "display_name": "Maximum Process File Descriptors", "name": "rlimit_fds", "value": null }, { "desc": "Whether to suppress the results of the Host Health heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: Host Health", "name": "role_health_suppression_kafka_kafka_broker_host_health", "value": "false" }, { "desc": "A list of one or more directories in which Kafka data is stored. Each new partition created is placed in the directory that currently has the fewest partitions. Each directory should be on its own separate drive.", "display_name": "Data Directories", "name": "log.dirs", "value": "/var/local/kafka/data" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Segment File Size parameter.", "display_name": "Suppress Parameter Validation: Segment File Size", "name": "role_config_suppression_log.segment.bytes", "value": "false" }, { "desc": "Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager.", "display_name": "Cgroup CPU Shares", "name": "rm_cpu_shares", "value": "1024" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Logging Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Logging Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_log4j_safety_valve", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Broker ID parameter.", "display_name": "Suppress Parameter Validation: Broker ID", "name": "role_config_suppression_broker.id", "value": "false" }, { "desc": "The password for the Kafka Broker JKS keystore file.", "display_name": "Kafka Broker TLS/SSL Server JKS Keystore File Password", "name": "ssl_server_keystore_password", "value": "" }, { "desc": "Whether to suppress the results of the File Descriptors heath test. The results of suppressed health tests are ignored when computing the overall health of the associated host, role or service, so suppressed health tests will not generate alerts.", "display_name": "Suppress Health Test: File Descriptors", "name": "role_health_suppression_kafka_kafka_broker_file_descriptor", "value": "false" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) parameter.", "display_name": "Suppress Parameter Validation: Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)", "name": "role_config_suppression_kafka_broker_role_env_safety_valve", "value": "false" }, { "desc": "For advanced use only. A string to be inserted into ssl.properties for this role only.", "display_name": "Kafka Broker Advanced Configuration Snippet (Safety Valve) for ssl.properties", "name": "ssl.properties_role_safety_valve", "value": null }, { "desc": "Encrypt communication between clients and Kafka Broker using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).", "display_name": "Enable TLS/SSL for Kafka Broker", "name": "ssl_enabled", "value": "false" }, { "desc": "Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit.", "display_name": "Cgroup Memory Hard Limit", "name": "rm_memory_hard_limit", "value": "-1" }, { "desc": "ID uniquely identifying each broker. Never set this property at the group level; it should always be overridden on instance level.", "display_name": "Broker ID", "name": "broker.id", "value": null }, { "desc": "Maximum number of connections allowed from each IP address.", "display_name": "Maximum Connections per IP Address", "name": "max.connections.per.ip", "value": null }, { "desc": "Enables the health test that the Kafka Broker's process state is consistent with the role configuration", "display_name": "Kafka Broker Process Health Test", "name": "kafka_broker_scm_health_enabled", "value": "true" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Number of I/O Threads parameter.", "display_name": "Suppress Parameter Validation: Number of I/O Threads", "name": "role_config_suppression_num.io.threads", "value": "false" }, { "desc": "The password for the Kafka Broker TLS/SSL Certificate Trust Store File. This password is not required to access the trust store; this field can be left blank. This password provides optional integrity checking of the file. The contents of trust stores are certificates, and certificates are public information.", "display_name": "Kafka Broker TLS/SSL Certificate Trust Store Password", "name": "ssl_client_truststore_password", "value": "" }, { "desc": "Whether to suppress configuration warnings produced by the built-in parameter validation for the Java Heap Size of Broker parameter.", "display_name": "Suppress Parameter Validation: Java Heap Size of Broker", "name": "role_config_suppression_broker_max_heap_size", "value": "false" }, { "desc": "Client authentication mode for SSL connections. Default is none, could be set to \"required\", i.e., client authentication is required or to \"requested\", i.e., client authentication is requested and client without certificates can still connect.", "display_name": "SSL Client Authentication", "name": "ssl.client.auth", "value": "none" }, { "desc": "Authenticate a SASL connection with zookeeper, if Kerberos authentication is enabled. It also allows a broker to set SASL ACL on zookeeper nodes which locks these nodes down so that only kafka broker can modify.", "display_name": "Authenticate Zookeeper Connection", "name": "authenticate.zookeeper.connection", "value": "true" } ]sahara-8.0.0/sahara/plugins/cdh/plugin_utils.py0000666000175100017510000004373313245514472021573 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # This file only contains utils not related to cm_api, while in # cloudera_utils the functions are cm_api involved. import os import telnetlib # nosec from oslo_log import log as logging from sahara.conductor import resource as res from sahara import context from sahara import exceptions as exc from sahara.i18n import _ from sahara.plugins.cdh import commands as cmd from sahara.plugins.cdh import db_helper as dh from sahara.plugins import recommendations_utils as ru from sahara.plugins import utils as u from sahara.swift import swift_helper from sahara.utils import cluster_progress_ops as cpo from sahara.utils import edp as edp_u from sahara.utils import poll_utils from sahara.utils import types PATH_TO_CORE_SITE_XML = '/etc/hadoop/conf/core-site.xml' HADOOP_LIB_DIR = '/usr/lib/hadoop-mapreduce' CM_API_PORT = 7180 LOG = logging.getLogger(__name__) AUTO_CONFIGURATION_SCHEMA = { 'node_configs': { 'yarn.scheduler.minimum-allocation-mb': ( 'RESOURCEMANAGER', 'yarn_scheduler_minimum_allocation_mb'), 'mapreduce.reduce.memory.mb': ( 'YARN_GATEWAY', 'mapreduce_reduce_memory_mb'), 'mapreduce.map.memory.mb': ( 'YARN_GATEWAY', 'mapreduce_map_memory_mb',), 'yarn.scheduler.maximum-allocation-mb': ( 'RESOURCEMANAGER', 'yarn_scheduler_maximum_allocation_mb'), 'yarn.app.mapreduce.am.command-opts': ( 'YARN_GATEWAY', 'yarn_app_mapreduce_am_command_opts'), 'yarn.nodemanager.resource.memory-mb': ( 'NODEMANAGER', 'yarn_nodemanager_resource_memory_mb'), 'mapreduce.task.io.sort.mb': ( 'YARN_GATEWAY', 'io_sort_mb'), 'mapreduce.map.java.opts': ( 'YARN_GATEWAY', 'mapreduce_map_java_opts'), 'mapreduce.reduce.java.opts': ( 'YARN_GATEWAY', 'mapreduce_reduce_java_opts'), 'yarn.app.mapreduce.am.resource.mb': ( 'YARN_GATEWAY', 'yarn_app_mapreduce_am_resource_mb') }, 'cluster_configs': { 'dfs.replication': ('HDFS', 'dfs_replication') } } class CDHPluginAutoConfigsProvider(ru.HadoopAutoConfigsProvider): def get_datanode_name(self): return 'HDFS_DATANODE' class AbstractPluginUtils(object): def __init__(self): # c_helper will be defined in derived classes. self.c_helper = None def get_role_name(self, instance, service): # NOTE: role name must match regexp "[_A-Za-z][-_A-Za-z0-9]{0,63}" shortcuts = { 'AGENT': 'A', 'ALERTPUBLISHER': 'AP', 'CATALOGSERVER': 'ICS', 'DATANODE': 'DN', 'EVENTSERVER': 'ES', 'HBASE_INDEXER': 'LHBI', 'HIVEMETASTORE': 'HVM', 'HIVESERVER2': 'HVS', 'HOSTMONITOR': 'HM', 'IMPALAD': 'ID', 'JOBHISTORY': 'JS', 'JOURNALNODE': 'JN', 'KAFKA_BROKER': 'KB', 'KMS': 'KMS', 'MASTER': 'M', 'NAMENODE': 'NN', 'NODEMANAGER': 'NM', 'OOZIE_SERVER': 'OS', 'REGIONSERVER': 'RS', 'RESOURCEMANAGER': 'RM', 'SECONDARYNAMENODE': 'SNN', 'SENTRY_SERVER': 'SNT', 'SERVER': 'S', 'SERVICEMONITOR': 'SM', 'SOLR_SERVER': 'SLR', 'SPARK_YARN_HISTORY_SERVER': 'SHS', 'SQOOP_SERVER': 'S2S', 'STATESTORE': 'ISS', 'WEBHCAT': 'WHC', 'HDFS_GATEWAY': 'HG', 'YARN_GATEWAY': 'YG' } return '%s_%s' % (shortcuts.get(service, service), instance.hostname().replace('-', '_')) def get_manager(self, cluster): return u.get_instance(cluster, 'CLOUDERA_MANAGER') def get_namenode(self, cluster): return u.get_instance(cluster, "HDFS_NAMENODE") def get_datanodes(self, cluster): return u.get_instances(cluster, 'HDFS_DATANODE') def get_hdfs_nodes(self, cluster, instances=None): instances = instances if instances else u.get_instances(cluster) return u.instances_with_services( instances, ["HDFS_DATANODE", "HDFS_NAMENODE", "HDFS_SECONDARYNAMENODE"]) def get_secondarynamenode(self, cluster): return u.get_instance(cluster, 'HDFS_SECONDARYNAMENODE') def get_historyserver(self, cluster): return u.get_instance(cluster, 'YARN_JOBHISTORY') def get_resourcemanager(self, cluster): return u.get_instance(cluster, 'YARN_RESOURCEMANAGER') def get_nodemanagers(self, cluster): return u.get_instances(cluster, 'YARN_NODEMANAGER') def get_oozie(self, cluster): return u.get_instance(cluster, 'OOZIE_SERVER') def get_hive_metastore(self, cluster): return u.get_instance(cluster, 'HIVE_METASTORE') def get_hive_servers(self, cluster): return u.get_instances(cluster, 'HIVE_SERVER2') def get_hue(self, cluster): return u.get_instance(cluster, 'HUE_SERVER') def get_spark_historyserver(self, cluster): return u.get_instance(cluster, 'SPARK_YARN_HISTORY_SERVER') def get_zookeepers(self, cluster): return u.get_instances(cluster, 'ZOOKEEPER_SERVER') def get_hbase_master(self, cluster): return u.get_instance(cluster, 'HBASE_MASTER') def get_sentry(self, cluster): return u.get_instance(cluster, 'SENTRY_SERVER') def get_flumes(self, cluster): return u.get_instances(cluster, 'FLUME_AGENT') def get_solrs(self, cluster): return u.get_instances(cluster, 'SOLR_SERVER') def get_sqoop(self, cluster): return u.get_instance(cluster, 'SQOOP_SERVER') def get_hbase_indexers(self, cluster): return u.get_instances(cluster, 'KEY_VALUE_STORE_INDEXER') def get_catalogserver(self, cluster): return u.get_instance(cluster, 'IMPALA_CATALOGSERVER') def get_statestore(self, cluster): return u.get_instance(cluster, 'IMPALA_STATESTORE') def get_impalads(self, cluster): return u.get_instances(cluster, 'IMPALAD') def get_kms(self, cluster): return u.get_instances(cluster, 'KMS') def get_jns(self, cluster): return u.get_instances(cluster, 'HDFS_JOURNALNODE') def get_stdb_rm(self, cluster): return u.get_instance(cluster, 'YARN_STANDBYRM') def get_kafka_brokers(self, cluster): return u.get_instances(cluster, 'KAFKA_BROKER') def convert_process_configs(self, configs): p_dict = { "CLOUDERA": ['MANAGER'], "NAMENODE": ['NAMENODE'], "DATANODE": ['DATANODE'], "SECONDARYNAMENODE": ['SECONDARYNAMENODE'], "RESOURCEMANAGER": ['RESOURCEMANAGER'], "NODEMANAGER": ['NODEMANAGER'], "JOBHISTORY": ['JOBHISTORY'], "OOZIE": ['OOZIE_SERVER'], "HIVESERVER": ['HIVESERVER2'], "HIVEMETASTORE": ['HIVEMETASTORE'], "WEBHCAT": ['WEBHCAT'], "HUE": ['HUE_SERVER'], "SPARK_ON_YARN": ['SPARK_YARN_HISTORY_SERVER'], "ZOOKEEPER": ['SERVER'], "MASTER": ['MASTER'], "REGIONSERVER": ['REGIONSERVER'], "FLUME": ['AGENT'], "CATALOGSERVER": ['CATALOGSERVER'], "STATESTORE": ['STATESTORE'], "IMPALAD": ['IMPALAD'], "KS_INDEXER": ['HBASE_INDEXER'], "SENTRY": ['SENTRY_SERVER'], "SOLR": ['SOLR_SERVER'], "SQOOP": ['SQOOP_SERVER'], "KMS": ['KMS'], "YARN_GATEWAY": ['YARN_GATEWAY'], "HDFS_GATEWAY": ['HDFS_GATEWAY'], "JOURNALNODE": ['JOURNALNODE'], "KAFKA": ['KAFKA_BROKER'] } if isinstance(configs, res.Resource): configs = configs.to_dict() for k in configs.keys(): if k in p_dict.keys(): item = configs[k] del configs[k] newkey = p_dict[k][0] configs[newkey] = item return res.Resource(configs) def convert_role_showname(self, showname): # Yarn ResourceManager and Standby ResourceManager will # be converted to ResourceManager. name_dict = { 'CLOUDERA_MANAGER': 'MANAGER', 'HDFS_NAMENODE': 'NAMENODE', 'HDFS_DATANODE': 'DATANODE', 'HDFS_JOURNALNODE': 'JOURNALNODE', 'HDFS_SECONDARYNAMENODE': 'SECONDARYNAMENODE', 'YARN_RESOURCEMANAGER': 'RESOURCEMANAGER', 'YARN_STANDBYRM': 'RESOURCEMANAGER', 'YARN_NODEMANAGER': 'NODEMANAGER', 'YARN_JOBHISTORY': 'JOBHISTORY', 'OOZIE_SERVER': 'OOZIE_SERVER', 'HIVE_SERVER2': 'HIVESERVER2', 'HIVE_METASTORE': 'HIVEMETASTORE', 'HIVE_WEBHCAT': 'WEBHCAT', 'HUE_SERVER': 'HUE_SERVER', 'SPARK_YARN_HISTORY_SERVER': 'SPARK_YARN_HISTORY_SERVER', 'ZOOKEEPER_SERVER': 'SERVER', 'HBASE_MASTER': 'MASTER', 'HBASE_REGIONSERVER': 'REGIONSERVER', 'FLUME_AGENT': 'AGENT', 'IMPALA_CATALOGSERVER': 'CATALOGSERVER', 'IMPALA_STATESTORE': 'STATESTORE', 'IMPALAD': 'IMPALAD', 'KEY_VALUE_STORE_INDEXER': 'HBASE_INDEXER', 'SENTRY_SERVER': 'SENTRY_SERVER', 'SOL_SERVER': 'SOLR_SERVER', 'SQOOP_SERVER': 'SQOOP_SERVER', } return name_dict.get(showname, showname) def install_packages(self, instances, packages): # instances non-empty cpo.add_provisioning_step( instances[0].cluster_id, _("Install packages"), len(instances)) with context.ThreadGroup() as tg: for i in instances: tg.spawn('cdh-inst-pkgs-%s' % i.instance_name, self._install_pkgs, i, packages) @cpo.event_wrapper(True) def _install_pkgs(self, instance, packages): with instance.remote() as r: cmd.install_packages(r, packages) def start_cloudera_agents(self, instances): # instances non-empty cpo.add_provisioning_step( instances[0].cluster_id, _("Start Cloudera Agents"), len(instances)) with context.ThreadGroup() as tg: for i in instances: tg.spawn('cdh-agent-start-%s' % i.instance_name, self._start_cloudera_agent, i) @cpo.event_wrapper(True) def _start_cloudera_agent(self, instance): mng_hostname = self.get_manager(instance.cluster).hostname() with instance.remote() as r: cmd.configure_agent(r, mng_hostname) cmd.start_agent(r) def configure_swift(self, cluster, instances=None): if self.c_helper.is_swift_enabled(cluster): if not instances: instances = u.get_instances(cluster) cpo.add_provisioning_step( cluster.id, _("Configure Swift"), len(instances)) with context.ThreadGroup() as tg: for i in instances: tg.spawn('cdh-swift-conf-%s' % i.instance_name, self._configure_swift_to_inst, i) swift_helper.install_ssl_certs(instances) @cpo.event_wrapper(True) def _configure_swift_to_inst(self, instance): cluster = instance.cluster swift_lib_remote_url = self.c_helper.get_swift_lib_url(cluster) with instance.remote() as r: if r.execute_command('ls %s/hadoop-openstack.jar' % HADOOP_LIB_DIR, raise_when_error=False)[0] != 0: r.execute_command('sudo curl %s -o %s/hadoop-openstack.jar' % ( swift_lib_remote_url, HADOOP_LIB_DIR)) def configure_sentry(self, cluster): manager = self.get_manager(cluster) with manager.remote() as r: dh.create_sentry_database(cluster, r) def put_hive_hdfs_xml(self, cluster): servers = self.get_hive_servers(cluster) with servers[0].remote() as r: conf_path = edp_u.get_hive_shared_conf_path('hdfs') r.execute_command( 'sudo su - -c "hadoop fs -mkdir -p %s" hdfs' % os.path.dirname(conf_path)) r.execute_command( 'sudo su - -c "hadoop fs -put /etc/hive/conf/hive-site.xml ' '%s" hdfs' % conf_path) def configure_hive(self, cluster): manager = self.get_manager(cluster) with manager.remote() as r: dh.create_hive_database(cluster, r) def install_extjs(self, cluster): extjs_remote_location = self.c_helper.get_extjs_lib_url(cluster) extjs_vm_location_dir = '/var/lib/oozie' extjs_vm_location_path = extjs_vm_location_dir + '/extjs.zip' with self.get_oozie(cluster).remote() as r: if r.execute_command('ls %s/ext-2.2' % extjs_vm_location_dir, raise_when_error=False)[0] != 0: r.execute_command('curl -L -o \'%s\' %s' % ( extjs_vm_location_path, extjs_remote_location), run_as_root=True) r.execute_command('unzip %s -d %s' % ( extjs_vm_location_path, extjs_vm_location_dir), run_as_root=True) def _check_cloudera_manager_started(self, manager): try: conn = telnetlib.Telnet(manager.management_ip, CM_API_PORT) conn.close() return True except IOError: return False @cpo.event_wrapper( True, step=_("Start Cloudera Manager"), param=('cluster', 1)) def _start_cloudera_manager(self, cluster, timeout_config): manager = self.get_manager(cluster) with manager.remote() as r: cmd.start_cloudera_db(r) cmd.start_manager(r) poll_utils.plugin_option_poll( cluster, self._check_cloudera_manager_started, timeout_config, _("Await starting Cloudera Manager"), 2, {'manager': manager}) def configure_os(self, instances): # instances non-empty cpo.add_provisioning_step( instances[0].cluster_id, _("Configure OS"), len(instances)) with context.ThreadGroup() as tg: for inst in instances: tg.spawn('cdh-repo-conf-%s' % inst.instance_name, self._configure_repo_from_inst, inst) @cpo.event_wrapper(True) def _configure_repo_from_inst(self, instance): LOG.debug("Configure repos from instance {instance}".format( instance=instance.instance_name)) cluster = instance.cluster with instance.remote() as r: if cmd.is_ubuntu_os(r): cdh5_key = ( self.c_helper.get_cdh5_key_url(cluster) or self.c_helper.DEFAULT_CDH5_UBUNTU_REPO_KEY_URL) cm5_key = ( self.c_helper.get_cm5_key_url(cluster) or self.c_helper.DEFAULT_CM5_UBUNTU_REPO_KEY_URL) kms_key = ( self.c_helper.get_kms_key_url(cluster) or self.c_helper.DEFAULT_KEY_TRUSTEE_UBUNTU_REPO_KEY_URL) cdh5_repo_content = self.c_helper.CDH5_UBUNTU_REPO cm5_repo_content = self.c_helper.CM5_UBUNTU_REPO kms_repo_url = self.c_helper.KEY_TRUSTEE_UBUNTU_REPO_URL cmd.write_ubuntu_repository(r, cdh5_repo_content, 'cdh') cmd.add_apt_key(r, cdh5_key) cmd.write_ubuntu_repository(r, cm5_repo_content, 'cm') cmd.add_apt_key(r, cm5_key) cmd.add_ubuntu_repository(r, kms_repo_url, 'kms') cmd.add_apt_key(r, kms_key) cmd.update_repository(r) if cmd.is_centos_os(r): cdh5_repo_content = self.c_helper.CDH5_CENTOS_REPO cm5_repo_content = self.c_helper.CM5_CENTOS_REPO kms_repo_url = self.c_helper.KEY_TRUSTEE_CENTOS_REPO_URL cmd.write_centos_repository(r, cdh5_repo_content, 'cdh') cmd.write_centos_repository(r, cm5_repo_content, 'cm') cmd.add_centos_repository(r, kms_repo_url, 'kms') cmd.update_repository(r) def _get_config_value(self, service, name, configs, cluster=None): if cluster: conf = cluster.cluster_configs if service in conf and name in conf[service]: return types.transform_to_num(conf[service][name]) for node_group in cluster.node_groups: conf = node_group.node_configs if service in conf and name in conf[service]: return types.transform_to_num(conf[service][name]) for config in configs: if config.applicable_target == service and config.name == name: return types.transform_to_num(config.default_value) raise exc.InvalidDataException( _("Unable to find config: applicable_target: {target}, name: " "{name}").format(target=service, name=name)) def recommend_configs(self, cluster, plugin_configs, scaling): provider = CDHPluginAutoConfigsProvider( AUTO_CONFIGURATION_SCHEMA, plugin_configs, cluster, scaling) provider.apply_recommended_configs() def start_cloudera_manager(self, cluster): self._start_cloudera_manager( cluster, self.c_helper.AWAIT_MANAGER_STARTING_TIMEOUT) def get_config_value(self, service, name, cluster=None): configs = self.c_helper.get_plugin_configs() return self._get_config_value(service, name, configs, cluster) sahara-8.0.0/sahara/plugins/cdh/client/0000775000175100017510000000000013245515026017741 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/client/services.py0000666000175100017510000005010213245514472022142 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. from oslo_serialization import jsonutils as json import six from sahara.plugins.cdh.client import role_config_groups from sahara.plugins.cdh.client import roles from sahara.plugins.cdh.client import types SERVICES_PATH = "/clusters/%s/services" SERVICE_PATH = "/clusters/%s/services/%s" ROLETYPES_CFG_KEY = 'roleTypeConfigs' def create_service(resource_root, name, service_type, cluster_name="default"): """Create a service :param resource_root: The root Resource object. :param name: Service name :param service_type: Service type :param cluster_name: Cluster name :return: An ApiService object """ apiservice = ApiService(resource_root, name, service_type) return types.call(resource_root.post, SERVICES_PATH % (cluster_name,), ApiService, True, data=[apiservice])[0] def get_service(resource_root, name, cluster_name="default"): """Lookup a service by name :param resource_root: The root Resource object. :param name: Service name :param cluster_name: Cluster name :return: An ApiService object """ return _get_service(resource_root, "%s/%s" % (SERVICES_PATH % (cluster_name,), name)) def _get_service(resource_root, path): return types.call(resource_root.get, path, ApiService) def get_all_services(resource_root, cluster_name="default", view=None): """Get all services :param resource_root: The root Resource object. :param cluster_name: Cluster name :return: A list of ApiService objects. """ return types.call(resource_root.get, SERVICES_PATH % (cluster_name,), ApiService, True, params=(dict(view=view) if view else None)) def delete_service(resource_root, name, cluster_name="default"): """Delete a service by name :param resource_root: The root Resource object. :param name: Service name :param cluster_name: Cluster name :return: The deleted ApiService object """ return types.call(resource_root.delete, "%s/%s" % (SERVICES_PATH % (cluster_name,), name), ApiService) class ApiService(types.BaseApiResource): _ATTRIBUTES = { 'name': None, 'type': None, 'displayName': None, 'serviceState': types.ROAttr(), 'healthSummary': types.ROAttr(), 'healthChecks': types.ROAttr(), 'clusterRef': types.ROAttr(types.ApiClusterRef), 'configStale': types.ROAttr(), 'configStalenessStatus': types.ROAttr(), 'clientConfigStalenessStatus': types.ROAttr(), 'serviceUrl': types.ROAttr(), 'maintenanceMode': types.ROAttr(), 'maintenanceOwners': types.ROAttr(), } def __init__(self, resource_root, name=None, type=None): types.BaseApiObject.init(self, resource_root, locals()) def __str__(self): return (": %s (cluster: %s)" % (self.name, self._get_cluster_name())) def _get_cluster_name(self): if hasattr(self, 'clusterRef') and self.clusterRef: return self.clusterRef.clusterName return None def _path(self): """Return the API path for this service This method assumes that lack of a cluster reference means that the object refers to the Cloudera Management Services instance. """ if self._get_cluster_name(): return SERVICE_PATH % (self._get_cluster_name(), self.name) else: return '/cm/service' def _role_cmd(self, cmd, roles, api_version=1): return self._post("roleCommands/" + cmd, types.ApiBulkCommandList, data=roles, api_version=api_version) def _parse_svc_config(self, json_dic, view=None): """Parse a json-decoded ApiServiceConfig dictionary into a 2-tuple :param json_dic: The json dictionary with the config data. :param view: View to materialize. :return: 2-tuple (service config dictionary, role type configurations) """ svc_config = types.json_to_config(json_dic, view == 'full') rt_configs = {} if ROLETYPES_CFG_KEY in json_dic: for rt_config in json_dic[ROLETYPES_CFG_KEY]: rt_configs[rt_config['roleType']] = types.json_to_config( rt_config, view == 'full') return (svc_config, rt_configs) def create_yarn_job_history_dir(self): """Create the Yarn job history directory :return: Reference to submitted command. :since: API v6 """ return self._cmd('yarnCreateJobHistoryDirCommand', api_version=6) def get_config(self, view=None): """Retrieve the service's configuration Retrieves both the service configuration and role type configuration for each of the service's supported role types. The role type configurations are returned as a dictionary, whose keys are the role type name, and values are the respective configuration dictionaries. The 'summary' view contains strings as the dictionary values. The full view contains types.ApiConfig instances as the values. :param view: View to materialize ('full' or 'summary') :return: 2-tuple (service config dictionary, role type configurations) """ path = self._path() + '/config' resp = self._get_resource_root().get( path, params=(dict(view=view) if view else None)) return self._parse_svc_config(resp, view) def update_config(self, svc_config, **rt_configs): """Update the service's configuration :param svc_config: Dictionary with service configuration to update. :param rt_configs: Dict of role type configurations to update. :return: 2-tuple (service config dictionary, role type configurations) """ path = self._path() + '/config' if svc_config: data = types.config_to_api_list(svc_config) else: data = {} if rt_configs: rt_list = [] for rt, cfg in six.iteritems(rt_configs): rt_data = types.config_to_api_list(cfg) rt_data['roleType'] = rt rt_list.append(rt_data) data[ROLETYPES_CFG_KEY] = rt_list resp = self._get_resource_root().put(path, data=json.dumps(data)) return self._parse_svc_config(resp) def create_role(self, role_name, role_type, host_id): """Create a role :param role_name: Role name :param role_type: Role type :param host_id: ID of the host to assign the role to :return: An ApiRole object """ return roles.create_role(self._get_resource_root(), self.name, role_type, role_name, host_id, self._get_cluster_name()) def delete_role(self, name): """Delete a role by name :param name: Role name :return: The deleted ApiRole object """ return roles.delete_role(self._get_resource_root(), self.name, name, self._get_cluster_name()) def get_roles_by_type(self, role_type, view=None): """Get all roles of a certain type in a service :param role_type: Role type :param view: View to materialize ('full' or 'summary') :return: A list of ApiRole objects. """ return roles.get_roles_by_type(self._get_resource_root(), self.name, role_type, self._get_cluster_name(), view) def get_all_role_config_groups(self): """Get a list of role configuration groups in the service :return: A list of ApiRoleConfigGroup objects. :since: API v3 """ return role_config_groups.get_all_role_config_groups( self._get_resource_root(), self.name, self._get_cluster_name()) def start(self): """Start a service :return: Reference to the submitted command. """ return self._cmd('start') def stop(self): """Stop a service :return: Reference to the submitted command. """ return self._cmd('stop') def restart(self): """Restart a service :return: Reference to the submitted command. """ return self._cmd('restart') def get_health_summary(self): return getattr(self, 'healthSummary', None) def get_health_checks_status(self): return getattr(self, 'healthChecks', None) def start_roles(self, *role_names): """Start a list of roles :param role_names: names of the roles to start. :return: List of submitted commands. """ return self._role_cmd('start', role_names) def create_hbase_root(self): """Create the root directory of an HBase service :return: Reference to the submitted command. """ return self._cmd('hbaseCreateRoot') def create_hdfs_tmp(self): """Create /tmp directory in HDFS Create the /tmp directory in HDFS with appropriate ownership and permissions. :return: Reference to the submitted command :since: API v2 """ return self._cmd('hdfsCreateTmpDir') def refresh(self, *role_names): """Execute the "refresh" command on a set of roles :param role_names: Names of the roles to refresh. :return: Reference to the submitted command. """ return self._role_cmd('refresh', role_names) def decommission(self, *role_names): """Decommission roles in a service :param role_names: Names of the roles to decommission. :return: Reference to the submitted command. """ return self._cmd('decommission', data=role_names) def deploy_client_config(self, *role_names): """Deploys client configuration to the hosts where roles are running :param role_names: Names of the roles to decommission. :return: Reference to the submitted command. """ return self._cmd('deployClientConfig', data=role_names) def format_hdfs(self, *namenodes): """Format NameNode instances of an HDFS service :param namenodes: Name of NameNode instances to format. :return: List of submitted commands. """ return self._role_cmd('hdfsFormat', namenodes) def install_oozie_sharelib(self): """Installs the Oozie ShareLib Oozie must be stopped before running this command. :return: Reference to the submitted command. :since: API v3 """ return self._cmd('installOozieShareLib', api_version=3) def create_oozie_db(self): """Creates the Oozie Database Schema in the configured database :return: Reference to the submitted command. :since: API v2 """ return self._cmd('createOozieDb', api_version=2) def upgrade_oozie_db(self): """Upgrade Oozie Database schema as part of a major version upgrade :return: Reference to the submitted command. :since: API v6 """ return self._cmd('oozieUpgradeDb', api_version=6) def create_hive_metastore_tables(self): """Creates the Hive metastore tables in the configured database Will do nothing if tables already exist. Will not perform an upgrade. :return: Reference to the submitted command. :since: API v3 """ return self._cmd('hiveCreateMetastoreDatabaseTables', api_version=3) def create_hive_warehouse(self): """Creates the Hive warehouse directory in HDFS :return: Reference to the submitted command. :since: API v3 """ return self._cmd('hiveCreateHiveWarehouse') def create_hive_userdir(self): """Creates the Hive user directory in HDFS :return: Reference to the submitted command. :since: API v4 """ return self._cmd('hiveCreateHiveUserDir') def enable_nn_ha(self, active_name, standby_host_id, nameservice, jns, standby_name_dir_list=None, qj_name=None, standby_name=None, active_fc_name=None, standby_fc_name=None, zk_service_name=None, force_init_znode=True, clear_existing_standby_name_dirs=True, clear_existing_jn_edits_dir=True): """Enable High Availability (HA) with Auto-Failover for HDFS NameNode @param active_name: Name of Active NameNode. @param standby_host_id: ID of host where Standby NameNode will be created. @param nameservice: Nameservice to be used while enabling HA. Optional if Active NameNode already has this config set. @param jns: List of Journal Nodes to be created during the command. Each element of the list must be a dict containing the following items: - jns['jnHostId']: ID of the host where the new JournalNode will be created. - jns['jnName']: Name of the JournalNode role (optional) - jns['jnEditsDir']: Edits dir of the JournalNode. Can be omitted if the config is already set at RCG level. @param standby_name_dir_list: List of directories for the new Standby NameNode. If not provided then it will use same dirs as Active NameNode. @param qj_name: Name of the journal located on each JournalNodes' filesystem. This can be optionally provided if the config hasn't been already set for the Active NameNode. If this isn't provided and Active NameNode doesn't also have the config, then nameservice is used by default. @param standby_name: Name of the Standby NameNode role to be created (Optional). @param active_fc_name: Name of the Active Failover Controller role to be created (Optional). @param standby_fc_name: Name of the Standby Failover Controller role to be created (Optional). @param zk_service_name: Name of the ZooKeeper service to use for auto- failover. If HDFS service already depends on a ZooKeeper service then that ZooKeeper service will be used for auto-failover and in that case this parameter can either be omitted or should be the same ZooKeeper service. @param force_init_znode: Indicates if the ZNode should be force initialized if it is already present. Useful while re-enabling High Availability. (Default: TRUE) @param clear_existing_standby_name_dirs: Indicates if the existing name directories for Standby NameNode should be cleared during the workflow. Useful while re-enabling High Availability. (Default: TRUE) @param clear_existing_jn_edits_dir: Indicates if the existing edits directories for the JournalNodes for the specified nameservice should be cleared during the workflow. Useful while re-enabling High Availability. (Default: TRUE) @return: Reference to the submitted command. @since: API v6 """ args = dict( activeNnName=active_name, standbyNnName=standby_name, standbyNnHostId=standby_host_id, standbyNameDirList=standby_name_dir_list, nameservice=nameservice, qjName=qj_name, activeFcName=active_fc_name, standbyFcName=standby_fc_name, zkServiceName=zk_service_name, forceInitZNode=force_init_znode, clearExistingStandbyNameDirs=clear_existing_standby_name_dirs, clearExistingJnEditsDir=clear_existing_jn_edits_dir, jns=jns ) return self._cmd('hdfsEnableNnHa', data=args, api_version=6) def enable_rm_ha(self, new_rm_host_id, zk_service_name=None): """Enable high availability for a YARN ResourceManager. @param new_rm_host_id: id of the host where the second ResourceManager will be added. @param zk_service_name: Name of the ZooKeeper service to use for auto- failover. If YARN service depends on a ZooKeeper service then that ZooKeeper service will be used for auto-failover and in that case this parameter can be omitted. @return: Reference to the submitted command. @since: API v6 """ args = dict( newRmHostId=new_rm_host_id, zkServiceName=zk_service_name ) return self._cmd('enableRmHa', data=args) class ApiServiceSetupInfo(ApiService): _ATTRIBUTES = { 'name': None, 'type': None, 'config': types.Attr(types.ApiConfig), 'roles': types.Attr(roles.ApiRole), } def __init__(self, name=None, type=None, config=None, roles=None): # The BaseApiObject expects a resource_root, which we don't care about resource_root = None # Unfortunately, the json key is called "type". So our input arg # needs to be called "type" as well, despite it being a python keyword. types.BaseApiObject.init(self, None, locals()) def set_config(self, config): """Set the service configuration :param config: A dictionary of config key/value """ if self.config is None: self.config = {} self.config.update(types.config_to_api_list(config)) def add_role_info(self, role_name, role_type, host_id, config=None): """Add a role info The role will be created along with the service setup. :param role_name: Role name :param role_type: Role type :param host_id: The host where the role should run :param config: (Optional) A dictionary of role config values """ if self.roles is None: self.roles = [] api_config_list = (config is not None and types.config_to_api_list(config) or None) self.roles.append({ 'name': role_name, 'type': role_type, 'hostRef': {'hostId': host_id}, 'config': api_config_list}) sahara-8.0.0/sahara/plugins/cdh/client/roles.py0000666000175100017510000001457513245514472021461 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. from sahara.plugins.cdh.client import types ROLES_PATH = "/clusters/%s/services/%s/roles" CM_ROLES_PATH = "/cm/service/roles" def _get_roles_path(cluster_name, service_name): if cluster_name: return ROLES_PATH % (cluster_name, service_name) else: return CM_ROLES_PATH def _get_role_path(cluster_name, service_name, role_name): path = _get_roles_path(cluster_name, service_name) return "%s/%s" % (path, role_name) def create_role(resource_root, service_name, role_type, role_name, host_id, cluster_name="default"): """Create a role :param resource_root: The root Resource object. :param service_name: Service name :param role_type: Role type :param role_name: Role name :param cluster_name: Cluster name :return: An ApiRole object """ apirole = ApiRole(resource_root, role_name, role_type, types.ApiHostRef(resource_root, host_id)) return types.call(resource_root.post, _get_roles_path(cluster_name, service_name), ApiRole, True, data=[apirole])[0] def get_role(resource_root, service_name, name, cluster_name="default"): """Lookup a role by name :param resource_root: The root Resource object. :param service_name: Service name :param name: Role name :param cluster_name: Cluster name :return: An ApiRole object """ return _get_role(resource_root, _get_role_path(cluster_name, service_name, name)) def _get_role(resource_root, path): return types.call(resource_root.get, path, ApiRole) def get_all_roles(resource_root, service_name, cluster_name="default", view=None): """Get all roles :param resource_root: The root Resource object. :param service_name: Service name :param cluster_name: Cluster name :return: A list of ApiRole objects. """ return types.call(resource_root.get, _get_roles_path(cluster_name, service_name), ApiRole, True, params=(dict(view=view) if view else None)) def get_roles_by_type(resource_root, service_name, role_type, cluster_name="default", view=None): """Get all roles of a certain type in a service :param resource_root: The root Resource object. :param service_name: Service name :param role_type: Role type :param cluster_name: Cluster name :return: A list of ApiRole objects. """ roles = get_all_roles(resource_root, service_name, cluster_name, view) return [r for r in roles if r.type == role_type] def delete_role(resource_root, service_name, name, cluster_name="default"): """Delete a role by name :param resource_root: The root Resource object. :param service_name: Service name :param name: Role name :param cluster_name: Cluster name :return: The deleted ApiRole object """ return types.call(resource_root.delete, _get_role_path(cluster_name, service_name, name), ApiRole) class ApiRole(types.BaseApiResource): _ATTRIBUTES = { 'name': None, 'type': None, 'hostRef': types.Attr(types.ApiHostRef), 'roleState': types.ROAttr(), 'healthSummary': types.ROAttr(), 'healthChecks': types.ROAttr(), 'serviceRef': types.ROAttr(types.ApiServiceRef), 'configStale': types.ROAttr(), 'configStalenessStatus': types.ROAttr(), 'haStatus': types.ROAttr(), 'roleUrl': types.ROAttr(), 'commissionState': types.ROAttr(), 'maintenanceMode': types.ROAttr(), 'maintenanceOwners': types.ROAttr(), 'roleConfigGroupRef': types.ROAttr(types.ApiRoleConfigGroupRef), 'zooKeeperServerMode': types.ROAttr(), } def __init__(self, resource_root, name=None, type=None, hostRef=None): types.BaseApiObject.init(self, resource_root, locals()) def __str__(self): return (": %s (cluster: %s; service: %s)" % (self.name, self.serviceRef.clusterName, self.serviceRef.serviceName)) def _path(self): return _get_role_path(self.serviceRef.clusterName, self.serviceRef.serviceName, self.name) def _get_log(self, log): path = "%s/logs/%s" % (self._path(), log) return self._get_resource_root().get(path) def get_commands(self, view=None): """Retrieve a list of running commands for this role :param view: View to materialize ('full' or 'summary') :return: A list of running commands. """ return self._get("commands", types.ApiCommand, True, params=(dict(view=view) if view else None)) def get_config(self, view=None): """Retrieve the role's configuration The 'summary' view contains strings as the dictionary values. The full view contains types.ApiConfig instances as the values. :param view: View to materialize ('full' or 'summary') :return: Dictionary with configuration data. """ return self._get_config("config", view) def update_config(self, config): """Update the role's configuration :param config: Dictionary with configuration to update. :return: Dictionary with updated configuration. """ return self._update_config("config", config) sahara-8.0.0/sahara/plugins/cdh/client/users.py0000666000175100017510000000372313245514472021467 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. from sahara.plugins.cdh.client import types USERS_PATH = "/users" def get_user(resource_root, username): """Look up a user by username. @param resource_root: The root Resource object @param username: Username to look up @return: An ApiUser object """ return types.call(resource_root.get, '%s/%s' % (USERS_PATH, username), ApiUser) def update_user(resource_root, user): """Update a user. Replaces the user's details with those provided. @param resource_root: The root Resource object @param user: An ApiUser object @return: An ApiUser object """ return types.call(resource_root.put, '%s/%s' % (USERS_PATH, user.name), ApiUser, data=user) class ApiUser(types.BaseApiResource): _ATTRIBUTES = { 'name': None, 'password': None, 'roles': None, } def __init__(self, resource_root, name=None, password=None, roles=None): types.BaseApiObject.init(self, resource_root, locals()) sahara-8.0.0/sahara/plugins/cdh/client/role_config_groups.py0000666000175100017510000000764113245514472024216 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. from sahara.plugins.cdh.client import types ROLE_CONFIG_GROUPS_PATH = "/clusters/%s/services/%s/roleConfigGroups" CM_ROLE_CONFIG_GROUPS_PATH = "/cm/service/roleConfigGroups" def _get_role_config_groups_path(cluster_name, service_name): if cluster_name: return ROLE_CONFIG_GROUPS_PATH % (cluster_name, service_name) else: return CM_ROLE_CONFIG_GROUPS_PATH def _get_role_config_group_path(cluster_name, service_name, name): path = _get_role_config_groups_path(cluster_name, service_name) return "%s/%s" % (path, name) def get_all_role_config_groups(resource_root, service_name, cluster_name="default"): """Get all role config groups in the specified service :param resource_root: The root Resource object. :param service_name: Service name. :param cluster_name: Cluster name. :return: A list of ApiRoleConfigGroup objects. :since: API v3 """ return types.call(resource_root.get, _get_role_config_groups_path(cluster_name, service_name), ApiRoleConfigGroup, True, api_version=3) class ApiRoleConfigGroup(types.BaseApiResource): _ATTRIBUTES = { 'name': None, 'displayName': None, 'roleType': None, 'config': types.Attr(types.ApiConfig), 'base': types.ROAttr(), 'serviceRef': types.ROAttr(types.ApiServiceRef), } def __init__(self, resource_root, name=None, displayName=None, roleType=None, config=None): types.BaseApiObject.init(self, resource_root, locals()) def __str__(self): return (": %s (cluster: %s; service: %s)" % (self.name, self.serviceRef.clusterName, self.serviceRef.serviceName)) def _api_version(self): return 3 def _path(self): return _get_role_config_group_path(self.serviceRef.clusterName, self.serviceRef.serviceName, self.name) def get_config(self, view=None): """Retrieve the group's configuration The 'summary' view contains strings as the dictionary values. The full view contains types.ApiConfig instances as the values. :param view: View to materialize ('full' or 'summary'). :return: Dictionary with configuration data. """ path = self._path() + '/config' resp = self._get_resource_root().get( path, params=(dict(view=view) if view else None)) return types.json_to_config(resp, view == 'full') def update_config(self, config): """Update the group's configuration :param config: Dictionary with configuration to update. :return: Dictionary with updated configuration. """ path = self._path() + '/config' resp = self._get_resource_root().put( path, data=types.config_to_json(config)) return types.json_to_config(resp) sahara-8.0.0/sahara/plugins/cdh/client/resource.py0000666000175100017510000001356613245514472022163 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. import posixpath import socket from oslo_log import log as logging from oslo_serialization import jsonutils as json import six from six.moves import urllib from sahara import context from sahara.i18n import _ from sahara.plugins.cdh import exceptions as ex LOG = logging.getLogger(__name__) class Resource(object): """Base Resource Encapsulates a resource, and provides actions to invoke on it. """ def __init__(self, client, relpath=""): """Constructor method :param client: A Client object. :param relpath: The relative path of the resource. """ self._client = client self._path = relpath.strip('/') self.retries = 3 self.retry_sleep = 3 @property def base_url(self): return self._client.base_url def _join_uri(self, relpath): if relpath is None: return self._path return self._path + posixpath.normpath('/' + relpath) def invoke(self, method, relpath=None, params=None, data=None, headers=None): """Invoke an API method :return: Raw body or JSON dictionary (if response content type is JSON). """ path = self._join_uri(relpath) resp = self._client.execute(method, path, params=params, data=data, headers=headers) try: body = resp.read() except Exception as ex: raise ex.CMApiException( _("Command %(method)s %(path)s failed: %(msg)s") % {'method': method, 'path': path, 'msg': six.text_type(ex)}) LOG.debug("{method} got response: {body}".format(method=method, body=body[:32])) # Is the response application/json? if (len(body) != 0 and resp.info().getmaintype() == "application" and resp.info().getsubtype() == "json"): try: json_dict = json.loads(body) return json_dict except Exception: LOG.error('JSON decode error: {body}'.format(body=body)) raise else: return body def get(self, relpath=None, params=None): """Invoke the GET method on a resource :param relpath: Optional. A relative path to this resource's path. :param params: Key-value data. :return: A dictionary of the JSON result. """ for retry in six.moves.xrange(self.retries + 1): if retry: context.sleep(self.retry_sleep) try: return self.invoke("GET", relpath, params) except (socket.error, urllib.error.URLError) as e: if "timed out" in six.text_type(e).lower(): if retry < self.retries: LOG.warning("Timeout issuing GET request for " "{path}. Will retry".format( path=self._join_uri(relpath))) else: LOG.warning("Timeout issuing GET request for " "{path}. No retries left".format( path=self._join_uri(relpath))) else: raise else: raise ex.CMApiException(_("Get retry max time reached.")) def delete(self, relpath=None, params=None): """Invoke the DELETE method on a resource :param relpath: Optional. A relative path to this resource's path. :param params: Key-value data. :return: A dictionary of the JSON result. """ return self.invoke("DELETE", relpath, params) def post(self, relpath=None, params=None, data=None, contenttype=None): """Invoke the POST method on a resource :param relpath: Optional. A relative path to this resource's path. :param params: Key-value data. :param data: Optional. Body of the request. :param contenttype: Optional. :return: A dictionary of the JSON result. """ return self.invoke("POST", relpath, params, data, self._make_headers(contenttype)) def put(self, relpath=None, params=None, data=None, contenttype=None): """Invoke the PUT method on a resource :param relpath: Optional. A relative path to this resource's path. :param params: Key-value data. :param data: Optional. Body of the request. :param contenttype: Optional. :return: A dictionary of the JSON result. """ return self.invoke("PUT", relpath, params, data, self._make_headers(contenttype)) def _make_headers(self, contenttype=None): if contenttype: return {'Content-Type': contenttype} return None sahara-8.0.0/sahara/plugins/cdh/client/clusters.py0000666000175100017510000002141213245514472022165 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. from sahara.i18n import _ from sahara.plugins.cdh.client import services from sahara.plugins.cdh.client import types from sahara.plugins.cdh import exceptions as ex CLUSTERS_PATH = "/clusters" def create_cluster(resource_root, name, version=None, fullVersion=None): """Create a cluster :param resource_root: The root Resource object. :param name: Cluster name :param version: Cluster CDH major version (eg: "CDH4") - The CDH minor version will be assumed to be the latest released version for CDH4, or 5.0 for CDH5. :param fullVersion: Cluster's full CDH version. (eg: "5.1.1") - If specified, 'version' will be ignored. - Since: v6 :return: An ApiCluster object """ if version is None and fullVersion is None: raise ex.CMApiVersionError( _("Either 'version' or 'fullVersion' must be specified")) if fullVersion is not None: api_version = 6 version = None else: api_version = 1 apicluster = ApiCluster(resource_root, name, version, fullVersion) return types.call(resource_root.post, CLUSTERS_PATH, ApiCluster, True, data=[apicluster], api_version=api_version)[0] def get_cluster(resource_root, name): """Lookup a cluster by name :param resource_root: The root Resource object. :param name: Cluster name :return: An ApiCluster object """ return types.call(resource_root.get, "%s/%s" % (CLUSTERS_PATH, name), ApiCluster) def get_all_clusters(resource_root, view=None): """Get all clusters :param resource_root: The root Resource object. :return: A list of ApiCluster objects. """ return types.call(resource_root.get, CLUSTERS_PATH, ApiCluster, True, params=(dict(view=view) if view else None)) class ApiCluster(types.BaseApiResource): _ATTRIBUTES = { 'name': None, 'clusterUrl': None, 'displayName': None, 'version': None, 'fullVersion': None, 'hostsUrl': types.ROAttr(), 'maintenanceMode': types.ROAttr(), 'maintenanceOwners': types.ROAttr(), 'entityStatus': types.ROAttr(), } def __init__(self, resource_root, name=None, version=None, fullVersion=None): types.BaseApiObject.init(self, resource_root, locals()) def _path(self): return "%s/%s" % (CLUSTERS_PATH, self.name) def get_service_types(self): """Get all service types supported by this cluster :return: A list of service types (strings) """ resp = self._get_resource_root().get(self._path() + '/serviceTypes') return resp[types.ApiList.LIST_KEY] def get_commands(self, view=None): """Retrieve a list of running commands for this cluster :param view: View to materialize ('full' or 'summary') :return: A list of running commands. """ return self._get("commands", types.ApiCommand, True, params=(dict(view=view) if view else None)) def create_service(self, name, service_type): """Create a service :param name: Service name :param service_type: Service type :return: An ApiService object """ return services.create_service(self._get_resource_root(), name, service_type, self.name) def get_service(self, name): """Lookup a service by name :param name: Service name :return: An ApiService object """ return services.get_service(self._get_resource_root(), name, self.name) def start(self): """Start all services in a cluster, respecting dependencies :return: Reference to the submitted command. """ return self._cmd('start') def restart(self, restart_only_stale_services=None, redeploy_client_configuration=None, restart_service_names=None): """Restart all services in the cluster. Services are restarted in the appropriate order given their dependencies. :param restart_only_stale_services: Only restart services that have stale configuration and their dependent services. Default is False. :param redeploy_client_configuration: Re-deploy client configuration for all services in the cluster. Default is False. :param restart_service_names: Only restart services that are specified and their dependent services. :return: Reference to the submitted command. """ if self._get_resource_root().version < 6: return self._cmd('restart') args = dict() args['restartOnlyStaleServices'] = restart_only_stale_services args['redeployClientConfiguration'] = redeploy_client_configuration if self._get_resource_root().version >= 11: args['restartServiceNames'] = restart_service_names return self._cmd('restart', data=args, api_version=6) def stop(self): """Stop all services in a cluster, respecting dependencies :return: Reference to the submitted command. """ return self._cmd('stop') def deploy_client_config(self): """Deploys Service client configuration to the hosts on the cluster :return: Reference to the submitted command. :since: API v2 """ return self._cmd('deployClientConfig') def first_run(self): """Prepare and start services in a cluster Perform all the steps needed to prepare each service in a cluster and start the services in order. :return: Reference to the submitted command. :since: API v7 """ return self._cmd('firstRun', None, api_version=7) def remove_host(self, hostId): """Removes the association of the host with the cluster :return: A ApiHostRef of the host that was removed. :since: API v3 """ return self._delete("hosts/" + hostId, types.ApiHostRef, api_version=3) def get_service_health_status(self): """Lookup a service health status by name :return: A dict with cluster health status """ health_dict = {} cl_services = services.get_all_services(self._get_resource_root(), cluster_name=self.name) for curr in cl_services: health_dict[curr.name] = { 'summary': curr.get_health_summary(), 'checks': curr.get_health_checks_status()} return health_dict def configure_for_kerberos(self, datanode_transceiver_port=None, datanode_web_port=None): """Command to configure the cluster to use Kerberos for authentication. This command will configure all relevant services on a cluster for Kerberos usage. This command will trigger a GenerateCredentials command to create Kerberos keytabs for all roles in the cluster. :param datanode_transceiver_port: The HDFS DataNode transceiver port to use. This will be applied to all DataNode role configuration groups. If not specified, this will default to 1004. :param datanode_web_port: The HDFS DataNode web port to use. This will be applied to all DataNode role configuration groups. If not specified, this will default to 1006. :return: Reference to the submitted command. :since: API v11 """ args = dict() if datanode_transceiver_port: args['datanodeTransceiverPort'] = datanode_transceiver_port if datanode_web_port: args['datanodeWebPort'] = datanode_web_port return self._cmd('configureForKerberos', data=args, api_version=11) sahara-8.0.0/sahara/plugins/cdh/client/__init__.py0000666000175100017510000000000013245514472022046 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/client/hosts.py0000666000175100017510000000567313245514472021474 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. import datetime from sahara.plugins.cdh.client import types HOSTS_PATH = "/hosts" def get_all_hosts(resource_root, view=None): """Get all hosts :param resource_root: The root Resource object. :return: A list of ApiHost objects. """ return types.call(resource_root.get, HOSTS_PATH, ApiHost, True, params=(dict(view=view) if view else None)) def delete_host(resource_root, host_id): """Delete a host by id :param resource_root: The root Resource object. :param host_id: Host id :return: The deleted ApiHost object """ return types.call(resource_root.delete, "%s/%s" % (HOSTS_PATH, host_id), ApiHost) class ApiHost(types.BaseApiResource): _ATTRIBUTES = { 'hostId': None, 'hostname': None, 'ipAddress': None, 'rackId': None, 'status': types.ROAttr(), 'lastHeartbeat': types.ROAttr(datetime.datetime), 'roleRefs': types.ROAttr(types.ApiRoleRef), 'healthSummary': types.ROAttr(), 'healthChecks': types.ROAttr(), 'hostUrl': types.ROAttr(), 'commissionState': types.ROAttr(), 'maintenanceMode': types.ROAttr(), 'maintenanceOwners': types.ROAttr(), 'numCores': types.ROAttr(), 'totalPhysMemBytes': types.ROAttr(), } def __init__(self, resource_root, hostId=None, hostname=None, ipAddress=None, rackId=None): types.BaseApiObject.init(self, resource_root, locals()) def __str__(self): return ": %s (%s)" % (self.hostId, self.ipAddress) def _path(self): return HOSTS_PATH + '/' + self.hostId def put_host(self): """Update this resource note (mionkin):Currently, according to Cloudera docs, only updating the rackId is supported. All other fields of the host will be ignored. :return: The updated object. """ return self._put('', ApiHost, data=self) sahara-8.0.0/sahara/plugins/cdh/client/http_client.py0000666000175100017510000001165413245514472022645 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. import posixpath from oslo_log import log as logging from oslo_serialization import jsonutils as json import six from six.moves import urllib from sahara.plugins.cdh import exceptions as ex LOG = logging.getLogger(__name__) class HttpClient(object): """Basic HTTP client tailored for rest APIs.""" def __init__(self, base_url, exc_class=ex.CMApiException): """Init Method :param base_url: The base url to the API. :param exc_class: An exception class to handle non-200 results. Creates an HTTP(S) client to connect to the Cloudera Manager API. """ self._base_url = base_url.rstrip('/') self._exc_class = exc_class self._headers = {} # Make a basic auth handler that does nothing. Set credentials later. self._passmgr = urllib.request.HTTPPasswordMgrWithDefaultRealm() authhandler = urllib.request.HTTPBasicAuthHandler(self._passmgr) # Make a cookie processor cookiejar = six.moves.http_cookiejar.CookieJar() self._opener = urllib.request.build_opener( urllib.request.HTTPErrorProcessor(), urllib.request.HTTPCookieProcessor(cookiejar), authhandler) def set_basic_auth(self, username, password, realm): """Set up basic auth for the client :param username: Login name. :param password: Login password. :param realm: The authentication realm. :return: The current object """ self._passmgr.add_password(realm, self._base_url, username, password) return self def set_headers(self, headers): """Add headers to the request :param headers: A dictionary with the key value pairs for the headers :return: The current object """ self._headers = headers return self @property def base_url(self): return self._base_url def _get_headers(self, headers): res = self._headers.copy() if headers: res.update(headers) return res def execute(self, http_method, path, params=None, data=None, headers=None): """Submit an HTTP request :param http_method: GET, POST, PUT, DELETE :param path: The path of the resource. :param params: Key-value parameter data. :param data: The data to attach to the body of the request. :param headers: The headers to set for this request. :return: The result of urllib.request.urlopen() """ # Prepare URL and params url = self._make_url(path, params) if http_method in ("GET", "DELETE"): if data is not None: LOG.warning("{method} method does not pass any data. " "Path {path}".format(method=http_method, path=path)) data = None # Setup the request request = urllib.request.Request(url, data) # Hack/workaround because urllib2 only does GET and POST request.get_method = lambda: http_method headers = self._get_headers(headers) for k, v in headers.items(): request.add_header(k, v) # Call it LOG.debug("Method: {method}, URL: {url}".format(method=http_method, url=url)) try: return self._opener.open(request) except urllib.error.HTTPError as ex: message = six.text_type(ex) try: json_body = json.loads(message) message = json_body['message'] except (ValueError, KeyError): pass # Ignore json parsing error raise self._exc_class(message) def _make_url(self, path, params): res = self._base_url if path: res += posixpath.normpath('/' + path.lstrip('/')) if params: param_str = urllib.parse.urlencode(params, True) res += '?' + param_str return res sahara-8.0.0/sahara/plugins/cdh/client/cms.py0000666000175100017510000000576513245514472021120 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. from sahara.plugins.cdh.client.services import ApiService from sahara.plugins.cdh.client import types class ClouderaManager(types.BaseApiResource): """The Cloudera Manager instance Provides access to CM configuration and services. """ def __init__(self, resource_root): types.BaseApiObject.init(self, resource_root) def _path(self): return '/cm' def create_mgmt_service(self, service_setup_info): """Setup the Cloudera Management Service :param service_setup_info: ApiServiceSetupInfo object. :return: The management service instance. """ return self._put("service", ApiService, data=service_setup_info) def get_service(self): """Return the Cloudera Management Services instance :return: An ApiService instance. """ return self._get("service", ApiService) def hosts_start_roles(self, host_names): """Start all the roles on the specified hosts :param host_names: List of names of hosts on which to start all roles. :return: Information about the submitted command. :since: API v2 """ return self._cmd('hostsStartRoles', data=host_names) def update_config(self, config): """Update the CM configuration. :param config: Dictionary with configuration to update. :return: Dictionary with updated configuration. """ return self._update_config("config", config) def import_admin_credentials(self, username, password): """Imports the KDC Account Manager credentials needed by Cloudera Manager to create kerberos principals needed by CDH services. :param username Username of the Account Manager. Full name including the Kerberos realm must be specified. :param password Password for the Account Manager. :return: Information about the submitted command. :since: API v7 """ return self._cmd('importAdminCredentials', params=dict( username=username, password=password)) sahara-8.0.0/sahara/plugins/cdh/client/types.py0000666000175100017510000005704213245514472021475 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. import copy import datetime import time from oslo_serialization import jsonutils as json from oslo_utils import reflection import six from sahara import context from sahara.i18n import _ from sahara.plugins.cdh import exceptions as ex class Attr(object): """Base Attribute Encapsulates information about an attribute in the JSON encoding of the object. It identifies properties of the attribute such as whether it's read-only, its type, etc. """ DATE_FMT = "%Y-%m-%dT%H:%M:%S.%fZ" def __init__(self, atype=None, rw=True, is_api_list=False): self._atype = atype self._is_api_list = is_api_list self.rw = rw def to_json(self, value, preserve_ro): """Returns the JSON encoding of the given attribute value If the value has a 'to_json_dict' object, that method is called. Otherwise, the following values are returned for each input type: - datetime.datetime: string with the API representation of a date. - dictionary: if 'atype' is ApiConfig, a list of ApiConfig objects. - python list: python list (or ApiList) with JSON encoding of items - the raw value otherwise """ if hasattr(value, 'to_json_dict'): return value.to_json_dict(preserve_ro) elif isinstance(value, dict) and self._atype == ApiConfig: return config_to_api_list(value) elif isinstance(value, datetime.datetime): return value.strftime(self.DATE_FMT) elif isinstance(value, list) or isinstance(value, tuple): if self._is_api_list: return ApiList(value).to_json_dict() else: return [self.to_json(x, preserve_ro) for x in value] else: return value def from_json(self, resource_root, data): """Parses the given JSON value into an appropriate python object This means: - a datetime.datetime if 'atype' is datetime.datetime - a converted config dictionary or config list if 'atype' is ApiConfig - if the attr is an API list, an ApiList with instances of 'atype' - an instance of 'atype' if it has a 'from_json_dict' method - a python list with decoded versions of the member objects if the input is a python list. - the raw value otherwise """ if data is None: return None if self._atype == datetime.datetime: return datetime.datetime.strptime(data, self.DATE_FMT) elif self._atype == ApiConfig: # ApiConfig is special. We want a python dictionary for summary # views, but an ApiList for full views. Try to detect each case # from the JSON data. if not data['items']: return {} first = data['items'][0] return json_to_config(data, len(first) == 2) elif self._is_api_list: return ApiList.from_json_dict(data, resource_root, self._atype) elif isinstance(data, list): return [self.from_json(resource_root, x) for x in data] elif hasattr(self._atype, 'from_json_dict'): return self._atype.from_json_dict(data, resource_root) else: return data class ROAttr(Attr): """Subclass that just defines the attribute as read-only.""" def __init__(self, atype=None, is_api_list=False): Attr.__init__(self, atype=atype, rw=False, is_api_list=is_api_list) def check_api_version(resource_root, min_version): """Check API version Checks if the resource_root's API version it at least the given minimum version. """ if resource_root.version < min_version: raise ex.CMApiVersionError( _("API version %(minv)s is required but %(acv)s is in use.") % {'minv': min_version, 'acv': resource_root.version}) def call(method, path, ret_type, ret_is_list=False, data=None, params=None, api_version=1): """Call a resource method Generic function for calling a resource method and automatically dealing with serialization of parameters and deserialization of return values. :param method: method to call (must be bound to a resource; e.g., "resource_root.get"). :param path: the full path of the API method to call. :param ret_type: return type of the call. :param ret_is_list: whether the return type is an ApiList. :param data: Optional data to send as payload to the call. :param params: Optional query parameters for the call. :param api_version: minimum API version for the call. """ check_api_version(method.im_self, api_version) if data is not None: data = json.dumps(Attr(is_api_list=True).to_json(data, False)) ret = method(path, data=data, params=params) else: ret = method(path, params=params) if ret_type is None: return elif ret_is_list: return ApiList.from_json_dict(ret, method.im_self, ret_type) elif isinstance(ret, list): return [ret_type.from_json_dict(x, method.im_self) for x in ret] else: return ret_type.from_json_dict(ret, method.im_self) class BaseApiObject(object): """The BaseApiObject helps with (de)serialization from/to JSON The derived class has two ways of defining custom attributes: - Overwriting the '_ATTRIBUTES' field with the attribute dictionary - Override the _get_attributes() method, in case static initialization of the above field is not possible. It's recommended that the _get_attributes() implementation do caching to avoid computing the dictionary on every invocation. The derived class's constructor must call the base class's init() static method. All constructor arguments (aside from self and resource_root) must be keywords arguments with default values (typically None), or from_json_dict() will not work. """ _ATTRIBUTES = {} _WHITELIST = ('_resource_root', '_attributes') @classmethod def _get_attributes(cls): """Get an attribute dictionary Returns a map of property names to attr instances (or None for default attribute behavior) describing the properties of the object. By default, this method will return the class's _ATTRIBUTES field. Classes can override this method to do custom initialization of the attributes when needed. """ return cls._ATTRIBUTES @staticmethod def init(obj, resource_root, attrs=None): """Wraper of real constructor Wraper around the real constructor to avoid issues with the 'self' argument. Call like this, from a subclass's constructor: - BaseApiObject.init(self, locals()) """ # This works around http://bugs.python.org/issue2646 # We use unicode strings as keys in kwargs. str_attrs = {} if attrs: for k, v in six.iteritems(attrs): if k not in ('self', 'resource_root'): str_attrs[k] = v BaseApiObject.__init__(obj, resource_root, **str_attrs) def __init__(self, resource_root, **attrs): """Init method Initializes internal state and sets all known writable properties of the object to None. Then initializes the properties given in the provided attributes dictionary. :param resource_root: API resource object. :param attrs: optional dictionary of attributes to set. This should only contain r/w attributes. """ self._resource_root = resource_root for name, attr in six.iteritems(self._get_attributes()): object.__setattr__(self, name, None) if attrs: self._set_attrs(attrs, from_json=False) def _set_attrs(self, attrs, allow_ro=False, from_json=True): """Set attributes from dictionary Sets all the attributes in the dictionary. Optionally, allows setting read-only attributes (e.g. when deserializing from JSON) and skipping JSON deserialization of values. """ for k, v in six.iteritems(attrs): attr = self._check_attr(k, allow_ro) if attr and from_json: v = attr.from_json(self._get_resource_root(), v) object.__setattr__(self, k, v) def __setattr__(self, name, val): if name not in BaseApiObject._WHITELIST: self._check_attr(name, False) object.__setattr__(self, name, val) def _check_attr(self, name, allow_ro): cls_name = reflection.get_class_name(self, fully_qualified=False) if name not in self._get_attributes(): raise ex.CMApiAttributeError( _('Invalid property %(attname)s for class %(classname)s.') % {'attname': name, 'classname': cls_name}) attr = self._get_attributes()[name] if not allow_ro and attr and not attr.rw: raise ex.CMApiAttributeError( _('Attribute %(attname)s of class %(classname)s ' 'is read only.') % {'attname': name, 'classname': cls_name}) return attr def _get_resource_root(self): return self._resource_root def _update(self, api_obj): """Copy state from api_obj to this object.""" if not isinstance(self, api_obj.__class__): raise ex.CMApiValueError( _("Class %(class1)s does not derive from %(class2)s; " "cannot update attributes.") % {'class1': self.__class__, 'class2': api_obj.__class__}) for name in self._get_attributes().keys(): try: val = getattr(api_obj, name) setattr(self, name, val) except AttributeError: pass def to_json_dict(self, preserve_ro=False): dic = {} for name, attr in six.iteritems(self._get_attributes()): if not preserve_ro and attr and not attr.rw: continue try: value = getattr(self, name) if value is not None: if attr: dic[name] = attr.to_json(value, preserve_ro) else: dic[name] = value except AttributeError: pass return dic def __str__(self): """Give a printable format of an attribute Default implementation of __str__. Uses the type name and the first attribute retrieved from the attribute map to create the string. """ cls_name = reflection.get_class_name(self, fully_qualified=False) name = list(self._get_attributes().keys())[0] value = getattr(self, name, None) return "<%s>: %s = %s" % (cls_name, name, value) @classmethod def from_json_dict(cls, dic, resource_root): obj = cls(resource_root) obj._set_attrs(dic, allow_ro=True) return obj class BaseApiResource(BaseApiObject): """Base ApiResource A specialization of BaseApiObject that provides some utility methods for resources. This class allows easier serialization / deserialization of parameters and return values. """ def _api_version(self): """Get API version Returns the minimum API version for this resource. Defaults to 1. """ return 1 def _path(self): """Get resource path Returns the path to the resource. e.g., for a service 'foo' in cluster 'bar', this should return '/clusters/bar/services/foo'. """ raise NotImplementedError def _require_min_api_version(self, version): """Check minimum version requirement Raise an exception if the version of the api is less than the given version. :param version: The minimum required version. """ actual_version = self._get_resource_root().version version = max(version, self._api_version()) if actual_version < version: raise ex.CMApiVersionError( _("API version %(minv)s is required but %(acv)s is in use.") % {'minv': version, 'acv': actual_version}) def _cmd(self, command, data=None, params=None, api_version=1): """Invoke a command on the resource Invokes a command on the resource. Commands are expected to be under the "commands/" sub-resource. """ return self._post("commands/" + command, ApiCommand, data=data, params=params, api_version=api_version) def _get_config(self, rel_path, view, api_version=1): """Get resource configurations Retrieves an ApiConfig list from the given relative path. """ self._require_min_api_version(api_version) params = dict(view=view) if view else None resp = self._get_resource_root().get(self._path() + '/' + rel_path, params=params) return json_to_config(resp, view == 'full') def _update_config(self, rel_path, config, api_version=1): self._require_min_api_version(api_version) resp = self._get_resource_root().put(self._path() + '/' + rel_path, data=config_to_json(config)) return json_to_config(resp, False) def _delete(self, rel_path, ret_type, ret_is_list=False, params=None, api_version=1): return self._call('delete', rel_path, ret_type, ret_is_list, None, params, api_version) def _get(self, rel_path, ret_type, ret_is_list=False, params=None, api_version=1): return self._call('get', rel_path, ret_type, ret_is_list, None, params, api_version) def _post(self, rel_path, ret_type, ret_is_list=False, data=None, params=None, api_version=1): return self._call('post', rel_path, ret_type, ret_is_list, data, params, api_version) def _put(self, rel_path, ret_type, ret_is_list=False, data=None, params=None, api_version=1): return self._call('put', rel_path, ret_type, ret_is_list, data, params, api_version) def _call(self, method, rel_path, ret_type, ret_is_list=False, data=None, params=None, api_version=1): path = self._path() if rel_path: path += '/' + rel_path return call(getattr(self._get_resource_root(), method), path, ret_type, ret_is_list, data, params, api_version) class ApiList(BaseApiObject): """A list of some api object""" LIST_KEY = "items" def __init__(self, objects, resource_root=None, **attrs): BaseApiObject.__init__(self, resource_root, **attrs) # Bypass checks in BaseApiObject.__setattr__ object.__setattr__(self, 'objects', objects) def __str__(self): return ("(%d): [%s]" % (len(self.objects), ", ".join([str(item) for item in self.objects]))) def to_json_dict(self, preserve_ro=False): ret = BaseApiObject.to_json_dict(self, preserve_ro) attr = Attr() ret[ApiList.LIST_KEY] = [attr.to_json(x, preserve_ro) for x in self.objects] return ret def __len__(self): return self.objects.__len__() def __iter__(self): return self.objects.__iter__() def __getitem__(self, i): return self.objects.__getitem__(i) def __getslice__(self, i, j): return self.objects.__getslice__(i, j) @classmethod def from_json_dict(cls, dic, resource_root, member_cls=None): if not member_cls: member_cls = cls._MEMBER_CLASS attr = Attr(atype=member_cls) items = [] if ApiList.LIST_KEY in dic: items = [attr.from_json(resource_root, x) for x in dic[ApiList.LIST_KEY]] ret = cls(items) # If the class declares custom attributes, populate them based on the # input dict. The check avoids extra overhead for the common case, # where we just have a plain list. _set_attrs() also does not # understand the "items" attribute, so it can't be in the input data. if cls._ATTRIBUTES: if ApiList.LIST_KEY in dic: dic = copy.copy(dic) del dic[ApiList.LIST_KEY] ret._set_attrs(dic, allow_ro=True) return ret class ApiHostRef(BaseApiObject): _ATTRIBUTES = { 'hostId': None, } def __init__(self, resource_root, hostId=None): BaseApiObject.init(self, resource_root, locals()) def __str__(self): return ": %s" % (self.hostId) class ApiServiceRef(BaseApiObject): _ATTRIBUTES = { 'clusterName': None, 'serviceName': None, 'peerName': None, } def __init__(self, resource_root, serviceName=None, clusterName=None, peerName=None): BaseApiObject.init(self, resource_root, locals()) class ApiClusterRef(BaseApiObject): _ATTRIBUTES = { 'clusterName': None, } def __init__(self, resource_root, clusterName=None): BaseApiObject.init(self, resource_root, locals()) class ApiRoleRef(BaseApiObject): _ATTRIBUTES = { 'clusterName': None, 'serviceName': None, 'roleName': None, } def __init__(self, resource_root, serviceName=None, roleName=None, clusterName=None): BaseApiObject.init(self, resource_root, locals()) class ApiRoleConfigGroupRef(BaseApiObject): _ATTRIBUTES = { 'roleConfigGroupName': None, } def __init__(self, resource_root, roleConfigGroupName=None): BaseApiObject.init(self, resource_root, locals()) class ApiCommand(BaseApiObject): SYNCHRONOUS_COMMAND_ID = -1 @classmethod def _get_attributes(cls): if not ('_ATTRIBUTES' in cls.__dict__): cls._ATTRIBUTES = { 'id': ROAttr(), 'name': ROAttr(), 'startTime': ROAttr(datetime.datetime), 'endTime': ROAttr(datetime.datetime), 'active': ROAttr(), 'success': ROAttr(), 'resultMessage': ROAttr(), 'clusterRef': ROAttr(ApiClusterRef), 'serviceRef': ROAttr(ApiServiceRef), 'roleRef': ROAttr(ApiRoleRef), 'hostRef': ROAttr(ApiHostRef), 'children': ROAttr(ApiCommand, is_api_list=True), 'parent': ROAttr(ApiCommand), 'resultDataUrl': ROAttr(), 'canRetry': ROAttr(), } return cls._ATTRIBUTES def __str__(self): return (": '%s' (id: %s; active: %s; success: %s)" % (self.name, self.id, self.active, self.success)) def _path(self): return '/commands/%d' % self.id def fetch(self): """Retrieve updated data about the command from the server :return: A new ApiCommand object. """ if self.id == ApiCommand.SYNCHRONOUS_COMMAND_ID: return self resp = self._get_resource_root().get(self._path()) return ApiCommand.from_json_dict(resp, self._get_resource_root()) def wait(self, timeout=None): """Wait for command to finish :param timeout: (Optional) Max amount of time (in seconds) to wait. Wait forever by default. :return: The final ApiCommand object, containing the last known state. The command may still be running in case of timeout. """ if self.id == ApiCommand.SYNCHRONOUS_COMMAND_ID: return self SLEEP_SEC = 5 if timeout is None: deadline = None else: deadline = time.time() + timeout while True: cmd = self.fetch() if not cmd.active: return cmd if deadline is not None: now = time.time() if deadline < now: return cmd else: context.sleep(min(SLEEP_SEC, deadline - now)) else: context.sleep(SLEEP_SEC) def abort(self): """Abort a running command :return: A new ApiCommand object with the updated information. """ if self.id == ApiCommand.SYNCHRONOUS_COMMAND_ID: return self path = self._path() + '/abort' resp = self._get_resource_root().post(path) return ApiCommand.from_json_dict(resp, self._get_resource_root()) class ApiBulkCommandList(ApiList): _ATTRIBUTES = { 'errors': ROAttr(), } _MEMBER_CLASS = ApiCommand # # Configuration helpers. # class ApiConfig(BaseApiObject): _ATTRIBUTES = { 'name': None, 'value': None, 'required': ROAttr(), 'default': ROAttr(), 'displayName': ROAttr(), 'description': ROAttr(), 'relatedName': ROAttr(), 'validationState': ROAttr(), 'validationMessage': ROAttr(), } def __init__(self, resource_root, name=None, value=None): BaseApiObject.init(self, resource_root, locals()) def __str__(self): return ": %s = %s" % (self.name, self.value) def config_to_api_list(dic): """Convert a python dictionary into an ApiConfig list Converts a python dictionary into a list containing the proper ApiConfig encoding for configuration data. :param dic: Key-value pairs to convert. :return: JSON dictionary of an ApiConfig list (*not* an ApiList). """ config = [] for k, v in six.iteritems(dic): config.append({'name': k, 'value': v}) return {ApiList.LIST_KEY: config} def config_to_json(dic): """Converts a python dictionary into a JSON payload The payload matches the expected "apiConfig list" type used to update configuration parameters using the API. :param dic: Key-value pairs to convert. :return: String with the JSON-encoded data. """ return json.dumps(config_to_api_list(dic)) def json_to_config(dic, full=False): """Converts a JSON-decoded config dictionary to a python dictionary When materializing the full view, the values in the dictionary will be instances of ApiConfig, instead of strings. :param dic: JSON-decoded config dictionary. :param full: Whether to materialize the full view of the config data. :return: Python dictionary with config data. """ config = {} for entry in dic['items']: k = entry['name'] if full: config[k] = ApiConfig.from_json_dict(entry, None) else: config[k] = entry.get('value') return config sahara-8.0.0/sahara/plugins/cdh/client/api_client.py0000666000175100017510000001163513245514472022436 0ustar zuulzuul00000000000000# Copyright (c) 2014 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # # The contents of this file are mainly copied from cm_api sources, # released by Cloudera. Codes not used by Sahara CDH plugin are removed. # You can find the original codes at # # https://github.com/cloudera/cm_api/tree/master/python/src/cm_api # # To satisfy the pep8 and python3 tests, we did some changes to the codes. # We also change some importings to use Sahara inherited classes. from sahara.plugins.cdh.client import clusters from sahara.plugins.cdh.client import cms from sahara.plugins.cdh.client import hosts from sahara.plugins.cdh.client import http_client from sahara.plugins.cdh.client import resource from sahara.plugins.cdh.client import users API_AUTH_REALM = "Cloudera Manager" API_CURRENT_VERSION = 8 class ApiResource(resource.Resource): """Top-level API Resource Resource object that provides methods for managing the top-level API resources. """ def __init__(self, server_host, server_port=None, username="admin", password="admin", use_tls=False, version=API_CURRENT_VERSION): """Creates a Resource object that provides API endpoints. :param server_host: The hostname of the Cloudera Manager server. :param server_port: The port of the server. Defaults to 7180 (http) or 7183 (https). :param username: Login name. :param password: Login password. :param use_tls: Whether to use tls (https). :param version: API version. :return: Resource object referring to the root. """ self._version = version protocol = "https" if use_tls else "http" if server_port is None: server_port = 7183 if use_tls else 7180 base_url = ("%s://%s:%s/api/v%s" % (protocol, server_host, server_port, version)) client = http_client.HttpClient(base_url) client.set_basic_auth(username, password, API_AUTH_REALM) client.set_headers({"Content-Type": "application/json"}) resource.Resource.__init__(self, client) @property def version(self): """Returns the API version (integer) being used.""" return self._version def get_cloudera_manager(self): """Returns a Cloudera Manager object.""" return cms.ClouderaManager(self) def create_cluster(self, name, version=None, fullVersion=None): """Create a new cluster :param name: Cluster name. :param version: Cluster major CDH version, e.g. 'CDH5'. Ignored if fullVersion is specified. :param fullVersion: Complete CDH version, e.g. '5.1.2'. Overrides major version if both specified. :return: The created cluster. """ return clusters.create_cluster(self, name, version, fullVersion) def get_all_clusters(self, view=None): """Retrieve a list of all clusters :param view: View to materialize ('full' or 'summary'). :return: A list of ApiCluster objects. """ return clusters.get_all_clusters(self, view) def get_cluster(self, name): """Look up a cluster by name :param name: Cluster name. :return: An ApiCluster object. """ return clusters.get_cluster(self, name) def delete_host(self, host_id): """Delete a host by id :param host_id: Host id :return: The deleted ApiHost object """ return hosts.delete_host(self, host_id) def get_all_hosts(self, view=None): """Get all hosts :param view: View to materialize ('full' or 'summary'). :return: A list of ApiHost objects. """ return hosts.get_all_hosts(self, view) def get_user(self, username): """Look up a user by username. @param username: Username to look up @return: An ApiUser object """ return users.get_user(self, username) def update_user(self, user): """Update a user detail profile. @param user: An ApiUser object @return: An ApiUser object """ return users.update_user(self, user) def get_service_health_status(self, cluster): """Get clusters service health status :param cluster: Cluster name. :return: A dict with cluster health status """ cluster = clusters.get_cluster(self, cluster) return cluster.get_service_health_status() sahara-8.0.0/sahara/plugins/cdh/abstractversionhandler.py0000666000175100017510000001653013245514472023617 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import six from sahara import conductor from sahara import context from sahara.plugins.cdh import db_helper as dh from sahara.plugins.cdh import health from sahara.plugins import kerberos @six.add_metaclass(abc.ABCMeta) class AbstractVersionHandler(object): @abc.abstractmethod def get_node_processes(self): return @abc.abstractmethod def get_plugin_configs(self): return @abc.abstractmethod def configure_cluster(self, cluster): return @abc.abstractmethod def start_cluster(self, cluster): return @abc.abstractmethod def validate(self, cluster): return @abc.abstractmethod def scale_cluster(self, cluster, instances): return @abc.abstractmethod def decommission_nodes(self, cluster, instances): return @abc.abstractmethod def validate_scaling(self, cluster, existing, additional): return @abc.abstractmethod def get_edp_engine(self, cluster, job_type): return @abc.abstractmethod def get_edp_job_types(self): return [] @abc.abstractmethod def get_edp_config_hints(self, job_type): return {} @abc.abstractmethod def get_open_ports(self, node_group): return def on_terminate_cluster(self, cluster): dh.delete_passwords_from_keymanager(cluster) @abc.abstractmethod def get_image_arguments(self): return NotImplemented @abc.abstractmethod def pack_image(self, hadoop_version, remote, test_only=False, image_arguments=None): pass @abc.abstractmethod def validate_images(self, cluster, test_only=False, image_arguments=None): pass class BaseVersionHandler(AbstractVersionHandler): def __init__(self): # Need to be specified in subclass self.config_helper = None # config helper self.cloudera_utils = None # ClouderaUtils self.deploy = None # to deploy self.edp_engine = None self.plugin_utils = None # PluginUtils self.validation = None # to validate def get_plugin_configs(self): result = self.config_helper.get_plugin_configs() result.extend(kerberos.get_config_list()) return result def get_node_processes(self): return { "CLOUDERA": ['CLOUDERA_MANAGER'], "HDFS": ['HDFS_NAMENODE', 'HDFS_DATANODE', 'HDFS_SECONDARYNAMENODE', 'HDFS_JOURNALNODE'], "YARN": ['YARN_RESOURCEMANAGER', 'YARN_NODEMANAGER', 'YARN_JOBHISTORY', 'YARN_STANDBYRM'], "OOZIE": ['OOZIE_SERVER'], "HIVE": ['HIVE_SERVER2', 'HIVE_METASTORE', 'HIVE_WEBHCAT'], "HUE": ['HUE_SERVER'], "SPARK_ON_YARN": ['SPARK_YARN_HISTORY_SERVER'], "ZOOKEEPER": ['ZOOKEEPER_SERVER'], "HBASE": ['HBASE_MASTER', 'HBASE_REGIONSERVER'], "FLUME": ['FLUME_AGENT'], "IMPALA": ['IMPALA_CATALOGSERVER', 'IMPALA_STATESTORE', 'IMPALAD'], "KS_INDEXER": ['KEY_VALUE_STORE_INDEXER'], "SOLR": ['SOLR_SERVER'], "SQOOP": ['SQOOP_SERVER'], "SENTRY": ['SENTRY_SERVER'], "KMS": ['KMS'], "KAFKA": ['KAFKA_BROKER'], "YARN_GATEWAY": [], "RESOURCEMANAGER": [], "NODEMANAGER": [], "JOBHISTORY": [], "HDFS_GATEWAY": [], 'DATANODE': [], 'NAMENODE': [], 'SECONDARYNAMENODE': [], 'JOURNALNODE': [], 'REGIONSERVER': [], 'MASTER': [], 'HIVEMETASTORE': [], 'HIVESERVER': [], 'WEBCAT': [], 'CATALOGSERVER': [], 'STATESTORE': [], 'IMPALAD': [], 'Kerberos': [], } def validate(self, cluster): self.validation.validate_cluster_creating(cluster) def configure_cluster(self, cluster): self.deploy.configure_cluster(cluster) conductor.API.cluster_update( context.ctx(), cluster, { 'info': self.cloudera_utils.get_cloudera_manager_info(cluster)}) def start_cluster(self, cluster): self.deploy.start_cluster(cluster) self._set_cluster_info(cluster) def decommission_nodes(self, cluster, instances): self.deploy.decommission_cluster(cluster, instances) def validate_scaling(self, cluster, existing, additional): self.validation.validate_existing_ng_scaling(cluster, existing) self.validation.validate_additional_ng_scaling(cluster, additional) def scale_cluster(self, cluster, instances): self.deploy.scale_cluster(cluster, instances) def _set_cluster_info(self, cluster): info = self.cloudera_utils.get_cloudera_manager_info(cluster) hue = self.cloudera_utils.pu.get_hue(cluster) if hue: info['Hue Dashboard'] = { 'Web UI': 'http://%s:8888' % hue.get_ip_or_dns_name() } ctx = context.ctx() conductor.API.cluster_update(ctx, cluster, {'info': info}) def get_edp_engine(self, cluster, job_type): oozie_type = self.edp_engine.EdpOozieEngine.get_supported_job_types() spark_type = self.edp_engine.EdpSparkEngine.get_supported_job_types() if job_type in oozie_type: return self.edp_engine.EdpOozieEngine(cluster) if job_type in spark_type: return self.edp_engine.EdpSparkEngine(cluster) return None def get_edp_job_types(self): return (self.edp_engine.EdpOozieEngine.get_supported_job_types() + self.edp_engine.EdpSparkEngine.get_supported_job_types()) def get_edp_config_hints(self, job_type): return self.edp_engine.EdpOozieEngine.get_possible_job_config(job_type) def get_open_ports(self, node_group): return self.deploy.get_open_ports(node_group) def recommend_configs(self, cluster, scaling): self.plugin_utils.recommend_configs( cluster, self.get_plugin_configs(), scaling) def get_health_checks(self, cluster): return health.get_health_checks(cluster, self.cloudera_utils) def get_image_arguments(self): if hasattr(self, 'images'): return self.images.get_image_arguments() else: return NotImplemented def pack_image(self, hadoop_version, remote, test_only=False, image_arguments=None): if hasattr(self, 'images'): self.images.pack_image( remote, test_only=test_only, image_arguments=image_arguments) def validate_images(self, cluster, test_only=False, image_arguments=None): if hasattr(self, 'images'): self.images.validate_images( cluster, test_only=test_only, image_arguments=image_arguments) sahara-8.0.0/sahara/plugins/cdh/db_resources/0000775000175100017510000000000013245515026021142 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/cdh/db_resources/create_hive_db.sql0000666000175100017510000000023013245514472024607 0ustar zuulzuul00000000000000CREATE ROLE hive LOGIN PASSWORD '%s'; CREATE DATABASE metastore OWNER hive encoding 'UTF8'; GRANT ALL PRIVILEGES ON DATABASE metastore TO hive; COMMIT; sahara-8.0.0/sahara/plugins/cdh/db_resources/create_sentry_db.sql0000666000175100017510000000023013245514472025200 0ustar zuulzuul00000000000000CREATE ROLE sentry LOGIN PASSWORD '%s'; CREATE DATABASE sentry OWNER sentry encoding 'UTF8'; GRANT ALL PRIVILEGES ON DATABASE sentry TO sentry; COMMIT; sahara-8.0.0/sahara/plugins/cdh/health.py0000666000175100017510000001201513245514472020307 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools from oslo_log import log as logging import six from sahara.i18n import _ from sahara.service.health import health_check_base LOG = logging.getLogger(__name__) class HealthStatusProvider(object): def __init__(self, cluster, cloudera_tools): self.cluster = cluster self.cloudera_tools = cloudera_tools self._data = None self._cluster_services = None self._exception_store = None self.get_health_status() def get_cluster_services(self): return self._cluster_services def is_cloudera_active(self): if self._exception_store: raise health_check_base.RedHealthError(self._exception_store) return _("Cloudera Manager is Active") def get_cloudera_health(self): cu = self.cloudera_tools api = cu.get_api_client(self.cluster) return api.get_service_health_status(self.cluster.name) def get_important_services(self): # will be overridable in future cu = self.cloudera_tools return [ cu.HDFS_SERVICE_NAME, cu.YARN_SERVICE_NAME, cu.OOZIE_SERVICE_NAME ] def get_health_status(self, service=None): if self._data is not None: return self._data.get(service, []) if service else self._data self._data = {} self._cluster_services = [] try: # all data already grouped by services self._data = self.get_cloudera_health() self._cluster_services = self._data.keys() except Exception as e: msg = _("Can't get response from Cloudera " "Manager") LOG.exception(msg) self._exception_store = _( "%(problem)s, reason: %(reason)s") % { 'problem': msg, 'reason': six.text_type(e)} class ClouderaManagerHealthCheck(health_check_base.BasicHealthCheck): def __init__(self, cluster, provider): self.provider = provider super(ClouderaManagerHealthCheck, self).__init__(cluster) def get_health_check_name(self): return _("Cloudera Manager health check") def is_available(self): return self.cluster.plugin_name == 'cdh' def check_health(self): return self.provider.is_cloudera_active() class ServiceHealthCheck(health_check_base.BasicHealthCheck): def __init__(self, cluster, provider, service): self.provider = provider self.service = service super(ServiceHealthCheck, self).__init__(cluster) def get_health_check_name(self): return _("CDH %s health check") % self.service def is_available(self): return self.cluster.plugin_name == 'cdh' def check_health(self): important_services = self.provider.get_important_services() observed_data = self.provider.get_health_status(self.service) imp_map = {'BAD': 'red', 'CONCERNING': 'yellow', 'GOOD': 'green'} summary = observed_data['summary'] checks = observed_data.get('checks', []) failed_checks = [] for check in checks: if check['summary'] != 'GOOD': failed_checks.append('%(name)s - %(summary)s state' % { 'name': check['name'], 'summary': check['summary'] }) additional_info = None if failed_checks: additional_info = _( "The following checks did not pass: %s") % ",".join( failed_checks) if self.service in important_services: overall = imp_map.get(summary, 'red') else: overall = 'green' if summary != 'GOOD': overall = 'yellow' msg = _("Cloudera Manager has responded that service is in " "the %s state") % summary if additional_info: msg = _("%(problem)s. %(description)s") % { 'problem': msg, 'description': additional_info} if overall == 'red': raise health_check_base.RedHealthError(msg) elif overall == 'yellow': raise health_check_base.YellowHealthError(msg) return msg def get_health_checks(cluster, cloudera_utils): provider = HealthStatusProvider(cluster, cloudera_utils) checks = [functools.partial( ClouderaManagerHealthCheck, provider=provider)] for service in provider.get_cluster_services(): checks.append(functools.partial( ServiceHealthCheck, provider=provider, service=service)) return checks sahara-8.0.0/sahara/plugins/cdh/commands.py0000666000175100017510000000662713245514472020657 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ from sahara.plugins import exceptions as ex def _root(remote, cmd, **kwargs): return remote.execute_command(cmd, run_as_root=True, **kwargs) def _get_os_distrib(remote): return remote.get_os_distrib() def is_centos_os(remote): return _get_os_distrib(remote) == 'centos' def is_ubuntu_os(remote): return _get_os_distrib(remote) == 'ubuntu' def is_pre_installed_cdh(remote): code, out = remote.execute_command('ls /etc/init.d/cloudera-scm-server', raise_when_error=False) return code == 0 def start_cloudera_db(remote): _root(remote, 'service cloudera-scm-server-db start') # for Hive access hive_access_param = 'host metastore hive 0.0.0.0/0 md5' remote.append_to_file('/var/lib/cloudera-scm-server-db/data/pg_hba.conf', hive_access_param, run_as_root=True) _root(remote, 'service cloudera-scm-server-db restart') def start_manager(remote): _root(remote, 'service cloudera-scm-server start') def configure_agent(remote, manager_address): remote.replace_remote_string('/etc/cloudera-scm-agent/config.ini', 'server_host=.*', 'server_host=%s' % manager_address) def start_agent(remote): _root(remote, 'service cloudera-scm-agent start') def install_packages(remote, packages, timeout=1800): distrib = _get_os_distrib(remote) if distrib == 'ubuntu': cmd = 'RUNLEVEL=1 apt-get install -y %s' elif distrib == 'centos': cmd = 'yum install -y %s' else: raise ex.HadoopProvisionError( _("OS on image is not supported by CDH plugin")) cmd = cmd % ' '.join(packages) _root(remote, cmd, timeout=timeout) def update_repository(remote): if is_ubuntu_os(remote): _root(remote, 'apt-get update') if is_centos_os(remote): _root(remote, 'yum clean all') def push_remote_file(remote, src, dst): cmd = 'curl %s -o %s' % (src, dst) _root(remote, cmd) def add_ubuntu_repository(r, repo_list_url, repo_name): push_remote_file(r, repo_list_url, '/etc/apt/sources.list.d/%s.list' % repo_name) def write_ubuntu_repository(r, repo_content, repo_name): r.write_file_to('/etc/apt/sources.list.d/%s.list' % repo_name, repo_content, run_as_root=True) def add_apt_key(remote, key_url): cmd = 'wget -qO - %s | apt-key add -' % key_url _root(remote, cmd) def add_centos_repository(r, repo_list_url, repo_name): push_remote_file(r, repo_list_url, '/etc/yum.repos.d/%s.repo' % repo_name) def write_centos_repository(r, repo_content, repo_name): r.write_file_to('/etc/yum.repos.d/%s.repo' % repo_name, repo_content, run_as_root=True) def start_mysql_server(remote): _root(remote, 'service mysql start') sahara-8.0.0/sahara/plugins/cdh/edp_engine.py0000666000175100017510000000767413245514472021156 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import exceptions as pl_ex from sahara.plugins import kerberos from sahara.plugins import utils as u from sahara.service.edp import hdfs_helper from sahara.service.edp.oozie import engine as edp_engine from sahara.service.edp.spark import engine as edp_spark_engine class EdpOozieEngine(edp_engine.OozieJobEngine): def __init__(self, cluster): super(EdpOozieEngine, self).__init__(cluster) # will be defined in derived classes self.cloudera_utils = None def get_client(self): if kerberos.is_kerberos_security_enabled(self.cluster): return super(EdpOozieEngine, self).get_remote_client() return super(EdpOozieEngine, self).get_client() def get_hdfs_user(self): return 'hdfs' def create_hdfs_dir(self, remote, dir_name): hdfs_helper.create_dir_hadoop2(remote, dir_name, self.get_hdfs_user()) def get_oozie_server_uri(self, cluster): oozie_ip = self.cloudera_utils.pu.get_oozie(cluster).management_ip return 'http://%s:11000/oozie' % oozie_ip def get_name_node_uri(self, cluster): if len(self.cloudera_utils.pu.get_jns(cluster)) > 0: return 'hdfs://%s' % self.cloudera_utils.NAME_SERVICE else: namenode_ip = self.cloudera_utils.pu.get_namenode(cluster).fqdn() return 'hdfs://%s:8020' % namenode_ip def get_resource_manager_uri(self, cluster): resourcemanager = self.cloudera_utils.pu.get_resourcemanager(cluster) return '%s:8032' % resourcemanager.fqdn() def get_oozie_server(self, cluster): return self.cloudera_utils.pu.get_oozie(cluster) def validate_job_execution(self, cluster, job, data): oo_count = u.get_instances_count(cluster, 'OOZIE_SERVER') if oo_count != 1: raise pl_ex.InvalidComponentCountException( 'OOZIE_SERVER', '1', oo_count) super(EdpOozieEngine, self).validate_job_execution(cluster, job, data) class EdpSparkEngine(edp_spark_engine.SparkJobEngine): edp_base_version = "" def __init__(self, cluster): super(EdpSparkEngine, self).__init__(cluster) self.master = u.get_instance(cluster, "SPARK_YARN_HISTORY_SERVER") self.plugin_params["spark-user"] = "sudo -u spark " self.plugin_params["spark-submit"] = "spark-submit" self.plugin_params["deploy-mode"] = "cluster" self.plugin_params["master"] = "yarn-cluster" driver_cp = u.get_config_value_or_default( "Spark", "Executor extra classpath", self.cluster) self.plugin_params["driver-class-path"] = driver_cp @classmethod def edp_supported(cls, version): return version >= cls.edp_base_version def validate_job_execution(self, cluster, job, data): if not self.edp_supported(cluster.hadoop_version): raise ex.InvalidDataException( _('Cloudera {base} or higher required to run {type}' 'jobs').format(base=self.edp_base_version, type=job.type)) shs_count = u.get_instances_count( cluster, 'SPARK_YARN_HISTORY_SERVER') if shs_count != 1: raise pl_ex.InvalidComponentCountException( 'SPARK_YARN_HISTORY_SERVER', '1', shs_count) super(EdpSparkEngine, self).validate_job_execution( cluster, job, data) sahara-8.0.0/sahara/plugins/images.py0000666000175100017510000012700013245514472017552 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils import abc import collections import copy import functools import itertools from os import path import jsonschema import six import yaml from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import exceptions as p_ex from sahara.utils import files def transform_exception(from_type, to_type, transform_func=None): """Decorator to transform exception types. :param from_type: The type of exception to catch and transform. :param to_type: The type of exception to raise instead. :param transform_func: A function to transform from_type into to_type, which must be of the form func(exc, to_type). Defaults to: lambda exc, new_type: new_type(exc.message) """ if not transform_func: transform_func = lambda exc, new_type: new_type(exc.message) def decorator(func): @functools.wraps(func) def handler(*args, **kwargs): try: func(*args, **kwargs) except from_type as exc: raise transform_func(exc, to_type) return handler return decorator def validate_instance(instance, validators, test_only=False, **kwargs): """Runs all validators against the specified instance. :param instance: An instance to validate. :param validators: A sequence of ImageValidators. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :raises ImageValidationError: If validation fails. """ with instance.remote() as remote: for validator in validators: validator.validate(remote, test_only=test_only, **kwargs) class ImageArgument(object): """An argument used by an image manifest.""" SPEC_SCHEMA = { "type": "object", "items": { "type": "object", "properties": { "target_variable": { "type": "string", "minLength": 1 }, "description": { "type": "string", "minLength": 1 }, "default": { "type": "string", "minLength": 1 }, "required": { "type": "boolean", "minLength": 1 }, "choices": { "type": "array", "minLength": 1, "items": { "type": "string" } } } } } @classmethod def from_spec(cls, spec): """Constructs and returns a set of arguments from a specification. :param spec: The specification for the argument set. :return: A dict of arguments built to the specification. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) arguments = {name: cls(name, arg.get('description'), arg.get('default'), arg.get('required'), arg.get('choices')) for name, arg in six.iteritems(spec)} reserved_names = ['distro', 'test_only'] for name, arg in six.iteritems(arguments): if name in reserved_names: raise p_ex.ImageValidationSpecificationError( _("The following argument names are reserved: " "{names}").format(reserved_names)) if not arg.default and not arg.required: raise p_ex.ImageValidationSpecificationError( _("Argument {name} is not required and must specify a " "default value.").format(name=arg.name)) if arg.choices and arg.default and arg.default not in arg.choices: raise p_ex.ImageValidationSpecificationError( _("Argument {name} specifies a default which is not one " "of its choices.").format(name=arg.name)) return arguments def __init__(self, name, description=None, default=None, required=False, choices=None): self.name = name self.description = description self.default = default self.required = required self.choices = choices @six.add_metaclass(abc.ABCMeta) class ImageValidator(object): """Validates the image spawned to an instance via a set of rules.""" @abc.abstractmethod def validate(self, remote, test_only=False, **kwargs): """Validates the image. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :raises ImageValidationError: If validation fails. """ pass @six.add_metaclass(abc.ABCMeta) class SaharaImageValidatorBase(ImageValidator): """Base class for Sahara's native image validation.""" DISTRO_KEY = 'distro' TEST_ONLY_KEY = 'test_only' ORDERED_VALIDATORS_SCHEMA = { "type": "array", "items": { "type": "object", "minProperties": 1, "maxProperties": 1 } } _DISTRO_FAMILES = { 'centos': 'redhat', 'centos7': 'redhat', 'fedora': 'redhat', 'redhat': 'redhat', 'redhatenterpriseserver': 'redhat', 'ubuntu': 'debian' } @staticmethod def get_validator_map(custom_validator_map=None): """Gets the map of validator name token to validator class. :param custom_validator_map: A map of validator names and classes to add to the ones Sahara provides by default. These will take precedence over the base validators in case of key overlap. :return: A map of validator names and classes. """ default_validator_map = { 'package': SaharaPackageValidator, 'script': SaharaScriptValidator, 'copy_script': SaharaCopyScriptValidator, 'any': SaharaAnyValidator, 'all': SaharaAllValidator, 'os_case': SaharaOSCaseValidator, 'argument_case': SaharaArgumentCaseValidator, 'argument_set': SaharaArgumentSetterValidator, } if custom_validator_map: default_validator_map.update(custom_validator_map) return default_validator_map @classmethod def from_yaml(cls, yaml_path, validator_map=None, resource_roots=None): """Constructs and returns a validator from the provided yaml file. :param yaml_path: The relative path to a yaml file. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A SaharaImageValidator built to the yaml specification. """ validator_map = validator_map or {} resource_roots = resource_roots or [] file_text = files.get_file_text(yaml_path) spec = yaml.safe_load(file_text) validator_map = cls.get_validator_map(validator_map) return cls.from_spec(spec, validator_map, resource_roots) @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Constructs and returns a validator from a specification object. :param spec: The specification for the validator. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A validator built to the specification. """ pass @classmethod def from_spec_list(cls, specs, validator_map, resource_roots): """Constructs a list of validators from a list of specifications. :param specs: A list of validator specifications, each of which will be a dict of size 1, where the key represents the validator type and the value respresents its specification. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A list of validators. """ validators = [] for spec in specs: validator_class, validator_spec = cls.get_class_from_spec( spec, validator_map) validators.append(validator_class.from_spec( validator_spec, validator_map, resource_roots)) return validators @classmethod def get_class_from_spec(cls, spec, validator_map): """Gets the class and specification from a validator dict. :param spec: A validator specification including its type: a dict of size 1, where the key represents the validator type and the value respresents its configuration. :param validator_map: A map of validator name to class. :return: A tuple of validator class and configuration. """ key, value = list(six.iteritems(spec))[0] validator_class = validator_map.get(key, None) if not validator_class: raise p_ex.ImageValidationSpecificationError( _("Validator type %s not found.") % validator_class) return validator_class, value class ValidationAttemptFailed(object): """An object representing a failed validation attempt. Primarily for use by the SaharaAnyValidator, which must aggregate failures for error exposition purposes. """ def __init__(self, exception): self.exception = exception def __bool__(self): return False def __nonzero__(self): return False def try_validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate, but returns rather than raising on failure. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. :return: True if successful, ValidationAttemptFailed object if failed. """ try: self.validate( remote, test_only=test_only, image_arguments=image_arguments, **kwargs) return True except p_ex.ImageValidationError as exc: return self.ValidationAttemptFailed(exc) class SaharaImageValidator(SaharaImageValidatorBase): """The root of any tree of SaharaImageValidators. This validator serves as the root of the tree for SaharaImageValidators, and provides any needed initialization (such as distro retrieval.) """ SPEC_SCHEMA = { "title": "SaharaImageValidator", "type": "object", "properties": { "validators": SaharaImageValidatorBase.ORDERED_VALIDATORS_SCHEMA }, "required": ["validators"] } def get_argument_list(self): return [argument for name, argument in six.iteritems(self.arguments)] @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Constructs and returns a validator from a specification object. :param spec: The specification for the validator: a dict containing the key "validators", which contains a list of validator specifications. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A SaharaImageValidator containing all specified validators. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) arguments_spec = spec.get('arguments', {}) arguments = ImageArgument.from_spec(arguments_spec) validators_spec = spec['validators'] validator = SaharaAllValidator.from_spec( validators_spec, validator_map, resource_roots) return cls(validator, arguments) def __init__(self, validator, arguments): """Constructor method. :param validator: A SaharaAllValidator containing the specified validators. """ self.validator = validator self.validators = validator.validators self.arguments = arguments @transform_exception(ex.RemoteCommandException, p_ex.ImageValidationError) def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate the image. Before deferring to contained validators, performs one-time setup steps such as distro discovery. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. :raises ImageValidationError: If validation fails. """ argument_values = {} for name, argument in six.iteritems(self.arguments): if name not in image_arguments: if argument.required: raise p_ex.ImageValidationError( _("Argument {name} is required for image " "processing.").format(name=name)) else: argument_values[name] = argument.default else: value = image_arguments[name] choices = argument.choices if choices and value not in choices: raise p_ex.ImageValidationError( _("Value for argument {name} must be one of " "{choices}.").format(name=name, choices=choices)) else: argument_values[name] = value argument_values[self.DISTRO_KEY] = remote.get_os_distrib() self.validator.validate(remote, test_only=test_only, image_arguments=argument_values) class SaharaPackageValidator(SaharaImageValidatorBase): """A validator that checks package installation state on the instance.""" class Package(object): def __init__(self, name, version=None): self.name = name self.version = version def __str__(self): return ("%s-%s" % (self.name, self.version) if self.version else self.name) _SINGLE_PACKAGE_SCHEMA = { "oneOf": [ { "type": "object", "minProperties": 1, "maxProperties": 1, "additionalProperties": { "type": "object", "properties": { "version": { "type": "string", "minLength": 1 }, } }, }, { "type": "string", "minLength": 1 } ] } SPEC_SCHEMA = { "title": "SaharaPackageValidator", "oneOf": [ _SINGLE_PACKAGE_SCHEMA, { "type": "array", "items": _SINGLE_PACKAGE_SCHEMA, "minLength": 1 } ] } @classmethod def _package_from_spec(cls, spec): """Builds a single package object from a specification. :param spec: May be a string or single-length dictionary of name to configuration values. :return: A package object. """ if isinstance(spec, six.string_types): return cls.Package(spec, None) else: package, properties = list(six.iteritems(spec))[0] version = properties.get('version', None) return cls.Package(package, version) @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Builds a package validator from a specification. :param spec: May be a string, a single-length dictionary of name to configuration values, or a list containing any number of either or both of the above. Configuration values may include: version: The version of the package to check and/or install. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A validator that will check that the specified package or packages are installed. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) packages = ([cls._package_from_spec(package_spec) for package_spec in spec] if isinstance(spec, list) else [cls._package_from_spec(spec)]) return cls(packages) def __init__(self, packages): self.packages = packages @transform_exception(ex.RemoteCommandException, p_ex.ImageValidationError) def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate package installation on the image. Even if test_only=False, attempts to verify previous package installation offline before using networked tools to validate or install new packages. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. :raises ImageValidationError: If validation fails. """ env_distro = image_arguments[self.DISTRO_KEY] env_family = self._DISTRO_FAMILES[env_distro] check, install = self._DISTRO_TOOLS[env_family] if not env_family: raise p_ex.ImageValidationError( _("Unknown distro: cannot verify or install packages.")) try: check(self, remote) except (ex.SubprocessException, ex.RemoteCommandException, RuntimeError): if not test_only: install(self, remote) check(self, remote) else: raise def _dpkg_check(self, remote): check_cmd = ("dpkg -s %s" % " ".join(str(package) for package in self.packages)) return _sudo(remote, check_cmd) def _rpm_check(self, remote): check_cmd = ("rpm -q %s" % " ".join(str(package) for package in self.packages)) return _sudo(remote, check_cmd) def _yum_install(self, remote): install_cmd = ( "yum install -y %s" % " ".join(str(package) for package in self.packages)) _sudo(remote, install_cmd) def _apt_install(self, remote): install_cmd = ( "apt-get -y install %s" % " ".join(str(package) for package in self.packages)) return _sudo(remote, install_cmd) _DISTRO_TOOLS = { "redhat": (_rpm_check, _yum_install), "debian": (_dpkg_check, _apt_install) } class SaharaScriptValidator(SaharaImageValidatorBase): """A validator that runs a script on the instance.""" _DEFAULT_ENV_VARS = [SaharaImageValidatorBase.TEST_ONLY_KEY, SaharaImageValidatorBase.DISTRO_KEY] SPEC_SCHEMA = { "title": "SaharaScriptValidator", "oneOf": [ { "type": "object", "minProperties": 1, "maxProperties": 1, "additionalProperties": { "type": "object", "properties": { "env_vars": { "type": "array", "items": { "type": "string" } }, "output": { "type": "string", "minLength": 1 }, "inline": { "type": "string", "minLength": 1 } }, } }, { "type": "string" } ] } @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Builds a script validator from a specification. :param spec: May be a string or a single-length dictionary of name to configuration values. Configuration values include: env_vars: A list of environment variable names to send to the script. output: A key into which to put the stdout of the script in the image_arguments of the validation run. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A validator that will run a script on the image. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) script_contents = None if isinstance(spec, six.string_types): script_path = spec env_vars, output_var = cls._DEFAULT_ENV_VARS, None else: script_path, properties = list(six.iteritems(spec))[0] env_vars = cls._DEFAULT_ENV_VARS + properties.get('env_vars', []) output_var = properties.get('output', None) script_contents = properties.get('inline') if not script_contents: for root in resource_roots: file_path = path.join(root, script_path) script_contents = files.try_get_file_text(file_path) if script_contents: break if not script_contents: raise p_ex.ImageValidationSpecificationError( _("Script %s not found in any resource roots.") % script_path) return SaharaScriptValidator(script_contents, env_vars, output_var) def __init__(self, script_contents, env_vars=None, output_var=None): """Constructor method. :param script_contents: A string representation of the script. :param env_vars: A list of environment variables to send to the script. :param output_var: A key into which to put the stdout of the script in the image_arguments of the validation run. :return: A SaharaScriptValidator. """ self.script_contents = script_contents self.env_vars = env_vars or [] self.output_var = output_var @transform_exception(ex.RemoteCommandException, p_ex.ImageValidationError) def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate by running a script on the image. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. Note that the key SIV_TEST_ONLY will be set to 1 if the script should test_only and 0 otherwise; all scripts should act on this input if possible. The key SIV_DISTRO will also contain the distro representation, per `lsb_release -is`. :raises ImageValidationError: If validation fails. """ arguments = copy.deepcopy(image_arguments) arguments[self.TEST_ONLY_KEY] = 1 if test_only else 0 script = "\n".join(["%(env_vars)s", "%(script)s"]) env_vars = "\n".join("export %s=%s" % (key, value) for (key, value) in six.iteritems(arguments) if key in self.env_vars) script = script % {"env_vars": env_vars, "script": self.script_contents} path = '/tmp/%s.sh' % uuidutils.generate_uuid() remote.write_file_to(path, script, run_as_root=True) _sudo(remote, 'chmod +x %s' % path) code, stdout = _sudo(remote, path) if self.output_var: image_arguments[self.output_var] = stdout class SaharaCopyScriptValidator(SaharaImageValidatorBase): """A validator that copy a script to the instance.""" SPEC_SCHEMA = { "title": "SaharaCopyScriptValidator", "oneOf": [ { "type": "object", "minProperties": 1, "maxProperties": 1, "additionalProperties": { "type": "object", "properties": { "output": { "type": "string", "minLength": 1 }, "inline": { "type": "string", "minLength": 1 } }, } }, { "type": "string" } ] } @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Builds a copy script validator from a specification. :param spec: May be a string or a single-length dictionary of name to configuration values. Configuration values include: env_vars: A list of environment variable names to send to the script. output: A key into which to put the stdout of the script in the image_arguments of the validation run. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A validator that will copy a script to the image. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) script_contents = None if isinstance(spec, six.string_types): script_path = spec output_var = None else: script_path, properties = list(six.iteritems(spec))[0] output_var = properties.get('output', None) script_contents = properties.get('inline') if not script_contents: for root in resource_roots: file_path = path.join(root, script_path) script_contents = files.try_get_file_text(file_path) if script_contents: break script_name = script_path.split('/')[2] if not script_contents: raise p_ex.ImageValidationSpecificationError( _("Script %s not found in any resource roots.") % script_path) return SaharaCopyScriptValidator(script_contents, script_name, output_var) def __init__(self, script_contents, script_name, output_var=None): """Constructor method. :param script_contents: A string representation of the script. :param output_var: A key into which to put the stdout of the script in the image_arguments of the validation run. :return: A SaharaScriptValidator. """ self.script_contents = script_contents self.script_name = script_name self.output_var = output_var @transform_exception(ex.RemoteCommandException, p_ex.ImageValidationError) def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate by running a script on the image. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. Note that the key SIV_TEST_ONLY will be set to 1 if the script should test_only and 0 otherwise; all scripts should act on this input if possible. The key SIV_DISTRO will also contain the distro representation, per `lsb_release -is`. :raises ImageValidationError: If validation fails. """ arguments = copy.deepcopy(image_arguments) arguments[self.TEST_ONLY_KEY] = 1 if test_only else 0 script = "\n".join(["%(script)s"]) script = script % {"script": self.script_contents} path = '/tmp/%s' % self.script_name remote.write_file_to(path, script, run_as_root=True) @six.add_metaclass(abc.ABCMeta) class SaharaAggregateValidator(SaharaImageValidatorBase): """An abstract class representing an ordered list of other validators.""" SPEC_SCHEMA = SaharaImageValidator.ORDERED_VALIDATORS_SCHEMA @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Builds the aggregate validator from a specification. :param spec: A list of validator definitions, each of which is a single-length dictionary of name to configuration values. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: An aggregate validator. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) validators = cls.from_spec_list(spec, validator_map, resource_roots) return cls(validators) def __init__(self, validators): self.validators = validators class SaharaAnyValidator(SaharaAggregateValidator): """A list of validators, only one of which must succeed.""" def _try_all(self, remote, test_only=False, image_arguments=None, **kwargs): results = [] for validator in self.validators: result = validator.try_validate(remote, test_only=test_only, image_arguments=image_arguments, **kwargs) results.append(result) if result: break return results def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate any of the contained validators. Note that if test_only=False, this validator will first run all contained validators using test_only=True, and succeed immediately should any pass validation. If all fail, it will only then run them using test_only=False, and again succeed immediately should any pass. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. :raises ImageValidationError: If validation fails. """ results = self._try_all(remote, test_only=True, image_arguments=image_arguments) if not test_only and not any(results): results = self._try_all(remote, test_only=False, image_arguments=image_arguments) if not any(results): raise p_ex.AllValidationsFailedError(result.exception for result in results) class SaharaAllValidator(SaharaAggregateValidator): """A list of validators, all of which must succeed.""" def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate all of the contained validators. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. :raises ImageValidationError: If validation fails. """ for validator in self.validators: validator.validate(remote, test_only=test_only, image_arguments=image_arguments) class SaharaOSCaseValidator(SaharaImageValidatorBase): """A validator which will take different actions depending on distro.""" _distro_tuple = collections.namedtuple('Distro', ['distro', 'validator']) SPEC_SCHEMA = { "type": "array", "minLength": 1, "items": { "type": "object", "minProperties": 1, "maxProperties": 1, "additionalProperties": SaharaImageValidator.ORDERED_VALIDATORS_SCHEMA, } } @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Builds an os_case validator from a specification. :param spec: A list of single-length dictionaries. The key of each is a distro or family name and the value under each key is a list of validators (all of which must succeed.) :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A SaharaOSCaseValidator. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) distros = itertools.chain(*(six.iteritems(distro_spec) for distro_spec in spec)) distros = [ cls._distro_tuple(key, SaharaAllValidator.from_spec( value, validator_map, resource_roots)) for (key, value) in distros] return cls(distros) def __init__(self, distros): """Constructor method. :param distros: A list of distro tuples (distro, list of validators). """ self.distros = distros def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate depending on distro. May match the OS by specific distro or by family (centos may match "centos" or "redhat", for instance.) If multiple keys match the distro, only the validators under the first matched key will be run. If no keys match, no validators are run, and validation proceeds. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. :raises ImageValidationError: If validation fails. """ env_distro = image_arguments[self.DISTRO_KEY] family = self._DISTRO_FAMILES.get(env_distro) matches = {env_distro, family} if family else {env_distro} for distro, validator in self.distros: if distro in matches: validator.validate( remote, test_only=test_only, image_arguments=image_arguments) break class SaharaArgumentCaseValidator(SaharaImageValidatorBase): """A validator which will take different actions depending on distro.""" SPEC_SCHEMA = { "type": "object", "properties": { "argument_name": { "type": "string", "minLength": 1 }, "cases": { "type": "object", "minProperties": 1, "additionalProperties": SaharaImageValidator.ORDERED_VALIDATORS_SCHEMA, }, }, "additionalProperties": False, "required": ["argument_name", "cases"] } @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Builds an argument_case validator from a specification. :param spec: A dictionary with two items: "argument_name", containing a string indicating the argument to be checked, and "cases", a dictionary. The key of each item in the dictionary is a value which may or may not match the argument value, and the value is a list of validators to be run in case it does. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A SaharaArgumentCaseValidator. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) argument_name = spec['argument_name'] cases = {key: SaharaAllValidator.from_spec( value, validator_map, resource_roots) for key, value in six.iteritems(spec['cases'])} return cls(argument_name, cases) def __init__(self, argument_name, cases): """Constructor method. :param argument_name: The name of an argument. :param cases: A dictionary of possible argument value to a sub-validator to run in case of a match. """ self.argument_name = argument_name self.cases = cases def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate depending on argument value. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. :raises ImageValidationError: If validation fails. """ arg = self.argument_name if arg not in image_arguments: raise p_ex.ImageValidationError( _("Argument {name} not found.").format(name=arg)) value = image_arguments[arg] if value in self.cases: self.cases[value].validate( remote, test_only=test_only, image_arguments=image_arguments) class SaharaArgumentSetterValidator(SaharaImageValidatorBase): """A validator which sets a specific argument to a specific value.""" SPEC_SCHEMA = { "type": "object", "properties": { "argument_name": { "type": "string", "minLength": 1 }, "value": { "type": "string", "minLength": 1 }, }, "additionalProperties": False, "required": ["argument_name", "value"] } @classmethod def from_spec(cls, spec, validator_map, resource_roots): """Builds an argument_set validator from a specification. :param spec: A dictionary with two items: "argument_name", containing a string indicating the argument to be set, and "value", a value to which to set that argument. :param validator_map: A map of validator name to class. :param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists. :return: A SaharaArgumentSetterValidator. """ jsonschema.validate(spec, cls.SPEC_SCHEMA) argument_name = spec['argument_name'] value = spec['value'] return cls(argument_name, value) def __init__(self, argument_name, value): """Constructor method. :param argument_name: The name of an argument. :param value: A value to which to set that argument. """ self.argument_name = argument_name self.value = value def validate(self, remote, test_only=False, image_arguments=None, **kwargs): """Attempts to validate depending on argument value. :param remote: A remote socket to the instance. :param test_only: If true, all validators will only verify that a desired state is present, and fail if it is not. If false, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds. :param image_arguments: A dictionary of image argument values keyed by argument name. """ image_arguments[self.argument_name] = self.value def _sudo(remote, cmd, **kwargs): return remote.execute_command(cmd, run_as_root=True, **kwargs) sahara-8.0.0/sahara/plugins/__init__.py0000666000175100017510000000000013245514472020032 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/0000775000175100017510000000000013245515027017354 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/versionfactory.py0000666000175100017510000000361313245514472023013 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import re from sahara.utils import general class VersionFactory(object): versions = None modules = None initialized = False @staticmethod def get_instance(): if not VersionFactory.initialized: src_dir = os.path.join(os.path.dirname(__file__), '') versions = ( [name[1:].replace('_', '.') for name in os.listdir(src_dir) if (os.path.isdir(os.path.join(src_dir, name)) and re.match(r'^v\d+_\d+_\d+$', name))]) versions.sort(key=general.natural_sort_key) VersionFactory.versions = versions VersionFactory.modules = {} for version in VersionFactory.versions: module_name = 'sahara.plugins.vanilla.v%s.versionhandler' % ( version.replace('.', '_')) module_class = getattr( __import__(module_name, fromlist=['sahara']), 'VersionHandler') module = module_class() VersionFactory.modules[version] = module VersionFactory.initialized = True return VersionFactory() def get_versions(self): return VersionFactory.versions def get_version_handler(self, version): return VersionFactory.modules[version] sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/0000775000175100017510000000000013245515027020353 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/versionhandler.py0000666000175100017510000001374413245514472023766 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from sahara import conductor from sahara import context from sahara.plugins import utils from sahara.plugins.vanilla import abstractversionhandler as avm from sahara.plugins.vanilla.hadoop2 import config as c from sahara.plugins.vanilla.hadoop2 import keypairs from sahara.plugins.vanilla.hadoop2 import recommendations_utils as ru from sahara.plugins.vanilla.hadoop2 import run_scripts as run from sahara.plugins.vanilla.hadoop2 import scaling as sc from sahara.plugins.vanilla.hadoop2 import starting_scripts as s_scripts from sahara.plugins.vanilla.hadoop2 import utils as u from sahara.plugins.vanilla.hadoop2 import validation as vl from sahara.plugins.vanilla import utils as vu from sahara.plugins.vanilla.v2_8_2 import config_helper as c_helper from sahara.plugins.vanilla.v2_8_2 import edp_engine from sahara.swift import swift_helper from sahara.utils import cluster as cluster_utils conductor = conductor.API CONF = cfg.CONF class VersionHandler(avm.AbstractVersionHandler): def __init__(self): self.pctx = { 'env_confs': c_helper.get_env_configs(), 'all_confs': c_helper.get_plugin_configs() } def get_plugin_configs(self): return self.pctx['all_confs'] def get_node_processes(self): return { "Hadoop": [], "MapReduce": ["historyserver"], "HDFS": ["namenode", "datanode", "secondarynamenode"], "YARN": ["resourcemanager", "nodemanager"], "JobFlow": ["oozie"], "Hive": ["hiveserver"], "Spark": ["spark history server"], "ZooKeeper": ["zookeeper"] } def validate(self, cluster): vl.validate_cluster_creating(self.pctx, cluster) def update_infra(self, cluster): pass def configure_cluster(self, cluster): c.configure_cluster(self.pctx, cluster) def start_cluster(self, cluster): keypairs.provision_keypairs(cluster) s_scripts.start_namenode(cluster) s_scripts.start_secondarynamenode(cluster) s_scripts.start_resourcemanager(cluster) run.start_dn_nm_processes(utils.get_instances(cluster)) run.await_datanodes(cluster) s_scripts.start_historyserver(cluster) s_scripts.start_oozie(self.pctx, cluster) s_scripts.start_hiveserver(self.pctx, cluster) s_scripts.start_zookeeper(cluster) swift_helper.install_ssl_certs(cluster_utils.get_instances(cluster)) self._set_cluster_info(cluster) s_scripts.start_spark(cluster) def decommission_nodes(self, cluster, instances): sc.decommission_nodes(self.pctx, cluster, instances) def validate_scaling(self, cluster, existing, additional): vl.validate_additional_ng_scaling(cluster, additional) vl.validate_existing_ng_scaling(self.pctx, cluster, existing) zk_ng = utils.get_node_groups(cluster, "zookeeper") if zk_ng: vl.validate_zookeeper_node_count(zk_ng, existing, additional) def scale_cluster(self, cluster, instances): keypairs.provision_keypairs(cluster, instances) sc.scale_cluster(self.pctx, cluster, instances) def _set_cluster_info(self, cluster): nn = vu.get_namenode(cluster) rm = vu.get_resourcemanager(cluster) hs = vu.get_historyserver(cluster) oo = vu.get_oozie(cluster) sp = vu.get_spark_history_server(cluster) info = {} if rm: info['YARN'] = { 'Web UI': 'http://%s:%s' % (rm.get_ip_or_dns_name(), '8088'), 'ResourceManager': 'http://%s:%s' % ( rm.get_ip_or_dns_name(), '8032') } if nn: info['HDFS'] = { 'Web UI': 'http://%s:%s' % (nn.get_ip_or_dns_name(), '50070'), 'NameNode': 'hdfs://%s:%s' % (nn.hostname(), '9000') } if oo: info['JobFlow'] = { 'Oozie': 'http://%s:%s' % (oo.get_ip_or_dns_name(), '11000') } if hs: info['MapReduce JobHistory Server'] = { 'Web UI': 'http://%s:%s' % (hs.get_ip_or_dns_name(), '19888') } if sp: info['Apache Spark'] = { 'Spark UI': 'http://%s:%s' % (sp.management_ip, '4040'), 'Spark History Server UI': 'http://%s:%s' % (sp.management_ip, '18080') } ctx = context.ctx() conductor.cluster_update(ctx, cluster, {'info': info}) def get_edp_engine(self, cluster, job_type): if job_type in edp_engine.EdpOozieEngine.get_supported_job_types(): return edp_engine.EdpOozieEngine(cluster) if job_type in edp_engine.EdpSparkEngine.get_supported_job_types(): return edp_engine.EdpSparkEngine(cluster) return None def get_edp_job_types(self): return (edp_engine.EdpOozieEngine.get_supported_job_types() + edp_engine.EdpSparkEngine.get_supported_job_types()) def get_edp_config_hints(self, job_type): return edp_engine.EdpOozieEngine.get_possible_job_config(job_type) def on_terminate_cluster(self, cluster): u.delete_oozie_password(cluster) keypairs.drop_key(cluster) def get_open_ports(self, node_group): return c.get_open_ports(node_group) def recommend_configs(self, cluster, scaling): ru.recommend_configs(cluster, self.get_plugin_configs(), scaling) sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/config_helper.py0000666000175100017510000000734613245514472023550 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six from sahara.plugins import provisioning as p from sahara.plugins.vanilla.hadoop2 import config_helper as c_helper from sahara.utils import xmlutils as x CONF = cfg.CONF CONF.import_opt("enable_data_locality", "sahara.topology.topology_helper") CORE_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_8_2/resources/core-default.xml') HDFS_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_8_2/resources/hdfs-default.xml') MAPRED_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_8_2/resources/mapred-default.xml') YARN_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_8_2/resources/yarn-default.xml') OOZIE_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_8_2/resources/oozie-default.xml') HIVE_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_8_2/resources/hive-default.xml') XML_CONFS = { "Hadoop": [CORE_DEFAULT], "HDFS": [HDFS_DEFAULT], "YARN": [YARN_DEFAULT], "MapReduce": [MAPRED_DEFAULT], "JobFlow": [OOZIE_DEFAULT], "Hive": [HIVE_DEFAULT] } ENV_CONFS = { "YARN": { 'ResourceManager Heap Size': 1024, 'NodeManager Heap Size': 1024 }, "HDFS": { 'NameNode Heap Size': 1024, 'SecondaryNameNode Heap Size': 1024, 'DataNode Heap Size': 1024 }, "MapReduce": { 'JobHistoryServer Heap Size': 1024 }, "JobFlow": { 'Oozie Heap Size': 1024 } } # Initialise plugin Hadoop configurations PLUGIN_XML_CONFIGS = c_helper.init_xml_configs(XML_CONFS) PLUGIN_ENV_CONFIGS = c_helper.init_env_configs(ENV_CONFS) def _init_all_configs(): configs = [] configs.extend(PLUGIN_XML_CONFIGS) configs.extend(PLUGIN_ENV_CONFIGS) configs.extend(c_helper.PLUGIN_GENERAL_CONFIGS) configs.extend(_get_spark_configs()) configs.extend(_get_zookeeper_configs()) return configs def _get_spark_configs(): spark_configs = [] for service, config_items in six.iteritems(c_helper.SPARK_CONFS): for item in config_items['OPTIONS']: cfg = p.Config(name=item["name"], description=item["description"], default_value=item["default"], applicable_target=service, scope="cluster", is_optional=True, priority=item["priority"]) spark_configs.append(cfg) return spark_configs def _get_zookeeper_configs(): zk_configs = [] for service, config_items in six.iteritems(c_helper.ZOOKEEPER_CONFS): for item in config_items['OPTIONS']: cfg = p.Config(name=item["name"], description=item["description"], default_value=item["default"], applicable_target=service, scope="cluster", is_optional=True, priority=item["priority"]) zk_configs.append(cfg) return zk_configs PLUGIN_CONFIGS = _init_all_configs() def get_plugin_configs(): return PLUGIN_CONFIGS def get_xml_configs(): return PLUGIN_XML_CONFIGS def get_env_configs(): return ENV_CONFS sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/__init__.py0000666000175100017510000000000013245514472022457 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/edp_engine.py0000666000175100017510000000663413245514472023040 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import utils as plugin_utils from sahara.plugins.vanilla import confighints_helper as ch_helper from sahara.plugins.vanilla.hadoop2 import edp_engine from sahara.plugins.vanilla import utils as v_utils from sahara.service.edp.spark import engine as edp_spark_engine from sahara.utils import edp class EdpOozieEngine(edp_engine.EdpOozieEngine): @staticmethod def get_possible_job_config(job_type): if edp.compare_job_type(job_type, edp.JOB_TYPE_HIVE): return {'job_config': ch_helper.get_possible_hive_config_from( 'plugins/vanilla/v2_8_2/resources/hive-default.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_MAPREDUCE_STREAMING): return {'job_config': ch_helper.get_possible_mapreduce_config_from( 'plugins/vanilla/v2_8_2/resources/mapred-default.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_PIG): return {'job_config': ch_helper.get_possible_pig_config_from( 'plugins/vanilla/v2_8_2/resources/mapred-default.xml')} return edp_engine.EdpOozieEngine.get_possible_job_config(job_type) class EdpSparkEngine(edp_spark_engine.SparkJobEngine): edp_base_version = "2.8.2" def __init__(self, cluster): super(EdpSparkEngine, self).__init__(cluster) self.master = plugin_utils.get_instance(cluster, "spark history server") self.plugin_params["spark-user"] = "sudo -u hadoop " self.plugin_params["spark-submit"] = os.path.join( plugin_utils.get_config_value_or_default( "Spark", "Spark home", self.cluster), "bin/spark-submit") self.plugin_params["deploy-mode"] = "cluster" self.plugin_params["master"] = "yarn" driver_cp = plugin_utils.get_config_value_or_default( "Spark", "Executor extra classpath", self.cluster) self.plugin_params["driver-class-path"] = driver_cp @staticmethod def edp_supported(version): return version >= EdpSparkEngine.edp_base_version @staticmethod def job_type_supported(job_type): return (job_type in edp_spark_engine.SparkJobEngine.get_supported_job_types()) def validate_job_execution(self, cluster, job, data): if (not self.edp_supported(cluster.hadoop_version) or not v_utils.get_spark_history_server(cluster)): raise ex.InvalidDataException( _('Spark {base} or higher required to run {type} jobs').format( base=EdpSparkEngine.edp_base_version, type=job.type)) super(EdpSparkEngine, self).validate_job_execution(cluster, job, data) sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/0000775000175100017510000000000013245515027022365 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/hive-default.xml0000666000175100017510000076704513245514472025514 0ustar zuulzuul00000000000000 hive.exec.script.wrapper hive.exec.plan hive.exec.stagingdir .hive-staging Directory name that will be created inside table locations in order to support HDFS encryption. This is replaces ${hive.exec.scratchdir} for query results with the exception of read-only tables. In all cases ${hive.exec.scratchdir} is still used for other temporary files, such as job plans. hive.exec.scratchdir /tmp/hive HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}. hive.repl.rootdir /user/hive/repl/ HDFS root dir for all replication dumps. hive.repl.cm.enabled false Turn on ChangeManager, so delete files will go to cmrootdir. hive.repl.cmrootdir /user/hive/cmroot/ Root dir for ChangeManager, used for deleted files. hive.repl.cm.retain 24h Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is hour if not specified. Time to retain removed files in cmrootdir. hive.repl.cm.interval 3600s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Inteval for cmroot cleanup thread. hive.exec.local.scratchdir ${system:java.io.tmpdir}/${system:user.name} Local scratch space for Hive jobs hive.downloaded.resources.dir ${system:java.io.tmpdir}/${hive.session.id}_resources Temporary local directory for added resources in the remote file system. hive.scratch.dir.permission 700 The permission for the user specific scratch directories that get created. hive.exec.submitviachild false hive.exec.submit.local.task.via.child true Determines whether local tasks (typically mapjoin hashtable generation phase) runs in separate JVM (true recommended) or not. Avoids the overhead of spawning new JVM, but can lead to out-of-memory issues. hive.exec.script.maxerrsize 100000 Maximum number of bytes a script is allowed to emit to standard error (per map-reduce task). This prevents runaway scripts from filling logs partitions to capacity hive.exec.script.allow.partial.consumption false When enabled, this option allows a user script to exit successfully without consuming all the data from the standard input. stream.stderr.reporter.prefix reporter: Streaming jobs that log to standard error with this prefix can log counter or status information. stream.stderr.reporter.enabled true Enable consumption of status and counter messages for streaming jobs. hive.exec.compress.output false This controls whether the final outputs of a query (to a local/HDFS file or a Hive table) is compressed. The compression codec and other options are determined from Hadoop config variables mapred.output.compress* hive.exec.compress.intermediate false This controls whether intermediate files produced by Hive between multiple map-reduce jobs are compressed. The compression codec and other options are determined from Hadoop config variables mapred.output.compress* hive.intermediate.compression.codec hive.intermediate.compression.type hive.exec.reducers.bytes.per.reducer 256000000 size per reducer.The default is 256Mb, i.e if the input size is 1G, it will use 4 reducers. hive.exec.reducers.max 1009 max number of reducers will be used. If the one specified in the configuration parameter mapred.reduce.tasks is negative, Hive will use this one as the max number of reducers when automatically determine number of reducers. hive.exec.pre.hooks Comma-separated list of pre-execution hooks to be invoked for each statement. A pre-execution hook is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.exec.post.hooks Comma-separated list of post-execution hooks to be invoked for each statement. A post-execution hook is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.exec.failure.hooks Comma-separated list of on-failure hooks to be invoked for each statement. An on-failure hook is specified as the name of Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.exec.query.redactor.hooks Comma-separated list of hooks to be invoked for each query which can tranform the query before it's placed in the job.xml file. Must be a Java class which extends from the org.apache.hadoop.hive.ql.hooks.Redactor abstract class. hive.client.stats.publishers Comma-separated list of statistics publishers to be invoked on counters on each job. A client stats publisher is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.stats.ClientStatsPublisher interface. hive.ats.hook.queue.capacity 64 Queue size for the ATS Hook executor. If the number of outstanding submissions to the ATS executor exceed this amount, the Hive ATS Hook will not try to log queries to ATS. hive.exec.parallel false Whether to execute jobs in parallel hive.exec.parallel.thread.number 8 How many jobs at most can be executed in parallel hive.mapred.reduce.tasks.speculative.execution true Whether speculative execution for reducers should be turned on. hive.exec.counters.pull.interval 1000 The interval with which to poll the JobTracker for the counters the running job. The smaller it is the more load there will be on the jobtracker, the higher it is the less granular the caught will be. hive.exec.dynamic.partition true Whether or not to allow dynamic partitions in DML/DDL. hive.exec.dynamic.partition.mode strict In strict mode, the user must specify at least one static partition in case the user accidentally overwrites all partitions. In nonstrict mode all partitions are allowed to be dynamic. hive.exec.max.dynamic.partitions 1000 Maximum number of dynamic partitions allowed to be created in total. hive.exec.max.dynamic.partitions.pernode 100 Maximum number of dynamic partitions allowed to be created in each mapper/reducer node. hive.exec.max.created.files 100000 Maximum number of HDFS files created by all mappers/reducers in a MapReduce job. hive.exec.default.partition.name __HIVE_DEFAULT_PARTITION__ The default partition name in case the dynamic partition column value is null/empty string or any other values that cannot be escaped. This value must not contain any special character used in HDFS URI (e.g., ':', '%', '/' etc). The user has to be aware that the dynamic partition value should not contain this value to avoid confusions. hive.lockmgr.zookeeper.default.partition.name __HIVE_DEFAULT_ZOOKEEPER_PARTITION__ hive.exec.show.job.failure.debug.info true If a job fails, whether to provide a link in the CLI to the task with the most failures, along with debugging hints if applicable. hive.exec.job.debug.capture.stacktraces true Whether or not stack traces parsed from the task logs of a sampled failed task for each failed job should be stored in the SessionState hive.exec.job.debug.timeout 30000 hive.exec.tasklog.debug.timeout 20000 hive.output.file.extension String used as a file extension for output files. If not set, defaults to the codec extension for text files (e.g. ".gz"), or no extension otherwise. hive.exec.mode.local.auto false Let Hive determine whether to run in local mode automatically hive.exec.mode.local.auto.inputbytes.max 134217728 When hive.exec.mode.local.auto is true, input bytes should less than this for local mode. hive.exec.mode.local.auto.input.files.max 4 When hive.exec.mode.local.auto is true, the number of tasks should less than this for local mode. hive.exec.drop.ignorenonexistent true Do not report an error if DROP TABLE/VIEW/Index/Function specifies a non-existent table/view/index/function hive.ignore.mapjoin.hint true Ignore the mapjoin hint hive.file.max.footer 100 maximum number of lines for footer user can define for a table file hive.resultset.use.unique.column.names true Make column names unique in the result set by qualifying column names with table alias if needed. Table alias will be added to column names for queries of type "select *" or if query explicitly uses table alias "select r1.x..". fs.har.impl org.apache.hadoop.hive.shims.HiveHarFileSystem The implementation for accessing Hadoop Archives. Note that this won't be applicable to Hadoop versions less than 0.20 hive.metastore.warehouse.dir /user/hive/warehouse location of default database for the warehouse hive.metastore.uris Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore. hive.metastore.client.capability.check true Whether to check client capabilities for potentially breaking API usage. hive.metastore.fastpath false Used to avoid all of the proxies and object copies in the metastore. Note, if this is set, you MUST use a local metastore (hive.metastore.uris must be empty) otherwise undefined and most likely undesired behavior will result hive.metastore.fshandler.threads 15 Number of threads to be allocated for metastore handler for fs operations. hive.metastore.hbase.catalog.cache.size 50000 Maximum number of objects we will place in the hbase metastore catalog cache. The objects will be divided up by types that we need to cache. hive.metastore.hbase.aggregate.stats.cache.size 10000 Maximum number of aggregate stats nodes that we will place in the hbase metastore aggregate stats cache. hive.metastore.hbase.aggregate.stats.max.partitions 10000 Maximum number of partitions that are aggregated per cache node. hive.metastore.hbase.aggregate.stats.false.positive.probability 0.01 Maximum false positive probability for the Bloom Filter used in each aggregate stats cache node (default 1%). hive.metastore.hbase.aggregate.stats.max.variance 0.1 Maximum tolerable variance in number of partitions between a cached node and our request (default 10%). hive.metastore.hbase.cache.ttl 600s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Number of seconds for a cached node to be active in the cache before they become stale. hive.metastore.hbase.cache.max.writer.wait 5000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Number of milliseconds a writer will wait to acquire the writelock before giving up. hive.metastore.hbase.cache.max.reader.wait 1000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Number of milliseconds a reader will wait to acquire the readlock before giving up. hive.metastore.hbase.cache.max.full 0.9 Maximum cache full % after which the cache cleaner thread kicks in. hive.metastore.hbase.cache.clean.until 0.8 The cleaner thread cleans until cache reaches this % full size. hive.metastore.hbase.connection.class org.apache.hadoop.hive.metastore.hbase.VanillaHBaseConnection Class used to connection to HBase hive.metastore.hbase.aggr.stats.cache.entries 10000 How many in stats objects to cache in memory hive.metastore.hbase.aggr.stats.memory.ttl 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Number of seconds stats objects live in memory after they are read from HBase. hive.metastore.hbase.aggr.stats.invalidator.frequency 5s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. How often the stats cache scans its HBase entries and looks for expired entries hive.metastore.hbase.aggr.stats.hbase.ttl 604800s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Number of seconds stats entries live in HBase cache after they are created. They may be invalided by updates or partition drops before this. Default is one week. hive.metastore.hbase.file.metadata.threads 1 Number of threads to use to read file metadata in background to cache it. hive.metastore.connect.retries 3 Number of retries while opening a connection to metastore hive.metastore.failure.retries 1 Number of retries upon failure of Thrift metastore calls hive.metastore.port 9083 Hive metastore listener port hive.metastore.client.connect.retry.delay 1s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Number of seconds for the client to wait between consecutive connection attempts hive.metastore.client.socket.timeout 600s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. MetaStore Client socket timeout in seconds hive.metastore.client.socket.lifetime 0s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. MetaStore Client socket lifetime in seconds. After this time is exceeded, client reconnects on the next MetaStore operation. A value of 0s means the connection has an infinite lifetime. javax.jdo.option.ConnectionPassword mine password to use against metastore database hive.metastore.ds.connection.url.hook Name of the hook to use for retrieving the JDO connection URL. If empty, the value in javax.jdo.option.ConnectionURL is used javax.jdo.option.Multithreaded true Set this to true if multiple threads access metastore through JDO concurrently. javax.jdo.option.ConnectionURL jdbc:derby:;databaseName=metastore_db;create=true JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/db?ssl=true for postgres database. hive.metastore.dbaccess.ssl.properties Comma-separated SSL properties for metastore to access database when JDO connection URL enables SSL access. e.g. javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd. hive.hmshandler.retry.attempts 10 The number of times to retry a HMSHandler call if there were a connection error. hive.hmshandler.retry.interval 2000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. The time between HMSHandler retry attempts on failure. hive.hmshandler.force.reload.conf false Whether to force reloading of the HMSHandler configuration (including the connection URL, before the next metastore query that accesses the datastore. Once reloaded, this value is reset to false. Used for testing only. hive.metastore.server.max.message.size 104857600 Maximum message size in bytes a HMS will accept. hive.metastore.server.min.threads 200 Minimum number of worker threads in the Thrift server's pool. hive.metastore.server.max.threads 1000 Maximum number of worker threads in the Thrift server's pool. hive.metastore.server.tcp.keepalive true Whether to enable TCP keepalive for the metastore server. Keepalive will prevent accumulation of half-open connections. hive.metastore.archive.intermediate.original _INTERMEDIATE_ORIGINAL Intermediate dir suffixes used for archiving. Not important what they are, as long as collisions are avoided hive.metastore.archive.intermediate.archived _INTERMEDIATE_ARCHIVED hive.metastore.archive.intermediate.extracted _INTERMEDIATE_EXTRACTED hive.metastore.kerberos.keytab.file The path to the Kerberos Keytab file containing the metastore Thrift server's service principal. hive.metastore.kerberos.principal hive-metastore/_HOST@EXAMPLE.COM The service principal for the metastore Thrift server. The special string _HOST will be replaced automatically with the correct host name. hive.metastore.sasl.enabled false If true, the metastore Thrift interface will be secured with SASL. Clients must authenticate with Kerberos. hive.metastore.thrift.framed.transport.enabled false If true, the metastore Thrift interface will use TFramedTransport. When false (default) a standard TTransport is used. hive.metastore.thrift.compact.protocol.enabled false If true, the metastore Thrift interface will use TCompactProtocol. When false (default) TBinaryProtocol will be used. Setting it to true will break compatibility with older clients running TBinaryProtocol. hive.metastore.token.signature The delegation token service name to match when selecting a token from the current user's tokens. hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.MemoryTokenStore The delegation token store implementation. Set to org.apache.hadoop.hive.thrift.ZooKeeperTokenStore for load-balanced cluster. hive.cluster.delegation.token.store.zookeeper.connectString The ZooKeeper token store connect string. You can re-use the configuration value set in hive.zookeeper.quorum, by leaving this parameter unset. hive.cluster.delegation.token.store.zookeeper.znode /hivedelegation The root path for token store data. Note that this is used by both HiveServer2 and MetaStore to store delegation Token. One directory gets created for each of them. The final directory names would have the servername appended to it (HIVESERVER2, METASTORE). hive.cluster.delegation.token.store.zookeeper.acl ACL for token store entries. Comma separated list of ACL entries. For example: sasl:hive/host1@MY.DOMAIN:cdrwa,sasl:hive/host2@MY.DOMAIN:cdrwa Defaults to all permissions for the hiveserver2/metastore process user. hive.metastore.cache.pinobjtypes Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order List of comma separated metastore object types that should be pinned in the cache datanucleus.connectionPoolingType BONECP Expects one of [bonecp, dbcp, hikaricp, none]. Specify connection pool library for datanucleus datanucleus.connectionPool.maxPoolSize 10 Specify the maximum number of connections in the connection pool. Note: The configured size will be used by 2 connection pools (TxnHandler and ObjectStore). When configuring the max connection pool size, it is recommended to take into account the number of metastore instances and the number of HiveServer2 instances configured with embedded metastore. To get optimal performance, set config to meet the following condition (2 * pool_size * metastore_instances + 2 * pool_size * HS2_instances_with_embedded_metastore) = (2 * physical_core_count + hard_disk_count). datanucleus.rdbms.initializeColumnInfo NONE initializeColumnInfo setting for DataNucleus; set to NONE at least on Postgres. datanucleus.schema.validateTables false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.schema.validateColumns false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.schema.validateConstraints false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.storeManagerType rdbms metadata store type datanucleus.schema.autoCreateAll false Auto creates necessary schema on a startup if one doesn't exist. Set this to false, after creating it once.To enable auto create also set hive.metastore.schema.verification=false. Auto creation is not recommended for production use cases, run schematool command instead. hive.metastore.schema.verification true Enforce metastore schema version consistency. True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures proper metastore schema migration. (Default) False: Warn if the version information stored in metastore doesn't match with one from in Hive jars. hive.metastore.schema.verification.record.version false When true the current MS version is recorded in the VERSION table. If this is disabled and verification is enabled the MS will be unusable. datanucleus.transactionIsolation read-committed Default transaction isolation level for identity generation. datanucleus.cache.level2 false Use a level 2 cache. Turn this off if metadata is changed independently of Hive metastore server datanucleus.cache.level2.type none datanucleus.identifierFactory datanucleus1 Name of the identifier factory to use when generating table/column names etc. 'datanucleus1' is used for backward compatibility with DataNucleus v1 datanucleus.rdbms.useLegacyNativeValueStrategy true datanucleus.plugin.pluginRegistryBundleCheck LOG Defines what happens when plugin bundles are found and are duplicated [EXCEPTION|LOG|NONE] hive.metastore.batch.retrieve.max 300 Maximum number of objects (tables/partitions) can be retrieved from metastore in one batch. The higher the number, the less the number of round trips is needed to the Hive metastore server, but it may also cause higher memory requirement at the client side. hive.metastore.batch.retrieve.table.partition.max 1000 Maximum number of objects that metastore internally retrieves in one batch. hive.metastore.init.hooks A comma separated list of hooks to be invoked at the beginning of HMSHandler initialization. An init hook is specified as the name of Java class which extends org.apache.hadoop.hive.metastore.MetaStoreInitListener. hive.metastore.pre.event.listeners List of comma separated listeners for metastore events. hive.metastore.event.listeners A comma separated list of Java classes that implement the org.apache.hadoop.hive.metastore.MetaStoreEventListener interface. The metastore event and corresponding listener method will be invoked in separate JDO transactions. Alternatively, configure hive.metastore.transactional.event.listeners to ensure both are invoked in same JDO transaction. hive.metastore.transactional.event.listeners A comma separated list of Java classes that implement the org.apache.hadoop.hive.metastore.MetaStoreEventListener interface. Both the metastore event and corresponding listener method will be invoked in the same JDO transaction. hive.metastore.event.db.listener.timetolive 86400s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. time after which events will be removed from the database listener queue hive.metastore.authorization.storage.checks false Should the metastore do authorization checks against the underlying storage (usually hdfs) for operations like drop-partition (disallow the drop-partition if the user in question doesn't have permissions to delete the corresponding directory on the storage). hive.metastore.authorization.storage.check.externaltable.drop true Should StorageBasedAuthorization check permission of the storage before dropping external table. StorageBasedAuthorization already does this check for managed table. For external table however, anyone who has read permission of the directory could drop external table, which is surprising. The flag is set to false by default to maintain backward compatibility. hive.metastore.event.clean.freq 0s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Frequency at which timer task runs to purge expired events in metastore. hive.metastore.event.expiry.duration 0s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Duration after which events expire from events table hive.metastore.event.message.factory org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory Factory class for making encoding and decoding messages in the events generated. hive.metastore.execute.setugi true In unsecure mode, setting this property to true will cause the metastore to execute DFS operations using the client's reported user and group permissions. Note that this property must be set on both the client and server sides. Further note that its best effort. If client sets its to true and server sets it to false, client setting will be ignored. hive.metastore.partition.name.whitelist.pattern Partition names will be checked against this regex pattern and rejected if not matched. hive.metastore.integral.jdo.pushdown false Allow JDO query pushdown for integral partition columns in metastore. Off by default. This improves metastore perf for integral columns, especially if there's a large number of partitions. However, it doesn't work correctly with integral values that are not normalized (e.g. have leading zeroes, like 0012). If metastore direct SQL is enabled and works, this optimization is also irrelevant. hive.metastore.try.direct.sql true Whether the Hive metastore should try to use direct SQL queries instead of the DataNucleus for certain read paths. This can improve metastore performance when fetching many partitions or column statistics by orders of magnitude; however, it is not guaranteed to work on all RDBMS-es and all versions. In case of SQL failures, the metastore will fall back to the DataNucleus, so it's safe even if SQL doesn't work for all queries on your datastore. If all SQL queries fail (for example, your metastore is backed by MongoDB), you might want to disable this to save the try-and-fall-back cost. hive.metastore.direct.sql.batch.size 0 Batch size for partition and other object retrieval from the underlying DB in direct SQL. For some DBs like Oracle and MSSQL, there are hardcoded or perf-based limitations that necessitate this. For DBs that can handle the queries, this isn't necessary and may impede performance. -1 means no batching, 0 means automatic batching. hive.metastore.try.direct.sql.ddl true Same as hive.metastore.try.direct.sql, for read statements within a transaction that modifies metastore data. Due to non-standard behavior in Postgres, if a direct SQL select query has incorrect syntax or something similar inside a transaction, the entire transaction will fail and fall-back to DataNucleus will not be possible. You should disable the usage of direct SQL inside transactions if that happens in your case. hive.direct.sql.max.query.length 100 The maximum size of a query string (in KB). hive.direct.sql.max.elements.in.clause 1000 The maximum number of values in a IN clause. Once exceeded, it will be broken into multiple OR separated IN clauses. hive.direct.sql.max.elements.values.clause 1000 The maximum number of values in a VALUES clause for INSERT statement. hive.metastore.orm.retrieveMapNullsAsEmptyStrings false Thrift does not support nulls in maps, so any nulls present in maps retrieved from ORM must either be pruned or converted to empty strings. Some backing dbs such as Oracle persist empty strings as nulls, so we should set this parameter if we wish to reverse that behaviour. For others, pruning is the correct behaviour hive.metastore.disallow.incompatible.col.type.changes true If true (default is false), ALTER TABLE operations which change the type of a column (say STRING) to an incompatible type (say MAP) are disallowed. RCFile default SerDe (ColumnarSerDe) serializes the values in such a way that the datatypes can be converted from string to any type. The map is also serialized as a string, which can be read as a string as well. However, with any binary serialization, this is not true. Blocking the ALTER TABLE prevents ClassCastExceptions when subsequently trying to access old partitions. Primitive types like INT, STRING, BIGINT, etc., are compatible with each other and are not blocked. See HIVE-4409 for more details. hive.metastore.limit.partition.request -1 This limits the number of partitions that can be requested from the metastore for a given table. The default value "-1" means no limit. hive.table.parameters.default Default property values for newly created tables hive.ddl.createtablelike.properties.whitelist Table Properties to copy over when executing a Create Table Like. hive.metastore.rawstore.impl org.apache.hadoop.hive.metastore.ObjectStore Name of the class that implements org.apache.hadoop.hive.metastore.rawstore interface. This class is used to store and retrieval of raw metadata objects such as table, database hive.metastore.txn.store.impl org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler Name of class that implements org.apache.hadoop.hive.metastore.txn.TxnStore. This class is used to store and retrieve transactions and locks javax.jdo.option.ConnectionDriverName org.apache.derby.jdbc.EmbeddedDriver Driver class name for a JDBC metastore javax.jdo.PersistenceManagerFactoryClass org.datanucleus.api.jdo.JDOPersistenceManagerFactory class implementing the jdo persistence hive.metastore.expression.proxy org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore javax.jdo.option.DetachAllOnCommit true Detaches all objects from session so that they can be used after transaction is committed javax.jdo.option.NonTransactionalRead true Reads outside of transactions javax.jdo.option.ConnectionUserName APP Username to use against metastore database hive.metastore.end.function.listeners List of comma separated listeners for the end of metastore functions. hive.metastore.partition.inherit.table.properties List of comma separated keys occurring in table properties which will get inherited to newly created partitions. * implies all the keys will get inherited. hive.metastore.filter.hook org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl Metastore hook class for filtering the metadata read results. If hive.security.authorization.manageris set to instance of HiveAuthorizerFactory, then this value is ignored. hive.metastore.dml.events false If true, the metastore will be asked to fire events for DML operations hive.metastore.client.drop.partitions.using.expressions true Choose whether dropping partitions with HCatClient pushes the partition-predicate to the metastore, or drops partitions iteratively hive.metastore.aggregate.stats.cache.enabled true Whether aggregate stats caching is enabled or not. hive.metastore.aggregate.stats.cache.size 10000 Maximum number of aggregate stats nodes that we will place in the metastore aggregate stats cache. hive.metastore.aggregate.stats.cache.max.partitions 10000 Maximum number of partitions that are aggregated per cache node. hive.metastore.aggregate.stats.cache.fpp 0.01 Maximum false positive probability for the Bloom Filter used in each aggregate stats cache node (default 1%). hive.metastore.aggregate.stats.cache.max.variance 0.01 Maximum tolerable variance in number of partitions between a cached node and our request (default 1%). hive.metastore.aggregate.stats.cache.ttl 600s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Number of seconds for a cached node to be active in the cache before they become stale. hive.metastore.aggregate.stats.cache.max.writer.wait 5000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Number of milliseconds a writer will wait to acquire the writelock before giving up. hive.metastore.aggregate.stats.cache.max.reader.wait 1000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Number of milliseconds a reader will wait to acquire the readlock before giving up. hive.metastore.aggregate.stats.cache.max.full 0.9 Maximum cache full % after which the cache cleaner thread kicks in. hive.metastore.aggregate.stats.cache.clean.until 0.8 The cleaner thread cleans until cache reaches this % full size. hive.metastore.metrics.enabled false Enable metrics on the metastore. hive.metastore.initial.metadata.count.enabled true Enable a metadata count at metastore startup for metrics. hive.metastore.use.SSL false Set this to true for using SSL encryption in HMS server. hive.metastore.keystore.path Metastore SSL certificate keystore location. hive.metastore.keystore.password Metastore SSL certificate keystore password. hive.metastore.truststore.path Metastore SSL certificate truststore location. hive.metastore.truststore.password Metastore SSL certificate truststore password. hive.metadata.export.location When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, it is the location to which the metadata will be exported. The default is an empty string, which results in the metadata being exported to the current user's home directory on HDFS. hive.metadata.move.exported.metadata.to.trash true When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, this setting determines if the metadata that is exported will subsequently be moved to the user's trash directory alongside the dropped table data. This ensures that the metadata will be cleaned up along with the dropped table data. hive.cli.errors.ignore false hive.cli.print.current.db false Whether to include the current database in the Hive prompt. hive.cli.prompt hive Command line prompt configuration value. Other hiveconf can be used in this configuration value. Variable substitution will only be invoked at the Hive CLI startup. hive.cli.pretty.output.num.cols -1 The number of columns to use when formatting output generated by the DESCRIBE PRETTY table_name command. If the value of this property is -1, then Hive will use the auto-detected terminal width. hive.metastore.fs.handler.class org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl hive.session.id hive.session.silent false hive.session.history.enabled false Whether to log Hive query, query plan, runtime statistics etc. hive.query.string Query being executed (might be multiple per a session) hive.query.id ID for query being executed (might be multiple per a session) hive.jobname.length 50 max jobname length hive.jar.path The location of hive_cli.jar that is used when submitting jobs in a separate jvm. hive.aux.jars.path The location of the plugin jars that contain implementations of user defined functions and serdes. hive.reloadable.aux.jars.path The locations of the plugin jars, which can be a comma-separated folders or jars. Jars can be renewed by executing reload command. And these jars can be used as the auxiliary classes like creating a UDF or SerDe. hive.added.files.path This an internal parameter. hive.added.jars.path This an internal parameter. hive.added.archives.path This an internal parameter. hive.auto.progress.timeout 0s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. How long to run autoprogressor for the script/UDTF operators. Set to 0 for forever. hive.script.auto.progress false Whether Hive Transform/Map/Reduce Clause should automatically send progress information to TaskTracker to avoid the task getting killed because of inactivity. Hive sends progress information when the script is outputting to stderr. This option removes the need of periodically producing stderr messages, but users should be cautious because this may prevent infinite loops in the scripts to be killed by TaskTracker. hive.script.operator.id.env.var HIVE_SCRIPT_OPERATOR_ID Name of the environment variable that holds the unique script operator ID in the user's transform function (the custom mapper/reducer that the user has specified in the query) hive.script.operator.truncate.env false Truncate each environment variable for external script in scripts operator to 20KB (to fit system limits) hive.script.operator.env.blacklist hive.txn.valid.txns,hive.script.operator.env.blacklist Comma separated list of keys from the configuration file not to convert to environment variables when envoking the script operator hive.strict.checks.large.query false Enabling strict large query checks disallows the following: Orderby without limit. No partition being picked up for a query against partitioned table. Note that these checks currently do not consider data size, only the query pattern. hive.strict.checks.type.safety true Enabling strict type safety checks disallows the following: Comparing bigints and strings. Comparing bigints and doubles. hive.strict.checks.cartesian.product true Enabling strict Cartesian join checks disallows the following: Cartesian product (cross join). hive.strict.checks.bucketing true Enabling strict bucketing checks disallows the following: Load into bucketed tables. hive.mapred.mode Deprecated; use hive.strict.checks.* settings instead. hive.alias hive.map.aggr true Whether to use map-side aggregation in Hive Group By queries hive.groupby.skewindata false Whether there is skew in data to optimize group by queries hive.join.emit.interval 1000 How many rows in the right-most join operand Hive should buffer before emitting the join result. hive.join.cache.size 25000 How many rows in the joining tables (except the streaming table) should be cached in memory. hive.cbo.enable true Flag to control enabling Cost Based Optimizations using Calcite framework. hive.cbo.cnf.maxnodes -1 When converting to conjunctive normal form (CNF), fail ifthe expression exceeds this threshold; the threshold is expressed in terms of number of nodes (leaves andinterior nodes). -1 to not set up a threshold. hive.cbo.returnpath.hiveop false Flag to control calcite plan to hive operator conversion hive.cbo.costmodel.extended false Flag to control enabling the extended cost model based onCPU, IO and cardinality. Otherwise, the cost model is based on cardinality. hive.cbo.costmodel.cpu 0.000001 Default cost of a comparison hive.cbo.costmodel.network 150.0 Default cost of a transfering a byte over network; expressed as multiple of CPU cost hive.cbo.costmodel.local.fs.write 4.0 Default cost of writing a byte to local FS; expressed as multiple of NETWORK cost hive.cbo.costmodel.local.fs.read 4.0 Default cost of reading a byte from local FS; expressed as multiple of NETWORK cost hive.cbo.costmodel.hdfs.write 10.0 Default cost of writing a byte to HDFS; expressed as multiple of Local FS write cost hive.cbo.costmodel.hdfs.read 1.5 Default cost of reading a byte from HDFS; expressed as multiple of Local FS read cost hive.cbo.show.warnings true Toggle display of CBO warnings like missing column stats hive.transpose.aggr.join false push aggregates through join hive.optimize.semijoin.conversion true convert group by followed by inner equi join into semijoin hive.order.columnalignment true Flag to control whether we want to try to aligncolumns in operators such as Aggregate or Join so that we try to reduce the number of shuffling stages hive.materializedview.rewriting false Whether to try to rewrite queries using the materialized views enabled for rewriting hive.materializedview.fileformat ORC Expects one of [none, textfile, sequencefile, rcfile, orc]. Default file format for CREATE MATERIALIZED VIEW statement hive.materializedview.serde org.apache.hadoop.hive.ql.io.orc.OrcSerde Default SerDe used for materialized views hive.mapjoin.bucket.cache.size 100 hive.mapjoin.optimized.hashtable true Whether Hive should use memory-optimized hash table for MapJoin. Only works on Tez and Spark, because memory-optimized hashtable cannot be serialized. hive.mapjoin.optimized.hashtable.probe.percent 0.5 Probing space percentage of the optimized hashtable hive.mapjoin.hybridgrace.hashtable true Whether to use hybridgrace hash join as the join method for mapjoin. Tez only. hive.mapjoin.hybridgrace.memcheckfrequency 1024 For hybrid grace hash join, how often (how many rows apart) we check if memory is full. This number should be power of 2. hive.mapjoin.hybridgrace.minwbsize 524288 For hybrid graceHash join, the minimum write buffer size used by optimized hashtable. Default is 512 KB. hive.mapjoin.hybridgrace.minnumpartitions 16 ForHybrid grace hash join, the minimum number of partitions to create. hive.mapjoin.optimized.hashtable.wbsize 8388608 Optimized hashtable (see hive.mapjoin.optimized.hashtable) uses a chain of buffers to store data. This is one buffer size. HT may be slightly faster if this is larger, but for small joins unnecessary memory will be allocated and then trimmed. hive.mapjoin.hybridgrace.bloomfilter true Whether to use BloomFilter in Hybrid grace hash join to minimize unnecessary spilling. hive.smbjoin.cache.rows 10000 How many rows with the same key value should be cached in memory per smb joined table. hive.groupby.mapaggr.checkinterval 100000 Number of rows after which size of the grouping keys/aggregation classes is performed hive.map.aggr.hash.percentmemory 0.5 Portion of total memory to be used by map-side group aggregation hash table hive.mapjoin.followby.map.aggr.hash.percentmemory 0.3 Portion of total memory to be used by map-side group aggregation hash table, when this group by is followed by map join hive.map.aggr.hash.force.flush.memory.threshold 0.9 The max memory to be used by map-side group aggregation hash table. If the memory usage is higher than this number, force to flush data hive.map.aggr.hash.min.reduction 0.5 Hash aggregation will be turned off if the ratio between hash table size and input rows is bigger than this number. Set to 1 to make sure hash aggregation is never turned off. hive.multigroupby.singlereducer true Whether to optimize multi group by query to generate single M/R job plan. If the multi group by query has common group by keys, it will be optimized to generate single M/R job. hive.map.groupby.sorted true If the bucketing/sorting properties of the table exactly match the grouping key, whether to perform the group by in the mapper by using BucketizedHiveInputFormat. The only downside to this is that it limits the number of mappers to the number of files. hive.groupby.position.alias false Whether to enable using Column Position Alias in Group By hive.orderby.position.alias true Whether to enable using Column Position Alias in Order By hive.groupby.orderby.position.alias false Whether to enable using Column Position Alias in Group By or Order By (deprecated). Use hive.orderby.position.alias or hive.groupby.position.alias instead hive.new.job.grouping.set.cardinality 30 Whether a new map-reduce job should be launched for grouping sets/rollups/cubes. For a query like: select a, b, c, count(1) from T group by a, b, c with rollup; 4 rows are created per row: (a, b, c), (a, b, null), (a, null, null), (null, null, null). This can lead to explosion across map-reduce boundary if the cardinality of T is very high, and map-side aggregation does not do a very good job. This parameter decides if Hive should add an additional map-reduce job. If the grouping set cardinality (4 in the example above), is more than this value, a new MR job is added under the assumption that the original group by will reduce the data size. hive.groupby.limit.extrastep true This parameter decides if Hive should create new MR job for sorting final output hive.exec.copyfile.maxnumfiles 1 Maximum number of files Hive uses to do sequential HDFS copies between directories.Distributed copies (distcp) will be used instead for larger numbers of files so that copies can be done faster. hive.exec.copyfile.maxsize 33554432 Maximum file size (in bytes) that Hive uses to do single HDFS copies between directories.Distributed copies (distcp) will be used instead for bigger files so that copies can be done faster. hive.udtf.auto.progress false Whether Hive should automatically send progress information to TaskTracker when using UDTF's to prevent the task getting killed because of inactivity. Users should be cautious because this may prevent TaskTracker from killing tasks with infinite loops. hive.default.fileformat TextFile Expects one of [textfile, sequencefile, rcfile, orc, parquet]. Default file format for CREATE TABLE statement. Users can explicitly override it by CREATE TABLE ... STORED AS [FORMAT] hive.default.fileformat.managed none Expects one of [none, textfile, sequencefile, rcfile, orc, parquet]. Default file format for CREATE TABLE statement applied to managed tables only. External tables will be created with format specified by hive.default.fileformat. Leaving this null will result in using hive.default.fileformat for all tables. hive.query.result.fileformat SequenceFile Expects one of [textfile, sequencefile, rcfile, llap]. Default file format for storing result of the query. hive.fileformat.check true Whether to check file format or not when loading data files hive.default.rcfile.serde org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe The default SerDe Hive will use for the RCFile format hive.default.serde org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe The default SerDe Hive will use for storage formats that do not specify a SerDe. hive.serdes.using.metastore.for.schema org.apache.hadoop.hive.ql.io.orc.OrcSerde,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe,org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe,org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe,org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe,org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe SerDes retrieving schema from metastore. This is an internal parameter. hive.querylog.location ${system:java.io.tmpdir}/${system:user.name} Location of Hive run time structured log file hive.querylog.enable.plan.progress true Whether to log the plan's progress every time a job's progress is checked. These logs are written to the location specified by hive.querylog.location hive.querylog.plan.progress.interval 60000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. The interval to wait between logging the plan's progress. If there is a whole number percentage change in the progress of the mappers or the reducers, the progress is logged regardless of this value. The actual interval will be the ceiling of (this value divided by the value of hive.exec.counters.pull.interval) multiplied by the value of hive.exec.counters.pull.interval I.e. if it is not divide evenly by the value of hive.exec.counters.pull.interval it will be logged less frequently than specified. This only has an effect if hive.querylog.enable.plan.progress is set to true. hive.script.serde org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe The default SerDe for transmitting input data to and reading output data from the user scripts. hive.script.recordreader org.apache.hadoop.hive.ql.exec.TextRecordReader The default record reader for reading data from the user scripts. hive.script.recordwriter org.apache.hadoop.hive.ql.exec.TextRecordWriter The default record writer for writing data to the user scripts. hive.transform.escape.input false This adds an option to escape special chars (newlines, carriage returns and tabs) when they are passed to the user script. This is useful if the Hive tables can contain data that contains special characters. hive.binary.record.max.length 1000 Read from a binary stream and treat each hive.binary.record.max.length bytes as a record. The last record before the end of stream can have less than hive.binary.record.max.length bytes hive.mapred.local.mem 0 mapper/reducer memory in local mode hive.mapjoin.smalltable.filesize 25000000 The threshold for the input file size of the small tables; if the file size is smaller than this threshold, it will try to convert the common join into map join hive.exec.schema.evolution true Use schema evolution to convert self-describing file format's data to the schema desired by the reader. hive.transactional.events.mem 10000000 Vectorized ACID readers can often load all the delete events from all the delete deltas into memory to optimize for performance. To prevent out-of-memory errors, this is a rough heuristic that limits the total number of delete events that can be loaded into memory at once. Roughly it has been set to 10 million delete events per bucket (~160 MB). hive.sample.seednumber 0 A number used to percentage sampling. By changing this number, user will change the subsets of data sampled. hive.test.mode false Whether Hive is running in test mode. If yes, it turns on sampling and prefixes the output tablename. hive.test.mode.prefix test_ In test mode, specfies prefixes for the output table hive.test.mode.samplefreq 32 In test mode, specfies sampling frequency for table, which is not bucketed, For example, the following query: INSERT OVERWRITE TABLE dest SELECT col1 from src would be converted to INSERT OVERWRITE TABLE test_dest SELECT col1 from src TABLESAMPLE (BUCKET 1 out of 32 on rand(1)) hive.test.mode.nosamplelist In test mode, specifies comma separated table names which would not apply sampling hive.test.dummystats.aggregator internal variable for test hive.test.dummystats.publisher internal variable for test hive.test.currenttimestamp current timestamp for test hive.test.rollbacktxn false For testing only. Will mark every ACID transaction aborted hive.test.fail.compaction false For testing only. Will cause CompactorMR to fail. hive.test.fail.heartbeater false For testing only. Will cause Heartbeater to fail. hive.merge.mapfiles true Merge small files at the end of a map-only job hive.merge.mapredfiles false Merge small files at the end of a map-reduce job hive.merge.tezfiles false Merge small files at the end of a Tez DAG hive.merge.sparkfiles false Merge small files at the end of a Spark DAG Transformation hive.merge.size.per.task 256000000 Size of merged files at the end of the job hive.merge.smallfiles.avgsize 16000000 When the average output file size of a job is less than this number, Hive will start an additional map-reduce job to merge the output files into bigger files. This is only done for map-only jobs if hive.merge.mapfiles is true, and for map-reduce jobs if hive.merge.mapredfiles is true. hive.merge.rcfile.block.level true hive.merge.orcfile.stripe.level true When hive.merge.mapfiles, hive.merge.mapredfiles or hive.merge.tezfiles is enabled while writing a table with ORC file format, enabling this config will do stripe-level fast merge for small ORC files. Note that enabling this config will not honor the padding tolerance config (hive.exec.orc.block.padding.tolerance). hive.exec.rcfile.use.explicit.header true If this is set the header for RCFiles will simply be RCF. If this is not set the header will be that borrowed from sequence files, e.g. SEQ- followed by the input and output RCFile formats. hive.exec.rcfile.use.sync.cache true hive.io.rcfile.record.interval 2147483647 hive.io.rcfile.column.number.conf 0 hive.io.rcfile.tolerate.corruptions false hive.io.rcfile.record.buffer.size 4194304 parquet.memory.pool.ratio 0.5 Maximum fraction of heap that can be used by Parquet file writers in one task. It is for avoiding OutOfMemory error in tasks. Work with Parquet 1.6.0 and above. This config parameter is defined in Parquet, so that it does not start with 'hive.'. hive.parquet.timestamp.skip.conversion true Current Hive implementation of parquet stores timestamps to UTC, this flag allows skipping of the conversionon reading parquet files from other tools hive.int.timestamp.conversion.in.seconds false Boolean/tinyint/smallint/int/bigint value is interpreted as milliseconds during the timestamp conversion. Set this flag to true to interpret the value as seconds to be consistent with float/double. hive.exec.orc.base.delta.ratio 8 The ratio of base writer and delta writer in terms of STRIPE_SIZE and BUFFER_SIZE. hive.exec.orc.split.strategy HYBRID Expects one of [hybrid, bi, etl]. This is not a user level config. BI strategy is used when the requirement is to spend less time in split generation as opposed to query execution (split generation does not read or cache file footers). ETL strategy is used when spending little more time in split generation is acceptable (split generation reads and caches file footers). HYBRID chooses between the above strategies based on heuristics. hive.orc.splits.ms.footer.cache.enabled false Whether to enable using file metadata cache in metastore for ORC file footers. hive.orc.splits.ms.footer.cache.ppd.enabled true Whether to enable file footer cache PPD (hive.orc.splits.ms.footer.cache.enabled must also be set to true for this to work). hive.orc.splits.include.file.footer false If turned on splits generated by orc will include metadata about the stripes in the file. This data is read remotely (from the client or HS2 machine) and sent to all the tasks. hive.orc.splits.directory.batch.ms 0 How long, in ms, to wait to batch input directories for processing during ORC split generation. 0 means process directories individually. This can increase the number of metastore calls if metastore metadata cache is used. hive.orc.splits.include.fileid true Include file ID in splits on file systems that support it. hive.orc.splits.allow.synthetic.fileid true Allow synthetic file ID in splits on file systems that don't have a native one. hive.orc.cache.stripe.details.mem.size 256Mb Expects a byte size value with unit (blank for bytes, kb, mb, gb, tb, pb). Maximum size of orc splits cached in the client. hive.orc.compute.splits.num.threads 10 How many threads orc should use to create splits in parallel. hive.orc.cache.use.soft.references false By default, the cache that ORC input format uses to store orc file footer use hard references for the cached object. Setting this to true can help avoid out of memory issues under memory pressure (in some cases) at the cost of slight unpredictability in overall query performance. hive.io.sarg.cache.max.weight.mb 10 The max weight allowed for the SearchArgument Cache. By default, the cache allows a max-weight of 10MB, after which entries will be evicted. hive.lazysimple.extended_boolean_literal false LazySimpleSerde uses this property to determine if it treats 'T', 't', 'F', 'f', '1', and '0' as extened, legal boolean literal, in addition to 'TRUE' and 'FALSE'. The default is false, which means only 'TRUE' and 'FALSE' are treated as legal boolean literal. hive.optimize.skewjoin false Whether to enable skew join optimization. The algorithm is as follows: At runtime, detect the keys with a large skew. Instead of processing those keys, store them temporarily in an HDFS directory. In a follow-up map-reduce job, process those skewed keys. The same key need not be skewed for all the tables, and so, the follow-up map-reduce job (for the skewed keys) would be much faster, since it would be a map-join. hive.optimize.dynamic.partition.hashjoin false Whether to enable dynamically partitioned hash join optimization. This setting is also dependent on enabling hive.auto.convert.join hive.auto.convert.join true Whether Hive enables the optimization about converting common join into mapjoin based on the input file size hive.auto.convert.join.noconditionaltask true Whether Hive enables the optimization about converting common join into mapjoin based on the input file size. If this parameter is on, and the sum of size for n-1 of the tables/partitions for a n-way join is smaller than the specified size, the join is directly converted to a mapjoin (there is no conditional task). hive.auto.convert.join.noconditionaltask.size 10000000 If hive.auto.convert.join.noconditionaltask is off, this parameter does not take affect. However, if it is on, and the sum of size for n-1 of the tables/partitions for a n-way join is smaller than this size, the join is directly converted to a mapjoin(there is no conditional task). The default is 10MB hive.auto.convert.join.use.nonstaged false For conditional joins, if input stream from a small alias can be directly applied to join operator without filtering or projection, the alias need not to be pre-staged in distributed cache via mapred local task. Currently, this is not working with vectorization or tez execution engine. hive.skewjoin.key 100000 Determine if we get a skew key in join. If we see more than the specified number of rows with the same key in join operator, we think the key as a skew join key. hive.skewjoin.mapjoin.map.tasks 10000 Determine the number of map task used in the follow up map join job for a skew join. It should be used together with hive.skewjoin.mapjoin.min.split to perform a fine grained control. hive.skewjoin.mapjoin.min.split 33554432 Determine the number of map task at most used in the follow up map join job for a skew join by specifying the minimum split size. It should be used together with hive.skewjoin.mapjoin.map.tasks to perform a fine grained control. hive.heartbeat.interval 1000 Send a heartbeat after this interval - used by mapjoin and filter operators hive.limit.row.max.size 100000 When trying a smaller subset of data for simple LIMIT, how much size we need to guarantee each row to have at least. hive.limit.optimize.limit.file 10 When trying a smaller subset of data for simple LIMIT, maximum number of files we can sample. hive.limit.optimize.enable false Whether to enable to optimization to trying a smaller subset of data for simple LIMIT first. hive.limit.optimize.fetch.max 50000 Maximum number of rows allowed for a smaller subset of data for simple LIMIT, if it is a fetch query. Insert queries are not restricted by this limit. hive.limit.pushdown.memory.usage 0.1 Expects value between 0.0f and 1.0f. The fraction of available memory to be used for buffering rows in Reducesink operator for limit pushdown optimization. hive.limit.query.max.table.partition -1 This controls how many partitions can be scanned for each partitioned table. The default value "-1" means no limit. (DEPRECATED: Please use hive.metastore.limit.partition.request in the metastore instead.) hive.auto.convert.join.hashtable.max.entries 40000000 If hive.auto.convert.join.noconditionaltask is off, this parameter does not take affect. However, if it is on, and the predicated number of entries in hashtable for a given join input is larger than this number, the join will not be converted to a mapjoin. The value "-1" means no limit. hive.hashtable.key.count.adjustment 1.0 Adjustment to mapjoin hashtable size derived from table and column statistics; the estimate of the number of keys is divided by this value. If the value is 0, statistics are not usedand hive.hashtable.initialCapacity is used instead. hive.hashtable.initialCapacity 100000 Initial capacity of mapjoin hashtable if statistics are absent, or if hive.hashtable.key.count.adjustment is set to 0 hive.hashtable.loadfactor 0.75 hive.mapjoin.followby.gby.localtask.max.memory.usage 0.55 This number means how much memory the local task can take to hold the key/value into an in-memory hash table when this map join is followed by a group by. If the local task's memory usage is more than this number, the local task will abort by itself. It means the data of the small table is too large to be held in memory. hive.mapjoin.localtask.max.memory.usage 0.9 This number means how much memory the local task can take to hold the key/value into an in-memory hash table. If the local task's memory usage is more than this number, the local task will abort by itself. It means the data of the small table is too large to be held in memory. hive.mapjoin.check.memory.rows 100000 The number means after how many rows processed it needs to check the memory usage hive.debug.localtask false hive.input.format org.apache.hadoop.hive.ql.io.CombineHiveInputFormat The default input format. Set this to HiveInputFormat if you encounter problems with CombineHiveInputFormat. hive.tez.input.format org.apache.hadoop.hive.ql.io.HiveInputFormat The default input format for tez. Tez groups splits in the AM. hive.tez.container.size -1 By default Tez will spawn containers of the size of a mapper. This can be used to overwrite. hive.tez.cpu.vcores -1 By default Tez will ask for however many cpus map-reduce is configured to use per container. This can be used to overwrite. hive.tez.java.opts By default Tez will use the Java options from map tasks. This can be used to overwrite. hive.tez.log.level INFO The log level to use for tasks executing as part of the DAG. Used only if hive.tez.java.opts is used to configure Java options. hive.tez.hs2.user.access true Whether to grant access to the hs2/hive user for queries hive.query.name This named is used by Tez to set the dag name. This name in turn will appear on the Tez UI representing the work that was done. hive.optimize.bucketingsorting true Don't create a reducer for enforcing bucketing/sorting for queries of the form: insert overwrite table T2 select * from T1; where T1 and T2 are bucketed/sorted by the same keys into the same number of buckets. hive.mapred.partitioner org.apache.hadoop.hive.ql.io.DefaultHivePartitioner hive.enforce.sortmergebucketmapjoin false If the user asked for sort-merge bucketed map-side join, and it cannot be performed, should the query fail or not ? hive.enforce.bucketmapjoin false If the user asked for bucketed map-side join, and it cannot be performed, should the query fail or not ? For example, if the buckets in the tables being joined are not a multiple of each other, bucketed map-side join cannot be performed, and the query will fail if hive.enforce.bucketmapjoin is set to true. hive.auto.convert.sortmerge.join false Will the join be automatically converted to a sort-merge join, if the joined tables pass the criteria for sort-merge join. hive.auto.convert.sortmerge.join.reduce.side true Whether hive.auto.convert.sortmerge.join (if enabled) should be applied to reduce side. hive.auto.convert.sortmerge.join.bigtable.selection.policy org.apache.hadoop.hive.ql.optimizer.AvgPartitionSizeBasedBigTableSelectorForAutoSMJ The policy to choose the big table for automatic conversion to sort-merge join. By default, the table with the largest partitions is assigned the big table. All policies are: . based on position of the table - the leftmost table is selected org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSMJ. . based on total size (all the partitions selected in the query) of the table org.apache.hadoop.hive.ql.optimizer.TableSizeBasedBigTableSelectorForAutoSMJ. . based on average size (all the partitions selected in the query) of the table org.apache.hadoop.hive.ql.optimizer.AvgPartitionSizeBasedBigTableSelectorForAutoSMJ. New policies can be added in future. hive.auto.convert.sortmerge.join.to.mapjoin false If hive.auto.convert.sortmerge.join is set to true, and a join was converted to a sort-merge join, this parameter decides whether each table should be tried as a big table, and effectively a map-join should be tried. That would create a conditional task with n+1 children for a n-way join (1 child for each table as the big table), and the backup task will be the sort-merge join. In some cases, a map-join would be faster than a sort-merge join, if there is no advantage of having the output bucketed and sorted. For example, if a very big sorted and bucketed table with few files (say 10 files) are being joined with a very small sorter and bucketed table with few files (10 files), the sort-merge join will only use 10 mappers, and a simple map-only join might be faster if the complete small table can fit in memory, and a map-join can be performed. hive.exec.script.trust false hive.exec.rowoffset false Whether to provide the row offset virtual column hive.optimize.index.filter false Whether to enable automatic use of indexes hive.optimize.index.autoupdate false Whether to update stale indexes automatically hive.optimize.ppd true Whether to enable predicate pushdown hive.optimize.ppd.windowing true Whether to enable predicate pushdown through windowing hive.ppd.recognizetransivity true Whether to transitively replicate predicate filters over equijoin conditions. hive.ppd.remove.duplicatefilters true During query optimization, filters may be pushed down in the operator tree. If this config is true only pushed down filters remain in the operator tree, and the original filter is removed. If this config is false, the original filter is also left in the operator tree at the original place. hive.optimize.point.lookup true Whether to transform OR clauses in Filter operators into IN clauses hive.optimize.point.lookup.min 31 Minimum number of OR clauses needed to transform into IN clauses hive.optimize.partition.columns.separate true Extract partition columns from IN clauses hive.optimize.constant.propagation true Whether to enable constant propagation optimizer hive.optimize.remove.identity.project true Removes identity project from operator tree hive.optimize.metadataonly false Whether to eliminate scans of the tables from which no columns are selected. Note that, when selecting from empty tables with data files, this can produce incorrect results, so it's disabled by default. It works correctly for normal tables. hive.optimize.null.scan true Dont scan relations which are guaranteed to not generate any rows hive.optimize.ppd.storage true Whether to push predicates down to storage handlers hive.optimize.groupby true Whether to enable the bucketed group by from bucketed partitions/tables. hive.optimize.bucketmapjoin false Whether to try bucket mapjoin hive.optimize.bucketmapjoin.sortedmerge false Whether to try sorted bucket merge map join hive.optimize.reducededuplication true Remove extra map-reduce jobs if the data is already clustered by the same key which needs to be used again. This should always be set to true. Since it is a new feature, it has been made configurable. hive.optimize.reducededuplication.min.reducer 4 Reduce deduplication merges two RSs by moving key/parts/reducer-num of the child RS to parent RS. That means if reducer-num of the child RS is fixed (order by or forced bucketing) and small, it can make very slow, single MR. The optimization will be automatically disabled if number of reducers would be less than specified value. hive.optimize.sort.dynamic.partition false When enabled dynamic partitioning column will be globally sorted. This way we can keep only one record writer open for each partition value in the reducer thereby reducing the memory pressure on reducers. hive.optimize.sampling.orderby false Uses sampling on order-by clause for parallel execution. hive.optimize.sampling.orderby.number 1000 Total number of samples to be obtained. hive.optimize.sampling.orderby.percent 0.1 Expects value between 0.0f and 1.0f. Probability with which a row will be chosen. hive.optimize.distinct.rewrite true When applicable this optimization rewrites distinct aggregates from a single stage to multi-stage aggregation. This may not be optimal in all cases. Ideally, whether to trigger it or not should be cost based decision. Until Hive formalizes cost model for this, this is config driven. hive.optimize.union.remove false Whether to remove the union and push the operators between union and the filesink above union. This avoids an extra scan of the output by union. This is independently useful for union queries, and specially useful when hive.optimize.skewjoin.compiletime is set to true, since an extra union is inserted. The merge is triggered if either of hive.merge.mapfiles or hive.merge.mapredfiles is set to true. If the user has set hive.merge.mapfiles to true and hive.merge.mapredfiles to false, the idea was the number of reducers are few, so the number of files anyway are small. However, with this optimization, we are increasing the number of files possibly by a big margin. So, we merge aggressively. hive.optimize.correlation false exploit intra-query correlations. hive.optimize.limittranspose false Whether to push a limit through left/right outer join or union. If the value is true and the size of the outer input is reduced enough (as specified in hive.optimize.limittranspose.reduction), the limit is pushed to the outer input or union; to remain semantically correct, the limit is kept on top of the join or the union too. hive.optimize.limittranspose.reductionpercentage 1.0 When hive.optimize.limittranspose is true, this variable specifies the minimal reduction of the size of the outer input of the join or input of the union that we should get in order to apply the rule. hive.optimize.limittranspose.reductiontuples 0 When hive.optimize.limittranspose is true, this variable specifies the minimal reduction in the number of tuples of the outer input of the join or the input of the union that you should get in order to apply the rule. hive.optimize.filter.stats.reduction false Whether to simplify comparison expressions in filter operators using column stats hive.optimize.skewjoin.compiletime false Whether to create a separate plan for skewed keys for the tables in the join. This is based on the skewed keys stored in the metadata. At compile time, the plan is broken into different joins: one for the skewed keys, and the other for the remaining keys. And then, a union is performed for the 2 joins generated above. So unless the same skewed key is present in both the joined tables, the join for the skewed key will be performed as a map-side join. The main difference between this parameter and hive.optimize.skewjoin is that this parameter uses the skew information stored in the metastore to optimize the plan at compile time itself. If there is no skew information in the metadata, this parameter will not have any affect. Both hive.optimize.skewjoin.compiletime and hive.optimize.skewjoin should be set to true. Ideally, hive.optimize.skewjoin should be renamed as hive.optimize.skewjoin.runtime, but not doing so for backward compatibility. If the skew information is correctly stored in the metadata, hive.optimize.skewjoin.compiletime would change the query plan to take care of it, and hive.optimize.skewjoin will be a no-op. hive.optimize.cte.materialize.threshold -1 If the number of references to a CTE clause exceeds this threshold, Hive will materialize it before executing the main query block. -1 will disable this feature. hive.optimize.index.filter.compact.minsize 5368709120 Minimum size (in bytes) of the inputs on which a compact index is automatically used. hive.optimize.index.filter.compact.maxsize -1 Maximum size (in bytes) of the inputs on which a compact index is automatically used. A negative number is equivalent to infinity. hive.index.compact.query.max.entries 10000000 The maximum number of index entries to read during a query that uses the compact index. Negative value is equivalent to infinity. hive.index.compact.query.max.size 10737418240 The maximum number of bytes that a query using the compact index can read. Negative value is equivalent to infinity. hive.index.compact.binary.search true Whether or not to use a binary search to find the entries in an index table that match the filter, where possible hive.stats.autogather true A flag to gather statistics (only basic) automatically during the INSERT OVERWRITE command. hive.stats.column.autogather false A flag to gather column statistics automatically. hive.stats.dbclass fs Expects one of the pattern in [custom, fs]. The storage that stores temporary Hive statistics. In filesystem based statistics collection ('fs'), each task writes statistics it has collected in a file on the filesystem, which will be aggregated after the job has finished. Supported values are fs (filesystem) and custom as defined in StatsSetupConst.java. hive.stats.default.publisher The Java class (implementing the StatsPublisher interface) that is used by default if hive.stats.dbclass is custom type. hive.stats.default.aggregator The Java class (implementing the StatsAggregator interface) that is used by default if hive.stats.dbclass is custom type. hive.stats.atomic false whether to update metastore stats only if all stats are available hive.client.stats.counters Subset of counters that should be of interest for hive.client.stats.publishers (when one wants to limit their publishing). Non-display names should be used hive.stats.reliable false Whether queries will fail because stats cannot be collected completely accurately. If this is set to true, reading/writing from/into a partition may fail because the stats could not be computed accurately. hive.analyze.stmt.collect.partlevel.stats true analyze table T compute statistics for columns. Queries like these should compute partitionlevel stats for partitioned table even when no part spec is specified. hive.stats.gather.num.threads 10 Number of threads used by partialscan/noscan analyze command for partitioned tables. This is applicable only for file formats that implement StatsProvidingRecordReader (like ORC). hive.stats.collect.tablekeys false Whether join and group by keys on tables are derived and maintained in the QueryPlan. This is useful to identify how tables are accessed and to determine if they should be bucketed. hive.stats.collect.scancols false Whether column accesses are tracked in the QueryPlan. This is useful to identify how tables are accessed and to determine if there are wasted columns that can be trimmed. hive.stats.ndv.error 20.0 Standard error expressed in percentage. Provides a tradeoff between accuracy and compute cost. A lower value for error indicates higher accuracy and a higher compute cost. hive.metastore.stats.ndv.tuner 0.0 Provides a tunable parameter between the lower bound and the higher bound of ndv for aggregate ndv across all the partitions. The lower bound is equal to the maximum of ndv of all the partitions. The higher bound is equal to the sum of ndv of all the partitions. Its value should be between 0.0 (i.e., choose lower bound) and 1.0 (i.e., choose higher bound) hive.metastore.stats.ndv.densityfunction false Whether to use density function to estimate the NDV for the whole table based on the NDV of partitions hive.stats.max.variable.length 100 To estimate the size of data flowing through operators in Hive/Tez(for reducer estimation etc.), average row size is multiplied with the total number of rows coming out of each operator. Average row size is computed from average column size of all columns in the row. In the absence of column statistics, for variable length columns (like string, bytes etc.), this value will be used. For fixed length columns their corresponding Java equivalent sizes are used (float - 4 bytes, double - 8 bytes etc.). hive.stats.list.num.entries 10 To estimate the size of data flowing through operators in Hive/Tez(for reducer estimation etc.), average row size is multiplied with the total number of rows coming out of each operator. Average row size is computed from average column size of all columns in the row. In the absence of column statistics and for variable length complex columns like list, the average number of entries/values can be specified using this config. hive.stats.map.num.entries 10 To estimate the size of data flowing through operators in Hive/Tez(for reducer estimation etc.), average row size is multiplied with the total number of rows coming out of each operator. Average row size is computed from average column size of all columns in the row. In the absence of column statistics and for variable length complex columns like map, the average number of entries/values can be specified using this config. hive.stats.fetch.partition.stats true Annotation of operator tree with statistics information requires partition level basic statistics like number of rows, data size and file size. Partition statistics are fetched from metastore. Fetching partition statistics for each needed partition can be expensive when the number of partitions is high. This flag can be used to disable fetching of partition statistics from metastore. When this flag is disabled, Hive will make calls to filesystem to get file sizes and will estimate the number of rows from row schema. hive.stats.fetch.column.stats false Annotation of operator tree with statistics information requires column statistics. Column statistics are fetched from metastore. Fetching column statistics for each needed column can be expensive when the number of columns is high. This flag can be used to disable fetching of column statistics from metastore. hive.stats.join.factor 1.1 Hive/Tez optimizer estimates the data size flowing through each of the operators. JOIN operator uses column statistics to estimate the number of rows flowing out of it and hence the data size. In the absence of column statistics, this factor determines the amount of rows that flows out of JOIN operator. hive.stats.deserialization.factor 1.0 Hive/Tez optimizer estimates the data size flowing through each of the operators. In the absence of basic statistics like number of rows and data size, file size is used to estimate the number of rows and data size. Since files in tables/partitions are serialized (and optionally compressed) the estimates of number of rows and data size cannot be reliably determined. This factor is multiplied with the file size to account for serialization and compression. hive.stats.filter.in.factor 1.0 Currently column distribution is assumed to be uniform. This can lead to overestimation/underestimation in the number of rows filtered by a certain operator, which in turn might lead to overprovision or underprovision of resources. This factor is applied to the cardinality estimation of IN clauses in filter operators. hive.support.concurrency false Whether Hive supports concurrency control or not. A ZooKeeper instance must be up and running when using zookeeper Hive lock manager hive.lock.manager org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager hive.lock.numretries 100 The number of times you want to try to get all the locks hive.unlock.numretries 10 The number of times you want to retry to do one unlock hive.lock.sleep.between.retries 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. The time should be in between 0 sec (exclusive) and 9223372036854775807 sec (exclusive). The maximum sleep time between various retries hive.lock.mapred.only.operation false This param is to control whether or not only do lock on queries that need to execute at least one mapred job. hive.zookeeper.quorum List of ZooKeeper servers to talk to. This is needed for: 1. Read/write locks - when hive.lock.manager is set to org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager, 2. When HiveServer2 supports service discovery via Zookeeper. 3. For delegation token storage if zookeeper store is used, if hive.cluster.delegation.token.store.zookeeper.connectString is not set 4. LLAP daemon registry service hive.zookeeper.client.port 2181 The port of ZooKeeper servers to talk to. If the list of Zookeeper servers specified in hive.zookeeper.quorum does not contain port numbers, this value is used. hive.zookeeper.session.timeout 1200000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. ZooKeeper client's session timeout (in milliseconds). The client is disconnected, and as a result, all locks released, if a heartbeat is not sent in the timeout. hive.zookeeper.namespace hive_zookeeper_namespace The parent node under which all ZooKeeper nodes are created. hive.zookeeper.clean.extra.nodes false Clean extra nodes at the end of the session. hive.zookeeper.connection.max.retries 3 Max number of times to retry when connecting to the ZooKeeper server. hive.zookeeper.connection.basesleeptime 1000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Initial amount of time (in milliseconds) to wait between retries when connecting to the ZooKeeper server when using ExponentialBackoffRetry policy. hive.txn.manager org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager Set to org.apache.hadoop.hive.ql.lockmgr.DbTxnManager as part of turning on Hive transactions, which also requires appropriate settings for hive.compactor.initiator.on, hive.compactor.worker.threads, hive.support.concurrency (true), and hive.exec.dynamic.partition.mode (nonstrict). The default DummyTxnManager replicates pre-Hive-0.13 behavior and provides no transactions. hive.txn.strict.locking.mode true In strict mode non-ACID resources use standard R/W lock semantics, e.g. INSERT will acquire exclusive lock. In nonstrict mode, for non-ACID resources, INSERT will only acquire shared lock, which allows two concurrent writes to the same partition but still lets lock manager prevent DROP TABLE etc. when the table is being written to hive.txn.timeout 300s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. time after which transactions are declared aborted if the client has not sent a heartbeat. hive.txn.heartbeat.threadpool.size 5 The number of threads to use for heartbeating. For Hive CLI, 1 is enough. For HiveServer2, we need a few hive.txn.manager.dump.lock.state.on.acquire.timeout false Set this to true so that when attempt to acquire a lock on resource times out, the current state of the lock manager is dumped to log file. This is for debugging. See also hive.lock.numretries and hive.lock.sleep.between.retries. hive.txn.operational.properties 0 Sets the operational properties that control the appropriate behavior for various versions of the Hive ACID subsystem. Setting it to zero will turn on the legacy mode for ACID, while setting it to one will enable a split-update feature found in the newer version of Hive ACID subsystem. Mostly it is intended to be used as an internal property for future versions of ACID. (See HIVE-14035 for details.) hive.max.open.txns 100000 Maximum number of open transactions. If current open transactions reach this limit, future open transaction requests will be rejected, until this number goes below the limit. hive.count.open.txns.interval 1s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Time in seconds between checks to count open transactions. hive.txn.max.open.batch 1000 Maximum number of transactions that can be fetched in one call to open_txns(). This controls how many transactions streaming agents such as Flume or Storm open simultaneously. The streaming agent then writes that number of entries into a single file (per Flume agent or Storm bolt). Thus increasing this value decreases the number of delta files created by streaming agents. But it also increases the number of open transactions that Hive has to track at any given time, which may negatively affect read performance. hive.txn.retryable.sqlex.regex Comma separated list of regular expression patterns for SQL state, error code, and error message of retryable SQLExceptions, that's suitable for the metastore DB. For example: Can't serialize.*,40001$,^Deadlock,.*ORA-08176.* The string that the regex will be matched against is of the following form, where ex is a SQLException: ex.getMessage() + " (SQLState=" + ex.getSQLState() + ", ErrorCode=" + ex.getErrorCode() + ")" hive.compactor.initiator.on false Whether to run the initiator and cleaner threads on this metastore instance or not. Set this to true on one instance of the Thrift metastore service as part of turning on Hive transactions. For a complete list of parameters required for turning on transactions, see hive.txn.manager. hive.compactor.worker.threads 0 How many compactor worker threads to run on this metastore instance. Set this to a positive number on one or more instances of the Thrift metastore service as part of turning on Hive transactions. For a complete list of parameters required for turning on transactions, see hive.txn.manager. Worker threads spawn MapReduce jobs to do compactions. They do not do the compactions themselves. Increasing the number of worker threads will decrease the time it takes tables or partitions to be compacted once they are determined to need compaction. It will also increase the background load on the Hadoop cluster as more MapReduce jobs will be running in the background. hive.compactor.worker.timeout 86400s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Time in seconds after which a compaction job will be declared failed and the compaction re-queued. hive.compactor.check.interval 300s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Time in seconds between checks to see if any tables or partitions need to be compacted. This should be kept high because each check for compaction requires many calls against the NameNode. Decreasing this value will reduce the time it takes for compaction to be started for a table or partition that requires compaction. However, checking if compaction is needed requires several calls to the NameNode for each table or partition that has had a transaction done on it since the last major compaction. So decreasing this value will increase the load on the NameNode. hive.compactor.delta.num.threshold 10 Number of delta directories in a table or partition that will trigger a minor compaction. hive.compactor.delta.pct.threshold 0.1 Percentage (fractional) size of the delta files relative to the base that will trigger a major compaction. (1.0 = 100%, so the default 0.1 = 10%.) hive.compactor.max.num.delta 500 Maximum number of delta files that the compactor will attempt to handle in a single job. hive.compactor.abortedtxn.threshold 1000 Number of aborted transactions involving a given table or partition that will trigger a major compaction. hive.compactor.initiator.failed.compacts.threshold 2 Expects value between 1 and 20. Number of consecutive compaction failures (per table/partition) after which automatic compactions will not be scheduled any more. Note that this must be less than hive.compactor.history.retention.failed. hive.compactor.cleaner.run.interval 5000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Time between runs of the cleaner thread hive.compactor.job.queue Used to specify name of Hadoop queue to which Compaction jobs will be submitted. Set to empty string to let Hadoop choose the queue. hive.compactor.history.retention.succeeded 3 Expects value between 0 and 100. Determines how many successful compaction records will be retained in compaction history for a given table/partition. hive.compactor.history.retention.failed 3 Expects value between 0 and 100. Determines how many failed compaction records will be retained in compaction history for a given table/partition. hive.compactor.history.retention.attempted 2 Expects value between 0 and 100. Determines how many attempted compaction records will be retained in compaction history for a given table/partition. hive.compactor.history.reaper.interval 2m Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Determines how often compaction history reaper runs hive.timedout.txn.reaper.start 100s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Time delay of 1st reaper run after metastore start hive.timedout.txn.reaper.interval 180s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Time interval describing how often the reaper runs hive.writeset.reaper.interval 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Frequency of WriteSet reaper runs hive.merge.cardinality.check true Set to true to ensure that each SQL Merge statement ensures that for each row in the target table there is at most 1 matching row in the source table per SQL Specification. hive.druid.indexer.segments.granularity DAY Expects one of the pattern in [YEAR, MONTH, WEEK, DAY, HOUR, MINUTE, SECOND]. Granularity for the segments created by the Druid storage handler hive.druid.indexer.partition.size.max 5000000 Maximum number of records per segment partition hive.druid.indexer.memory.rownum.max 75000 Maximum number of records in memory while storing data in Druid hive.druid.broker.address.default localhost:8082 Address of the Druid broker. If we are querying Druid from Hive, this address needs to be declared hive.druid.coordinator.address.default localhost:8081 Address of the Druid coordinator. It is used to check the load status of newly created segments hive.druid.select.distribute true If it is set to true, we distribute the execution of Druid Select queries. Concretely, we retrieve the result for Select queries directly from the Druid nodes containing the segments data. In particular, first we contact the Druid broker node to obtain the nodes containing the segments for the given query, and then we contact those nodes to retrieve the results for the query. If it is set to false, we do not execute the Select queries in a distributed fashion. Instead, results for those queries are returned by the Druid broker node. hive.druid.select.threshold 10000 Takes only effect when hive.druid.select.distribute is set to false. When we can split a Select query, this is the maximum number of rows that we try to retrieve per query. In order to do that, we obtain the estimated size for the complete result. If the number of records of the query results is larger than this threshold, we split the query in total number of rows/threshold parts across the time dimension. Note that we assume the records to be split uniformly across the time dimension. hive.druid.http.numConnection 20 Number of connections used by the HTTP client. hive.druid.http.read.timeout PT1M Read timeout period for the HTTP client in ISO8601 format (for example P2W, P3M, PT1H30M, PT0.750S), default is period of 1 minute. hive.druid.sleep.time PT10S Sleep time between retries in ISO8601 format (for example P2W, P3M, PT1H30M, PT0.750S), default is period of 10 seconds. hive.druid.basePersistDirectory Local temporary directory used to persist intermediate indexing state, will default to JVM system property java.io.tmpdir. hive.druid.storage.storageDirectory /druid/segments druid deep storage location. hive.druid.metadata.base druid Default prefix for metadata tables hive.druid.metadata.db.type mysql Expects one of the pattern in [mysql, postgresql]. Type of the metadata database. hive.druid.metadata.username Username to connect to Type of the metadata DB. hive.druid.metadata.password Password to connect to Type of the metadata DB. hive.druid.metadata.uri URI to connect to the database (for example jdbc:mysql://hostname:port/DBName). hive.druid.working.directory /tmp/workingDirectory Default hdfs working directory used to store some intermediate metadata hive.druid.maxTries 5 Maximum number of retries before giving up hive.druid.passiveWaitTimeMs 30000 Wait time in ms default to 30 seconds. hive.hbase.wal.enabled true Whether writes to HBase should be forced to the write-ahead log. Disabling this improves HBase write performance at the risk of lost writes in case of a crash. hive.hbase.generatehfiles false True when HBaseStorageHandler should generate hfiles instead of operate against the online table. hive.hbase.snapshot.name The HBase table snapshot name to use. hive.hbase.snapshot.restoredir /tmp The directory in which to restore the HBase table snapshot. hive.archive.enabled false Whether archiving operations are permitted hive.optimize.index.groupby false Whether to enable optimization of group-by queries using Aggregate indexes. hive.fetch.task.conversion more Expects one of [none, minimal, more]. Some select queries can be converted to single FETCH task minimizing latency. Currently the query should be single sourced not having any subquery and should not have any aggregations or distincts (which incurs RS), lateral views and joins. 0. none : disable hive.fetch.task.conversion 1. minimal : SELECT STAR, FILTER on partition columns, LIMIT only 2. more : SELECT, FILTER, LIMIT only (support TABLESAMPLE and virtual columns) hive.fetch.task.conversion.threshold 1073741824 Input threshold for applying hive.fetch.task.conversion. If target table is native, input length is calculated by summation of file lengths. If it's not native, storage handler for the table can optionally implement org.apache.hadoop.hive.ql.metadata.InputEstimator interface. hive.fetch.task.aggr false Aggregation queries with no group-by clause (for example, select count(*) from src) execute final aggregations in single reduce task. If this is set true, Hive delegates final aggregation stage to fetch task, possibly decreasing the query time. hive.compute.query.using.stats true When set to true Hive will answer a few queries like count(1) purely using stats stored in metastore. For basic stats collection turn on the config hive.stats.autogather to true. For more advanced stats collection need to run analyze table queries. hive.fetch.output.serde org.apache.hadoop.hive.serde2.DelimitedJSONSerDe The SerDe used by FetchTask to serialize the fetch output. hive.cache.expr.evaluation true If true, the evaluation result of a deterministic expression referenced twice or more will be cached. For example, in a filter condition like '.. where key + 10 = 100 or key + 10 = 0' the expression 'key + 10' will be evaluated/cached once and reused for the following expression ('key + 10 = 0'). Currently, this is applied only to expressions in select or filter operators. hive.variable.substitute true This enables substitution using syntax like ${var} ${system:var} and ${env:var}. hive.variable.substitute.depth 40 The maximum replacements the substitution engine will do. hive.conf.validation true Enables type checking for registered Hive configurations hive.semantic.analyzer.hook hive.security.authorization.enabled false enable or disable the Hive client authorization hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory The Hive client authorization manager class name. The user defined authorization class should implement interface org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider. hive.security.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator hive client authenticator manager class name. The user defined authenticator should implement interface org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider. hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.DefaultHiveMetastoreAuthorizationProvider Names of authorization manager classes (comma separated) to be used in the metastore for authorization. The user defined authorization class should implement interface org.apache.hadoop.hive.ql.security.authorization.HiveMetastoreAuthorizationProvider. All authorization manager classes have to successfully authorize the metastore API call for the command execution to be allowed. hive.security.metastore.authorization.auth.reads true If this is true, metastore authorizer authorizes read actions on database, table hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator authenticator manager class name to be used in the metastore for authentication. The user defined authenticator should implement interface org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider. hive.security.authorization.createtable.user.grants the privileges automatically granted to some users whenever a table gets created. An example like "userX,userY:select;userZ:create" will grant select privilege to userX and userY, and grant create privilege to userZ whenever a new table created. hive.security.authorization.createtable.group.grants the privileges automatically granted to some groups whenever a table gets created. An example like "groupX,groupY:select;groupZ:create" will grant select privilege to groupX and groupY, and grant create privilege to groupZ whenever a new table created. hive.security.authorization.createtable.role.grants the privileges automatically granted to some roles whenever a table gets created. An example like "roleX,roleY:select;roleZ:create" will grant select privilege to roleX and roleY, and grant create privilege to roleZ whenever a new table created. hive.security.authorization.createtable.owner.grants The privileges automatically granted to the owner whenever a table gets created. An example like "select,drop" will grant select and drop privilege to the owner of the table. Note that the default gives the creator of a table no access to the table (but see HIVE-8067). hive.security.authorization.task.factory org.apache.hadoop.hive.ql.parse.authorization.HiveAuthorizationTaskFactoryImpl Authorization DDL task factory implementation hive.security.authorization.sqlstd.confwhitelist List of comma separated Java regexes. Configurations parameters that match these regexes can be modified by user when SQL standard authorization is enabled. To get the default value, use the 'set <param>' command. Note that the hive.conf.restricted.list checks are still enforced after the white list check hive.security.authorization.sqlstd.confwhitelist.append List of comma separated Java regexes, to be appended to list set in hive.security.authorization.sqlstd.confwhitelist. Using this list instead of updating the original list means that you can append to the defaults set by SQL standard authorization instead of replacing it entirely. hive.cli.print.header false Whether to print the names of the columns in query output. hive.cli.tez.session.async true Whether to start Tez session in background when running CLI with Tez, allowing CLI to be available earlier. hive.error.on.empty.partition false Whether to throw an exception if dynamic partition insert generates empty results. hive.index.compact.file internal variable hive.index.blockfilter.file internal variable hive.index.compact.file.ignore.hdfs false When true the HDFS location stored in the index file will be ignored at runtime. If the data got moved or the name of the cluster got changed, the index data should still be usable. hive.exim.uri.scheme.whitelist hdfs,pfile,file,s3,s3a A comma separated list of acceptable URI schemes for import and export. hive.exim.strict.repl.tables true Parameter that determines if 'regular' (non-replication) export dumps can be imported on to tables that are the target of replication. If this parameter is set, regular imports will check if the destination table(if it exists) has a 'repl.last.id' set on it. If so, it will fail. hive.repl.task.factory org.apache.hive.hcatalog.api.repl.exim.EximReplicationTaskFactory Parameter that can be used to override which ReplicationTaskFactory will be used to instantiate ReplicationTask events. Override for third party repl plugins hive.mapper.cannot.span.multiple.partitions false hive.rework.mapredwork false should rework the mapred work or not. This is first introduced by SymlinkTextInputFormat to replace symlink files with real paths at compile time. hive.exec.concatenate.check.index true If this is set to true, Hive will throw error when doing 'alter table tbl_name [partSpec] concatenate' on a table/partition that has indexes on it. The reason the user want to set this to true is because it can help user to avoid handling all index drop, recreation, rebuild work. This is very helpful for tables with thousands of partitions. hive.io.exception.handlers A list of io exception handler class names. This is used to construct a list exception handlers to handle exceptions thrown by record readers hive.log4j.file Hive log4j configuration file. If the property is not set, then logging will be initialized using hive-log4j2.properties found on the classpath. If the property is set, the value must be a valid URI (java.net.URI, e.g. "file:///tmp/my-logging.xml"), which you can then extract a URL from and pass to PropertyConfigurator.configure(URL). hive.exec.log4j.file Hive log4j configuration file for execution mode(sub command). If the property is not set, then logging will be initialized using hive-exec-log4j2.properties found on the classpath. If the property is set, the value must be a valid URI (java.net.URI, e.g. "file:///tmp/my-logging.xml"), which you can then extract a URL from and pass to PropertyConfigurator.configure(URL). hive.async.log.enabled true Whether to enable Log4j2's asynchronous logging. Asynchronous logging can give significant performance improvement as logging will be handled in separate thread that uses LMAX disruptor queue for buffering log messages. Refer https://logging.apache.org/log4j/2.x/manual/async.html for benefits and drawbacks. hive.log.explain.output false Whether to log explain output for every query. When enabled, will log EXPLAIN EXTENDED output for the query at INFO log4j log level. hive.explain.user true Whether to show explain result at user level. When enabled, will log EXPLAIN output for the query at user level. hive.autogen.columnalias.prefix.label _c String used as a prefix when auto generating column alias. By default the prefix label will be appended with a column position number to form the column alias. Auto generation would happen if an aggregate function is used in a select clause without an explicit alias. hive.autogen.columnalias.prefix.includefuncname false Whether to include function name in the column alias auto generated by Hive. hive.service.metrics.class org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics Expects one of [org.apache.hadoop.hive.common.metrics.metrics2.codahalemetrics, org.apache.hadoop.hive.common.metrics.legacymetrics]. Hive metrics subsystem implementation class. hive.service.metrics.reporter JSON_FILE, JMX Reporter type for metric class org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics, comma separated list of JMX, CONSOLE, JSON_FILE, HADOOP2 hive.service.metrics.file.location /tmp/report.json For metric class org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics JSON_FILE reporter, the location of local JSON metrics file. This file will get overwritten at every interval. hive.service.metrics.file.frequency 5s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. For metric class org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics JSON_FILE reporter, the frequency of updating JSON metrics file. hive.service.metrics.hadoop2.frequency 30s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. For metric class org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics HADOOP2 reporter, the frequency of updating the HADOOP2 metrics system. hive.service.metrics.hadoop2.component hive Component name to provide to Hadoop2 Metrics system. Ideally 'hivemetastore' for the MetaStore and and 'hiveserver2' for HiveServer2. hive.exec.perf.logger org.apache.hadoop.hive.ql.log.PerfLogger The class responsible for logging client side performance metrics. Must be a subclass of org.apache.hadoop.hive.ql.log.PerfLogger hive.start.cleanup.scratchdir false To cleanup the Hive scratchdir when starting the Hive Server hive.scratchdir.lock false To hold a lock file in scratchdir to prevent to be removed by cleardanglingscratchdir hive.insert.into.multilevel.dirs false Where to insert into multilevel directories like "insert directory '/HIVEFT25686/chinna/' from table" hive.warehouse.subdir.inherit.perms true Set this to false if the table directories should be created with the permissions derived from dfs umask instead of inheriting the permission of the warehouse or database directory. hive.insert.into.external.tables true whether insert into external tables is allowed hive.exec.temporary.table.storage default Expects one of [memory, ssd, default]. Define the storage policy for temporary tables.Choices between memory, ssd and default hive.query.lifetime.hooks A comma separated list of hooks which implement QueryLifeTimeHook. These will be triggered before/after query compilation and before/after query execution, in the order specified hive.exec.driver.run.hooks A comma separated list of hooks which implement HiveDriverRunHook. Will be run at the beginning and end of Driver.run, these will be run in the order specified. hive.ddl.output.format The data format to use for DDL output. One of "text" (for human readable text) or "json" (for a json object). hive.entity.separator @ Separator used to construct names of tables and partitions. For example, dbname@tablename@partitionname hive.entity.capture.transform false Compiler to capture transform URI referred in the query hive.display.partition.cols.separately true In older Hive version (0.10 and earlier) no distinction was made between partition columns or non-partition columns while displaying columns in describe table. From 0.12 onwards, they are displayed separately. This flag will let you get old behavior, if desired. See, test-case in patch for HIVE-6689. hive.ssl.protocol.blacklist SSLv2,SSLv3 SSL Versions to disable for all Hive Servers hive.server2.clear.dangling.scratchdir false Clear dangling scratch dir periodically in HS2 hive.server2.clear.dangling.scratchdir.interval 1800s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Interval to clear dangling scratch dir periodically in HS2 hive.server2.sleep.interval.between.start.attempts 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. The time should be in between 0 msec (inclusive) and 9223372036854775807 msec (inclusive). Amount of time to sleep between HiveServer2 start attempts. Primarily meant for tests hive.server2.max.start.attempts 30 Expects value bigger than 0. Number of times HiveServer2 will attempt to start before exiting. The sleep interval between retries is determined by hive.server2.sleep.interval.between.start.attempts The default of 30 will keep trying for 30 minutes. hive.server2.support.dynamic.service.discovery false Whether HiveServer2 supports dynamic service discovery for its clients. To support this, each instance of HiveServer2 currently uses ZooKeeper to register itself, when it is brought up. JDBC/ODBC clients should use the ZooKeeper ensemble: hive.zookeeper.quorum in their connection string. hive.server2.zookeeper.namespace hiveserver2 The parent node in ZooKeeper used by HiveServer2 when supporting dynamic service discovery. hive.server2.zookeeper.publish.configs true Whether we should publish HiveServer2's configs to ZooKeeper. hive.server2.global.init.file.location ${env:HIVE_CONF_DIR} Either the location of a HS2 global init file or a directory containing a .hiverc file. If the property is set, the value must be a valid path to an init file or directory where the init file is located. hive.server2.transport.mode binary Expects one of [binary, http]. Transport mode of HiveServer2. hive.server2.thrift.bind.host Bind host on which to run the HiveServer2 Thrift service. hive.driver.parallel.compilation false Whether to enable parallel compilation of the queries between sessions and within the same session on HiveServer2. The default is false. hive.server2.compile.lock.timeout 0s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Number of seconds a request will wait to acquire the compile lock before giving up. Setting it to 0s disables the timeout. hive.server2.parallel.ops.in.session true Whether to allow several parallel operations (such as SQL statements) in one session. hive.server2.webui.host 0.0.0.0 The host address the HiveServer2 WebUI will listen on hive.server2.webui.port 10002 The port the HiveServer2 WebUI will listen on. This can beset to 0 or a negative integer to disable the web UI hive.server2.webui.max.threads 50 The max HiveServer2 WebUI threads hive.server2.webui.use.ssl false Set this to true for using SSL encryption for HiveServer2 WebUI. hive.server2.webui.keystore.path SSL certificate keystore location for HiveServer2 WebUI. hive.server2.webui.keystore.password SSL certificate keystore password for HiveServer2 WebUI. hive.server2.webui.use.spnego false If true, the HiveServer2 WebUI will be secured with SPNEGO. Clients must authenticate with Kerberos. hive.server2.webui.spnego.keytab The path to the Kerberos Keytab file containing the HiveServer2 WebUI SPNEGO service principal. hive.server2.webui.spnego.principal HTTP/_HOST@EXAMPLE.COM The HiveServer2 WebUI SPNEGO service principal. The special string _HOST will be replaced automatically with the value of hive.server2.webui.host or the correct host name. hive.server2.webui.max.historic.queries 25 The maximum number of past queries to show in HiverSever2 WebUI. hive.server2.tez.default.queues A list of comma separated values corresponding to YARN queues of the same name. When HiveServer2 is launched in Tez mode, this configuration needs to be set for multiple Tez sessions to run in parallel on the cluster. hive.server2.tez.sessions.per.default.queue 1 A positive integer that determines the number of Tez sessions that should be launched on each of the queues specified by "hive.server2.tez.default.queues". Determines the parallelism on each queue. hive.server2.tez.initialize.default.sessions false This flag is used in HiveServer2 to enable a user to use HiveServer2 without turning on Tez for HiveServer2. The user could potentially want to run queries over Tez without the pool of sessions. hive.server2.tez.session.lifetime 162h Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is hour if not specified. The lifetime of the Tez sessions launched by HS2 when default sessions are enabled. Set to 0 to disable session expiration. hive.server2.tez.session.lifetime.jitter 3h Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is hour if not specified. The jitter for Tez session lifetime; prevents all the sessions from restarting at once. hive.server2.tez.sessions.init.threads 16 If hive.server2.tez.initialize.default.sessions is enabled, the maximum number of threads to use to initialize the default sessions. hive.server2.tez.sessions.restricted.configs The configuration settings that cannot be set when submitting jobs to HiveServer2. If any of these are set to values different from those in the server configuration, an exception will be thrown. hive.server2.tez.sessions.custom.queue.allowed true Expects one of [true, false, ignore]. Whether Tez session pool should allow submitting queries to custom queues. The options are true, false (error out), ignore (accept the query but ignore the queue setting). hive.server2.logging.operation.enabled true When true, HS2 will save operation logs and make them available for clients hive.server2.logging.operation.log.location ${system:java.io.tmpdir}/${system:user.name}/operation_logs Top level directory where operation logs are stored if logging functionality is enabled hive.server2.logging.operation.level EXECUTION Expects one of [none, execution, performance, verbose]. HS2 operation logging mode available to clients to be set at session level. For this to work, hive.server2.logging.operation.enabled should be set to true. NONE: Ignore any logging EXECUTION: Log completion of tasks PERFORMANCE: Execution + Performance logs VERBOSE: All logs hive.server2.metrics.enabled false Enable metrics on the HiveServer2. hive.server2.thrift.http.port 10001 Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'http'. hive.server2.thrift.http.path cliservice Path component of URL endpoint when in HTTP mode. hive.server2.thrift.max.message.size 104857600 Maximum message size in bytes a HS2 server will accept. hive.server2.thrift.http.max.idle.time 1800s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Maximum idle time for a connection on the server when in HTTP mode. hive.server2.thrift.http.worker.keepalive.time 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Keepalive time for an idle http worker thread. When the number of workers exceeds min workers, excessive threads are killed after this time interval. hive.server2.thrift.http.request.header.size 6144 Request header size in bytes, when using HTTP transport mode. Jetty defaults used. hive.server2.thrift.http.response.header.size 6144 Response header size in bytes, when using HTTP transport mode. Jetty defaults used. hive.server2.thrift.http.cookie.auth.enabled true When true, HiveServer2 in HTTP transport mode, will use cookie based authentication mechanism. hive.server2.thrift.http.cookie.max.age 86400s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Maximum age in seconds for server side cookie used by HS2 in HTTP mode. hive.server2.thrift.http.cookie.domain Domain for the HS2 generated cookies hive.server2.thrift.http.cookie.path Path for the HS2 generated cookies hive.server2.thrift.http.cookie.is.secure true Deprecated: Secure attribute of the HS2 generated cookie (this is automatically enabled for SSL enabled HiveServer2). hive.server2.thrift.http.cookie.is.httponly true HttpOnly attribute of the HS2 generated cookie. hive.server2.thrift.port 10000 Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'binary'. hive.server2.thrift.sasl.qop auth Expects one of [auth, auth-int, auth-conf]. Sasl QOP value; set it to one of following values to enable higher levels of protection for HiveServer2 communication with clients. Setting hadoop.rpc.protection to a higher level than HiveServer2 does not make sense in most situations. HiveServer2 ignores hadoop.rpc.protection in favor of hive.server2.thrift.sasl.qop. "auth" - authentication only (default) "auth-int" - authentication plus integrity protection "auth-conf" - authentication plus integrity and confidentiality protection This is applicable only if HiveServer2 is configured to use Kerberos authentication. hive.server2.thrift.min.worker.threads 5 Minimum number of Thrift worker threads hive.server2.thrift.max.worker.threads 500 Maximum number of Thrift worker threads hive.server2.thrift.exponential.backoff.slot.length 100ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Binary exponential backoff slot time for Thrift clients during login to HiveServer2, for retries until hitting Thrift client timeout hive.server2.thrift.login.timeout 20s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Timeout for Thrift clients during login to HiveServer2 hive.server2.thrift.worker.keepalive.time 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Keepalive time (in seconds) for an idle worker thread. When the number of workers exceeds min workers, excessive threads are killed after this time interval. hive.server2.async.exec.threads 100 Number of threads in the async thread pool for HiveServer2 hive.server2.async.exec.shutdown.timeout 10s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. How long HiveServer2 shutdown will wait for async threads to terminate. hive.server2.async.exec.wait.queue.size 100 Size of the wait queue for async thread pool in HiveServer2. After hitting this limit, the async thread pool will reject new requests. hive.server2.async.exec.keepalive.time 10s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Time that an idle HiveServer2 async thread (from the thread pool) will wait for a new task to arrive before terminating hive.server2.async.exec.async.compile false Whether to enable compiling async query asynchronously. If enabled, it is unknown if the query will have any resultset before compilation completed. hive.server2.long.polling.timeout 5000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Time that HiveServer2 will wait before responding to asynchronous calls that use long polling hive.session.impl.classname Classname for custom implementation of hive session hive.session.impl.withugi.classname Classname for custom implementation of hive session with UGI hive.server2.authentication NONE Expects one of [nosasl, none, ldap, kerberos, pam, custom]. Client authentication types. NONE: no authentication check LDAP: LDAP/AD based authentication KERBEROS: Kerberos/GSSAPI authentication CUSTOM: Custom authentication provider (Use with property hive.server2.custom.authentication.class) PAM: Pluggable authentication module NOSASL: Raw transport hive.server2.allow.user.substitution true Allow alternate user to be specified as part of HiveServer2 open connection request. hive.server2.authentication.kerberos.keytab Kerberos keytab file for server principal hive.server2.authentication.kerberos.principal Kerberos server principal hive.server2.authentication.spnego.keytab keytab file for SPNego principal, optional, typical value would look like /etc/security/keytabs/spnego.service.keytab, This keytab would be used by HiveServer2 when Kerberos security is enabled and HTTP transport mode is used. This needs to be set only if SPNEGO is to be used in authentication. SPNego authentication would be honored only if valid hive.server2.authentication.spnego.principal and hive.server2.authentication.spnego.keytab are specified. hive.server2.authentication.spnego.principal SPNego service principal, optional, typical value would look like HTTP/_HOST@EXAMPLE.COM SPNego service principal would be used by HiveServer2 when Kerberos security is enabled and HTTP transport mode is used. This needs to be set only if SPNEGO is to be used in authentication. hive.server2.authentication.ldap.url LDAP connection URL(s), this value could contain URLs to mutiple LDAP servers instances for HA, each LDAP URL is separated by a SPACE character. URLs are used in the order specified until a connection is successful. hive.server2.authentication.ldap.baseDN LDAP base DN hive.server2.authentication.ldap.Domain hive.server2.authentication.ldap.groupDNPattern COLON-separated list of patterns to use to find DNs for group entities in this directory. Use %s where the actual group name is to be substituted for. For example: CN=%s,CN=Groups,DC=subdomain,DC=domain,DC=com. hive.server2.authentication.ldap.groupFilter COMMA-separated list of LDAP Group names (short name not full DNs). For example: HiveAdmins,HadoopAdmins,Administrators hive.server2.authentication.ldap.userDNPattern COLON-separated list of patterns to use to find DNs for users in this directory. Use %s where the actual group name is to be substituted for. For example: CN=%s,CN=Users,DC=subdomain,DC=domain,DC=com. hive.server2.authentication.ldap.userFilter COMMA-separated list of LDAP usernames (just short names, not full DNs). For example: hiveuser,impalauser,hiveadmin,hadoopadmin hive.server2.authentication.ldap.guidKey uid LDAP attribute name whose values are unique in this LDAP server. For example: uid or CN. hive.server2.authentication.ldap.groupMembershipKey member LDAP attribute name on the group object that contains the list of distinguished names for the user, group, and contact objects that are members of the group. For example: member, uniqueMember or memberUid hive.server2.authentication.ldap.userMembershipKey LDAP attribute name on the user object that contains groups of which the user is a direct member, except for the primary group, which is represented by the primaryGroupId. For example: memberOf hive.server2.authentication.ldap.groupClassKey groupOfNames LDAP attribute name on the group entry that is to be used in LDAP group searches. For example: group, groupOfNames or groupOfUniqueNames. hive.server2.authentication.ldap.customLDAPQuery A full LDAP query that LDAP Atn provider uses to execute against LDAP Server. If this query returns a null resultset, the LDAP Provider fails the Authentication request, succeeds if the user is part of the resultset.For example: (&(objectClass=group)(objectClass=top)(instanceType=4)(cn=Domain*)) (&(objectClass=person)(|(sAMAccountName=admin)(|(memberOf=CN=Domain Admins,CN=Users,DC=domain,DC=com)(memberOf=CN=Administrators,CN=Builtin,DC=domain,DC=com)))) hive.server2.custom.authentication.class Custom authentication class. Used when property 'hive.server2.authentication' is set to 'CUSTOM'. Provided class must be a proper implementation of the interface org.apache.hive.service.auth.PasswdAuthenticationProvider. HiveServer2 will call its Authenticate(user, passed) method to authenticate requests. The implementation may optionally implement Hadoop's org.apache.hadoop.conf.Configurable class to grab Hive's Configuration object. hive.server2.authentication.pam.services List of the underlying pam services that should be used when auth type is PAM A file with the same name must exist in /etc/pam.d hive.server2.enable.doAs true Setting this property to true will have HiveServer2 execute Hive operations as the user making the calls to it. hive.server2.table.type.mapping CLASSIC Expects one of [classic, hive]. This setting reflects how HiveServer2 will report the table types for JDBC and other client implementations that retrieve the available tables and supported table types HIVE : Exposes Hive's native table types like MANAGED_TABLE, EXTERNAL_TABLE, VIRTUAL_VIEW CLASSIC : More generic types like TABLE and VIEW hive.server2.session.hook hive.server2.use.SSL false Set this to true for using SSL encryption in HiveServer2. hive.server2.keystore.path SSL certificate keystore location. hive.server2.keystore.password SSL certificate keystore password. hive.server2.map.fair.scheduler.queue true If the YARN fair scheduler is configured and HiveServer2 is running in non-impersonation mode, this setting determines the user for fair scheduler queue mapping. If set to true (default), the logged-in user determines the fair scheduler queue for submitted jobs, so that map reduce resource usage can be tracked by user. If set to false, all Hive jobs go to the 'hive' user's queue. hive.server2.builtin.udf.whitelist Comma separated list of builtin udf names allowed in queries. An empty whitelist allows all builtin udfs to be executed. The udf black list takes precedence over udf white list hive.server2.builtin.udf.blacklist Comma separated list of udfs names. These udfs will not be allowed in queries. The udf black list takes precedence over udf white list hive.allow.udf.load.on.demand false Whether enable loading UDFs from metastore on demand; this is mostly relevant for HS2 and was the default behavior before Hive 1.2. Off by default. hive.server2.session.check.interval 6h Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. The time should be bigger than or equal to 3000 msec. The check interval for session/operation timeout, which can be disabled by setting to zero or negative value. hive.server2.close.session.on.disconnect true Session will be closed when connection is closed. Set this to false to have session outlive its parent connection. hive.server2.idle.session.timeout 7d Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Session will be closed when it's not accessed for this duration, which can be disabled by setting to zero or negative value. hive.server2.idle.operation.timeout 5d Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Operation will be closed when it's not accessed for this duration of time, which can be disabled by setting to zero value. With positive value, it's checked for operations in terminal state only (FINISHED, CANCELED, CLOSED, ERROR). With negative value, it's checked for all of the operations regardless of state. hive.server2.idle.session.check.operation true Session will be considered to be idle only if there is no activity, and there is no pending operation. This setting takes effect only if session idle timeout (hive.server2.idle.session.timeout) and checking (hive.server2.session.check.interval) are enabled. hive.server2.thrift.client.retry.limit 1 Number of retries upon failure of Thrift HiveServer2 calls hive.server2.thrift.client.connect.retry.limit 1 Number of retries while opening a connection to HiveServe2 hive.server2.thrift.client.retry.delay.seconds 1s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Number of seconds for the HiveServer2 thrift client to wait between consecutive connection attempts. Also specifies the time to wait between retrying thrift calls upon failures hive.server2.thrift.client.user anonymous Username to use against thrift client hive.server2.thrift.client.password anonymous Password to use against thrift client hive.server2.thrift.resultset.serialize.in.tasks false Whether we should serialize the Thrift structures used in JDBC ResultSet RPC in task nodes. We use SequenceFile and ThriftJDBCBinarySerDe to read and write the final results if this is true. hive.server2.thrift.resultset.max.fetch.size 10000 Max number of rows sent in one Fetch RPC call by the server to the client. hive.server2.thrift.resultset.default.fetch.size 1000 The number of rows sent in one Fetch RPC call by the server to the client, if not specified by the client. hive.server2.xsrf.filter.enabled false If enabled, HiveServer2 will block any requests made to it over http if an X-XSRF-HEADER header is not present hive.security.command.whitelist set,reset,dfs,add,list,delete,reload,compile Comma separated list of non-SQL Hive commands users are authorized to execute hive.server2.job.credential.provider.path If set, this configuration property should provide a comma-separated list of URLs that indicates the type and location of providers to be used by hadoop credential provider API. It provides HiveServer2 the ability to provide job-specific credential providers for jobs run using MR and Spark execution engines. This functionality has not been tested against Tez. hive.mv.files.thread 15 Expects a byte size value with unit (blank for bytes, kb, mb, gb, tb, pb). The size should be in between 0Pb (inclusive) and 1Kb (inclusive). Number of threads used to move files in move task. Set it to 0 to disable multi-threaded file moves. This parameter is also used by MSCK to check tables. hive.load.dynamic.partitions.thread 15 Expects a byte size value with unit (blank for bytes, kb, mb, gb, tb, pb). The size should be in between 1 bytes (inclusive) and 1Kb (inclusive). Number of threads used to load dynamic partitions. hive.multi.insert.move.tasks.share.dependencies false If this is set all move tasks for tables/partitions (not directories) at the end of a multi-insert query will only begin once the dependencies for all these move tasks have been met. Advantages: If concurrency is enabled, the locks will only be released once the query has finished, so with this config enabled, the time when the table/partition is generated will be much closer to when the lock on it is released. Disadvantages: If concurrency is not enabled, with this disabled, the tables/partitions which are produced by this query and finish earlier will be available for querying much earlier. Since the locks are only released once the query finishes, this does not apply if concurrency is enabled. hive.exec.infer.bucket.sort false If this is set, when writing partitions, the metadata will include the bucketing/sorting properties with which the data was written if any (this will not overwrite the metadata inherited from the table if the table is bucketed/sorted) hive.exec.infer.bucket.sort.num.buckets.power.two false If this is set, when setting the number of reducers for the map reduce task which writes the final output files, it will choose a number which is a power of two, unless the user specifies the number of reducers to use using mapred.reduce.tasks. The number of reducers may be set to a power of two, only to be followed by a merge task meaning preventing anything from being inferred. With hive.exec.infer.bucket.sort set to true: Advantages: If this is not set, the number of buckets for partitions will seem arbitrary, which means that the number of mappers used for optimized joins, for example, will be very low. With this set, since the number of buckets used for any partition is a power of two, the number of mappers used for optimized joins will be the least number of buckets used by any partition being joined. Disadvantages: This may mean a much larger or much smaller number of reducers being used in the final map reduce job, e.g. if a job was originally going to take 257 reducers, it will now take 512 reducers, similarly if the max number of reducers is 511, and a job was going to use this many, it will now use 256 reducers. hive.optimize.listbucketing false Enable list bucketing optimizer. Default value is false so that we disable it by default. hive.server.read.socket.timeout 10s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Timeout for the HiveServer to close the connection if no response from the client. By default, 10 seconds. hive.server.tcp.keepalive true Whether to enable TCP keepalive for the Hive Server. Keepalive will prevent accumulation of half-open connections. hive.decode.partition.name false Whether to show the unquoted partition names in query results. hive.execution.engine mr Expects one of [mr, tez, spark]. Chooses execution engine. Options are: mr (Map reduce, default), tez, spark. While MR remains the default engine for historical reasons, it is itself a historical engine and is deprecated in Hive 2 line. It may be removed without further warning. hive.execution.mode container Expects one of [container, llap]. Chooses whether query fragments will run in container or in llap hive.jar.directory This is the location hive in tez mode will look for to find a site wide installed hive instance. hive.user.install.directory /user/ If hive (in tez mode only) cannot find a usable hive jar in "hive.jar.directory", it will upload the hive jar to "hive.user.install.directory/user.name" and use it to run queries. hive.vectorized.execution.enabled false This flag should be set to true to enable vectorized mode of query execution. The default value is false. hive.vectorized.execution.reduce.enabled true This flag should be set to true to enable vectorized mode of the reduce-side of query execution. The default value is true. hive.vectorized.execution.reduce.groupby.enabled true This flag should be set to true to enable vectorized mode of the reduce-side GROUP BY query execution. The default value is true. hive.vectorized.execution.mapjoin.native.enabled true This flag should be set to true to enable native (i.e. non-pass through) vectorization of queries using MapJoin. The default value is true. hive.vectorized.execution.mapjoin.native.multikey.only.enabled false This flag should be set to true to restrict use of native vector map join hash tables to the MultiKey in queries using MapJoin. The default value is false. hive.vectorized.execution.mapjoin.minmax.enabled false This flag should be set to true to enable vector map join hash tables to use max / max filtering for integer join queries using MapJoin. The default value is false. hive.vectorized.execution.mapjoin.overflow.repeated.threshold -1 The number of small table rows for a match in vector map join hash tables where we use the repeated field optimization in overflow vectorized row batch for join queries using MapJoin. A value of -1 means do use the join result optimization. Otherwise, threshold value can be 0 to maximum integer. hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled false This flag should be set to true to enable use of native fast vector map join hash tables in queries using MapJoin. The default value is false. hive.vectorized.groupby.checkinterval 100000 Number of entries added to the group by aggregation hash before a recomputation of average entry size is performed. hive.vectorized.groupby.maxentries 1000000 Max number of entries in the vector group by aggregation hashtables. Exceeding this will trigger a flush irrelevant of memory pressure condition. hive.vectorized.groupby.flush.percent 0.1 Percent of entries in the group by aggregation hash flushed when the memory threshold is exceeded. hive.vectorized.execution.reducesink.new.enabled true This flag should be set to true to enable the new vectorization of queries using ReduceSink. iThe default value is true. hive.vectorized.use.vectorized.input.format true This flag should be set to true to enable vectorizing with vectorized input file format capable SerDe. The default value is true. hive.vectorized.use.vector.serde.deserialize true This flag should be set to true to enable vectorizing rows using vector deserialize. The default value is true. hive.vectorized.use.row.serde.deserialize false This flag should be set to true to enable vectorizing using row deserialize. The default value is false. hive.vectorized.adaptor.usage.mode all Expects one of [none, chosen, all]. Specifies the extent to which the VectorUDFAdaptor will be used for UDFs that do not have a cooresponding vectorized class. 0. none : disable any usage of VectorUDFAdaptor 1. chosen : use VectorUDFAdaptor for a small set of UDFs that were choosen for good performance 2. all : use VectorUDFAdaptor for all UDFs hive.typecheck.on.insert true This property has been extended to control whether to check, convert, and normalize partition value to conform to its column type in partition operations including but not limited to insert, such as alter, describe etc. hive.hadoop.classpath For Windows OS, we need to pass HIVE_HADOOP_CLASSPATH Java parameter while starting HiveServer2 using "-hiveconf hive.hadoop.classpath=%HIVE_LIB%". hive.rpc.query.plan false Whether to send the query plan via local resource or RPC hive.compute.splits.in.am true Whether to generate the splits locally or in the AM (tez only) hive.tez.input.generate.consistent.splits true Whether to generate consistent split locations when generating splits in the AM hive.prewarm.enabled false Enables container prewarm for Tez/Spark (Hadoop 2 only) hive.prewarm.numcontainers 10 Controls the number of containers to prewarm for Tez/Spark (Hadoop 2 only) hive.stageid.rearrange none Expects one of [none, idonly, traverse, execution]. hive.explain.dependency.append.tasktype false hive.counters.group.name HIVE The name of counter group for internal Hive variables (CREATED_FILE, FATAL_ERROR, etc.) hive.support.quoted.identifiers column Expects one of [none, column]. Whether to use quoted identifier. 'none' or 'column' can be used. none: default(past) behavior. Implies only alphaNumeric and underscore are valid characters in identifiers. column: implies column names can contain any character. hive.support.special.characters.tablename true This flag should be set to true to enable support for special characters in table names. When it is set to false, only [a-zA-Z_0-9]+ are supported. The only supported special character right now is '/'. This flag applies only to quoted table names. The default value is true. hive.users.in.admin.role Comma separated list of users who are in admin role for bootstrapping. More users can be added in ADMIN role later. hive.compat 0.12 Enable (configurable) deprecated behaviors by setting desired level of backward compatibility. Setting to 0.12: Maintains division behavior: int / int = double hive.convert.join.bucket.mapjoin.tez false Whether joins can be automatically converted to bucket map joins in hive when tez is used as the execution engine. hive.exec.check.crossproducts true Check if a plan contains a Cross Product. If there is one, output a warning to the Session's console. hive.localize.resource.wait.interval 5000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Time to wait for another thread to localize the same resource for hive-tez. hive.localize.resource.num.wait.attempts 5 The number of attempts waiting for localizing a resource in hive-tez. hive.tez.auto.reducer.parallelism false Turn on Tez' auto reducer parallelism feature. When enabled, Hive will still estimate data sizes and set parallelism estimates. Tez will sample source vertices' output sizes and adjust the estimates at runtime as necessary. hive.tez.max.partition.factor 2.0 When auto reducer parallelism is enabled this factor will be used to over-partition data in shuffle edges. hive.tez.min.partition.factor 0.25 When auto reducer parallelism is enabled this factor will be used to put a lower limit to the number of reducers that tez specifies. hive.tez.bucket.pruning false When pruning is enabled, filters on bucket columns will be processed by filtering the splits against a bitset of included buckets. This needs predicates produced by hive.optimize.ppd and hive.optimize.index.filters. hive.tez.bucket.pruning.compat true When pruning is enabled, handle possibly broken inserts due to negative hashcodes. This occasionally doubles the data scan cost, but is default enabled for safety hive.tez.dynamic.partition.pruning true When dynamic pruning is enabled, joins on partition keys will be processed by sending events from the processing vertices to the Tez application master. These events will be used to prune unnecessary partitions. hive.tez.dynamic.partition.pruning.max.event.size 1048576 Maximum size of events sent by processors in dynamic pruning. If this size is crossed no pruning will take place. hive.tez.dynamic.partition.pruning.max.data.size 104857600 Maximum total data size of events in dynamic pruning. hive.tez.dynamic.semijoin.reduction true When dynamic semijoin is enabled, shuffle joins will perform a leaky semijoin before shuffle. This requires hive.tez.dynamic.partition.pruning to be enabled. hive.tez.min.bloom.filter.entries 1000000 Bloom filter should be of at min certain size to be effective hive.tez.max.bloom.filter.entries 100000000 Bloom filter should be of at max certain size to be effective hive.tez.bloom.filter.factor 2.0 Bloom filter should be a multiple of this factor with nDV hive.tez.bigtable.minsize.semijoin.reduction 1000000 Big table for runtime filteting should be of atleast this size hive.tez.dynamic.semijoin.reduction.threshold 0.5 Only perform semijoin optimization if the estimated benefit at or above this fraction of the target table hive.tez.smb.number.waves 0.5 The number of waves in which to run the SMB join. Account for cluster being occupied. Ideally should be 1 wave. hive.tez.exec.print.summary false Display breakdown of execution steps, for every query executed by the shell. hive.tez.exec.inplace.progress true Updates tez job execution progress in-place in the terminal when hive-cli is used. hive.server2.in.place.progress true Allows hive server 2 to send progress bar update information. This is currently available only if the execution engine is tez. hive.spark.exec.inplace.progress true Updates spark job execution progress in-place in the terminal. hive.tez.container.max.java.heap.fraction 0.8 This is to override the tez setting with the same name hive.tez.task.scale.memory.reserve-fraction.min 0.3 This is to override the tez setting tez.task.scale.memory.reserve-fraction hive.tez.task.scale.memory.reserve.fraction.max 0.5 The maximum fraction of JVM memory which Tez will reserve for the processor hive.tez.task.scale.memory.reserve.fraction -1.0 The customized fraction of JVM memory which Tez will reserve for the processor hive.llap.io.enabled Whether the LLAP IO layer is enabled. hive.llap.io.nonvector.wrapper.enabled true Whether the LLAP IO layer is enabled for non-vectorized queries that read inputs that can be vectorized hive.llap.io.memory.mode cache Expects one of [cache, none]. LLAP IO memory usage; 'cache' (the default) uses data and metadata cache with a custom off-heap allocator, 'none' doesn't use either (this mode may result in significant performance degradation) hive.llap.io.allocator.alloc.min 256Kb Expects a byte size value with unit (blank for bytes, kb, mb, gb, tb, pb). Minimum allocation possible from LLAP buddy allocator. Allocations below that are padded to minimum allocation. For ORC, should generally be the same as the expected compression buffer size, or next lowest power of 2. Must be a power of 2. hive.llap.io.allocator.alloc.max 16Mb Expects a byte size value with unit (blank for bytes, kb, mb, gb, tb, pb). Maximum allocation possible from LLAP buddy allocator. For ORC, should be as large as the largest expected ORC compression buffer size. Must be a power of 2. hive.llap.io.metadata.fraction 0.1 Temporary setting for on-heap metadata cache fraction of xmx, set to avoid potential heap problems on very large datasets when on-heap metadata cache takes over everything. -1 managed metadata and data together (which is more flexible). This setting will be removed (in effect become -1) once ORC metadata cache is moved off-heap. hive.llap.io.allocator.arena.count 8 Arena count for LLAP low-level cache; cache will be allocated in the steps of (size/arena_count) bytes. This size must be <= 1Gb and >= max allocation; if it is not the case, an adjusted size will be used. Using powers of 2 is recommended. hive.llap.io.memory.size 1Gb Expects a byte size value with unit (blank for bytes, kb, mb, gb, tb, pb). Maximum size for IO allocator or ORC low-level cache. hive.llap.io.allocator.direct true Whether ORC low-level cache should use direct allocation. hive.llap.io.allocator.mmap false Whether ORC low-level cache should use memory mapped allocation (direct I/O). This is recommended to be used along-side NVDIMM (DAX) or NVMe flash storage. hive.llap.io.allocator.mmap.path /tmp Expects a writable directory on the local filesystem. The directory location for mapping NVDIMM/NVMe flash storage into the ORC low-level cache. hive.llap.io.use.lrfu true Whether ORC low-level cache should use LRFU cache policy instead of default (FIFO). hive.llap.io.lrfu.lambda 0.01 Lambda for ORC low-level cache LRFU cache policy. Must be in [0, 1]. 0 makes LRFU behave like LFU, 1 makes it behave like LRU, values in between balance accordingly. hive.llap.cache.allow.synthetic.fileid false Whether LLAP cache should use synthetic file ID if real one is not available. Systems like HDFS, Isilon, etc. provide a unique file/inode ID. On other FSes (e.g. local FS), the cache would not work by default because LLAP is unable to uniquely track the files; enabling this setting allows LLAP to generate file ID from the path, size and modification time, which is almost certain to identify file uniquely. However, if you use a FS without file IDs and rewrite files a lot (or are paranoid), you might want to avoid this setting. hive.llap.orc.gap.cache true Whether LLAP cache for ORC should remember gaps in ORC compression buffer read estimates, to avoid re-reading the data that was read once and discarded because it is unneeded. This is only necessary for ORC files written before HIVE-9660. hive.llap.io.use.fileid.path true Whether LLAP should use fileId (inode)-based path to ensure better consistency for the cases of file overwrites. This is supported on HDFS. hive.llap.io.encode.enabled true Whether LLAP should try to re-encode and cache data for non-ORC formats. This is used on LLAP Server side to determine if the infrastructure for that is initialized. hive.llap.io.encode.formats org.apache.hadoop.mapred.TextInputFormat, The table input formats for which LLAP IO should re-encode and cache data. Comma-separated list. hive.llap.io.encode.alloc.size 256Kb Expects a byte size value with unit (blank for bytes, kb, mb, gb, tb, pb). Allocation size for the buffers used to cache encoded data from non-ORC files. Must be a power of two between hive.llap.io.allocator.alloc.min and hive.llap.io.allocator.alloc.max. hive.llap.io.encode.vector.serde.enabled true Whether LLAP should use vectorized SerDe reader to read text data when re-encoding. hive.llap.io.encode.vector.serde.async.enabled true Whether LLAP should use async mode in vectorized SerDe reader to read text data. hive.llap.io.encode.slice.row.count 100000 Row count to use to separate cache slices when reading encoded data from row-based inputs into LLAP cache, if this feature is enabled. hive.llap.io.encode.slice.lrr true Whether to separate cache slices when reading encoded data from text inputs via MR MR LineRecordRedader into LLAP cache, if this feature is enabled. Safety flag. hive.llap.io.orc.time.counters true Whether to enable time counters for LLAP IO layer (time spent in HDFS, etc.) hive.llap.auto.allow.uber false Whether or not to allow the planner to run vertices in the AM. hive.llap.auto.enforce.tree true Enforce that all parents are in llap, before considering vertex hive.llap.auto.enforce.vectorized true Enforce that inputs are vectorized, before considering vertex hive.llap.auto.enforce.stats true Enforce that col stats are available, before considering vertex hive.llap.auto.max.input.size 10737418240 Check input size, before considering vertex (-1 disables check) hive.llap.auto.max.output.size 1073741824 Check output size, before considering vertex (-1 disables check) hive.llap.skip.compile.udf.check false Whether to skip the compile-time check for non-built-in UDFs when deciding whether to execute tasks in LLAP. Skipping the check allows executing UDFs from pre-localized jars in LLAP; if the jars are not pre-localized, the UDFs will simply fail to load. hive.llap.allow.permanent.fns true Whether LLAP decider should allow permanent UDFs. hive.llap.execution.mode none Expects one of [auto, none, all, map, only]. Chooses whether query fragments will run in container or in llap hive.llap.object.cache.enabled true Cache objects (plans, hashtables, etc) in llap hive.llap.io.decoding.metrics.percentiles.intervals 30 Comma-delimited set of integers denoting the desired rollover intervals (in seconds) for percentile latency metrics on the LLAP daemon IO decoding time. hive.llap.queue.metrics.percentiles.intervals hive.llap.io.threadpool.size 10 Specify the number of threads to use for low-level IO thread pool. hive.llap.daemon.service.principal The name of the LLAP daemon's service principal. hive.llap.daemon.keytab.file The path to the Kerberos Keytab file containing the LLAP daemon's service principal. hive.llap.zk.sm.principal The name of the principal to use to talk to ZooKeeper for ZooKeeper SecretManager. hive.llap.zk.sm.keytab.file The path to the Kerberos Keytab file containing the principal to use to talk to ZooKeeper for ZooKeeper SecretManager. hive.llap.webui.spnego.keytab The path to the Kerberos Keytab file containing the LLAP WebUI SPNEGO principal. Typical value would look like /etc/security/keytabs/spnego.service.keytab. hive.llap.webui.spnego.principal The LLAP WebUI SPNEGO service principal. Configured similarly to hive.server2.webui.spnego.principal hive.llap.task.principal The name of the principal to use to run tasks. By default, the clients are required to provide tokens to access HDFS/etc. hive.llap.task.keytab.file The path to the Kerberos Keytab file containing the principal to use to run tasks. By default, the clients are required to provide tokens to access HDFS/etc. hive.llap.zk.sm.connectionString ZooKeeper connection string for ZooKeeper SecretManager. hive.llap.zk.registry.user In the LLAP ZooKeeper-based registry, specifies the username in the Zookeeper path. This should be the hive user or whichever user is running the LLAP daemon. hive.llap.zk.registry.namespace In the LLAP ZooKeeper-based registry, overrides the ZK path namespace. Note that using this makes the path management (e.g. setting correct ACLs) your responsibility. hive.llap.daemon.acl * The ACL for LLAP daemon. hive.llap.daemon.acl.blocked The deny ACL for LLAP daemon. hive.llap.management.acl * The ACL for LLAP daemon management. hive.llap.management.acl.blocked The deny ACL for LLAP daemon management. hive.llap.remote.token.requires.signing true Expects one of [false, except_llap_owner, true]. Whether the token returned from LLAP management API should require fragment signing. True by default; can be disabled to allow CLI to get tokens from LLAP in a secure cluster by setting it to true or 'except_llap_owner' (the latter returns such tokens to everyone except the user LLAP cluster is authenticating under). hive.llap.daemon.delegation.token.lifetime 14d Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. LLAP delegation token lifetime, in seconds if specified without a unit. hive.llap.management.rpc.port 15004 RPC port for LLAP daemon management service. hive.llap.auto.auth false Whether or not to set Hadoop configs to enable auth in LLAP web app. hive.llap.daemon.rpc.num.handlers 5 Number of RPC handlers for LLAP daemon. hive.llap.daemon.work.dirs Working directories for the daemon. This should not be set if running as a YARN application via Slider. It must be set when not running via Slider on YARN. If the value is set when running as a Slider YARN application, the specified value will be used. hive.llap.daemon.yarn.shuffle.port 15551 YARN shuffle port for LLAP-daemon-hosted shuffle. hive.llap.daemon.yarn.container.mb -1 llap server yarn container size in MB. Used in LlapServiceDriver and package.py hive.llap.daemon.queue.name Queue name within which the llap slider application will run. Used in LlapServiceDriver and package.py hive.llap.daemon.container.id ContainerId of a running LlapDaemon. Used to publish to the registry hive.llap.daemon.nm.address NM Address host:rpcPort for the NodeManager on which the instance of the daemon is running. Published to the llap registry. Should never be set by users hive.llap.daemon.shuffle.dir.watcher.enabled false TODO doc hive.llap.daemon.am.liveness.heartbeat.interval.ms 10000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Tez AM-LLAP heartbeat interval (milliseconds). This needs to be below the task timeout interval, but otherwise as high as possible to avoid unnecessary traffic. hive.llap.am.liveness.connection.timeout.ms 10000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Amount of time to wait on connection failures to the AM from an LLAP daemon before considering the AM to be dead. hive.llap.am.use.fqdn false Whether to use FQDN of the AM machine when submitting work to LLAP. hive.llap.am.liveness.connection.sleep.between.retries.ms 2000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Sleep duration while waiting to retry connection failures to the AM from the daemon for the general keep-alive thread (milliseconds). hive.llap.task.scheduler.timeout.seconds 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Amount of time to wait before failing the query when there are no llap daemons running (alive) in the cluster. hive.llap.daemon.num.executors 4 Number of executors to use in LLAP daemon; essentially, the number of tasks that can be executed in parallel. hive.llap.daemon.am-reporter.max.threads 4 Maximum number of threads to be used for AM reporter. If this is lower than number of executors in llap daemon, it would be set to number of executors at runtime. hive.llap.daemon.rpc.port 0 The LLAP daemon RPC port. hive.llap.daemon.memory.per.instance.mb 4096 The total amount of memory to use for the executors inside LLAP (in megabytes). hive.llap.daemon.xmx.headroom 5% The total amount of heap memory set aside by LLAP and not used by the executors. Can be specified as size (e.g. '512Mb'), or percentage (e.g. '5%'). Note that the latter is derived from the total daemon XMX, which can be different from the total executor memory if the cache is on-heap; although that's not the default configuration. hive.llap.daemon.vcpus.per.instance 4 The total number of vcpus to use for the executors inside LLAP. hive.llap.daemon.num.file.cleaner.threads 1 Number of file cleaner threads in LLAP. hive.llap.file.cleanup.delay.seconds 300s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. How long to delay before cleaning up query files in LLAP (in seconds, for debugging). hive.llap.daemon.service.hosts Explicitly specified hosts to use for LLAP scheduling. Useful for testing. By default, YARN registry is used. hive.llap.daemon.service.refresh.interval.sec 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. LLAP YARN registry service list refresh delay, in seconds. hive.llap.daemon.communicator.num.threads 10 Number of threads to use in LLAP task communicator in Tez AM. hive.llap.daemon.download.permanent.fns false Whether LLAP daemon should localize the resources for permanent UDFs. hive.llap.task.scheduler.node.reenable.min.timeout.ms 200ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Minimum time after which a previously disabled node will be re-enabled for scheduling, in milliseconds. This may be modified by an exponential back-off if failures persist. hive.llap.task.scheduler.node.reenable.max.timeout.ms 10000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Maximum time after which a previously disabled node will be re-enabled for scheduling, in milliseconds. This may be modified by an exponential back-off if failures persist. hive.llap.task.scheduler.node.disable.backoff.factor 1.5 Backoff factor on successive blacklists of a node due to some failures. Blacklist times start at the min timeout and go up to the max timeout based on this backoff factor. hive.llap.task.scheduler.num.schedulable.tasks.per.node 0 The number of tasks the AM TaskScheduler will try allocating per node. 0 indicates that this should be picked up from the Registry. -1 indicates unlimited capacity; positive values indicate a specific bound. hive.llap.task.scheduler.locality.delay 0ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. The time should be in between -1 msec (inclusive) and 9223372036854775807 msec (inclusive). Amount of time to wait before allocating a request which contains location information, to a location other than the ones requested. Set to -1 for an infinite delay, 0for no delay. hive.llap.daemon.task.preemption.metrics.intervals 30,60,300 Comma-delimited set of integers denoting the desired rollover intervals (in seconds) for percentile latency metrics. Used by LLAP daemon task scheduler metrics for time taken to kill task (due to pre-emption) and useful time wasted by the task that is about to be preempted. hive.llap.daemon.task.scheduler.wait.queue.size 10 LLAP scheduler maximum queue size. hive.llap.daemon.wait.queue.comparator.class.name org.apache.hadoop.hive.llap.daemon.impl.comparator.ShortestJobFirstComparator The priority comparator to use for LLAP scheduler prioroty queue. The built-in options are org.apache.hadoop.hive.llap.daemon.impl.comparator.ShortestJobFirstComparator and .....FirstInFirstOutComparator hive.llap.daemon.task.scheduler.enable.preemption true Whether non-finishable running tasks (e.g. a reducer waiting for inputs) should be preempted by finishable tasks inside LLAP scheduler. hive.llap.task.communicator.connection.timeout.ms 16000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Connection timeout (in milliseconds) before a failure to an LLAP daemon from Tez AM. hive.llap.task.communicator.listener.thread-count 30 The number of task communicator listener threads. hive.llap.task.communicator.connection.sleep.between.retries.ms 2000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Sleep duration (in milliseconds) to wait before retrying on error when obtaining a connection to LLAP daemon from Tez AM. hive.llap.daemon.web.port 15002 LLAP daemon web UI port. hive.llap.daemon.web.ssl false Whether LLAP daemon web UI should use SSL. hive.llap.client.consistent.splits false Whether to setup split locations to match nodes on which llap daemons are running, instead of using the locations provided by the split itself. If there is no llap daemon running, fall back to locations provided by the split. This is effective only if hive.execution.mode is llap hive.llap.validate.acls true Whether LLAP should reject permissive ACLs in some cases (e.g. its own management protocol or ZK paths), similar to how ssh refuses a key with bad access permissions. hive.llap.daemon.output.service.port 15003 LLAP daemon output service port hive.llap.daemon.output.stream.timeout 120s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. The timeout for the client to connect to LLAP output service and start the fragment output after sending the fragment. The fragment will fail if its output is not claimed. hive.llap.daemon.output.service.send.buffer.size 131072 Send buffer size to be used by LLAP daemon output service hive.llap.daemon.output.service.max.pending.writes 8 Maximum number of queued writes allowed per connection when sending data via the LLAP output service to external clients. hive.llap.enable.grace.join.in.llap false Override if grace join should be allowed to run in llap. hive.llap.hs2.coordinator.enabled true Whether to create the LLAP coordinator; since execution engine and container vs llap settings are both coming from job configs, we don't know at start whether this should be created. Default true. hive.llap.daemon.logger query-routing Expects one of [query-routing, rfa, console]. logger used for llap-daemons. hive.spark.use.op.stats true Whether to use operator stats to determine reducer parallelism for Hive on Spark. If this is false, Hive will use source table stats to determine reducer parallelism for all first level reduce tasks, and the maximum reducer parallelism from all parents for all the rest (second level and onward) reducer tasks. hive.spark.use.file.size.for.mapjoin false If this is set to true, mapjoin optimization in Hive/Spark will use source file sizes associated with TableScan operator on the root of operator tree, instead of using operator statistics. hive.spark.client.future.timeout 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Timeout for requests from Hive client to remote Spark driver. hive.spark.job.monitor.timeout 60s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Timeout for job monitor to get Spark job state. hive.spark.client.connect.timeout 1000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Timeout for remote Spark driver in connecting back to Hive client. hive.spark.client.server.connect.timeout 90000ms Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified. Timeout for handshake between Hive client and remote Spark driver. Checked by both processes. hive.spark.client.secret.bits 256 Number of bits of randomness in the generated secret for communication between Hive client and remote Spark driver. Rounded down to the nearest multiple of 8. hive.spark.client.rpc.threads 8 Maximum number of threads for remote Spark driver's RPC event loop. hive.spark.client.rpc.max.size 52428800 Maximum message size in bytes for communication between Hive client and remote Spark driver. Default is 50MB. hive.spark.client.channel.log.level Channel logging level for remote Spark driver. One of {DEBUG, ERROR, INFO, TRACE, WARN}. hive.spark.client.rpc.sasl.mechanisms DIGEST-MD5 Name of the SASL mechanism to use for authentication. hive.spark.client.rpc.server.address The server address of HiverServer2 host to be used for communication between Hive client and remote Spark driver. Default is empty, which means the address will be determined in the same way as for hive.server2.thrift.bind.host.This is only necessary if the host has mutiple network addresses and if a different network address other than hive.server2.thrift.bind.host is to be used. hive.spark.client.rpc.server.port A list of port ranges which can be used by RPC server with the format of 49152-49222,49228 and a random one is selected from the list. Default is empty, which randomly selects one port from all available ones. hive.spark.dynamic.partition.pruning false When dynamic pruning is enabled, joins on partition keys will be processed by writing to a temporary HDFS file, and read later for removing unnecessary partitions. hive.spark.dynamic.partition.pruning.max.data.size 104857600 Maximum total data size in dynamic pruning. hive.spark.use.groupby.shuffle true Spark groupByKey transformation has better performance but uses unbounded memory.Turn this off when there is a memory issue. hive.reorder.nway.joins true Runs reordering of tables within single n-way join (i.e.: picks streamtable) hive.merge.nway.joins true Merge adjacent joins into a single n-way join hive.log.every.n.records 0 Expects value bigger than 0. If value is greater than 0 logs in fixed intervals of size n rather than exponentially. hive.msck.path.validation throw Expects one of [throw, skip, ignore]. The approach msck should take with HDFS directories that are partition-like but contain unsupported characters. 'throw' (an exception) is the default; 'skip' will skip the invalid directories and still repair the others; 'ignore' will skip the validation (legacy behavior, causes bugs in many cases) hive.msck.repair.batch.size 0 Batch size for the msck repair command. If the value is greater than zero, it will execute batch wise with the configured batch size. The default value is zero. Zero means it will execute directly (Not batch wise) hive.server2.llap.concurrent.queries -1 The number of queries allowed in parallel via llap. Negative number implies 'infinite'. hive.tez.enable.memory.manager true Enable memory manager for tez hive.hash.table.inflation.factor 2.0 Expected inflation factor between disk/in memory representation of hash tables hive.log.trace.id Log tracing id that can be used by upstream clients for tracking respective logs. Truncated to 64 characters. Defaults to use auto-generated session id. hive.conf.restricted.list hive.security.authenticator.manager,hive.security.authorization.manager,hive.security.metastore.authorization.manager,hive.security.metastore.authenticator.manager,hive.users.in.admin.role,hive.server2.xsrf.filter.enabled,hive.security.authorization.enabled,hive.server2.authentication.ldap.baseDN,hive.server2.authentication.ldap.url,hive.server2.authentication.ldap.Domain,hive.server2.authentication.ldap.groupDNPattern,hive.server2.authentication.ldap.groupFilter,hive.server2.authentication.ldap.userDNPattern,hive.server2.authentication.ldap.userFilter,hive.server2.authentication.ldap.groupMembershipKey,hive.server2.authentication.ldap.userMembershipKey,hive.server2.authentication.ldap.groupClassKey,hive.server2.authentication.ldap.customLDAPQuery Comma separated list of configuration options which are immutable at runtime hive.conf.hidden.list javax.jdo.option.ConnectionPassword,hive.server2.keystore.password,fs.s3.awsAccessKeyId,fs.s3.awsSecretAccessKey,fs.s3n.awsAccessKeyId,fs.s3n.awsSecretAccessKey,fs.s3a.access.key,fs.s3a.secret.key,fs.s3a.proxy.password Comma separated list of configuration options which should not be read by normal user like passwords hive.conf.internal.variable.list hive.added.files.path,hive.added.jars.path,hive.added.archives.path Comma separated list of variables which are used internally and should not be configurable. hive.query.timeout.seconds 0s Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified. Timeout for Running Query in seconds. A nonpositive value means infinite. If the query timeout is also set by thrift API call, the smaller one will be taken. hive.exec.input.listing.max.threads 0 Expects a byte size value with unit (blank for bytes, kb, mb, gb, tb, pb). The size should be in between 0Pb (inclusive) and 1Kb (inclusive). Maximum number of threads that Hive uses to list file information from file systems (recommended > 1 for blobstore). hive.blobstore.supported.schemes s3,s3a,s3n Comma-separated list of supported blobstore schemes. hive.blobstore.use.blobstore.as.scratchdir false Enable the use of scratch directories directly on blob storage systems (it may cause performance penalties). hive.blobstore.optimizations.enabled true This parameter enables a number of optimizations when running on blobstores: (1) If hive.blobstore.use.blobstore.as.scratchdir is false, force the last Hive job to write to the blobstore. This is a performance optimization that forces the final FileSinkOperator to write to the blobstore. See HIVE-15121 for details. sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/create_hive_db.sql0000666000175100017510000000066113245514472026041 0ustar zuulzuul00000000000000CREATE DATABASE metastore; USE metastore; SOURCE /opt/hive/scripts/metastore/upgrade/mysql/hive-schema-2.3.0.mysql.sql; CREATE USER 'hive'@'localhost' IDENTIFIED BY '{{password}}'; REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hive'@'localhost'; GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'localhost' IDENTIFIED BY '{{password}}'; GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%' IDENTIFIED BY '{{password}}'; FLUSH PRIVILEGES; exit sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/core-default.xml0000666000175100017510000023172113245514472025474 0ustar zuulzuul00000000000000 hadoop.common.configuration.version 0.23.0 version of this configuration file hadoop.tmp.dir /tmp/hadoop-${user.name} A base for other temporary directories. io.native.lib.available true Controls whether to use native libraries for bz2 and zlib compression codecs or not. The property does not control any other native libraries. hadoop.http.filter.initializers org.apache.hadoop.http.lib.StaticUserWebFilter A comma separated list of class names. Each class in the list must extend org.apache.hadoop.http.FilterInitializer. The corresponding Filter will be initialized. Then, the Filter will be applied to all user facing jsp and servlet web pages. The ordering of the list defines the ordering of the filters. hadoop.security.authorization false Is service-level authorization enabled? hadoop.security.instrumentation.requires.admin false Indicates if administrator ACLs are required to access instrumentation servlets (JMX, METRICS, CONF, STACKS). hadoop.security.authentication simple Possible values are simple (no authentication), and kerberos hadoop.security.group.mapping org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback Class for user to group mapping (get groups for a given user) for ACL. The default implementation, org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, will determine if the Java Native Interface (JNI) is available. If JNI is available the implementation will use the API within hadoop to resolve a list of groups for a user. If JNI is not available then the shell implementation, ShellBasedUnixGroupsMapping, is used. This implementation shells out to the Linux/Unix environment with the bash -c groups command to resolve a list of groups for a user. hadoop.security.dns.interface The name of the Network Interface from which the service should determine its host name for Kerberos login. e.g. eth2. In a multi-homed environment, the setting can be used to affect the _HOST subsitution in the service Kerberos principal. If this configuration value is not set, the service will use its default hostname as returned by InetAddress.getLocalHost().getCanonicalHostName(). Most clusters will not require this setting. hadoop.security.dns.nameserver The host name or IP address of the name server (DNS) which a service Node should use to determine its own host name for Kerberos Login. Requires hadoop.security.dns.interface. Most clusters will not require this setting. hadoop.security.dns.log-slow-lookups.enabled false Time name lookups (via SecurityUtil) and log them if they exceed the configured threshold. hadoop.security.dns.log-slow-lookups.threshold.ms 1000 If slow lookup logging is enabled, this threshold is used to decide if a lookup is considered slow enough to be logged. hadoop.security.groups.cache.secs 300 This is the config controlling the validity of the entries in the cache containing the user->group mapping. When this duration has expired, then the implementation of the group mapping provider is invoked to get the groups of the user and then cached back. hadoop.security.groups.negative-cache.secs 30 Expiration time for entries in the the negative user-to-group mapping caching, in seconds. This is useful when invalid users are retrying frequently. It is suggested to set a small value for this expiration, since a transient error in group lookup could temporarily lock out a legitimate user. Set this to zero or negative value to disable negative user-to-group caching. hadoop.security.groups.cache.warn.after.ms 5000 If looking up a single user to group takes longer than this amount of milliseconds, we will log a warning message. hadoop.security.groups.cache.background.reload false Whether to reload expired user->group mappings using a background thread pool. If set to true, a pool of hadoop.security.groups.cache.background.reload.threads is created to update the cache in the background. hadoop.security.groups.cache.background.reload.threads 3 Only relevant if hadoop.security.groups.cache.background.reload is true. Controls the number of concurrent background user->group cache entry refreshes. Pending refresh requests beyond this value are queued and processed when a thread is free. hadoop.security.group.mapping.ldap.connection.timeout.ms 60000 This property is the connection timeout (in milliseconds) for LDAP operations. If the LDAP provider doesn't establish a connection within the specified period, it will abort the connect attempt. Non-positive value means no LDAP connection timeout is specified in which case it waits for the connection to establish until the underlying network times out. hadoop.security.group.mapping.ldap.read.timeout.ms 60000 This property is the read timeout (in milliseconds) for LDAP operations. If the LDAP provider doesn't get a LDAP response within the specified period, it will abort the read attempt. Non-positive value means no read timeout is specified in which case it waits for the response infinitely. hadoop.security.group.mapping.ldap.url The URL of the LDAP server to use for resolving user groups when using the LdapGroupsMapping user to group mapping. hadoop.security.group.mapping.ldap.ssl false Whether or not to use SSL when connecting to the LDAP server. hadoop.security.group.mapping.ldap.ssl.keystore File path to the SSL keystore that contains the SSL certificate required by the LDAP server. hadoop.security.group.mapping.ldap.ssl.keystore.password.file The path to a file containing the password of the LDAP SSL keystore. IMPORTANT: This file should be readable only by the Unix user running the daemons. hadoop.security.group.mapping.ldap.bind.user The distinguished name of the user to bind as when connecting to the LDAP server. This may be left blank if the LDAP server supports anonymous binds. hadoop.security.group.mapping.ldap.bind.password.file The path to a file containing the password of the bind user. IMPORTANT: This file should be readable only by the Unix user running the daemons. hadoop.security.group.mapping.ldap.base The search base for the LDAP connection. This is a distinguished name, and will typically be the root of the LDAP directory. hadoop.security.group.mapping.ldap.search.filter.user (&(objectClass=user)(sAMAccountName={0})) An additional filter to use when searching for LDAP users. The default will usually be appropriate for Active Directory installations. If connecting to an LDAP server with a non-AD schema, this should be replaced with (&(objectClass=inetOrgPerson)(uid={0}). {0} is a special string used to denote where the username fits into the filter. If the LDAP server supports posixGroups, Hadoop can enable the feature by setting the value of this property to "posixAccount" and the value of the hadoop.security.group.mapping.ldap.search.filter.group property to "posixGroup". hadoop.security.group.mapping.ldap.search.filter.group (objectClass=group) An additional filter to use when searching for LDAP groups. This should be changed when resolving groups against a non-Active Directory installation. See the description of hadoop.security.group.mapping.ldap.search.filter.user to enable posixGroups support. hadoop.security.group.mapping.ldap.search.attr.member member The attribute of the group object that identifies the users that are members of the group. The default will usually be appropriate for any LDAP installation. hadoop.security.group.mapping.ldap.search.attr.group.name cn The attribute of the group object that identifies the group name. The default will usually be appropriate for all LDAP systems. hadoop.security.group.mapping.ldap.posix.attr.uid.name uidNumber The attribute of posixAccount to use when groups for membership. Mostly useful for schemas wherein groups have memberUids that use an attribute other than uidNumber. hadoop.security.group.mapping.ldap.posix.attr.gid.name gidNumber The attribute of posixAccount indicating the group id. hadoop.security.group.mapping.ldap.directory.search.timeout 10000 The attribute applied to the LDAP SearchControl properties to set a maximum time limit when searching and awaiting a result. Set to 0 if infinite wait period is desired. Default is 10 seconds. Units in milliseconds. hadoop.security.group.mapping.providers Comma separated of names of other providers to provide user to group mapping. Used by CompositeGroupsMapping. hadoop.security.group.mapping.providers.combined true true or false to indicate whether groups from the providers are combined or not. The default value is true. If true, then all the providers will be tried to get groups and all the groups are combined to return as the final results. Otherwise, providers are tried one by one in the configured list order, and if any groups are retrieved from any provider, then the groups will be returned without trying the left ones. hadoop.security.service.user.name.key For those cases where the same RPC protocol is implemented by multiple servers, this configuration is required for specifying the principal name to use for the service when the client wishes to make an RPC call. hadoop.security.uid.cache.secs 14400 This is the config controlling the validity of the entries in the cache containing the userId to userName and groupId to groupName used by NativeIO getFstat(). hadoop.rpc.protection authentication A comma-separated list of protection values for secured sasl connections. Possible values are authentication, integrity and privacy. authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled. hadoop.security.saslproperties.resolver.class can be used to override the hadoop.rpc.protection for a connection at the server side. hadoop.security.saslproperties.resolver.class SaslPropertiesResolver used to resolve the QOP used for a connection. If not specified, the full set of values specified in hadoop.rpc.protection is used while determining the QOP used for the connection. If a class is specified, then the QOP values returned by the class will be used while determining the QOP used for the connection. hadoop.security.sensitive-config-keys secret$ password$ ssl.keystore.pass$ fs.s3.*[Ss]ecret.?[Kk]ey fs.azure.account.key.* credential$ oauth.*token$ hadoop.security.sensitive-config-keys A comma-separated or multi-line list of regular expressions to match configuration keys that should be redacted where appropriate, for example, when logging modified properties during a reconfiguration, private credentials should not be logged. hadoop.workaround.non.threadsafe.getpwuid true Some operating systems or authentication modules are known to have broken implementations of getpwuid_r and getpwgid_r, such that these calls are not thread-safe. Symptoms of this problem include JVM crashes with a stack trace inside these functions. If your system exhibits this issue, enable this configuration parameter to include a lock around the calls as a workaround. An incomplete list of some systems known to have this issue is available at http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations hadoop.kerberos.kinit.command kinit Used to periodically renew Kerberos credentials when provided to Hadoop. The default setting assumes that kinit is in the PATH of users running the Hadoop client. Change this to the absolute path to kinit if this is not the case. hadoop.kerberos.min.seconds.before.relogin 60 The minimum time between relogin attempts for Kerberos, in seconds. hadoop.security.auth_to_local Maps kerberos principals to local user names io.file.buffer.size 4096 The size of buffer for use in sequence files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. io.bytes.per.checksum 512 The number of bytes per checksum. Must not be larger than io.file.buffer.size. io.skip.checksum.errors false If true, when a checksum error is encountered while reading a sequence file, entries are skipped, instead of throwing an exception. io.compression.codecs A comma-separated list of the compression codec classes that can be used for compression/decompression. In addition to any classes specified with this property (which take precedence), codec classes on the classpath are discovered using a Java ServiceLoader. io.compression.codec.bzip2.library system-native The native-code library to be used for compression and decompression by the bzip2 codec. This library could be specified either by by name or the full pathname. In the former case, the library is located by the dynamic linker, usually searching the directories specified in the environment variable LD_LIBRARY_PATH. The value of "system-native" indicates that the default system library should be used. To indicate that the algorithm should operate entirely in Java, specify "java-builtin". io.serializations org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization A list of serialization classes that can be used for obtaining serializers and deserializers. io.seqfile.local.dir ${hadoop.tmp.dir}/io/local The local directory where sequence file stores intermediate data files during merge. May be a comma-separated list of directories on different devices in order to spread disk i/o. Directories that do not exist are ignored. io.map.index.skip 0 Number of index entries to skip between each entry. Zero by default. Setting this to values larger than zero can facilitate opening large MapFiles using less memory. io.map.index.interval 128 MapFile consist of two files - data file (tuples) and index file (keys). For every io.map.index.interval records written in the data file, an entry (record-key, data-file-position) is written in the index file. This is to allow for doing binary search later within the index file to look up records by their keys and get their closest positions in the data file. fs.defaultFS file:/// The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem. fs.default.name file:/// Deprecated. Use (fs.defaultFS) property instead fs.trash.interval 0 Number of minutes after which the checkpoint gets deleted. If zero, the trash feature is disabled. This option may be configured both on the server and the client. If trash is disabled server side then the client side configuration is checked. If trash is enabled on the server side then the value configured on the server is used and the client configuration value is ignored. fs.trash.checkpoint.interval 0 Number of minutes between trash checkpoints. Should be smaller or equal to fs.trash.interval. If zero, the value is set to the value of fs.trash.interval. Every time the checkpointer runs it creates a new checkpoint out of current and removes checkpoints created more than fs.trash.interval minutes ago. fs.protected.directories A comma-separated list of directories which cannot be deleted even by the superuser unless they are empty. This setting can be used to guard important system directories against accidental deletion due to administrator error. fs.AbstractFileSystem.file.impl org.apache.hadoop.fs.local.LocalFs The AbstractFileSystem for file: uris. fs.AbstractFileSystem.har.impl org.apache.hadoop.fs.HarFs The AbstractFileSystem for har: uris. fs.AbstractFileSystem.hdfs.impl org.apache.hadoop.fs.Hdfs The FileSystem for hdfs: uris. fs.AbstractFileSystem.viewfs.impl org.apache.hadoop.fs.viewfs.ViewFs The AbstractFileSystem for view file system for viewfs: uris (ie client side mount table:). fs.AbstractFileSystem.ftp.impl org.apache.hadoop.fs.ftp.FtpFs The FileSystem for Ftp: uris. fs.AbstractFileSystem.webhdfs.impl org.apache.hadoop.fs.WebHdfs The FileSystem for webhdfs: uris. fs.AbstractFileSystem.swebhdfs.impl org.apache.hadoop.fs.SWebHdfs The FileSystem for swebhdfs: uris. fs.ftp.host 0.0.0.0 FTP filesystem connects to this server fs.ftp.host.port 21 FTP filesystem connects to fs.ftp.host on this port fs.df.interval 60000 Disk usage statistics refresh interval in msec. fs.du.interval 600000 File space usage statistics refresh interval in msec. fs.s3.block.size 67108864 Block size to use when writing files to S3. fs.s3.buffer.dir ${hadoop.tmp.dir}/s3 Determines where on the local filesystem the s3:/s3n: filesystem should store files before sending them to S3 (or after retrieving them from S3). fs.s3.maxRetries 4 The maximum number of retries for reading or writing files to S3, before we signal failure to the application. fs.s3.sleepTimeSeconds 10 The number of seconds to sleep between each S3 retry. fs.swift.impl org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem The implementation class of the OpenStack Swift Filesystem fs.automatic.close true By default, FileSystem instances are automatically closed at program exit using a JVM shutdown hook. Setting this property to false disables this behavior. This is an advanced option that should only be used by server applications requiring a more carefully orchestrated shutdown sequence. fs.s3n.block.size 67108864 Block size to use when reading files using the native S3 filesystem (s3n: URIs). fs.s3n.multipart.uploads.enabled false Setting this property to true enables multiple uploads to native S3 filesystem. When uploading a file, it is split into blocks if the size is larger than fs.s3n.multipart.uploads.block.size. fs.s3n.multipart.uploads.block.size 67108864 The block size for multipart uploads to native S3 filesystem. Default size is 64MB. fs.s3n.multipart.copy.block.size 5368709120 The block size for multipart copy in native S3 filesystem. Default size is 5GB. fs.s3n.server-side-encryption-algorithm Specify a server-side encryption algorithm for S3. Unset by default, and the only other currently allowable value is AES256. fs.s3a.access.key AWS access key ID used by S3A file system. Omit for IAM role-based or provider-based authentication. fs.s3a.secret.key AWS secret key used by S3A file system. Omit for IAM role-based or provider-based authentication. fs.s3a.aws.credentials.provider Comma-separated class names of credential provider classes which implement com.amazonaws.auth.AWSCredentialsProvider. These are loaded and queried in sequence for a valid set of credentials. Each listed class must implement one of the following means of construction, which are attempted in order: 1. a public constructor accepting java.net.URI and org.apache.hadoop.conf.Configuration, 2. a public static method named getInstance that accepts no arguments and returns an instance of com.amazonaws.auth.AWSCredentialsProvider, or 3. a public default constructor. Specifying org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider allows anonymous access to a publicly accessible S3 bucket without any credentials. Please note that allowing anonymous access to an S3 bucket compromises security and therefore is unsuitable for most use cases. It can be useful for accessing public data sets without requiring AWS credentials. If unspecified, then the default list of credential provider classes, queried in sequence, is: 1. org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider: supports static configuration of AWS access key ID and secret access key. See also fs.s3a.access.key and fs.s3a.secret.key. 2. com.amazonaws.auth.EnvironmentVariableCredentialsProvider: supports configuration of AWS access key ID and secret access key in environment variables named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, as documented in the AWS SDK. 3. org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider: a shared instance of com.amazonaws.auth.InstanceProfileCredentialsProvider from the AWS SDK, which supports use of instance profile credentials if running in an EC2 VM. Using this shared instance potentially reduces load on the EC2 instance metadata service for multi-threaded applications. fs.s3a.session.token Session token, when using org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider as one of the providers. fs.s3a.security.credential.provider.path Optional comma separated list of credential providers, a list which is prepended to that set in hadoop.security.credential.provider.path fs.s3a.connection.maximum 15 Controls the maximum number of simultaneous connections to S3. fs.s3a.connection.ssl.enabled true Enables or disables SSL connections to S3. fs.s3a.endpoint AWS S3 endpoint to connect to. An up-to-date list is provided in the AWS Documentation: regions and endpoints. Without this property, the standard region (s3.amazonaws.com) is assumed. fs.s3a.path.style.access false Enable S3 path style access ie disabling the default virtual hosting behaviour. Useful for S3A-compliant storage providers as it removes the need to set up DNS for virtual hosting. fs.s3a.proxy.host Hostname of the (optional) proxy server for S3 connections. fs.s3a.proxy.port Proxy server port. If this property is not set but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with the value of fs.s3a.connection.ssl.enabled). fs.s3a.proxy.username Username for authenticating with proxy server. fs.s3a.proxy.password Password for authenticating with proxy server. fs.s3a.proxy.domain Domain for authenticating with proxy server. fs.s3a.proxy.workstation Workstation for authenticating with proxy server. fs.s3a.attempts.maximum 20 How many times we should retry commands on transient errors. fs.s3a.connection.establish.timeout 5000 Socket connection setup timeout in milliseconds. fs.s3a.connection.timeout 200000 Socket connection timeout in milliseconds. fs.s3a.socket.send.buffer 8192 Socket send buffer hint to amazon connector. Represented in bytes. fs.s3a.socket.recv.buffer 8192 Socket receive buffer hint to amazon connector. Represented in bytes. fs.s3a.paging.maximum 5000 How many keys to request from S3 when doing directory listings at a time. fs.s3a.threads.max 10 The total number of threads available in the filesystem for data uploads *or any other queued filesystem operation*. fs.s3a.threads.keepalivetime 60 Number of seconds a thread can be idle before being terminated. fs.s3a.max.total.tasks 5 The number of operations which can be queued for execution fs.s3a.multipart.size 100M How big (in bytes) to split upload or copy operations up into. A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. fs.s3a.multipart.threshold 2147483647 How big (in bytes) to split upload or copy operations up into. This also controls the partition size in renamed files, as rename() involves copying the source file(s). A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. fs.s3a.multiobjectdelete.enable true When enabled, multiple single-object delete requests are replaced by a single 'delete multiple objects'-request, reducing the number of requests. Beware: legacy S3-compatible object stores might not support this request. fs.s3a.acl.default Set a canned ACL for newly created and copied objects. Value may be Private, PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead, or BucketOwnerFullControl. fs.s3a.multipart.purge false True if you want to purge existing multipart uploads that may not have been completed/aborted correctly. The corresponding purge age is defined in fs.s3a.multipart.purge.age. If set, when the filesystem is instantiated then all outstanding uploads older than the purge age will be terminated -across the entire bucket. This will impact multipart uploads by other applications and users. so should be used sparingly, with an age value chosen to stop failed uploads, without breaking ongoing operations. fs.s3a.multipart.purge.age 86400 Minimum age in seconds of multipart uploads to purge. fs.s3a.server-side-encryption-algorithm Specify a server-side encryption algorithm for s3a: file system. Unset by default, and the only other currently allowable value is AES256. fs.s3a.signing-algorithm Override the default signing algorithm so legacy implementations can still be used fs.s3a.block.size 32M Block size to use when reading files using s3a: file system. A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. fs.s3a.buffer.dir ${hadoop.tmp.dir}/s3a Comma separated list of directories that will be used to buffer file uploads to. fs.s3a.fast.upload false Use the incremental block-based fast upload mechanism with the buffering mechanism set in fs.s3a.fast.upload.buffer. fs.s3a.fast.upload.buffer disk The buffering mechanism to use when using S3A fast upload (fs.s3a.fast.upload=true). Values: disk, array, bytebuffer. This configuration option has no effect if fs.s3a.fast.upload is false. "disk" will use the directories listed in fs.s3a.buffer.dir as the location(s) to save data prior to being uploaded. "array" uses arrays in the JVM heap "bytebuffer" uses off-heap memory within the JVM. Both "array" and "bytebuffer" will consume memory in a single stream up to the number of blocks set by: fs.s3a.multipart.size * fs.s3a.fast.upload.active.blocks. If using either of these mechanisms, keep this value low The total number of threads performing work across all threads is set by fs.s3a.threads.max, with fs.s3a.max.total.tasks values setting the number of queued work items. fs.s3a.fast.upload.active.blocks 4 Maximum Number of blocks a single output stream can have active (uploading, or queued to the central FileSystem instance's pool of queued operations. This stops a single stream overloading the shared thread pool. fs.s3a.readahead.range 64K Bytes to read ahead during a seek() before closing and re-opening the S3 HTTP connection. This option will be overridden if any call to setReadahead() is made to an open stream. A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. fs.s3a.user.agent.prefix Sets a custom value that will be prepended to the User-Agent header sent in HTTP requests to the S3 back-end by S3AFileSystem. The User-Agent header always includes the Hadoop version number followed by a string generated by the AWS SDK. An example is "User-Agent: Hadoop 2.8.0, aws-sdk-java/1.10.6". If this optional property is set, then its value is prepended to create a customized User-Agent. For example, if this configuration property was set to "MyApp", then an example of the resulting User-Agent would be "User-Agent: MyApp, Hadoop 2.8.0, aws-sdk-java/1.10.6". fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem The implementation class of the S3A Filesystem fs.AbstractFileSystem.s3a.impl org.apache.hadoop.fs.s3a.S3A The implementation class of the S3A AbstractFileSystem. fs.adl.impl org.apache.hadoop.fs.adl.AdlFileSystem fs.AbstractFileSystem.adl.impl org.apache.hadoop.fs.adl.Adl io.seqfile.compress.blocksize 1000000 The minimum block size for compression in block compressed SequenceFiles. io.mapfile.bloom.size 1048576 The size of BloomFilter-s used in BloomMapFile. Each time this many keys is appended the next BloomFilter will be created (inside a DynamicBloomFilter). Larger values minimize the number of filters, which slightly increases the performance, but may waste too much space if the total number of keys is usually much smaller than this number. io.mapfile.bloom.error.rate 0.005 The rate of false positives in BloomFilter-s used in BloomMapFile. As this value decreases, the size of BloomFilter-s increases exponentially. This value is the probability of encountering false positives (default is 0.5%). hadoop.util.hash.type murmur The default implementation of Hash. Currently this can take one of the two values: 'murmur' to select MurmurHash and 'jenkins' to select JenkinsHash. ipc.client.idlethreshold 4000 Defines the threshold number of connections after which connections will be inspected for idleness. ipc.client.kill.max 10 Defines the maximum number of clients to disconnect in one go. ipc.client.connection.maxidletime 10000 The maximum time in msec after which a client will bring down the connection to the server. ipc.client.connect.max.retries 10 Indicates the number of retries a client will make to establish a server connection. ipc.client.connect.retry.interval 1000 Indicates the number of milliseconds a client will wait for before retrying to establish a server connection. ipc.client.connect.timeout 20000 Indicates the number of milliseconds a client will wait for the socket to establish a server connection. ipc.client.connect.max.retries.on.timeouts 45 Indicates the number of retries a client will make on socket timeout to establish a server connection. ipc.client.tcpnodelay true Use TCP_NODELAY flag to bypass Nagle's algorithm transmission delays. ipc.client.low-latency false Use low-latency QoS markers for IPC connections. ipc.client.ping true Send a ping to the server when timeout on reading the response, if set to true. If no failure is detected, the client retries until at least a byte is read or the time given by ipc.client.rpc-timeout.ms is passed. ipc.ping.interval 60000 Timeout on waiting response from server, in milliseconds. The client will send ping when the interval is passed without receiving bytes, if ipc.client.ping is set to true. ipc.client.rpc-timeout.ms 0 Timeout on waiting response from server, in milliseconds. If ipc.client.ping is set to true and this rpc-timeout is greater than the value of ipc.ping.interval, the effective value of the rpc-timeout is rounded up to multiple of ipc.ping.interval. ipc.server.listen.queue.size 128 Indicates the length of the listen queue for servers accepting client connections. ipc.server.log.slow.rpc false This setting is useful to troubleshoot performance issues for various services. If this value is set to true then we log requests that fall into 99th percentile as well as increment RpcSlowCalls counter. ipc.maximum.data.length 67108864 This indicates the maximum IPC message length (bytes) that can be accepted by the server. Messages larger than this value are rejected by the immediately to avoid possible OOMs. This setting should rarely need to be changed. ipc.maximum.response.length 134217728 This indicates the maximum IPC message length (bytes) that can be accepted by the client. Messages larger than this value are rejected immediately to avoid possible OOMs. This setting should rarely need to be changed. Set to 0 to disable. hadoop.security.impersonation.provider.class A class which implements ImpersonationProvider interface, used to authorize whether one user can impersonate a specific user. If not specified, the DefaultImpersonationProvider will be used. If a class is specified, then that class will be used to determine the impersonation capability. hadoop.rpc.socket.factory.class.default org.apache.hadoop.net.StandardSocketFactory Default SocketFactory to use. This parameter is expected to be formatted as "package.FactoryClassName". hadoop.rpc.socket.factory.class.ClientProtocol SocketFactory to use to connect to a DFS. If null or empty, use hadoop.rpc.socket.class.default. This socket factory is also used by DFSClient to create sockets to DataNodes. hadoop.socks.server Address (host:port) of the SOCKS server to be used by the SocksSocketFactory. net.topology.node.switch.mapping.impl org.apache.hadoop.net.ScriptBasedMapping The default implementation of the DNSToSwitchMapping. It invokes a script specified in net.topology.script.file.name to resolve node names. If the value for net.topology.script.file.name is not set, the default value of DEFAULT_RACK is returned for all node names. net.topology.impl org.apache.hadoop.net.NetworkTopology The default implementation of NetworkTopology which is classic three layer one. net.topology.script.file.name The script name that should be invoked to resolve DNS names to NetworkTopology names. Example: the script would take host.foo.bar as an argument, and return /rack1 as the output. net.topology.script.number.args 100 The max number of args that the script configured with net.topology.script.file.name should be run with. Each arg is an IP address. net.topology.table.file.name The file name for a topology file, which is used when the net.topology.node.switch.mapping.impl property is set to org.apache.hadoop.net.TableMapping. The file format is a two column text file, with columns separated by whitespace. The first column is a DNS or IP address and the second column specifies the rack where the address maps. If no entry corresponding to a host in the cluster is found, then /default-rack is assumed. file.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. file.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than file.stream-buffer-size file.client-write-packet-size 65536 Packet size for clients to write file.blocksize 67108864 Block size file.replication 1 Replication factor s3.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. s3.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than s3.stream-buffer-size s3.client-write-packet-size 65536 Packet size for clients to write s3.blocksize 67108864 Block size s3.replication 3 Replication factor s3native.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. s3native.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than s3native.stream-buffer-size s3native.client-write-packet-size 65536 Packet size for clients to write s3native.blocksize 67108864 Block size s3native.replication 3 Replication factor ftp.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. ftp.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than ftp.stream-buffer-size ftp.client-write-packet-size 65536 Packet size for clients to write ftp.blocksize 67108864 Block size ftp.replication 3 Replication factor tfile.io.chunk.size 1048576 Value chunk size in bytes. Default to 1MB. Values of the length less than the chunk size is guaranteed to have known value length in read time (See also TFile.Reader.Scanner.Entry.isValueLengthKnown()). tfile.fs.output.buffer.size 262144 Buffer size used for FSDataOutputStream in bytes. tfile.fs.input.buffer.size 262144 Buffer size used for FSDataInputStream in bytes. hadoop.http.authentication.type simple Defines authentication used for Oozie HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# hadoop.http.authentication.token.validity 36000 Indicates how long (in seconds) an authentication token is valid before it has to be renewed. hadoop.http.authentication.signature.secret.file ${user.home}/hadoop-http-auth-signature-secret The signature secret for signing the authentication tokens. The same secret should be used for JT/NN/DN/TT configurations. hadoop.http.authentication.cookie.domain The domain to use for the HTTP cookie that stores the authentication token. In order to authentiation to work correctly across all Hadoop nodes web-consoles the domain must be correctly set. IMPORTANT: when using IP addresses, browsers ignore cookies with domain settings. For this setting to work properly all nodes in the cluster must be configured to generate URLs with hostname.domain names on it. hadoop.http.authentication.simple.anonymous.allowed true Indicates if anonymous requests are allowed when using 'simple' authentication. hadoop.http.authentication.kerberos.principal HTTP/_HOST@LOCALHOST Indicates the Kerberos principal to be used for HTTP endpoint. The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification. hadoop.http.authentication.kerberos.keytab ${user.home}/hadoop.keytab Location of the keytab file with the credentials for the principal. Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop. Enable/disable the cross-origin (CORS) filter. hadoop.http.cross-origin.enabled false Comma separated list of origins that are allowed for web services needing cross-origin (CORS) support. Wildcards (*) and patterns allowed hadoop.http.cross-origin.allowed-origins * Comma separated list of methods that are allowed for web services needing cross-origin (CORS) support. hadoop.http.cross-origin.allowed-methods GET,POST,HEAD Comma separated list of headers that are allowed for web services needing cross-origin (CORS) support. hadoop.http.cross-origin.allowed-headers X-Requested-With,Content-Type,Accept,Origin The number of seconds a pre-flighted request can be cached for web services needing cross-origin (CORS) support. hadoop.http.cross-origin.max-age 1800 dfs.ha.fencing.methods List of fencing methods to use for service fencing. May contain builtin methods (eg shell and sshfence) or user-defined method. dfs.ha.fencing.ssh.connect-timeout 30000 SSH connection timeout, in milliseconds, to use with the builtin sshfence fencer. dfs.ha.fencing.ssh.private-key-files The SSH private key files to use with the builtin sshfence fencer. The user name to filter as, on static web filters while rendering content. An example use is the HDFS web UI (user to be used for browsing files). hadoop.http.staticuser.user dr.who ha.zookeeper.quorum A list of ZooKeeper server addresses, separated by commas, that are to be used by the ZKFailoverController in automatic failover. ha.zookeeper.session-timeout.ms 5000 The session timeout to use when the ZKFC connects to ZooKeeper. Setting this value to a lower value implies that server crashes will be detected more quickly, but risks triggering failover too aggressively in the case of a transient error or network blip. ha.zookeeper.parent-znode /hadoop-ha The ZooKeeper znode under which the ZK failover controller stores its information. Note that the nameservice ID is automatically appended to this znode, so it is not normally necessary to configure this, even in a federated environment. ha.zookeeper.acl world:anyone:rwcda A comma-separated list of ZooKeeper ACLs to apply to the znodes used by automatic failover. These ACLs are specified in the same format as used by the ZooKeeper CLI. If the ACL itself contains secrets, you may instead specify a path to a file, prefixed with the '@' symbol, and the value of this configuration will be loaded from within. ha.zookeeper.auth A comma-separated list of ZooKeeper authentications to add when connecting to ZooKeeper. These are specified in the same format as used by the "addauth" command in the ZK CLI. It is important that the authentications specified here are sufficient to access znodes with the ACL specified in ha.zookeeper.acl. If the auths contain secrets, you may instead specify a path to a file, prefixed with the '@' symbol, and the value of this configuration will be loaded from within. hadoop.ssl.keystores.factory.class org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory The keystores factory to use for retrieving certificates. hadoop.ssl.require.client.cert false Whether client certificates are required hadoop.ssl.hostname.verifier DEFAULT The hostname verifier to provide for HttpsURLConnections. Valid values are: DEFAULT, STRICT, STRICT_I6, DEFAULT_AND_LOCALHOST and ALLOW_ALL hadoop.ssl.server.conf ssl-server.xml Resource file from which ssl server keystore information will be extracted. This file is looked up in the classpath, typically it should be in Hadoop conf/ directory. hadoop.ssl.client.conf ssl-client.xml Resource file from which ssl client keystore information will be extracted This file is looked up in the classpath, typically it should be in Hadoop conf/ directory. hadoop.ssl.enabled false Deprecated. Use dfs.http.policy and yarn.http.policy instead. hadoop.ssl.enabled.protocols TLSv1 Protocols supported by the ssl. hadoop.jetty.logs.serve.aliases true Enable/Disable aliases serving from jetty fs.permissions.umask-mode 022 The umask used when creating files and directories. Can be in octal or in symbolic. Examples are: "022" (octal for u=rwx,g=r-x,o=r-x in symbolic), or "u=rwx,g=rwx,o=" (symbolic for 007 in octal). ha.health-monitor.connect-retry-interval.ms 1000 How often to retry connecting to the service. ha.health-monitor.check-interval.ms 1000 How often to check the service. ha.health-monitor.sleep-after-disconnect.ms 1000 How long to sleep after an unexpected RPC error. ha.health-monitor.rpc-timeout.ms 45000 Timeout for the actual monitorHealth() calls. ha.failover-controller.new-active.rpc-timeout.ms 60000 Timeout that the FC waits for the new active to become active ha.failover-controller.graceful-fence.rpc-timeout.ms 5000 Timeout that the FC waits for the old active to go to standby ha.failover-controller.graceful-fence.connection.retries 1 FC connection retries for graceful fencing ha.failover-controller.cli-check.rpc-timeout.ms 20000 Timeout that the CLI (manual) FC waits for monitorHealth, getServiceState ipc.client.fallback-to-simple-auth-allowed false When a client is configured to attempt a secure connection, but attempts to connect to an insecure server, that server may instruct the client to switch to SASL SIMPLE (unsecure) authentication. This setting controls whether or not the client will accept this instruction from the server. When false (the default), the client will not allow the fallback to SIMPLE authentication, and will abort the connection. fs.client.resolve.remote.symlinks true Whether to resolve symlinks when accessing a remote Hadoop filesystem. Setting this to false causes an exception to be thrown upon encountering a symlink. This setting does not apply to local filesystems, which automatically resolve local symlinks. nfs.exports.allowed.hosts * rw By default, the export can be mounted by any client. The value string contains machine name and access privilege, separated by whitespace characters. The machine name format can be a single host, a Java regular expression, or an IPv4 address. The access privilege uses rw or ro to specify read/write or read-only access of the machines to exports. If the access privilege is not provided, the default is read-only. Entries are separated by ";". For example: "192.168.0.0/22 rw ; host.*\.example\.com ; host1.test.org ro;". Only the NFS gateway needs to restart after this property is updated. hadoop.user.group.static.mapping.overrides dr.who=; Static mapping of user to groups. This will override the groups if available in the system for the specified user. In otherwords, groups look-up will not happen for these users, instead groups mapped in this configuration will be used. Mapping should be in this format. user1=group1,group2;user2=;user3=group2; Default, "dr.who=;" will consider "dr.who" as user without groups. rpc.metrics.quantile.enable false Setting this property to true and rpc.metrics.percentiles.intervals to a comma-separated list of the granularity in seconds, the 50/75/90/95/99th percentile latency for rpc queue/processing time in milliseconds are added to rpc metrics. rpc.metrics.percentiles.intervals A comma-separated list of the granularity in seconds for the metrics which describe the 50/75/90/95/99th percentile latency for rpc queue/processing time. The metrics are outputted if rpc.metrics.quantile.enable is set to true. hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE The prefix for a given crypto codec, contains a comma-separated list of implementation classes for a given crypto codec (eg EXAMPLECIPHERSUITE). The first implementation will be used if available, others are fallbacks. hadoop.security.crypto.codec.classes.aes.ctr.nopadding org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec Comma-separated list of crypto codec implementations for AES/CTR/NoPadding. The first implementation will be used if available, others are fallbacks. hadoop.security.crypto.cipher.suite AES/CTR/NoPadding Cipher suite for crypto codec. hadoop.security.crypto.jce.provider The JCE provider name used in CryptoCodec. hadoop.security.crypto.buffer.size 8192 The buffer size used by CryptoInputStream and CryptoOutputStream. hadoop.security.java.secure.random.algorithm SHA1PRNG The java secure random algorithm. hadoop.security.secure.random.impl Implementation of secure random. hadoop.security.random.device.file.path /dev/urandom OS security random device file path. hadoop.security.key.provider.path The KeyProvider to use when managing zone keys, and interacting with encryption keys when reading and writing to an encryption zone. For hdfs clients, the provider path will be same as namenode's provider path. fs.har.impl.disable.cache true Don't cache 'har' filesystem instances. hadoop.security.kms.client.authentication.retry-count 1 Number of time to retry connecting to KMS on authentication failure hadoop.security.kms.client.encrypted.key.cache.size 500 Size of the EncryptedKeyVersion cache Queue for each key hadoop.security.kms.client.encrypted.key.cache.low-watermark 0.3f If size of the EncryptedKeyVersion cache Queue falls below the low watermark, this cache queue will be scheduled for a refill hadoop.security.kms.client.encrypted.key.cache.num.refill.threads 2 Number of threads to use for refilling depleted EncryptedKeyVersion cache Queues hadoop.security.kms.client.encrypted.key.cache.expiry 43200000 Cache expiry time for a Key, after which the cache Queue for this key will be dropped. Default = 12hrs ipc.server.max.connections 0 The maximum number of concurrent connections a server is allowed to accept. If this limit is exceeded, incoming connections will first fill the listen queue and then may go to an OS-specific listen overflow queue. The client may fail or timeout, but the server can avoid running out of file descriptors using this feature. 0 means no limit. Is the registry enabled in the YARN Resource Manager? If true, the YARN RM will, as needed. create the user and system paths, and purge service records when containers, application attempts and applications complete. If false, the paths must be created by other means, and no automatic cleanup of service records will take place. hadoop.registry.rm.enabled false The root zookeeper node for the registry hadoop.registry.zk.root /registry Zookeeper session timeout in milliseconds hadoop.registry.zk.session.timeout.ms 60000 Zookeeper connection timeout in milliseconds hadoop.registry.zk.connection.timeout.ms 15000 Zookeeper connection retry count before failing hadoop.registry.zk.retry.times 5 hadoop.registry.zk.retry.interval.ms 1000 Zookeeper retry limit in milliseconds, during exponential backoff. This places a limit even if the retry times and interval limit, combined with the backoff policy, result in a long retry period hadoop.registry.zk.retry.ceiling.ms 60000 List of hostname:port pairs defining the zookeeper quorum binding for the registry hadoop.registry.zk.quorum localhost:2181 Key to set if the registry is secure. Turning it on changes the permissions policy from "open access" to restrictions on kerberos with the option of a user adding one or more auth key pairs down their own tree. hadoop.registry.secure false A comma separated list of Zookeeper ACL identifiers with system access to the registry in a secure cluster. These are given full access to all entries. If there is an "@" at the end of a SASL entry it instructs the registry client to append the default kerberos domain. hadoop.registry.system.acls sasl:yarn@, sasl:mapred@, sasl:hdfs@ The kerberos realm: used to set the realm of system principals which do not declare their realm, and any other accounts that need the value. If empty, the default realm of the running process is used. If neither are known and the realm is needed, then the registry service/client will fail. hadoop.registry.kerberos.realm Key to define the JAAS context. Used in secure mode hadoop.registry.jaas.context Client Enable hdfs shell commands to display warnings if (fs.defaultFS) property is not set. hadoop.shell.missing.defaultFs.warning false hadoop.shell.safely.delete.limit.num.files 100 Used by -safely option of hadoop fs shell -rm command to avoid accidental deletion of large directories. When enabled, the -rm command requires confirmation if the number of files to be deleted is greater than this limit. The default limit is 100 files. The warning is disabled if the limit is 0 or the -safely is not specified in -rm command. fs.client.htrace.sampler.classes The class names of the HTrace Samplers to use for Hadoop filesystem clients. hadoop.htrace.span.receiver.classes The class names of the Span Receivers to use for Hadoop. fs.adl.impl org.apache.hadoop.fs.adl.AdlFileSystem fs.AbstractFileSystem.adl.impl org.apache.hadoop.fs.adl.Adl adl.feature.ownerandgroup.enableupn false When true : User and Group in FileStatus/AclStatus response is represented as user friendly name as per Azure AD profile. When false (default) : User and Group in FileStatus/AclStatus response is represented by the unique identifier from Azure AD profile (Object ID as GUID). For optimal performance, false is recommended. fs.adl.oauth2.access.token.provider.type ClientCredential Defines Azure Active Directory OAuth2 access token provider type. Supported types are ClientCredential, RefreshToken, and Custom. The ClientCredential type requires property fs.adl.oauth2.client.id, fs.adl.oauth2.credential, and fs.adl.oauth2.refresh.url. The RefreshToken type requires property fs.adl.oauth2.client.id and fs.adl.oauth2.refresh.token. The Custom type requires property fs.adl.oauth2.access.token.provider. fs.adl.oauth2.client.id The OAuth2 client id. fs.adl.oauth2.credential The OAuth2 access key. fs.adl.oauth2.refresh.url The OAuth2 token endpoint. fs.adl.oauth2.refresh.token The OAuth2 refresh token. fs.adl.oauth2.access.token.provider The class name of the OAuth2 access token provider. hadoop.caller.context.enabled false When the feature is enabled, additional fields are written into name-node audit log records for auditing coarse granularity operations. hadoop.caller.context.max.size 128 The maximum bytes a caller context string can have. If the passed caller context is longer than this maximum bytes, client will truncate it before sending to server. Note that the server may have a different maximum size, and will truncate the caller context to the maximum size it allows. hadoop.caller.context.signature.max.size 40 The caller's signature (optional) is for offline validation. If the signature exceeds the maximum allowed bytes in server, the caller context will be abandoned, in which case the caller context will not be recorded in audit logs. sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/README.rst0000666000175100017510000000254213245514472024064 0ustar zuulzuul00000000000000Apache Hadoop Configurations for Sahara ======================================= This directory contains default XML configuration files: * core-default.xml * hdfs-default.xml * mapred-default.xml * yarn-default.xml * oozie-default.xml * hive-default.xml These files are applied for Sahara's plugin of Apache Hadoop version 2.8.2 Files were taken from here: * `core-default.xml `_ * `hdfs-default.xml `_ * `yarn-default.xml `_ * `mapred-default.xml `_ * `oozie-default.xml `_ XML configs are used to expose default Hadoop configurations to the users through Sahara's REST API. It allows users to override some config values which will be pushed to the provisioned VMs running Hadoop services as part of appropriate xml config. sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/yarn-default.xml0000666000175100017510000027760313245514472025526 0ustar zuulzuul00000000000000 Factory to create client IPC classes. yarn.ipc.client.factory.class Factory to create server IPC classes. yarn.ipc.server.factory.class Factory to create serializeable records. yarn.ipc.record.factory.class RPC class implementation yarn.ipc.rpc.class org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC The hostname of the RM. yarn.resourcemanager.hostname 0.0.0.0 The address of the applications manager interface in the RM. yarn.resourcemanager.address ${yarn.resourcemanager.hostname}:8032 The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This is most useful for making RM listen to all interfaces by setting to 0.0.0.0. yarn.resourcemanager.bind-host The number of threads used to handle applications manager requests. yarn.resourcemanager.client.thread-count 50 Number of threads used to launch/cleanup AM. yarn.resourcemanager.amlauncher.thread-count 50 Retry times to connect with NM. yarn.resourcemanager.nodemanager-connect-retries 10 Timeout in milliseconds when YARN dispatcher tries to drain the events. Typically, this happens when service is stopping. e.g. RM drains the ATS events dispatcher when stopping. yarn.dispatcher.drain-events.timeout 300000 The expiry interval for application master reporting. yarn.am.liveness-monitor.expiry-interval-ms 600000 The Kerberos principal for the resource manager. yarn.resourcemanager.principal The address of the scheduler interface. yarn.resourcemanager.scheduler.address ${yarn.resourcemanager.hostname}:8030 Number of threads to handle scheduler interface. yarn.resourcemanager.scheduler.client.thread-count 50 This configures the HTTP endpoint for Yarn Daemons.The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https yarn.http.policy HTTP_ONLY The http address of the RM web application. If only a host is provided as the value, the webapp will be served on a random port. yarn.resourcemanager.webapp.address ${yarn.resourcemanager.hostname}:8088 The https address of the RM web application. If only a host is provided as the value, the webapp will be served on a random port. yarn.resourcemanager.webapp.https.address ${yarn.resourcemanager.hostname}:8090 The Kerberos keytab file to be used for spnego filter for the RM web interface. yarn.resourcemanager.webapp.spnego-keytab-file The Kerberos principal to be used for spnego filter for the RM web interface. yarn.resourcemanager.webapp.spnego-principal Add button to kill application in the RM Application view. yarn.resourcemanager.webapp.ui-actions.enabled true yarn.resourcemanager.resource-tracker.address ${yarn.resourcemanager.hostname}:8031 Are acls enabled. yarn.acl.enable false Are reservation acls enabled. yarn.acl.reservation-enable false ACL of who can be admin of the YARN cluster. yarn.admin.acl * The address of the RM admin interface. yarn.resourcemanager.admin.address ${yarn.resourcemanager.hostname}:8033 Number of threads used to handle RM admin interface. yarn.resourcemanager.admin.client.thread-count 1 Maximum time to wait to establish connection to ResourceManager. yarn.resourcemanager.connect.max-wait.ms 900000 How often to try connecting to the ResourceManager. yarn.resourcemanager.connect.retry-interval.ms 30000 The maximum number of application attempts. It's a global setting for all application masters. Each application master can specify its individual maximum number of application attempts via the API, but the individual number cannot be more than the global upper bound. If it is, the resourcemanager will override it. The default number is set to 2, to allow at least one retry for AM. yarn.resourcemanager.am.max-attempts 2 How often to check that containers are still alive. yarn.resourcemanager.container.liveness-monitor.interval-ms 600000 The keytab for the resource manager. yarn.resourcemanager.keytab /etc/krb5.keytab Flag to enable override of the default kerberos authentication filter with the RM authentication filter to allow authentication using delegation tokens(fallback to kerberos if the tokens are missing). Only applicable when the http authentication type is kerberos. yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled true Flag to enable cross-origin (CORS) support in the RM. This flag requires the CORS filter initializer to be added to the filter initializers list in core-site.xml. yarn.resourcemanager.webapp.cross-origin.enabled false How long to wait until a node manager is considered dead. yarn.nm.liveness-monitor.expiry-interval-ms 600000 Path to file with nodes to include. yarn.resourcemanager.nodes.include-path Path to file with nodes to exclude. yarn.resourcemanager.nodes.exclude-path The expiry interval for node IP caching. -1 disables the caching yarn.resourcemanager.node-ip-cache.expiry-interval-secs -1 Number of threads to handle resource tracker calls. yarn.resourcemanager.resource-tracker.client.thread-count 50 The class to use as the resource scheduler. yarn.resourcemanager.scheduler.class org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this will throw a InvalidResourceRequestException. yarn.scheduler.minimum-allocation-mb 1024 The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this will throw a InvalidResourceRequestException. yarn.scheduler.maximum-allocation-mb 8192 The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this will throw a InvalidResourceRequestException. yarn.scheduler.minimum-allocation-vcores 1 The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this will throw a InvalidResourceRequestException. yarn.scheduler.maximum-allocation-vcores 4 Used by node labels. If set to true, the port should be included in the node name. Only usable if your scheduler supports node labels. yarn.scheduler.include-port-in-node-name false Enable RM to recover state after starting. If true, then yarn.resourcemanager.store.class must be specified. yarn.resourcemanager.recovery.enabled false Should RM fail fast if it encounters any errors. By defalt, it points to ${yarn.fail-fast}. Errors include: 1) exceptions when state-store write/read operations fails. yarn.resourcemanager.fail-fast ${yarn.fail-fast} Should YARN fail fast if it encounters any errors. This is a global config for all other components including RM,NM etc. If no value is set for component-specific config (e.g yarn.resourcemanager.fail-fast), this value will be the default. yarn.fail-fast false Enable RM work preserving recovery. This configuration is private to YARN for experimenting the feature. yarn.resourcemanager.work-preserving-recovery.enabled true Set the amount of time RM waits before allocating new containers on work-preserving-recovery. Such wait period gives RM a chance to settle down resyncing with NMs in the cluster on recovery, before assigning new containers to applications. yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms 10000 The class to use as the persistent store. If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore is used, the store is implicitly fenced; meaning a single ResourceManager is able to use the store at any point in time. More details on this implicit fencing, along with setting up appropriate ACLs is discussed under yarn.resourcemanager.zk-state-store.root-node.acl. yarn.resourcemanager.store.class org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore When automatic failover is enabled, number of zookeeper operation retry times in ActiveStandbyElector yarn.resourcemanager.ha.failover-controller.active-standby-elector.zk.retries The maximum number of completed applications RM state store keeps, less than or equals to ${yarn.resourcemanager.max-completed-applications}. By default, it equals to ${yarn.resourcemanager.max-completed-applications}. This ensures that the applications kept in the state store are consistent with the applications remembered in RM memory. Any values larger than ${yarn.resourcemanager.max-completed-applications} will be reset to ${yarn.resourcemanager.max-completed-applications}. Note that this value impacts the RM recovery performance.Typically, a smaller value indicates better performance on RM recovery. yarn.resourcemanager.state-store.max-completed-applications ${yarn.resourcemanager.max-completed-applications} Host:Port of the ZooKeeper server to be used by the RM. This must be supplied when using the ZooKeeper based implementation of the RM state store and/or embedded automatic failover in a HA setting. yarn.resourcemanager.zk-address Number of times RM tries to connect to ZooKeeper. yarn.resourcemanager.zk-num-retries 1000 Retry interval in milliseconds when connecting to ZooKeeper. When HA is enabled, the value here is NOT used. It is generated automatically from yarn.resourcemanager.zk-timeout-ms and yarn.resourcemanager.zk-num-retries. yarn.resourcemanager.zk-retry-interval-ms 1000 Full path of the ZooKeeper znode where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore as the value for yarn.resourcemanager.store.class yarn.resourcemanager.zk-state-store.parent-path /rmstore ZooKeeper session timeout in milliseconds. Session expiration is managed by the ZooKeeper cluster itself, not by the client. This value is used by the cluster to determine when the client's session expires. Expirations happens when the cluster does not hear from the client within the specified session timeout period (i.e. no heartbeat). yarn.resourcemanager.zk-timeout-ms 10000 ACL's to be used for ZooKeeper znodes. yarn.resourcemanager.zk-acl world:anyone:rwcda ACLs to be used for the root znode when using ZKRMStateStore in a HA scenario for fencing. ZKRMStateStore supports implicit fencing to allow a single ResourceManager write-access to the store. For fencing, the ResourceManagers in the cluster share read-write-admin privileges on the root node, but the Active ResourceManager claims exclusive create-delete permissions. By default, when this property is not set, we use the ACLs from yarn.resourcemanager.zk-acl for shared admin access and rm-address:random-number for username-based exclusive create-delete access. This property allows users to set ACLs of their choice instead of using the default mechanism. For fencing to work, the ACLs should be carefully set differently on each ResourceManger such that all the ResourceManagers have shared admin access and the Active ResourceManger takes over (exclusively) the create-delete access. yarn.resourcemanager.zk-state-store.root-node.acl Specify the auths to be used for the ACL's specified in both the yarn.resourcemanager.zk-acl and yarn.resourcemanager.zk-state-store.root-node.acl properties. This takes a comma-separated list of authentication mechanisms, each of the form 'scheme:auth' (the same syntax used for the 'addAuth' command in the ZK CLI). yarn.resourcemanager.zk-auth URI pointing to the location of the FileSystem path where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore as the value for yarn.resourcemanager.store.class yarn.resourcemanager.fs.state-store.uri ${hadoop.tmp.dir}/yarn/system/rmstore hdfs client retry policy specification. hdfs client retry is always enabled. Specified in pairs of sleep-time and number-of-retries and (t0, n0), (t1, n1), ..., the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on. yarn.resourcemanager.fs.state-store.retry-policy-spec 2000, 500 the number of retries to recover from IOException in FileSystemRMStateStore. yarn.resourcemanager.fs.state-store.num-retries 0 Retry interval in milliseconds in FileSystemRMStateStore. yarn.resourcemanager.fs.state-store.retry-interval-ms 1000 Local path where the RM state will be stored when using org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore as the value for yarn.resourcemanager.store.class yarn.resourcemanager.leveldb-state-store.path ${hadoop.tmp.dir}/yarn/system/rmstore The time in seconds between full compactions of the leveldb database. Setting the interval to zero disables the full compaction cycles. yarn.resourcemanager.leveldb-state-store.compaction-interval-secs 3600 Enable RM high-availability. When enabled, (1) The RM starts in the Standby mode by default, and transitions to the Active mode when prompted to. (2) The nodes in the RM ensemble are listed in yarn.resourcemanager.ha.rm-ids (3) The id of each RM either comes from yarn.resourcemanager.ha.id if yarn.resourcemanager.ha.id is explicitly specified or can be figured out by matching yarn.resourcemanager.address.{id} with local address (4) The actual physical addresses come from the configs of the pattern - {rpc-config}.{id} yarn.resourcemanager.ha.enabled false Enable automatic failover. By default, it is enabled only when HA is enabled yarn.resourcemanager.ha.automatic-failover.enabled true Enable embedded automatic failover. By default, it is enabled only when HA is enabled. The embedded elector relies on the RM state store to handle fencing, and is primarily intended to be used in conjunction with ZKRMStateStore. yarn.resourcemanager.ha.automatic-failover.embedded true The base znode path to use for storing leader information, when using ZooKeeper based leader election. yarn.resourcemanager.ha.automatic-failover.zk-base-path /yarn-leader-election Name of the cluster. In a HA setting, this is used to ensure the RM participates in leader election for this cluster and ensures it does not affect other clusters yarn.resourcemanager.cluster-id The list of RM nodes in the cluster when HA is enabled. See description of yarn.resourcemanager.ha .enabled for full details on how this is used. yarn.resourcemanager.ha.rm-ids The id (string) of the current RM. When HA is enabled, this is an optional config. The id of current RM can be set by explicitly specifying yarn.resourcemanager.ha.id or figured out by matching yarn.resourcemanager.address.{id} with local address See description of yarn.resourcemanager.ha.enabled for full details on how this is used. yarn.resourcemanager.ha.id When HA is enabled, the class to be used by Clients, AMs and NMs to failover to the Active RM. It should extend org.apache.hadoop.yarn.client.RMFailoverProxyProvider yarn.client.failover-proxy-provider org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider When HA is enabled, the max number of times FailoverProxyProvider should attempt failover. When set, this overrides the yarn.resourcemanager.connect.max-wait.ms. When not set, this is inferred from yarn.resourcemanager.connect.max-wait.ms. yarn.client.failover-max-attempts When HA is enabled, the sleep base (in milliseconds) to be used for calculating the exponential delay between failovers. When set, this overrides the yarn.resourcemanager.connect.* settings. When not set, yarn.resourcemanager.connect.retry-interval.ms is used instead. yarn.client.failover-sleep-base-ms When HA is enabled, the maximum sleep time (in milliseconds) between failovers. When set, this overrides the yarn.resourcemanager.connect.* settings. When not set, yarn.resourcemanager.connect.retry-interval.ms is used instead. yarn.client.failover-sleep-max-ms When HA is enabled, the number of retries per attempt to connect to a ResourceManager. In other words, it is the ipc.client.connect.max.retries to be used during failover attempts yarn.client.failover-retries 0 When HA is enabled, the number of retries per attempt to connect to a ResourceManager on socket timeouts. In other words, it is the ipc.client.connect.max.retries.on.timeouts to be used during failover attempts yarn.client.failover-retries-on-socket-timeouts 0 The maximum number of completed applications RM keeps. yarn.resourcemanager.max-completed-applications 10000 Interval at which the delayed token removal thread runs yarn.resourcemanager.delayed.delegation-token.removal-interval-ms 30000 If true, ResourceManager will have proxy-user privileges. Use case: In a secure cluster, YARN requires the user hdfs delegation-tokens to do localization and log-aggregation on behalf of the user. If this is set to true, ResourceManager is able to request new hdfs delegation tokens on behalf of the user. This is needed by long-running-service, because the hdfs tokens will eventually expire and YARN requires new valid tokens to do localization and log-aggregation. Note that to enable this use case, the corresponding HDFS NameNode has to configure ResourceManager as the proxy-user so that ResourceManager can itself ask for new tokens on behalf of the user when tokens are past their max-life-time. yarn.resourcemanager.proxy-user-privileges.enabled false Interval for the roll over for the master key used to generate application tokens yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs 86400 Interval for the roll over for the master key used to generate container tokens. It is expected to be much greater than yarn.nm.liveness-monitor.expiry-interval-ms and yarn.resourcemanager.rm.container-allocation.expiry-interval-ms. Otherwise the behavior is undefined. yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs 86400 The heart-beat interval in milliseconds for every NodeManager in the cluster. yarn.resourcemanager.nodemanagers.heartbeat-interval-ms 1000 The minimum allowed version of a connecting nodemanager. The valid values are NONE (no version checking), EqualToRM (the nodemanager's version is equal to or greater than the RM version), or a Version String. yarn.resourcemanager.nodemanager.minimum.version NONE Enable a set of periodic monitors (specified in yarn.resourcemanager.scheduler.monitor.policies) that affect the scheduler. yarn.resourcemanager.scheduler.monitor.enable false The list of SchedulingEditPolicy classes that interact with the scheduler. A particular module may be incompatible with the scheduler, other policies, or a configuration of either. yarn.resourcemanager.scheduler.monitor.policies org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy The class to use as the configuration provider. If org.apache.hadoop.yarn.LocalConfigurationProvider is used, the local configuration will be loaded. If org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider is used, the configuration which will be loaded should be uploaded to remote File system first. yarn.resourcemanager.configuration.provider-class org.apache.hadoop.yarn.LocalConfigurationProvider The value specifies the file system (e.g. HDFS) path where ResourceManager loads configuration if yarn.resourcemanager.configuration.provider-class is set to org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider. yarn.resourcemanager.configuration.file-system-based-store /yarn/conf The setting that controls whether yarn system metrics is published on the timeline server or not by RM. yarn.resourcemanager.system-metrics-publisher.enabled false Number of worker threads that send the yarn system metrics data. yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size 10 Number of diagnostics/failure messages can be saved in RM for log aggregation. It also defines the number of diagnostics/failure messages can be shown in log aggregation web ui. yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory 10 RM DelegationTokenRenewer thread count yarn.resourcemanager.delegation-token-renewer.thread-count 50 RM secret key update interval in ms yarn.resourcemanager.delegation.key.update-interval 86400000 RM delegation token maximum lifetime in ms yarn.resourcemanager.delegation.token.max-lifetime 604800000 RM delegation token update interval in ms yarn.resourcemanager.delegation.token.renew-interval 86400000 Thread pool size for RMApplicationHistoryWriter. yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size 10 Comma-separated list of values (in minutes) for schedule queue related metrics. yarn.resourcemanager.metrics.runtime.buckets 60,300,1440 Interval for the roll over for the master key used to generate NodeManager tokens. It is expected to be set to a value much larger than yarn.nm.liveness-monitor.expiry-interval-ms. yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs 86400 Flag to enable the ResourceManager reservation system. yarn.resourcemanager.reservation-system.enable false The Java class to use as the ResourceManager reservation system. By default, is set to org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityReservationSystem when using CapacityScheduler and is set to org.apache.hadoop.yarn.server.resourcemanager.reservation.FairReservationSystem when using FairScheduler. yarn.resourcemanager.reservation-system.class The plan follower policy class name to use for the ResourceManager reservation system. By default, is set to org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacitySchedulerPlanFollower is used when using CapacityScheduler, and is set to org.apache.hadoop.yarn.server.resourcemanager.reservation.FairSchedulerPlanFollower when using FairScheduler. yarn.resourcemanager.reservation-system.plan.follower Step size of the reservation system in ms yarn.resourcemanager.reservation-system.planfollower.time-step 1000 The expiry interval for a container yarn.resourcemanager.rm.container-allocation.expiry-interval-ms 600000 The hostname of the NM. yarn.nodemanager.hostname 0.0.0.0 The address of the container manager in the NM. yarn.nodemanager.address ${yarn.nodemanager.hostname}:0 The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is most useful for making NM listen to all interfaces by setting to 0.0.0.0. yarn.nodemanager.bind-host Environment variables that should be forwarded from the NodeManager's environment to the container's. yarn.nodemanager.admin-env MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX Environment variables that containers may override rather than use NodeManager's default. yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME who will execute(launch) the containers. yarn.nodemanager.container-executor.class org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor Number of threads container manager uses. yarn.nodemanager.container-manager.thread-count 20 Number of threads used in cleanup. yarn.nodemanager.delete.thread-count 4 Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory. To diagnose Yarn application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications' log directories is configurable with the yarn.nodemanager.log-dirs property (see also below). yarn.nodemanager.delete.debug-delay-sec 0 Keytab for NM. yarn.nodemanager.keytab /etc/krb5.keytab List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this. yarn.nodemanager.local-dirs ${hadoop.tmp.dir}/nm-local-dir It limits the maximum number of files which will be localized in a single local directory. If the limit is reached then sub-directories will be created and new files will be localized in them. If it is set to a value less than or equal to 36 [which are sub-directories (0-9 and then a-z)] then NodeManager will fail to start. For example; [for public cache] if this is configured with a value of 40 ( 4 files + 36 sub-directories) and the local-dir is "/tmp/local-dir1" then it will allow 4 files to be created directly inside "/tmp/local-dir1/filecache". For files that are localized further it will create a sub-directory "0" inside "/tmp/local-dir1/filecache" and will localize files inside it until it becomes full. If a file is removed from a sub-directory that is marked full, then that sub-directory will be used back again to localize files. yarn.nodemanager.local-cache.max-files-per-directory 8192 Address where the localizer IPC is. yarn.nodemanager.localizer.address ${yarn.nodemanager.hostname}:8040 Interval in between cache cleanups. yarn.nodemanager.localizer.cache.cleanup.interval-ms 600000 Target size of localizer cache in MB, per nodemanager. It is a target retention size that only includes resources with PUBLIC and PRIVATE visibility and excludes resources with APPLICATION visibility yarn.nodemanager.localizer.cache.target-size-mb 10240 Number of threads to handle localization requests. yarn.nodemanager.localizer.client.thread-count 5 Number of threads to use for localization fetching. yarn.nodemanager.localizer.fetch.thread-count 4 yarn.nodemanager.container-localizer.java.opts -Xmx256m Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container. yarn.nodemanager.log-dirs ${yarn.log.dir}/userlogs Whether to enable log aggregation. Log aggregation collects each container's logs and moves these logs onto a file-system, for e.g. HDFS, after the application completes. Users can configure the "yarn.nodemanager.remote-app-log-dir" and "yarn.nodemanager.remote-app-log-dir-suffix" properties to determine where these logs are moved to. Users can access the logs via the Application Timeline Server. yarn.log-aggregation-enable false How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node. yarn.log-aggregation.retain-seconds -1 How long to wait between aggregated log retention checks. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful set this too small and you will spam the name node. yarn.log-aggregation.retain-check-interval-seconds -1 How long for ResourceManager to wait for NodeManager to report its log aggregation status. If waiting time of which the log aggregation status is reported from NodeManager exceeds the configured value, RM will report log aggregation status for this NodeManager as TIME_OUT yarn.log-aggregation-status.time-out.ms 600000 Time in seconds to retain user logs. Only applicable if log aggregation is disabled yarn.nodemanager.log.retain-seconds 10800 Where to aggregate logs to. yarn.nodemanager.remote-app-log-dir /tmp/logs The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam} yarn.nodemanager.remote-app-log-dir-suffix logs Generate additional logs about container launches. Currently, this creates a copy of the launch script and lists the directory contents of the container work dir. When listing directory contents, we follow symlinks to a max-depth of 5(including symlinks which point to outside the container work dir) which may lead to a slowness in launching containers. yarn.nodemanager.log-container-debug-info.enabled false Amount of physical memory, in MB, that can be allocated for containers. If set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically calculated(in case of Windows and Linux). In other cases, the default is 8192MB. yarn.nodemanager.resource.memory-mb -1 Amount of physical memory, in MB, that is reserved for non-YARN processes. This configuration is only used if yarn.nodemanager.resource.detect-hardware-capabilities is set to true and yarn.nodemanager.resource.memory-mb is -1. If set to -1, this amount is calculated as 20% of (system memory - 2*HADOOP_HEAPSIZE) yarn.nodemanager.resource.system-reserved-memory-mb -1 Whether physical memory limits will be enforced for containers. yarn.nodemanager.pmem-check-enabled true Whether virtual memory limits will be enforced for containers. yarn.nodemanager.vmem-check-enabled true Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio. yarn.nodemanager.vmem-pmem-ratio 2.1 Number of vcores that can be allocated for containers. This is used by the RM scheduler when allocating resources for containers. This is not used to limit the number of CPUs used by YARN containers. If it is set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically determined from the hardware in case of Windows and Linux. In other cases, number of vcores is 8 by default. yarn.nodemanager.resource.cpu-vcores -1 Flag to determine if logical processors(such as hyperthreads) should be counted as cores. Only applicable on Linux when yarn.nodemanager.resource.cpu-vcores is set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true. yarn.nodemanager.resource.count-logical-processors-as-cores false Multiplier to determine how to convert phyiscal cores to vcores. This value is used if yarn.nodemanager.resource.cpu-vcores is set to -1(which implies auto-calculate vcores) and yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The number of vcores will be calculated as number of CPUs * multiplier. yarn.nodemanager.resource.pcores-vcores-multiplier 1.0 Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU. yarn.nodemanager.resource.percentage-physical-cpu-limit 100 Enable auto-detection of node capabilities such as memory and CPU. yarn.nodemanager.resource.detect-hardware-capabilities false NM Webapp address. yarn.nodemanager.webapp.address ${yarn.nodemanager.hostname}:8042 The https adddress of the NM web application. yarn.nodemanager.webapp.https.address 0.0.0.0:8044 The Kerberos keytab file to be used for spnego filter for the NM web interface. yarn.nodemanager.webapp.spnego-keytab-file The Kerberos principal to be used for spnego filter for the NM web interface. yarn.nodemanager.webapp.spnego-principal How often to monitor the node and the containers. yarn.nodemanager.resource-monitor.interval-ms 3000 Class that calculates current resource utilization. yarn.nodemanager.resource-calculator.class How often to monitor containers. If not set, the value for yarn.nodemanager.resource-monitor.interval-ms will be used. yarn.nodemanager.container-monitor.interval-ms Class that calculates containers current resource utilization. If not set, the value for yarn.nodemanager.resource-calculator.class will be used. yarn.nodemanager.container-monitor.resource-calculator.class Frequency of running node health script. yarn.nodemanager.health-checker.interval-ms 600000 Script time out period. yarn.nodemanager.health-checker.script.timeout-ms 1200000 The health check script to run. yarn.nodemanager.health-checker.script.path The arguments to pass to the health check script. yarn.nodemanager.health-checker.script.opts Frequency of running disk health checker code. yarn.nodemanager.disk-health-checker.interval-ms 120000 The minimum fraction of number of disks to be healthy for the nodemanager to launch new containers. This correspond to both yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there are less number of healthy local-dirs (or log-dirs) available, then new containers will not be launched on this node. yarn.nodemanager.disk-health-checker.min-healthy-disks 0.25 The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the nodemanager will check for full disk. This applies to yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage 90.0 The low threshold percentage of disk space used when a bad disk is marked as good. Values can range from 0.0 to 100.0. This applies to yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. Note that if its value is more than yarn.nodemanager.disk-health-checker. max-disk-utilization-per-disk-percentage or not set, it will be set to the same value as yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage. yarn.nodemanager.disk-health-checker.disk-utilization-watermark-low-per-disk-percentage The minimum space that must be available on a disk for it to be used. This applies to yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb 0 The path to the Linux container executor. yarn.nodemanager.linux-container-executor.path The class which should help the LCE handle resources. yarn.nodemanager.linux-container-executor.resources-handler.class org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler The cgroups hierarchy under which to place YARN proccesses (cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured), then this cgroups hierarchy must already exist and be writable by the NodeManager user, otherwise the NodeManager may fail. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. yarn.nodemanager.linux-container-executor.cgroups.hierarchy /hadoop-yarn Whether the LCE should attempt to mount cgroups if not found. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. yarn.nodemanager.linux-container-executor.cgroups.mount false Where the LCE should attempt to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux distribution in use. This path must exist before the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and yarn.nodemanager.linux-container-executor.cgroups.mount is true. yarn.nodemanager.linux-container-executor.cgroups.mount-path Delay in ms between attempts to remove linux cgroup yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms 20 This determines which of the two modes that LCE should use on a non-secure cluster. If this value is set to true, then all containers will be launched as the user specified in yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If this value is set to false, then containers will run as the user who submitted the application. yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users true The UNIX user that containers will run as when Linux-container-executor is used in nonsecure mode (a use case for this is using cgroups) if the yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users is set to true. yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user nobody The allowed pattern for UNIX user names enforced by Linux-container-executor when used in nonsecure mode (use case for this is using cgroups). The default value is taken from /usr/sbin/adduser yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern ^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$ This flag determines whether apps should run with strict resource limits or be allowed to consume spare resources if they need them. For example, turning the flag on will restrict apps to use only their share of CPU, even if the node has spare CPU cycles. The default value is false i.e. use available resources. Please note that turning this flag on may reduce job throughput on the cluster. yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage false Comma separated list of runtimes that are allowed when using LinuxContainerExecutor. The allowed values are default and docker. yarn.nodemanager.runtime.linux.allowed-runtimes default This configuration setting determines the capabilities assigned to docker containers when they are launched. While these may not be case-sensitive from a docker perspective, it is best to keep these uppercase. yarn.nodemanager.runtime.linux.docker.capabilities CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE This configuration setting determines if privileged docker containers are allowed on this cluster. Use with extreme care. yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed false This configuration setting determines who is allowed to run privileged docker containers on this cluster. Use with extreme care. yarn.nodemanager.runtime.linux.docker.privileged-containers.acl This flag determines whether memory limit will be set for the Windows Job Object of the containers launched by the default container executor. yarn.nodemanager.windows-container.memory-limit.enabled false This flag determines whether CPU limit will be set for the Windows Job Object of the containers launched by the default container executor. yarn.nodemanager.windows-container.cpu-limit.enabled false Interval of time the linux container executor should try cleaning up cgroups entry when cleaning up a container. yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms 1000 The UNIX group that the linux-container-executor should run as. yarn.nodemanager.linux-container-executor.group T-file compression types used to compress aggregated logs. yarn.nodemanager.log-aggregation.compression-type none The kerberos principal for the node manager. yarn.nodemanager.principal A comma separated list of services where service name should only contain a-zA-Z0-9_ and can not start with numbers yarn.nodemanager.aux-services No. of ms to wait between sending a SIGTERM and SIGKILL to a container yarn.nodemanager.sleep-delay-before-sigkill.ms 250 Max time to wait for a process to come up when trying to cleanup a container yarn.nodemanager.process-kill-wait.ms 2000 The minimum allowed version of a resourcemanager that a nodemanager will connect to. The valid values are NONE (no version checking), EqualToNM (the resourcemanager's version is equal to or greater than the NM version), or a Version String. yarn.nodemanager.resourcemanager.minimum.version NONE Max number of threads in NMClientAsync to process container management events yarn.client.nodemanager-client-async.thread-pool-max-size 500 Max time to wait to establish a connection to NM yarn.client.nodemanager-connect.max-wait-ms 180000 Time interval between each attempt to connect to NM yarn.client.nodemanager-connect.retry-interval-ms 10000 Max time to wait for NM to connect to RM. When not set, proxy will fall back to use value of yarn.resourcemanager.connect.max-wait.ms. yarn.nodemanager.resourcemanager.connect.max-wait.ms Time interval between each NM attempt to connect to RM. When not set, proxy will fall back to use value of yarn.resourcemanager.connect.retry-interval.ms. yarn.nodemanager.resourcemanager.connect.retry-interval.ms Maximum number of proxy connections to cache for node managers. If set to a value greater than zero then the cache is enabled and the NMClient and MRAppMaster will cache the specified number of node manager proxies. There will be at max one proxy per node manager. Ex. configuring it to a value of 5 will make sure that client will at max have 5 proxies cached with 5 different node managers. These connections for these proxies will be timed out if idle for more than the system wide idle timeout period. Note that this could cause issues on large clusters as many connections could linger simultaneously and lead to a large number of connection threads. The token used for authentication will be used only at connection creation time. If a new token is received then the earlier connection should be closed in order to use the new token. This and (yarn.client.nodemanager-client-async.thread-pool-max-size) are related and should be in sync (no need for them to be equal). If the value of this property is zero then the connection cache is disabled and connections will use a zero idle timeout to prevent too many connection threads on large clusters. yarn.client.max-cached-nodemanagers-proxies 0 Enable the node manager to recover after starting yarn.nodemanager.recovery.enabled false The local filesystem directory in which the node manager will store state when recovery is enabled. yarn.nodemanager.recovery.dir ${hadoop.tmp.dir}/yarn-nm-recovery The time in seconds between full compactions of the NM state database. Setting the interval to zero disables the full compaction cycles. yarn.nodemanager.recovery.compaction-interval-secs 3600 Whether the nodemanager is running under supervision. A nodemanager that supports recovery and is running under supervision will not try to cleanup containers as it exits with the assumption it will be immediately be restarted and recover containers. yarn.nodemanager.recovery.supervised false Adjustment to the container OS scheduling priority. In Linux, passed directly to the nice command. yarn.nodemanager.container-executor.os.sched.priority.adjustment 0 Flag to enable container metrics yarn.nodemanager.container-metrics.enable true Container metrics flush period in ms. Set to -1 for flush on completion. yarn.nodemanager.container-metrics.period-ms -1 The delay time ms to unregister container metrics after completion. yarn.nodemanager.container-metrics.unregister-delay-ms 10000 Class used to calculate current container resource utilization. yarn.nodemanager.container-monitor.process-tree.class Flag to enable NodeManager disk health checker yarn.nodemanager.disk-health-checker.enable true Number of threads to use in NM log cleanup. Used when log aggregation is disabled. yarn.nodemanager.log.deletion-threads-count 4 The Windows group that the windows-container-executor should run as. yarn.nodemanager.windows-secure-container-executor.group yarn.nodemanager.docker-container-executor.exec-name /usr/bin/docker Name or path to the Docker client. The Docker image name to use for DockerContainerExecutor yarn.nodemanager.docker-container-executor.image-name mapreduce.job.hdfs-servers ${fs.defaultFS} yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler The kerberos principal for the proxy, if the proxy is not running as part of the RM. yarn.web-proxy.principal Keytab for WebAppProxy, if the proxy is not running as part of the RM. yarn.web-proxy.keytab The address for the web proxy as HOST:PORT, if this is not given then the proxy will run as part of the RM yarn.web-proxy.address CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries. When this value is empty, the following default CLASSPATH for YARN applications would be used. For Linux: $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, $HADOOP_YARN_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/* For Windows: %HADOOP_CONF_DIR%, %HADOOP_COMMON_HOME%/share/hadoop/common/*, %HADOOP_COMMON_HOME%/share/hadoop/common/lib/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/lib/* yarn.application.classpath Indicate what is the current version of the running timeline service. For example, if "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it means the cluster will and should bring up the timeline service v.1.5. On the client side, if the client uses the same version of timeline service, it should succeed. If the client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary. yarn.timeline-service.version 1.0f In the server side it indicates whether timeline service is enabled or not. And in the client side, users can enable it to indicate whether client wants to use timeline service. If it's enabled in the client side along with security, then yarn client tries to fetch the delegation tokens for the timeline server. yarn.timeline-service.enabled false The hostname of the timeline service web application. yarn.timeline-service.hostname 0.0.0.0 This is default address for the timeline server to start the RPC server. yarn.timeline-service.address ${yarn.timeline-service.hostname}:10200 The http address of the timeline service web application. yarn.timeline-service.webapp.address ${yarn.timeline-service.hostname}:8188 The https address of the timeline service web application. yarn.timeline-service.webapp.https.address ${yarn.timeline-service.hostname}:8190 The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively. This is most useful for making the service listen to all interfaces by setting to 0.0.0.0. yarn.timeline-service.bind-host Defines the max number of applications could be fetched using REST API or application history protocol and shown in timeline server web ui. yarn.timeline-service.generic-application-history.max-applications 10000 Store class name for timeline store. yarn.timeline-service.store-class org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore Enable age off of timeline store data. yarn.timeline-service.ttl-enable true Time to live for timeline store data in milliseconds. yarn.timeline-service.ttl-ms 604800000 Store file name for leveldb timeline store. yarn.timeline-service.leveldb-timeline-store.path ${hadoop.tmp.dir}/yarn/timeline Length of time to wait between deletion cycles of leveldb timeline store in milliseconds. yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms 300000 Size of read cache for uncompressed blocks for leveldb timeline store in bytes. yarn.timeline-service.leveldb-timeline-store.read-cache-size 104857600 Size of cache for recently read entity start times for leveldb timeline store in number of entities. yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size 10000 Size of cache for recently written entity start times for leveldb timeline store in number of entities. yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size 10000 Handler thread count to serve the client RPC requests. yarn.timeline-service.handler-thread-count 10 yarn.timeline-service.http-authentication.type simple Defines authentication used for the timeline server HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# yarn.timeline-service.http-authentication.simple.anonymous.allowed true Indicates if anonymous requests are allowed by the timeline server when using 'simple' authentication. The Kerberos principal for the timeline server. yarn.timeline-service.principal The Kerberos keytab for the timeline server. yarn.timeline-service.keytab /etc/krb5.keytab Comma separated list of UIs that will be hosted yarn.timeline-service.ui-names Default maximum number of retries for timeline service client and value -1 means no limit. yarn.timeline-service.client.max-retries 30 Client policy for whether timeline operations are non-fatal. Should the failure to obtain a delegation token be considered an application failure (option = false), or should the client attempt to continue to publish information without it (option=true) yarn.timeline-service.client.best-effort false Default retry time interval for timeline servive client. yarn.timeline-service.client.retry-interval-ms 1000 Enable timeline server to recover state after starting. If true, then yarn.timeline-service.state-store-class must be specified. yarn.timeline-service.recovery.enabled false Store class name for timeline state store. yarn.timeline-service.state-store-class org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore Store file name for leveldb state store. yarn.timeline-service.leveldb-state-store.path ${hadoop.tmp.dir}/yarn/timeline yarn.timeline-service.entity-group-fs-store.active-dir /tmp/entity-file-history/active HDFS path to store active application's timeline data yarn.timeline-service.entity-group-fs-store.done-dir /tmp/entity-file-history/done/ HDFS path to store done application's timeline data yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes Plugins that can translate a timeline entity read request into a list of timeline entity group ids, separated by commas. yarn.timeline-service.entity-group-fs-store.summary-store Summary storage for ATS v1.5 org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore yarn.timeline-service.entity-group-fs-store.scan-interval-seconds Scan interval for ATS v1.5 entity group file system storage reader.This value controls how frequent the reader will scan the HDFS active directory for application status. 60 yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds Scan interval for ATS v1.5 entity group file system storage cleaner.This value controls how frequent the reader will scan the HDFS done directory for stale application data. 3600 yarn.timeline-service.entity-group-fs-store.retain-seconds How long the ATS v1.5 entity group file system storage will keep an application's data in the done directory. 604800 yarn.timeline-service.entity-group-fs-store.leveldb-cache-read-cache-size Read cache size for the leveldb cache storage in ATS v1.5 plugin storage. 10485760 yarn.timeline-service.entity-group-fs-store.app-cache-size Size of the reader cache for ATS v1.5 reader. This value controls how many entity groups the ATS v1.5 server should cache. If the number of active read entity groups is greater than the number of caches items, some reads may return empty data. This value must be greater than 0. 10 yarn.timeline-service.client.fd-flush-interval-secs Flush interval for ATS v1.5 writer. This value controls how frequent the writer will flush the HDFS FSStream for the entity/domain. 10 yarn.timeline-service.client.fd-clean-interval-secs Scan interval for ATS v1.5 writer. This value controls how frequent the writer will scan the HDFS FSStream for the entity/domain. If the FSStream is stale for a long time, this FSStream will be close. 60 yarn.timeline-service.client.fd-retain-secs How long the ATS v1.5 writer will keep a FSStream open. If this fsstream does not write anything for this configured time, it will be close. 300 yarn.timeline-service.client.internal-timers-ttl-secs How long the internal Timer Tasks can be alive in writer. If there is no write operation for this configured time, the internal timer tasks will be close. 420 Whether the shared cache is enabled yarn.sharedcache.enabled false The root directory for the shared cache yarn.sharedcache.root-dir /sharedcache The level of nested directories before getting to the checksum directories. It must be non-negative. yarn.sharedcache.nested-level 3 The implementation to be used for the SCM store yarn.sharedcache.store.class org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore The implementation to be used for the SCM app-checker yarn.sharedcache.app-checker.class org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker A resource in the in-memory store is considered stale if the time since the last reference exceeds the staleness period. This value is specified in minutes. yarn.sharedcache.store.in-memory.staleness-period-mins 10080 Initial delay before the in-memory store runs its first check to remove dead initial applications. Specified in minutes. yarn.sharedcache.store.in-memory.initial-delay-mins 10 The frequency at which the in-memory store checks to remove dead initial applications. Specified in minutes. yarn.sharedcache.store.in-memory.check-period-mins 720 The address of the admin interface in the SCM (shared cache manager) yarn.sharedcache.admin.address 0.0.0.0:8047 The number of threads used to handle SCM admin interface (1 by default) yarn.sharedcache.admin.thread-count 1 The address of the web application in the SCM (shared cache manager) yarn.sharedcache.webapp.address 0.0.0.0:8788 The frequency at which a cleaner task runs. Specified in minutes. yarn.sharedcache.cleaner.period-mins 1440 Initial delay before the first cleaner task is scheduled. Specified in minutes. yarn.sharedcache.cleaner.initial-delay-mins 10 The time to sleep between processing each shared cache resource. Specified in milliseconds. yarn.sharedcache.cleaner.resource-sleep-ms 0 The address of the node manager interface in the SCM (shared cache manager) yarn.sharedcache.uploader.server.address 0.0.0.0:8046 The number of threads used to handle shared cache manager requests from the node manager (50 by default) yarn.sharedcache.uploader.server.thread-count 50 The address of the client interface in the SCM (shared cache manager) yarn.sharedcache.client-server.address 0.0.0.0:8045 The number of threads used to handle shared cache manager requests from clients (50 by default) yarn.sharedcache.client-server.thread-count 50 The algorithm used to compute checksums of files (SHA-256 by default) yarn.sharedcache.checksum.algo.impl org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl The replication factor for the node manager uploader for the shared cache (10 by default) yarn.sharedcache.nm.uploader.replication.factor 10 The number of threads used to upload files from a node manager instance (20 by default) yarn.sharedcache.nm.uploader.thread-count 20 ACL protocol for use in the Timeline server. security.applicationhistory.protocol.acl Set to true for MiniYARNCluster unit tests yarn.is.minicluster false Set for MiniYARNCluster unit tests to control resource monitoring yarn.minicluster.control-resource-monitoring false Set to false in order to allow MiniYARNCluster to run tests without port conflicts. yarn.minicluster.fixed.ports false Set to false in order to allow the NodeManager in MiniYARNCluster to use RPC to talk to the RM. yarn.minicluster.use-rpc false As yarn.nodemanager.resource.memory-mb property but for the NodeManager in a MiniYARNCluster. yarn.minicluster.yarn.nodemanager.resource.memory-mb 4096 Enable node labels feature yarn.node-labels.enabled false Retry policy used for FileSystem node label store. The policy is specified by N pairs of sleep-time in milliseconds and number-of-retries "s1,n1,s2,n2,...". yarn.node-labels.fs-store.retry-policy-spec 2000, 500 URI for NodeLabelManager. The default value is /tmp/hadoop-yarn-${user}/node-labels/ in the local filesystem. yarn.node-labels.fs-store.root-dir Set configuration type for node labels. Administrators can specify "centralized", "delegated-centralized" or "distributed". yarn.node-labels.configuration-type centralized When "yarn.node-labels.configuration-type" is configured with "distributed" in RM, Administrators can configure in NM the provider for the node labels by configuring this parameter. Administrators can configure "config", "script" or the class name of the provider. Configured class needs to extend org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider. If "config" is configured, then "ConfigurationNodeLabelsProvider" and if "script" is configured, then "ScriptNodeLabelsProvider" will be used. yarn.nodemanager.node-labels.provider When "yarn.nodemanager.node-labels.provider" is configured with "config", "Script" or the configured class extends AbstractNodeLabelsProvider, then periodically node labels are retrieved from the node labels provider. This configuration is to define the interval period. If -1 is configured then node labels are retrieved from provider only during initialization. Defaults to 10 mins. yarn.nodemanager.node-labels.provider.fetch-interval-ms 600000 Interval at which NM syncs its node labels with RM. NM will send its loaded labels every x intervals configured, along with heartbeat to RM. yarn.nodemanager.node-labels.resync-interval-ms 120000 When "yarn.nodemanager.node-labels.provider" is configured with "config" then ConfigurationNodeLabelsProvider fetches the partition label from this parameter. yarn.nodemanager.node-labels.provider.configured-node-partition When "yarn.nodemanager.node-labels.provider" is configured with "Script" then this configuration provides the timeout period after which it will interrupt the script which queries the Node labels. Defaults to 20 mins. yarn.nodemanager.node-labels.provider.fetch-timeout-ms 1200000 When node labels "yarn.node-labels.configuration-type" is of type "delegated-centralized", administrators should configure the class for fetching node labels by ResourceManager. Configured class needs to extend org.apache.hadoop.yarn.server.resourcemanager.nodelabels. RMNodeLabelsMappingProvider. yarn.resourcemanager.node-labels.provider When "yarn.node-labels.configuration-type" is configured with "delegated-centralized", then periodically node labels are retrieved from the node labels provider. This configuration is to define the interval. If -1 is configured then node labels are retrieved from provider only once for each node after it registers. Defaults to 30 mins. yarn.resourcemanager.node-labels.provider.fetch-interval-ms 1800000 The Node Label script to run. Script output Line starting with "NODE_PARTITION:" will be considered as Node Label Partition. In case of multiple lines have this pattern, then last one will be considered yarn.nodemanager.node-labels.provider.script.path The arguments to pass to the Node label script. yarn.nodemanager.node-labels.provider.script.opts The interval that the yarn client library uses to poll the completion status of the asynchronous API of application client protocol. yarn.client.application-client-protocol.poll-interval-ms 200 The duration (in ms) the YARN client waits for an expected state change to occur. -1 means unlimited wait time. yarn.client.application-client-protocol.poll-timeout-ms -1 RSS usage of a process computed via /proc/pid/stat is not very accurate as it includes shared pages of a process. /proc/pid/smaps provides useful information like Private_Dirty, Private_Clean, Shared_Dirty, Shared_Clean which can be used for computing more accurate RSS. When this flag is enabled, RSS is computed as Min(Shared_Dirty, Pss) + Private_Clean + Private_Dirty. It excludes read-only shared mappings in RSS computation. yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled false URL for log aggregation server yarn.log.server.url RM Application Tracking URL yarn.tracking.url.generator Class to be used for YarnAuthorizationProvider yarn.authorization-provider Defines how often NMs wake up to upload log files. The default value is -1. By default, the logs will be uploaded when the application is finished. By setting this configure, logs can be uploaded periodically when the application is running. The minimum rolling-interval-seconds can be set is 3600. yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds -1 Enable/disable intermediate-data encryption at YARN level. For now, this only is used by the FileSystemRMStateStore to setup right file-system security attributes. yarn.intermediate-data-encryption.enable false Flag to enable cross-origin (CORS) support in the NM. This flag requires the CORS filter initializer to be added to the filter initializers list in core-site.xml. yarn.nodemanager.webapp.cross-origin.enabled false Defines maximum application priority in a cluster. If an application is submitted with a priority higher than this value, it will be reset to this maximum value. yarn.cluster.max-application-priority 0 The default log aggregation policy class. Applications can override it via LogAggregationContext. This configuration can provide some cluster-side default behavior so that if the application doesn't specify any policy via LogAggregationContext administrators of the cluster can adjust the policy globally. yarn.nodemanager.log-aggregation.policy.class org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AllContainerLogAggregationPolicy The default parameters for the log aggregation policy. Applications can override it via LogAggregationContext. This configuration can provide some cluster-side default behavior so that if the application doesn't specify any policy via LogAggregationContext administrators of the cluster can adjust the policy globally. yarn.nodemanager.log-aggregation.policy.parameters Enable/Disable AMRMProxyService in the node manager. This service is used to intercept calls from the application masters to the resource manager. yarn.nodemanager.amrmproxy.enable false The address of the AMRMProxyService listener. yarn.nodemanager.amrmproxy.address 0.0.0.0:8048 The number of threads used to handle requests by the AMRMProxyService. yarn.nodemanager.amrmproxy.client.thread-count 25 The comma separated list of class names that implement the RequestInterceptor interface. This is used by the AMRMProxyService to create the request processing pipeline for applications. yarn.nodemanager.amrmproxy.interceptor-class.pipeline org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor Choose different implementation of node label's storage yarn.node-labels.fs-store.impl.class org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore The least amount of time(msec.) an inactive (decommissioned or shutdown) node can stay in the nodes list of the resourcemanager after being declared untracked. A node is marked untracked if and only if it is absent from both include and exclude nodemanager lists on the RM. All inactive nodes are checked twice per timeout interval or every 10 minutes, whichever is lesser, and marked appropriately. The same is done when refreshNodes command (graceful or otherwise) is invoked. yarn.resourcemanager.node-removal-untracked.timeout-ms 60000 sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/mapred-default.xml0000666000175100017510000020570613245514472026020 0ustar zuulzuul00000000000000 mapreduce.job.committer.setup.cleanup.needed true true, if job needs job-setup and job-cleanup. false, otherwise mapreduce.task.io.sort.factor 10 The number of streams to merge at once while sorting files. This determines the number of open file handles. mapreduce.task.io.sort.mb 100 The total amount of buffer memory to use while sorting files, in megabytes. By default, gives each merge stream 1MB, which should minimize seeks. mapreduce.map.sort.spill.percent 0.80 The soft limit in the serialization buffer. Once reached, a thread will begin to spill the contents to disk in the background. Note that collection will not block if this threshold is exceeded while a spill is already in progress, so spills may be larger than this threshold when it is set to less than .5 mapreduce.jobtracker.address local The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. mapreduce.local.clientfactory.class.name org.apache.hadoop.mapred.LocalClientFactory This the client factory that is responsible for creating local job runner client mapreduce.jobtracker.system.dir ${hadoop.tmp.dir}/mapred/system The directory where MapReduce stores control files. mapreduce.jobtracker.staging.root.dir ${hadoop.tmp.dir}/mapred/staging The root of the staging area for users' job files In practice, this should be the directory where users' home directories are located (usually /user) mapreduce.cluster.temp.dir ${hadoop.tmp.dir}/mapred/temp A shared directory for temporary files. mapreduce.job.maps 2 The default number of map tasks per job. Ignored when mapreduce.framework.name is "local". mapreduce.job.reduces 1 The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapreduce.framework.name is "local". mapreduce.job.running.map.limit 0 The maximum number of simultaneous map tasks per job. There is no limit if this value is 0 or negative. mapreduce.job.running.reduce.limit 0 The maximum number of simultaneous reduce tasks per job. There is no limit if this value is 0 or negative. mapreduce.job.reducer.preempt.delay.sec 0 The threshold (in seconds) after which an unsatisfied mapper request triggers reducer preemption when there is no anticipated headroom. If set to 0 or a negative value, the reducer is preempted as soon as lack of headroom is detected. Default is 0. mapreduce.job.reducer.unconditional-preempt.delay.sec 300 The threshold (in seconds) after which an unsatisfied mapper request triggers a forced reducer preemption irrespective of the anticipated headroom. By default, it is set to 5 mins. Setting it to 0 leads to immediate reducer preemption. Setting to -1 disables this preemption altogether. mapreduce.job.max.split.locations 10 The max number of block locations to store for each split for locality calculation. mapreduce.job.split.metainfo.maxsize 10000000 The maximum permissible size of the split metainfo file. The MapReduce ApplicationMaster won't attempt to read submitted split metainfo files bigger than this configured value. No limits if set to -1. mapreduce.map.maxattempts 4 Expert: The maximum number of attempts per map task. In other words, framework will try to execute a map task these many number of times before giving up on it. mapreduce.reduce.maxattempts 4 Expert: The maximum number of attempts per reduce task. In other words, framework will try to execute a reduce task these many number of times before giving up on it. mapreduce.reduce.shuffle.fetch.retry.enabled ${yarn.nodemanager.recovery.enabled} Set to enable fetch retry during host restart. mapreduce.reduce.shuffle.fetch.retry.interval-ms 1000 Time of interval that fetcher retry to fetch again when some non-fatal failure happens because of some events like NM restart. mapreduce.reduce.shuffle.fetch.retry.timeout-ms 30000 Timeout value for fetcher to retry to fetch again when some non-fatal failure happens because of some events like NM restart. mapreduce.reduce.shuffle.retry-delay.max.ms 60000 The maximum number of ms the reducer will delay before retrying to download map data. mapreduce.reduce.shuffle.parallelcopies 5 The default number of parallel transfers run by reduce during the copy(shuffle) phase. mapreduce.reduce.shuffle.connect.timeout 180000 Expert: The maximum amount of time (in milli seconds) reduce task spends in trying to connect to a remote node for getting map output. mapreduce.reduce.shuffle.read.timeout 180000 Expert: The maximum amount of time (in milli seconds) reduce task waits for map output data to be available for reading after obtaining connection. mapreduce.shuffle.listen.queue.size 128 The length of the shuffle server listen queue. mapreduce.shuffle.connection-keep-alive.enable false set to true to support keep-alive connections. mapreduce.shuffle.connection-keep-alive.timeout 5 The number of seconds a shuffle client attempts to retain http connection. Refer "Keep-Alive: timeout=" header in Http specification mapreduce.task.timeout 600000 The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string. A value of 0 disables the timeout. mapreduce.map.memory.mb 1024 The amount of memory to request from the scheduler for each map task. mapreduce.map.cpu.vcores 1 The number of virtual cores to request from the scheduler for each map task. mapreduce.reduce.memory.mb 1024 The amount of memory to request from the scheduler for each reduce task. mapreduce.reduce.cpu.vcores 1 The number of virtual cores to request from the scheduler for each reduce task. mapred.child.java.opts -Xmx200m Java opts for the task processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. mapred.child.env User added environment variables for the task processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit nodemanager's B env variable on Unix. 3) B=%B%;c This is inherit nodemanager's B env variable on Windows. mapreduce.admin.user.env Expert: Additional execution environment entries for map and reduce task processes. This is not an additive property. You must preserve the original value if you want your map and reduce tasks to have access to native libraries (compression, etc). When this value is empty, the command to set execution envrionment will be OS dependent: For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native. For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin. yarn.app.mapreduce.am.log.level INFO The logging level for the MR ApplicationMaster. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting here could be overriden if "mapreduce.job.log4j-properties-file" is set. mapreduce.map.log.level INFO The logging level for the map task. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting here could be overridden if "mapreduce.job.log4j-properties-file" is set. mapreduce.reduce.log.level INFO The logging level for the reduce task. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting here could be overridden if "mapreduce.job.log4j-properties-file" is set. mapreduce.map.cpu.vcores 1 The number of virtual cores required for each map task. mapreduce.reduce.cpu.vcores 1 The number of virtual cores required for each reduce task. mapreduce.reduce.merge.inmem.threshold 1000 The threshold, in terms of the number of files for the in-memory merge process. When we accumulate threshold number of files we initiate the in-memory merge and spill to disk. A value of 0 or less than 0 indicates we want to DON'T have any threshold and instead depend only on the ramfs's memory consumption to trigger the merge. mapreduce.reduce.shuffle.merge.percent 0.66 The usage threshold at which an in-memory merge will be initiated, expressed as a percentage of the total memory allocated to storing in-memory map outputs, as defined by mapreduce.reduce.shuffle.input.buffer.percent. mapreduce.reduce.shuffle.input.buffer.percent 0.70 The percentage of memory to be allocated from the maximum heap size to storing map outputs during the shuffle. mapreduce.reduce.input.buffer.percent 0.0 The percentage of memory- relative to the maximum heap size- to retain map outputs during the reduce. When the shuffle is concluded, any remaining map outputs in memory must consume less than this threshold before the reduce can begin. mapreduce.reduce.shuffle.memory.limit.percent 0.25 Expert: Maximum percentage of the in-memory limit that a single shuffle can consume mapreduce.shuffle.ssl.enabled false Whether to use SSL for for the Shuffle HTTP endpoints. mapreduce.shuffle.ssl.file.buffer.size 65536 Buffer size for reading spills from file when using SSL. mapreduce.shuffle.max.connections 0 Max allowed connections for the shuffle. Set to 0 (zero) to indicate no limit on the number of connections. mapreduce.shuffle.max.threads 0 Max allowed threads for serving shuffle connections. Set to zero to indicate the default of 2 times the number of available processors (as reported by Runtime.availableProcessors()). Netty is used to serve requests, so a thread is not needed for each connection. mapreduce.shuffle.transferTo.allowed This option can enable/disable using nio transferTo method in the shuffle phase. NIO transferTo does not perform well on windows in the shuffle phase. Thus, with this configuration property it is possible to disable it, in which case custom transfer method will be used. Recommended value is false when running Hadoop on Windows. For Linux, it is recommended to set it to true. If nothing is set then the default value is false for Windows, and true for Linux. mapreduce.shuffle.transfer.buffer.size 131072 This property is used only if mapreduce.shuffle.transferTo.allowed is set to false. In that case, this property defines the size of the buffer used in the buffer copy code for the shuffle phase. The size of this buffer determines the size of the IO requests. mapreduce.reduce.markreset.buffer.percent 0.0 The percentage of memory -relative to the maximum heap size- to be used for caching values when using the mark-reset functionality. mapreduce.map.speculative true If true, then multiple instances of some map tasks may be executed in parallel. mapreduce.reduce.speculative true If true, then multiple instances of some reduce tasks may be executed in parallel. mapreduce.job.speculative.speculative-cap-running-tasks 0.1 The max percent (0-1) of running tasks that can be speculatively re-executed at any time. mapreduce.job.speculative.speculative-cap-total-tasks 0.01 The max percent (0-1) of all tasks that can be speculatively re-executed at any time. mapreduce.job.speculative.minimum-allowed-tasks 10 The minimum allowed tasks that can be speculatively re-executed at any time. mapreduce.job.speculative.retry-after-no-speculate 1000 The waiting time(ms) to do next round of speculation if there is no task speculated in this round. mapreduce.job.speculative.retry-after-speculate 15000 The waiting time(ms) to do next round of speculation if there are tasks speculated in this round. mapreduce.job.map.output.collector.class org.apache.hadoop.mapred.MapTask$MapOutputBuffer The MapOutputCollector implementation(s) to use. This may be a comma-separated list of class names, in which case the map task will try to initialize each of the collectors in turn. The first to successfully initialize will be used. mapreduce.job.speculative.slowtaskthreshold 1.0 The number of standard deviations by which a task's ave progress-rates must be lower than the average of all running tasks' for the task to be considered too slow. mapreduce.job.ubertask.enable false Whether to enable the small-jobs "ubertask" optimization, which runs "sufficiently small" jobs sequentially within a single JVM. "Small" is defined by the following maxmaps, maxreduces, and maxbytes settings. Note that configurations for application masters also affect the "Small" definition - yarn.app.mapreduce.am.resource.mb must be larger than both mapreduce.map.memory.mb and mapreduce.reduce.memory.mb, and yarn.app.mapreduce.am.resource.cpu-vcores must be larger than both mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores to enable ubertask. Users may override this value. mapreduce.job.ubertask.maxmaps 9 Threshold for number of maps, beyond which job is considered too big for the ubertasking optimization. Users may override this value, but only downward. mapreduce.job.ubertask.maxreduces 1 Threshold for number of reduces, beyond which job is considered too big for the ubertasking optimization. CURRENTLY THE CODE CANNOT SUPPORT MORE THAN ONE REDUCE and will ignore larger values. (Zero is a valid max, however.) Users may override this value, but only downward. mapreduce.job.ubertask.maxbytes Threshold for number of input bytes, beyond which job is considered too big for the ubertasking optimization. If no value is specified, dfs.block.size is used as a default. Be sure to specify a default value in mapred-site.xml if the underlying filesystem is not HDFS. Users may override this value, but only downward. mapreduce.job.emit-timeline-data false Specifies if the Application Master should emit timeline data to the timeline server. Individual jobs can override this value. mapreduce.input.fileinputformat.split.minsize 0 The minimum size chunk that map input should be split into. Note that some file formats may have minimum split sizes that take priority over this setting. mapreduce.input.fileinputformat.list-status.num-threads 1 The number of threads to use to list and fetch block locations for the specified input paths. Note: multiple threads should not be used if a custom non thread-safe path filter is used. mapreduce.input.lineinputformat.linespermap 1 When using NLineInputFormat, the number of lines of input data to include in each split. mapreduce.client.submit.file.replication 10 The replication level for submitted job files. This should be around the square root of the number of nodes. mapreduce.task.files.preserve.failedtasks false Should the files for failed tasks be kept. This should only be used on jobs that are failing, because the storage is never reclaimed. It also prevents the map outputs from being erased from the reduce directory as they are consumed. mapreduce.output.fileoutputformat.compress false Should the job outputs be compressed? mapreduce.output.fileoutputformat.compress.type RECORD If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK. mapreduce.output.fileoutputformat.compress.codec org.apache.hadoop.io.compress.DefaultCodec If the job outputs are compressed, how should they be compressed? mapreduce.map.output.compress false Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression. mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.DefaultCodec If the map outputs are compressed, how should they be compressed? map.sort.class org.apache.hadoop.util.QuickSort The default sort class for sorting keys. mapreduce.task.userlog.limit.kb 0 The maximum size of user-logs of each task in KB. 0 disables the cap. yarn.app.mapreduce.am.container.log.limit.kb 0 The maximum size of the MRAppMaster attempt container logs in KB. 0 disables the cap. yarn.app.mapreduce.task.container.log.backups 0 Number of backup files for task logs when using ContainerRollingLogAppender (CRLA). See org.apache.log4j.RollingFileAppender.maxBackupIndex. By default, ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA is enabled for tasks when both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.task.container.log.backups are greater than zero. yarn.app.mapreduce.am.container.log.backups 0 Number of backup files for the ApplicationMaster logs when using ContainerRollingLogAppender (CRLA). See org.apache.log4j.RollingFileAppender.maxBackupIndex. By default, ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA is enabled for the ApplicationMaster when both yarn.app.mapreduce.am.container.log.limit.kb and yarn.app.mapreduce.am.container.log.backups are greater than zero. yarn.app.mapreduce.shuffle.log.separate true If enabled ('true') logging generated by the client-side shuffle classes in a reducer will be written in a dedicated log file 'syslog.shuffle' instead of 'syslog'. yarn.app.mapreduce.shuffle.log.limit.kb 0 Maximum size of the syslog.shuffle file in kilobytes (0 for no limit). yarn.app.mapreduce.shuffle.log.backups 0 If yarn.app.mapreduce.shuffle.log.limit.kb and yarn.app.mapreduce.shuffle.log.backups are greater than zero then a ContainerRollngLogAppender is used instead of ContainerLogAppender for syslog.shuffle. See org.apache.log4j.RollingFileAppender.maxBackupIndex mapreduce.job.maxtaskfailures.per.tracker 3 The number of task-failures on a node manager of a given job after which new tasks of that job aren't assigned to it. It MUST be less than mapreduce.map.maxattempts and mapreduce.reduce.maxattempts otherwise the failed task will never be tried on a different node. mapreduce.client.output.filter FAILED The filter for controlling the output of the task's userlogs sent to the console of the JobClient. The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and ALL. mapreduce.client.completion.pollinterval 5000 The interval (in milliseconds) between which the JobClient polls the MapReduce ApplicationMaster for updates about job status. You may want to set this to a lower value to make tests run faster on a single node system. Adjusting this value in production may lead to unwanted client-server traffic. mapreduce.client.progressmonitor.pollinterval 1000 The interval (in milliseconds) between which the JobClient reports status to the console and checks for job completion. You may want to set this to a lower value to make tests run faster on a single node system. Adjusting this value in production may lead to unwanted client-server traffic. mapreduce.task.profile false To set whether the system should collect profiler information for some of the tasks in this job? The information is stored in the user log directory. The value is "true" if task profiling is enabled. mapreduce.task.profile.maps 0-2 To set the ranges of map tasks to profile. mapreduce.task.profile has to be set to true for the value to be accounted. mapreduce.task.profile.reduces 0-2 To set the ranges of reduce tasks to profile. mapreduce.task.profile has to be set to true for the value to be accounted. mapreduce.task.profile.params -agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s JVM profiler parameters used to profile map and reduce task attempts. This string may contain a single format specifier %s that will be replaced by the path to profile.out in the task attempt log directory. To specify different profiling options for map tasks and reduce tasks, more specific parameters mapreduce.task.profile.map.params and mapreduce.task.profile.reduce.params should be used. mapreduce.task.profile.map.params ${mapreduce.task.profile.params} Map-task-specific JVM profiler parameters. See mapreduce.task.profile.params mapreduce.task.profile.reduce.params ${mapreduce.task.profile.params} Reduce-task-specific JVM profiler parameters. See mapreduce.task.profile.params mapreduce.task.skip.start.attempts 2 The number of Task attempts AFTER which skip mode will be kicked off. When skip mode is kicked off, the tasks reports the range of records which it will process next, to the MR ApplicationMaster. So that on failures, the MR AM knows which ones are possibly the bad records. On further executions, those are skipped. mapreduce.map.skip.proc-count.auto-incr true The flag which if set to true, SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented by MapRunner after invoking the map function. This value must be set to false for applications which process the records asynchronously or buffer the input records. For example streaming. In such cases applications should increment this counter on their own. mapreduce.reduce.skip.proc-count.auto-incr true The flag which if set to true, SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented by framework after invoking the reduce function. This value must be set to false for applications which process the records asynchronously or buffer the input records. For example streaming. In such cases applications should increment this counter on their own. mapreduce.job.skip.outdir If no value is specified here, the skipped records are written to the output directory at _logs/skip. User can stop writing skipped records by giving the value "none". mapreduce.map.skip.maxrecords 0 The number of acceptable skip records surrounding the bad record PER bad record in mapper. The number includes the bad record as well. To turn the feature of detection/skipping of bad records off, set the value to 0. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to indicate that framework need not try to narrow down. Whatever records(depends on application) get skipped are acceptable. mapreduce.reduce.skip.maxgroups 0 The number of acceptable skip groups surrounding the bad group PER bad group in reducer. The number includes the bad group as well. To turn the feature of detection/skipping of bad groups off, set the value to 0. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to indicate that framework need not try to narrow down. Whatever groups(depends on application) get skipped are acceptable. mapreduce.ifile.readahead true Configuration key to enable/disable IFile readahead. mapreduce.ifile.readahead.bytes 4194304 Configuration key to set the IFile readahead length in bytes. mapreduce.job.queuename default Queue to which a job is submitted. This must match one of the queues defined in mapred-queues.xml for the system. Also, the ACL setup for the queue must allow the current user to submit a job to the queue. Before specifying a queue, ensure that the system is configured with the queue, and access is allowed for submitting jobs to the queue. mapreduce.job.tags Tags for the job that will be passed to YARN at submission time. Queries to YARN for applications can filter on these tags. mapreduce.cluster.local.dir ${hadoop.tmp.dir}/mapred/local The local directory where MapReduce stores intermediate data files. May be a comma-separated list of directories on different devices in order to spread disk i/o. Directories that do not exist are ignored. mapreduce.cluster.acls.enabled false Specifies whether ACLs should be checked for authorization of users for doing various queue and job level operations. ACLs are disabled by default. If enabled, access control checks are made by MapReduce ApplicationMaster when requests are made by users for queue operations like submit job to a queue and kill a job in the queue and job operations like viewing the job-details (See mapreduce.job.acl-view-job) or for modifying the job (See mapreduce.job.acl-modify-job) using Map/Reduce APIs, RPCs or via the console and web user interfaces. For enabling this flag, set to true in mapred-site.xml file of all MapReduce clients (MR job submitting nodes). mapreduce.job.acl-modify-job Job specific access-control list for 'modifying' the job. It is only used if authorization is enabled in Map/Reduce by setting the configuration property mapreduce.cluster.acls.enabled to true. This specifies the list of users and/or groups who can do modification operations on the job. For specifying a list of users and groups the format to use is "user1,user2 group1,group". If set to '*', it allows all users/groups to modify this job. If set to ' '(i.e. space), it allows none. This configuration is used to guard all the modifications with respect to this job and takes care of all the following operations: o killing this job o killing a task of this job, failing a task of this job o setting the priority of this job Each of these operations are also protected by the per-queue level ACL "acl-administer-jobs" configured via mapred-queues.xml. So a caller should have the authorization to satisfy either the queue-level ACL or the job-level ACL. Irrespective of this ACL configuration, (a) job-owner, (b) the user who started the cluster, (c) members of an admin configured supergroup configured via mapreduce.cluster.permissions.supergroup and (d) queue administrators of the queue to which this job was submitted to configured via acl-administer-jobs for the specific queue in mapred-queues.xml can do all the modification operations on a job. By default, nobody else besides job-owner, the user who started the cluster, members of supergroup and queue administrators can perform modification operations on a job. mapreduce.job.acl-view-job Job specific access-control list for 'viewing' the job. It is only used if authorization is enabled in Map/Reduce by setting the configuration property mapreduce.cluster.acls.enabled to true. This specifies the list of users and/or groups who can view private details about the job. For specifying a list of users and groups the format to use is "user1,user2 group1,group". If set to '*', it allows all users/groups to modify this job. If set to ' '(i.e. space), it allows none. This configuration is used to guard some of the job-views and at present only protects APIs that can return possibly sensitive information of the job-owner like o job-level counters o task-level counters o tasks' diagnostic information o task-logs displayed on the HistoryServer's web-UI and o job.xml showed by the HistoryServer's web-UI Every other piece of information of jobs is still accessible by any other user, for e.g., JobStatus, JobProfile, list of jobs in the queue, etc. Irrespective of this ACL configuration, (a) job-owner, (b) the user who started the cluster, (c) members of an admin configured supergroup configured via mapreduce.cluster.permissions.supergroup and (d) queue administrators of the queue to which this job was submitted to configured via acl-administer-jobs for the specific queue in mapred-queues.xml can do all the view operations on a job. By default, nobody else besides job-owner, the user who started the cluster, memebers of supergroup and queue administrators can perform view operations on a job. mapreduce.job.finish-when-all-reducers-done false Specifies whether the job should complete once all reducers have finished, regardless of whether there are still running mappers. mapreduce.job.token.tracking.ids.enabled false Whether to write tracking ids of tokens to job-conf. When true, the configuration property "mapreduce.job.token.tracking.ids" is set to the token-tracking-ids of the job mapreduce.job.token.tracking.ids When mapreduce.job.token.tracking.ids.enabled is set to true, this is set by the framework to the token-tracking-ids used by the job. mapreduce.task.merge.progress.records 10000 The number of records to process during merge before sending a progress notification to the MR ApplicationMaster. mapreduce.task.combine.progress.records 10000 The number of records to process during combine output collection before sending a progress notification. mapreduce.job.reduce.slowstart.completedmaps 0.05 Fraction of the number of maps in the job which should be complete before reduces are scheduled for the job. mapreduce.job.complete.cancel.delegation.tokens true if false - do not unregister/cancel delegation tokens from renewal, because same tokens may be used by spawned jobs mapreduce.shuffle.port 13562 Default port that the ShuffleHandler will run on. ShuffleHandler is a service run at the NodeManager to facilitate transfers of intermediate Map outputs to requesting Reducers. mapreduce.job.reduce.shuffle.consumer.plugin.class org.apache.hadoop.mapreduce.task.reduce.Shuffle Name of the class whose instance will be used to send shuffle requests by reducetasks of this job. The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin. mapreduce.job.node-label-expression All the containers of the Map Reduce job will be run with this node label expression. If the node-label-expression for job is not set, then it will use queue's default-node-label-expression for all job's containers. mapreduce.job.am.node-label-expression This is node-label configuration for Map Reduce Application Master container. If not configured it will make use of mapreduce.job.node-label-expression and if job's node-label expression is not configured then it will use queue's default-node-label-expression. mapreduce.map.node-label-expression This is node-label configuration for Map task containers. If not configured it will use mapreduce.job.node-label-expression and if job's node-label expression is not configured then it will use queue's default-node-label-expression. mapreduce.reduce.node-label-expression This is node-label configuration for Reduce task containers. If not configured it will use mapreduce.job.node-label-expression and if job's node-label expression is not configured then it will use queue's default-node-label-expression. mapreduce.job.counters.limit 120 Limit on the number of user counters allowed per job. mapreduce.framework.name local The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn. yarn.app.mapreduce.am.staging-dir /tmp/hadoop-yarn/staging The staging dir used while submitting jobs. mapreduce.am.max-attempts 2 The maximum number of application attempts. It is a application-specific setting. It should not be larger than the global number set by resourcemanager. Otherwise, it will be override. The default number is set to 2, to allow at least one retry for AM. mapreduce.job.end-notification.url Indicates url which will be called on completion of job to inform end status of job. User can give at most 2 variables with URI : $jobId and $jobStatus. If they are present in URI, then they will be replaced by their respective values. mapreduce.job.end-notification.retry.attempts 0 The number of times the submitter of the job wants to retry job end notification if it fails. This is capped by mapreduce.job.end-notification.max.attempts mapreduce.job.end-notification.retry.interval 1000 The number of milliseconds the submitter of the job wants to wait before job end notification is retried if it fails. This is capped by mapreduce.job.end-notification.max.retry.interval mapreduce.job.end-notification.max.attempts 5 true The maximum number of times a URL will be read for providing job end notification. Cluster administrators can set this to limit how long after end of a job, the Application Master waits before exiting. Must be marked as final to prevent users from overriding this. mapreduce.job.log4j-properties-file Used to override the default settings of log4j in container-log4j.properties for NodeManager. Like container-log4j.properties, it requires certain framework appenders properly defined in this overriden file. The file on the path will be added to distributed cache and classpath. If no-scheme is given in the path, it defaults to point to a log4j file on the local FS. mapreduce.job.end-notification.max.retry.interval 5000 true The maximum amount of time (in milliseconds) to wait before retrying job end notification. Cluster administrators can set this to limit how long the Application Master waits before exiting. Must be marked as final to prevent users from overriding this. yarn.app.mapreduce.am.env User added environment variables for the MR App Master processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit tasktracker's B env variable. yarn.app.mapreduce.am.admin.user.env Environment variables for the MR App Master processes for admin purposes. These values are set first and can be overridden by the user env (yarn.app.mapreduce.am.env) Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit app master's B env variable. yarn.app.mapreduce.am.command-opts -Xmx1024m Java opts for the MR App Master processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. yarn.app.mapreduce.am.admin-command-opts Java opts for the MR App Master processes for admin purposes. It will appears before the opts set by yarn.app.mapreduce.am.command-opts and thus its options can be overridden user. Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. yarn.app.mapreduce.am.job.task.listener.thread-count 30 The number of threads used to handle RPC calls in the MR AppMaster from remote tasks yarn.app.mapreduce.am.job.client.port-range Range of ports that the MapReduce AM can use when binding. Leave blank if you want all possible ports. For example 50000-50050,50100-50200 yarn.app.mapreduce.am.job.committer.cancel-timeout 60000 The amount of time in milliseconds to wait for the output committer to cancel an operation if the job is killed yarn.app.mapreduce.am.job.committer.commit-window 10000 Defines a time window in milliseconds for output commit operations. If contact with the RM has occurred within this window then commits are allowed, otherwise the AM will not allow output commits until contact with the RM has been re-established. mapreduce.fileoutputcommitter.algorithm.version 1 The file output committer algorithm version valid algorithm version number: 1 or 2 default to 1, which is the original algorithm In algorithm version 1, 1. commitTask will rename directory $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/ to $joboutput/_temporary/$appAttemptID/$taskID/ 2. recoverTask will also do a rename $joboutput/_temporary/$appAttemptID/$taskID/ to $joboutput/_temporary/($appAttemptID + 1)/$taskID/ 3. commitJob will merge every task output file in $joboutput/_temporary/$appAttemptID/$taskID/ to $joboutput/, then it will delete $joboutput/_temporary/ and write $joboutput/_SUCCESS It has a performance regression, which is discussed in MAPREDUCE-4815. If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. the commit is single-threaded and waits until all tasks have completed before commencing. algorithm version 2 will change the behavior of commitTask, recoverTask, and commitJob. 1. commitTask will rename all files in $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/ to $joboutput/ 2. recoverTask actually doesn't require to do anything, but for upgrade from version 1 to version 2 case, it will check if there are any files in $joboutput/_temporary/($appAttemptID - 1)/$taskID/ and rename them to $joboutput/ 3. commitJob can simply delete $joboutput/_temporary and write $joboutput/_SUCCESS This algorithm will reduce the output commit time for large jobs by having the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms 1000 The interval in ms at which the MR AppMaster should send heartbeats to the ResourceManager yarn.app.mapreduce.client-am.ipc.max-retries 3 The number of client retries to the AM - before reconnecting to the RM to fetch Application Status. yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts 3 The number of client retries on socket timeouts to the AM - before reconnecting to the RM to fetch Application Status. yarn.app.mapreduce.client.max-retries 3 The number of client retries to the RM/HS before throwing exception. This is a layer above the ipc. yarn.app.mapreduce.am.resource.mb 1536 The amount of memory the MR AppMaster needs. yarn.app.mapreduce.am.resource.cpu-vcores 1 The number of virtual CPU cores the MR AppMaster needs. yarn.app.mapreduce.am.hard-kill-timeout-ms 10000 Number of milliseconds to wait before the job client kills the application. yarn.app.mapreduce.client.job.max-retries 0 The number of retries the client will make for getJob and dependent calls. The default is 0 as this is generally only needed for non-HDFS DFS where additional, high level retries are required to avoid spurious failures during the getJob call. 30 is a good value for WASB yarn.app.mapreduce.client.job.retry-interval 2000 The delay between getJob retries in ms for retries configured with yarn.app.mapreduce.client.job.max-retries. CLASSPATH for MR applications. A comma-separated list of CLASSPATH entries. If mapreduce.application.framework is set then this must specify the appropriate classpath for that archive, and the name of the archive must be present in the classpath. If mapreduce.app-submission.cross-platform is false, platform-specific environment vairable expansion syntax would be used to construct the default CLASSPATH entries. For Linux: $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*. For Windows: %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*, %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*. If mapreduce.app-submission.cross-platform is true, platform-agnostic default CLASSPATH for MR applications would be used: {{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*, {{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/* Parameter expansion marker will be replaced by NodeManager on container launch based on the underlying OS accordingly. mapreduce.application.classpath If enabled, user can submit an application cross-platform i.e. submit an application from a Windows client to a Linux/Unix server or vice versa. mapreduce.app-submission.cross-platform false Path to the MapReduce framework archive. If set, the framework archive will automatically be distributed along with the job, and this path would normally reside in a public location in an HDFS filesystem. As with distributed cache files, this can be a URL with a fragment specifying the alias to use for the archive name. For example, hdfs:/mapred/framework/hadoop-mapreduce-2.1.1.tar.gz#mrframework would alias the localized archive as "mrframework". Note that mapreduce.application.classpath must include the appropriate classpath for the specified framework. The base name of the archive, or alias of the archive if an alias is used, must appear in the specified classpath. mapreduce.application.framework.path mapreduce.job.classloader false Whether to use a separate (isolated) classloader for user classes in the task JVM. mapreduce.job.classloader.system.classes Used to override the default definition of the system classes for the job classloader. The system classes are a comma-separated list of patterns that indicate whether to load a class from the system classpath, instead from the user-supplied JARs, when mapreduce.job.classloader is enabled. A positive pattern is defined as: 1. A single class name 'C' that matches 'C' and transitively all nested classes 'C$*' defined in C; 2. A package name ending with a '.' (e.g., "com.example.") that matches all classes from that package. A negative pattern is defined by a '-' in front of a positive pattern (e.g., "-com.example."). A class is considered a system class if and only if it matches one of the positive patterns and none of the negative ones. More formally: A class is a member of the inclusion set I if it matches one of the positive patterns. A class is a member of the exclusion set E if it matches one of the negative patterns. The set of system classes S = I \ E. mapreduce.jvm.system-properties-to-log os.name,os.version,java.home,java.runtime.version,java.vendor,java.version,java.vm.name,java.class.path,java.io.tmpdir,user.dir,user.name Comma-delimited list of system properties to log on mapreduce JVM start mapreduce.jobhistory.address 0.0.0.0:10020 MapReduce JobHistory Server IPC host:port mapreduce.jobhistory.webapp.address 0.0.0.0:19888 MapReduce JobHistory Server Web UI host:port mapreduce.jobhistory.webapp.https.address 0.0.0.0:19890 The https address the MapReduce JobHistory Server WebApp is on. mapreduce.jobhistory.keytab Location of the kerberos keytab file for the MapReduce JobHistory Server. /etc/security/keytab/jhs.service.keytab mapreduce.jobhistory.principal Kerberos principal name for the MapReduce JobHistory Server. jhs/_HOST@REALM.TLD mapreduce.jobhistory.intermediate-done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate mapreduce.jobhistory.done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done mapreduce.jobhistory.cleaner.enable true mapreduce.jobhistory.cleaner.interval-ms 86400000 How often the job history cleaner checks for files to delete, in milliseconds. Defaults to 86400000 (one day). Files are only deleted if they are older than mapreduce.jobhistory.max-age-ms. mapreduce.jobhistory.max-age-ms 604800000 Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week). mapreduce.jobhistory.client.thread-count 10 The number of threads to handle client API requests mapreduce.jobhistory.datestring.cache.size 200000 Size of the date string cache. Effects the number of directories which will be scanned to find a job. mapreduce.jobhistory.joblist.cache.size 20000 Size of the job list cache mapreduce.jobhistory.loadedjobs.cache.size 5 Size of the loaded job cache. This property is ignored if the property mapreduce.jobhistory.loadedtasks.cache.size is set to a positive value. mapreduce.jobhistory.loadedtasks.cache.size Change the job history cache limit to be set in terms of total task count. If the total number of tasks loaded exceeds this value, then the job cache will be shrunk down until it is under this limit (minimum 1 job in cache). If this value is empty or nonpositive then the cache reverts to using the property mapreduce.jobhistory.loadedjobs.cache.size as a job cache size. Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size property: 1) For every 100k of cache size, set the heap size of the Job History Server to 1.2GB. For example, mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB. 2) Make sure that the cache size is larger than the number of tasks required for the largest job run on the cluster. It might be a good idea to set the value slightly higher (say, 20%) in order to allow for job size growth. mapreduce.jobhistory.move.interval-ms 180000 Scan for history files to more from intermediate done dir to done dir at this frequency. mapreduce.jobhistory.move.thread-count 3 The number of threads used to move files. mapreduce.jobhistory.store.class The HistoryStorage class to use to cache history data. mapreduce.jobhistory.minicluster.fixed.ports false Whether to use fixed ports with the minicluster mapreduce.jobhistory.admin.address 0.0.0.0:10033 The address of the History server admin interface. mapreduce.jobhistory.admin.acl * ACL of who can be admin of the History server. mapreduce.jobhistory.recovery.enable false Enable the history server to store server state and recover server state upon startup. If enabled then mapreduce.jobhistory.recovery.store.class must be specified. mapreduce.jobhistory.recovery.store.class org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService The HistoryServerStateStoreService class to store history server state for recovery. mapreduce.jobhistory.recovery.store.fs.uri ${hadoop.tmp.dir}/mapred/history/recoverystore The URI where history server state will be stored if HistoryServerFileSystemStateStoreService is configured as the recovery storage class. mapreduce.jobhistory.recovery.store.leveldb.path ${hadoop.tmp.dir}/mapred/history/recoverystore The URI where history server state will be stored if HistoryServerLeveldbSystemStateStoreService is configured as the recovery storage class. mapreduce.jobhistory.http.policy HTTP_ONLY This configures the HTTP endpoint for JobHistoryServer web UI. The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https mapreduce.jobhistory.jobname.limit 50 Number of characters allowed for job name in Job History Server web page. File format the AM will use when generating the .jhist file. Valid values are "json" for text output and "binary" for faster parsing. mapreduce.jobhistory.jhist.format json yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size 10 The initial size of thread pool to launch containers in the app master. mapreduce.task.exit.timeout 60000 The number of milliseconds before a task will be terminated if it stays in finishing state for too long. After a task attempt completes from TaskUmbilicalProtocol's point of view, it will be transitioned to finishing state. That will give a chance for the task to exit by itself. mapreduce.task.exit.timeout.check-interval-ms 20000 The interval in milliseconds between which the MR framework checks if task attempts stay in finishing state for too long. mapreduce.task.local-fs.write-limit.bytes -1 Limit on the byte written to the local file system by each task. This limit only applies to writes that go through the Hadoop filesystem APIs within the task process (i.e.: writes that will update the local filesystem's BYTES_WRITTEN counter). It does not cover other writes such as logging, sideband writes from subprocesses (e.g.: streaming jobs), etc. Negative values disable the limit. default is -1 The list of job configuration properties whose value will be redacted. mapreduce.job.redacted-properties sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/hdfs-default.xml0000666000175100017510000033475613245514472025504 0ustar zuulzuul00000000000000 hadoop.hdfs.configuration.version 1 version of this configuration file dfs.namenode.rpc-address RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port. dfs.namenode.rpc-bind-host The actual address the RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.rpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0. dfs.namenode.servicerpc-address RPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port. If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default. dfs.namenode.servicerpc-bind-host The actual address the service RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.servicerpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0. dfs.namenode.lifeline.rpc-address NameNode RPC lifeline address. This is an optional separate RPC address that can be used to isolate health checks and liveness to protect against resource exhaustion in the main RPC handler pool. In the case of HA/Federation where multiple NameNodes exist, the name service ID is added to the name e.g. dfs.namenode.lifeline.rpc-address.ns1. The value of this property will take the form of nn-host1:rpc-port. If this property is not defined, then the NameNode will not start a lifeline RPC server. By default, the property is not defined. dfs.namenode.lifeline.rpc-bind-host The actual address the lifeline RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.lifeline.rpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0. dfs.namenode.secondary.http-address 0.0.0.0:50090 The secondary namenode http server address and port. dfs.namenode.secondary.https-address 0.0.0.0:50091 The secondary namenode HTTPS server address and port. dfs.datanode.address 0.0.0.0:50010 The datanode server address and port for data transfer. dfs.datanode.http.address 0.0.0.0:50075 The datanode http server address and port. dfs.datanode.ipc.address 0.0.0.0:50020 The datanode ipc server address and port. dfs.datanode.handler.count 10 The number of server threads for the datanode. dfs.namenode.http-address 0.0.0.0:50070 The address and the base port where the dfs namenode web ui will listen on. dfs.namenode.http-bind-host The actual adress the HTTP server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.http-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node HTTP server listen on all interfaces by setting it to 0.0.0.0. dfs.namenode.heartbeat.recheck-interval 300000 This time decides the interval to check for expired datanodes. With this value and dfs.heartbeat.interval, the interval of deciding the datanode is stale or not is also calculated. The unit of this configuration is millisecond. dfs.http.policy HTTP_ONLY Decide if HTTPS(SSL) is supported on HDFS This configures the HTTP endpoint for HDFS daemons: The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https - HTTP_AND_HTTPS : Service is provided both on http and https dfs.client.https.need-auth false Whether SSL client certificate authentication is required dfs.client.cached.conn.retry 3 The number of times the HDFS client will pull a socket from the cache. Once this number is exceeded, the client will try to create a new socket. dfs.https.server.keystore.resource ssl-server.xml Resource file from which ssl server keystore information will be extracted dfs.client.https.keystore.resource ssl-client.xml Resource file from which ssl client keystore information will be extracted dfs.datanode.https.address 0.0.0.0:50475 The datanode secure http server address and port. dfs.namenode.https-address 0.0.0.0:50470 The namenode secure http server address and port. dfs.namenode.https-bind-host The actual adress the HTTPS server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.https-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node HTTPS server listen on all interfaces by setting it to 0.0.0.0. dfs.datanode.dns.interface default The name of the Network Interface from which a data node should report its IP address. e.g. eth2. This setting may be required for some multi-homed nodes where the DataNodes are assigned multiple hostnames and it is desirable for the DataNodes to use a non-default hostname. Prefer using hadoop.security.dns.interface over dfs.datanode.dns.interface. dfs.datanode.dns.nameserver default The host name or IP address of the name server (DNS) which a DataNode should use to determine its own host name. Prefer using hadoop.security.dns.nameserver over dfs.datanode.dns.nameserver. dfs.namenode.backup.address 0.0.0.0:50100 The backup node server address and port. If the port is 0 then the server will start on a free port. dfs.namenode.backup.http-address 0.0.0.0:50105 The backup node http server address and port. If the port is 0 then the server will start on a free port. dfs.namenode.replication.considerLoad true Decide if chooseTarget considers the target's load or not dfs.namenode.replication.considerLoad.factor 2.0 The factor by which a node's load can exceed the average before being rejected for writes, only if considerLoad is true. dfs.default.chunk.view.size 32768 The number of bytes to view for a file on the browser. dfs.datanode.du.reserved 0 Reserved space in bytes per volume. Always leave this much space free for non dfs use. Specific storage type based reservation is also supported. The property can be followed with corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage. For example, reserved space for RAM_DISK storage can be configured using property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then dfs.datanode.du.reserved will be used. dfs.namenode.name.dir file://${hadoop.tmp.dir}/dfs/name Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. dfs.namenode.name.dir.restore false Set to true to enable NameNode to attempt recovering a previously failed dfs.namenode.name.dir. When enabled, a recovery of any failed directory is attempted during checkpoint. dfs.namenode.fs-limits.max-component-length 255 Defines the maximum number of bytes in UTF-8 encoding in each component of a path. A value of 0 will disable the check. dfs.namenode.fs-limits.max-directory-items 1048576 Defines the maximum number of items that a directory may contain. Cannot set the property to a value less than 1 or more than 6400000. dfs.namenode.fs-limits.min-block-size 1048576 Minimum block size in bytes, enforced by the Namenode at create time. This prevents the accidental creation of files with tiny block sizes (and thus many blocks), which can degrade performance. dfs.namenode.fs-limits.max-blocks-per-file 1048576 Maximum number of blocks per file, enforced by the Namenode on write. This prevents the creation of extremely large files which can degrade performance. dfs.namenode.edits.dir ${dfs.namenode.name.dir} Determines where on the local filesystem the DFS name node should store the transaction (edits) file. If this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. Default value is same as dfs.namenode.name.dir dfs.namenode.edits.dir.required This should be a subset of dfs.namenode.edits.dir, to ensure that the transaction (edits) file in these places is always up-to-date. dfs.namenode.shared.edits.dir A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster. dfs.namenode.edits.journal-plugin.qjournal org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager dfs.permissions.enabled true If "true", enable permission checking in HDFS. If "false", permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories. dfs.permissions.superusergroup supergroup The name of the group of super-users. The value should be a single group name. dfs.cluster.administrators ACL for the admins, this configuration is used to control who can access the default servlets in the namenode, etc. The value should be a comma separated list of users and groups. The user list comes first and is separated by a space followed by the group list, e.g. "user1,user2 group1,group2". Both users and groups are optional, so "user1", " group1", "", "user1 group1", "user1,user2 group1,group2" are all valid (note the leading space in " group1"). '*' grants access to all users and groups, e.g. '*', '* ' and ' *' are all valid. dfs.namenode.acls.enabled false Set to true to enable support for HDFS ACLs (Access Control Lists). By default, ACLs are disabled. When ACLs are disabled, the NameNode rejects all RPCs related to setting or getting ACLs. dfs.namenode.lazypersist.file.scrub.interval.sec 300 The NameNode periodically scans the namespace for LazyPersist files with missing blocks and unlinks them from the namespace. This configuration key controls the interval between successive scans. Set it to a negative value to disable this behavior. dfs.block.access.token.enable false If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanodes. dfs.block.access.key.update.interval 600 Interval in minutes at which namenode updates its access keys. dfs.block.access.token.lifetime 600 The lifetime of access tokens in minutes. dfs.datanode.data.dir file://${hadoop.tmp.dir}/dfs/data Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. The directories should be tagged with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS storage policies. The default storage type will be DISK if the directory does not have a storage type tagged explicitly. Directories that do not exist will be created if local filesystem permission allows. dfs.datanode.data.dir.perm 700 Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic. dfs.replication 3 Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. dfs.replication.max 512 Maximal block replication. dfs.namenode.replication.min 1 Minimal block replication. dfs.namenode.safemode.replication.min a separate minimum replication factor for calculating safe block count. This is an expert level setting. Setting this lower than the dfs.namenode.replication.min is not recommend and/or dangerous for production setups. When it's not set it takes value from dfs.namenode.replication.min dfs.blocksize 134217728 The default block size for new files, in bytes. You can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.), Or provide complete size in bytes (such as 134217728 for 128 MB). dfs.client.block.write.retries 3 The number of retries for writing blocks to the data nodes, before we signal failure to the application. dfs.client.block.write.replace-datanode-on-failure.enable true If there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. The feature is to add new datanodes to the pipeline. This is a site-wide property to enable/disable the feature. When the cluster size is extremely small, e.g. 3 nodes or less, cluster administrators may want to set the policy to NEVER in the default configuration file or disable this feature. Otherwise, users may experience an unusually high rate of pipeline failures since it is impossible to find new datanodes for replacement. See also dfs.client.block.write.replace-datanode-on-failure.policy dfs.client.block.write.replace-datanode-on-failure.policy DEFAULT This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. ALWAYS: always add a new datanode when an existing datanode is removed. NEVER: never add a new datanode. DEFAULT: Let r be the replication number. Let n be the number of existing datanodes. Add a new datanode only if r is greater than or equal to 3 and either (1) floor(r/2) is greater than or equal to n; or (2) r is greater than n and the block is hflushed/appended. dfs.client.block.write.replace-datanode-on-failure.best-effort false This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. Best effort means that the client will try to replace a failed datanode in write pipeline (provided that the policy is satisfied), however, it continues the write operation in case that the datanode replacement also fails. Suppose the datanode replacement fails. false: An exception should be thrown so that the write will fail. true : The write should be resumed with the remaining datandoes. Note that setting this property to true allows writing to a pipeline with a smaller number of datanodes. As a result, it increases the probability of data loss. dfs.blockreport.intervalMsec 21600000 Determines block reporting interval in milliseconds. dfs.blockreport.initialDelay 0 Delay for first block report in seconds. dfs.blockreport.split.threshold 1000000 If the number of blocks on the DataNode is below this threshold then it will send block reports for all Storage Directories in a single message. If the number of blocks exceeds this threshold then the DataNode will send block reports for each Storage Directory in separate messages. Set to zero to always split. dfs.namenode.max.full.block.report.leases 6 The maximum number of leases for full block reports that the NameNode will issue at any given time. This prevents the NameNode from being flooded with full block reports that use up all the RPC handler threads. This number should never be more than the number of RPC handler threads or less than 1. dfs.namenode.full.block.report.lease.length.ms 300000 The number of milliseconds that the NameNode will wait before invalidating a full block report lease. This prevents a crashed DataNode from permanently using up a full block report lease. dfs.datanode.directoryscan.interval 21600 Interval in seconds for Datanode to scan data directories and reconcile the difference between blocks in memory and on the disk. dfs.datanode.directoryscan.threads 1 How many threads should the threadpool used to compile reports for volumes in parallel have. dfs.datanode.directoryscan.throttle.limit.ms.per.sec 1000 The report compilation threads are limited to only running for a given number of milliseconds per second, as configured by the property. The limit is taken per thread, not in aggregate, e.g. setting a limit of 100ms for 4 compiler threads will result in each thread being limited to 100ms, not 25ms. Note that the throttle does not interrupt the report compiler threads, so the actual running time of the threads per second will typically be somewhat higher than the throttle limit, usually by no more than 20%. Setting this limit to 1000 disables compiler thread throttling. Only values between 1 and 1000 are valid. Setting an invalid value will result in the throttle being disbled and an error message being logged. 1000 is the default setting. dfs.heartbeat.interval 3 Determines datanode heartbeat interval in seconds. dfs.datanode.lifeline.interval.seconds Sets the interval in seconds between sending DataNode Lifeline Protocol messages from the DataNode to the NameNode. The value must be greater than the value of dfs.heartbeat.interval. If this property is not defined, then the default behavior is to calculate the interval as 3x the value of dfs.heartbeat.interval. Note that normal heartbeat processing may cause the DataNode to postpone sending lifeline messages if they are not required. Under normal operations with speedy heartbeat processing, it is possible that no lifeline messages will need to be sent at all. This property has no effect if dfs.namenode.lifeline.rpc-address is not defined. dfs.namenode.handler.count 10 The number of Namenode RPC server threads that listen to requests from clients. If dfs.namenode.servicerpc-address is not configured then Namenode RPC server threads listen to requests from all nodes. dfs.namenode.service.handler.count 10 The number of Namenode RPC server threads that listen to requests from DataNodes and from all other non-client nodes. dfs.namenode.service.handler.count will be valid only if dfs.namenode.servicerpc-address is configured. dfs.namenode.lifeline.handler.ratio 0.10 A ratio applied to the value of dfs.namenode.handler.count, which then provides the number of RPC server threads the NameNode runs for handling the lifeline RPC server. For example, if dfs.namenode.handler.count is 100, and dfs.namenode.lifeline.handler.factor is 0.10, then the NameNode starts 100 * 0.10 = 10 threads for handling the lifeline RPC server. It is common to tune the value of dfs.namenode.handler.count as a function of the number of DataNodes in a cluster. Using this property allows for the lifeline RPC server handler threads to be tuned automatically without needing to touch a separate property. Lifeline message processing is lightweight, so it is expected to require many fewer threads than the main NameNode RPC server. This property is not used if dfs.namenode.lifeline.handler.count is defined, which sets an absolute thread count. This property has no effect if dfs.namenode.lifeline.rpc-address is not defined. dfs.namenode.lifeline.handler.count Sets an absolute number of RPC server threads the NameNode runs for handling the DataNode Lifeline Protocol and HA health check requests from ZKFC. If this property is defined, then it overrides the behavior of dfs.namenode.lifeline.handler.ratio. By default, it is not defined. This property has no effect if dfs.namenode.lifeline.rpc-address is not defined. dfs.namenode.safemode.threshold-pct 0.999f Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.namenode.replication.min. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent. dfs.namenode.safemode.min.datanodes 0 Specifies the number of datanodes that must be considered alive before the name node exits safemode. Values less than or equal to 0 mean not to take the number of live datanodes into account when deciding whether to remain in safe mode during startup. Values greater than the number of datanodes in the cluster will make safe mode permanent. dfs.namenode.safemode.extension 30000 Determines extension of safe mode in milliseconds after the threshold level is reached. dfs.namenode.resource.check.interval 5000 The interval in milliseconds at which the NameNode resource checker runs. The checker calculates the number of the NameNode storage volumes whose available spaces are more than dfs.namenode.resource.du.reserved, and enters safemode if the number becomes lower than the minimum value specified by dfs.namenode.resource.checked.volumes.minimum. dfs.namenode.resource.du.reserved 104857600 The amount of space to reserve/require for a NameNode storage directory in bytes. The default is 100MB. dfs.namenode.resource.checked.volumes A list of local directories for the NameNode resource checker to check in addition to the local edits directories. dfs.namenode.resource.checked.volumes.minimum 1 The minimum number of redundant NameNode storage volumes required. dfs.datanode.balance.bandwidthPerSec 10m Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. You can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa)to specify the size (such as 128k, 512m, 1g, etc.). Or provide complete size in bytes (such as 134217728 for 128 MB). dfs.mover.max-no-move-interval 60000 If this specified amount of time has elapsed and no block has been moved out of a source DataNode, on more effort will be made to move blocks out of this DataNode in the current Mover iteration. dfs.hosts Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted. dfs.hosts.exclude Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded. dfs.namenode.max.objects 0 The maximum number of files, directories and blocks dfs supports. A value of zero indicates no limit to the number of objects that dfs supports. dfs.namenode.datanode.registration.ip-hostname-check true If true (the default), then the namenode requires that a connecting datanode's address must be resolved to a hostname. If necessary, a reverse DNS lookup is performed. All attempts to register a datanode from an unresolvable address are rejected. It is recommended that this setting be left on to prevent accidental registration of datanodes listed by hostname in the excludes file during a DNS outage. Only set this to false in environments where there is no infrastructure to support reverse DNS lookup. dfs.namenode.decommission.interval 30 Namenode periodicity in seconds to check if decommission is complete. dfs.namenode.decommission.blocks.per.interval 500000 The approximate number of blocks to process per decommission interval, as defined in dfs.namenode.decommission.interval. dfs.namenode.decommission.max.concurrent.tracked.nodes 100 The maximum number of decommission-in-progress datanodes nodes that will be tracked at one time by the namenode. Tracking a decommission-in-progress datanode consumes additional NN memory proportional to the number of blocks on the datnode. Having a conservative limit reduces the potential impact of decomissioning a large number of nodes at once. A value of 0 means no limit will be enforced. dfs.namenode.replication.interval 3 The periodicity in seconds with which the namenode computes replication work for datanodes. dfs.namenode.accesstime.precision 3600000 The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS. dfs.datanode.plugins Comma-separated list of datanode plug-ins to be activated. dfs.namenode.plugins Comma-separated list of namenode plug-ins to be activated. dfs.namenode.block-placement-policy.default.prefer-local-node true Controls how the default block placement policy places the first replica of a block. When true, it will prefer the node where the client is running. When false, it will prefer a node in the same rack as the client. Setting to false avoids situations where entire copies of large files end up on a single node, thus creating hotspots. dfs.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. dfs.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than dfs.stream-buffer-size dfs.client-write-packet-size 65536 Packet size for clients to write dfs.client.write.exclude.nodes.cache.expiry.interval.millis 600000 The maximum period to keep a DN in the excluded nodes list at a client. After this period, in milliseconds, the previously excluded node(s) will be removed automatically from the cache and will be considered good for block allocations again. Useful to lower or raise in situations where you keep a file open for very long periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance restarts. Defaults to 10 minutes. dfs.namenode.checkpoint.dir file://${hadoop.tmp.dir}/dfs/namesecondary Determines where on the local filesystem the DFS secondary name node should store the temporary images to merge. If this is a comma-delimited list of directories then the image is replicated in all of the directories for redundancy. dfs.namenode.checkpoint.edits.dir ${dfs.namenode.checkpoint.dir} Determines where on the local filesystem the DFS secondary name node should store the temporary edits to merge. If this is a comma-delimited list of directories then the edits is replicated in all of the directories for redundancy. Default value is same as dfs.namenode.checkpoint.dir dfs.namenode.checkpoint.period 3600 The number of seconds between two periodic checkpoints. dfs.namenode.checkpoint.txns 1000000 The Secondary NameNode or CheckpointNode will create a checkpoint of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless of whether 'dfs.namenode.checkpoint.period' has expired. dfs.namenode.checkpoint.check.period 60 The SecondaryNameNode and CheckpointNode will poll the NameNode every 'dfs.namenode.checkpoint.check.period' seconds to query the number of uncheckpointed transactions. dfs.namenode.checkpoint.max-retries 3 The SecondaryNameNode retries failed checkpointing. If the failure occurs while loading fsimage or replaying edits, the number of retries is limited by this variable. dfs.namenode.num.checkpoints.retained 2 The number of image checkpoint files (fsimage_*) that will be retained by the NameNode and Secondary NameNode in their storage directories. All edit logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained checkpoint will also be retained. dfs.namenode.num.extra.edits.retained 1000000 The number of extra transactions which should be retained beyond what is minimally necessary for a NN restart. It does not translate directly to file's age, or the number of files kept, but to the number of transactions (here "edits" means transactions). One edit file may contain several transactions (edits). During checkpoint, NameNode will identify the total number of edits to retain as extra by checking the latest checkpoint transaction value, subtracted by the value of this property. Then, it scans edits files to identify the older ones that don't include the computed range of retained transactions that are to be kept around, and purges them subsequently. The retainment can be useful for audit purposes or for an HA setup where a remote Standby Node may have been offline for some time and need to have a longer backlog of retained edits in order to start again. Typically each edit is on the order of a few hundred bytes, so the default of 1 million edits should be on the order of hundreds of MBs or low GBs. NOTE: Fewer extra edits may be retained than value specified for this setting if doing so would mean that more segments would be retained than the number configured by dfs.namenode.max.extra.edits.segments.retained. dfs.namenode.max.extra.edits.segments.retained 10000 The maximum number of extra edit log segments which should be retained beyond what is minimally necessary for a NN restart. When used in conjunction with dfs.namenode.num.extra.edits.retained, this configuration property serves to cap the number of extra edits files to a reasonable value. dfs.namenode.delegation.key.update-interval 86400000 The update interval for master key for delegation tokens in the namenode in milliseconds. dfs.namenode.delegation.token.max-lifetime 604800000 The maximum lifetime in milliseconds for which a delegation token is valid. dfs.namenode.delegation.token.renew-interval 86400000 The renewal interval for delegation token in milliseconds. dfs.datanode.failed.volumes.tolerated 0 The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown. dfs.image.compress false Should the dfs image be compressed? dfs.image.compression.codec org.apache.hadoop.io.compress.DefaultCodec If the dfs image is compressed, how should they be compressed? This has to be a codec defined in io.compression.codecs. dfs.image.transfer.timeout 60000 Socket timeout for image transfer in milliseconds. This timeout and the related dfs.image.transfer.bandwidthPerSec parameter should be configured such that normal image transfer can complete successfully. This timeout prevents client hangs when the sender fails during image transfer. This is socket timeout during image transfer. dfs.image.transfer.bandwidthPerSec 0 Maximum bandwidth used for regular image transfers (instead of bootstrapping the standby namenode), in bytes per second. This can help keep normal namenode operations responsive during checkpointing. The maximum bandwidth and timeout in dfs.image.transfer.timeout should be set such that normal image transfers can complete successfully. A default value of 0 indicates that throttling is disabled. The maximum bandwidth used for bootstrapping standby namenode is configured with dfs.image.transfer-bootstrap-standby.bandwidthPerSec. dfs.image.transfer-bootstrap-standby.bandwidthPerSec 0 Maximum bandwidth used for transferring image to bootstrap standby namenode, in bytes per second. A default value of 0 indicates that throttling is disabled. This default value should be used in most cases, to ensure timely HA operations. The maximum bandwidth used for regular image transfers is configured with dfs.image.transfer.bandwidthPerSec. dfs.image.transfer.chunksize 65536 Chunksize in bytes to upload the checkpoint. Chunked streaming is used to avoid internal buffering of contents of image file of huge size. dfs.namenode.support.allow.format true Does HDFS namenode allow itself to be formatted? You may consider setting this to false for any production cluster, to avoid any possibility of formatting a running DFS. dfs.datanode.max.transfer.threads 4096 Specifies the maximum number of threads to use for transferring data in and out of the DN. dfs.datanode.scan.period.hours 504 If this is positive, the DataNode will not scan any individual block more than once in the specified scan period. If this is negative, the block scanner is disabled. If this is set to zero, then the default value of 504 hours or 3 weeks is used. Prior versions of HDFS incorrectly documented that setting this key to zero will disable the block scanner. dfs.block.scanner.volume.bytes.per.second 1048576 If this is 0, the DataNode's block scanner will be disabled. If this is positive, this is the number of bytes per second that the DataNode's block scanner will try to scan from each volume. dfs.datanode.readahead.bytes 4194304 While reading block files, if the Hadoop native libraries are available, the datanode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader's position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the datanode will attempt to read ahead. This feature may be disabled by configuring this property to 0. If the native libraries are not available, this configuration has no effect. dfs.datanode.drop.cache.behind.reads false In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This behavior is automatically disabled for workloads which read only short sections of a block (e.g HBase random-IO workloads). This may improve performance for some workloads by freeing buffer cache space usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect. dfs.datanode.drop.cache.behind.writes false In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache space usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect. dfs.datanode.sync.behind.writes false If this configuration is enabled, the datanode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. If the Hadoop native libraries are not available, this configuration has no effect. dfs.client.failover.max.attempts 15 Expert only. The number of client failover attempts that should be made before the failover is considered failed. dfs.client.failover.sleep.base.millis 500 Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the base value used in the failover calculation. The first failover will retry immediately. The 2nd failover attempt will delay at least dfs.client.failover.sleep.base.millis milliseconds. And so on. dfs.client.failover.sleep.max.millis 15000 Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the maximum value to wait between failovers. Specifically, the time between two failover attempts will not exceed +/- 50% of dfs.client.failover.sleep.max.millis milliseconds. dfs.client.failover.connection.retries 0 Expert only. Indicates the number of retries a failover IPC client will make to establish a server connection. dfs.client.failover.connection.retries.on.timeouts 0 Expert only. The number of retry attempts a failover IPC client will make on socket timeout when establishing a server connection. dfs.client.datanode-restart.timeout 30 Expert only. The time to wait, in seconds, from reception of an datanode shutdown notification for quick restart, until declaring the datanode dead and invoking the normal recovery mechanisms. The notification is sent by a datanode when it is being shutdown using the shutdownDatanode admin command with the upgrade option. dfs.nameservices Comma-separated list of nameservices. dfs.nameservice.id The ID of this nameservice. If the nameservice ID is not configured or more than one nameservice is configured for dfs.nameservices it is determined automatically by matching the local node's address with the configured address. dfs.internal.nameservices Comma-separated list of nameservices that belong to this cluster. Datanode will report to all the nameservices in this list. By default this is set to the value of dfs.nameservices. dfs.ha.namenodes.EXAMPLENAMESERVICE The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). dfs.ha.namenode.id The ID of this namenode. If the namenode ID is not configured it is determined automatically by matching the local node's address with the configured address. dfs.ha.log-roll.period 120 How often, in seconds, the StandbyNode should ask the active to roll edit logs. Since the StandbyNode only reads from finalized log segments, the StandbyNode will only be as up-to-date as how often the logs are rolled. Note that failover triggers a log roll so the StandbyNode will be up to date before it becomes active. dfs.ha.tail-edits.period 60 How often, in seconds, the StandbyNode should check for new finalized log segments in the shared edits log. dfs.ha.automatic-failover.enabled false Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. dfs.client.use.datanode.hostname false Whether clients should use datanode hostnames when connecting to datanodes. dfs.datanode.use.datanode.hostname false Whether datanodes should use datanode hostnames when connecting to other datanodes for data transfer. dfs.client.local.interfaces A comma separated list of network interface names to use for data transfer between the client and datanodes. When creating a connection to read from or write to a datanode, the client chooses one of the specified interfaces at random and binds its socket to the IP of that interface. Individual names may be specified as either an interface name (eg "eth0"), a subinterface name (eg "eth0:0"), or an IP address (which may be specified using CIDR notation to match a range of IPs). dfs.datanode.shared.file.descriptor.paths /dev/shm,/tmp A comma-separated list of paths to use when creating file descriptors that will be shared between the DataNode and the DFSClient. Typically we use /dev/shm, so that the file descriptors will not be written to disk. Systems that don't have /dev/shm will fall back to /tmp by default. dfs.short.circuit.shared.memory.watcher.interrupt.check.ms 60000 The length of time in milliseconds that the short-circuit shared memory watcher will go between checking for java interruptions sent from other threads. This is provided mainly for unit tests. dfs.namenode.kerberos.principal The NameNode service principal. This is typically set to nn/_HOST@REALM.TLD. Each NameNode will substitute _HOST with its own fully qualified hostname at startup. The _HOST placeholder allows using the same configuration setting on both NameNodes in an HA setup. dfs.namenode.keytab.file The keytab file used by each NameNode daemon to login as its service principal. The principal name is configured with dfs.namenode.kerberos.principal. dfs.datanode.kerberos.principal The DataNode service principal. This is typically set to dn/_HOST@REALM.TLD. Each DataNode will substitute _HOST with its own fully qualified hostname at startup. The _HOST placeholder allows using the same configuration setting on all DataNodes. dfs.datanode.keytab.file The keytab file used by each DataNode daemon to login as its service principal. The principal name is configured with dfs.datanode.kerberos.principal. dfs.journalnode.kerberos.principal The JournalNode service principal. This is typically set to jn/_HOST@REALM.TLD. Each JournalNode will substitute _HOST with its own fully qualified hostname at startup. The _HOST placeholder allows using the same configuration setting on all JournalNodes. dfs.journalnode.keytab.file The keytab file used by each JournalNode daemon to login as its service principal. The principal name is configured with dfs.journalnode.kerberos.principal. dfs.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal} The server principal used by the NameNode for web UI SPNEGO authentication when Kerberos security is enabled. This is typically set to HTTP/_HOST@REALM.TLD The SPNEGO server principal begins with the prefix HTTP/ by convention. If the value is '*', the web server will attempt to login with every principal specified in the keytab file dfs.web.authentication.kerberos.keytab. dfs.journalnode.kerberos.internal.spnego.principal The server principal used by the JournalNode HTTP Server for SPNEGO authentication when Kerberos security is enabled. This is typically set to HTTP/_HOST@REALM.TLD. The SPNEGO server principal begins with the prefix HTTP/ by convention. If the value is '*', the web server will attempt to login with every principal specified in the keytab file dfs.web.authentication.kerberos.keytab. For most deployments this can be set to ${dfs.web.authentication.kerberos.principal} i.e use the value of dfs.web.authentication.kerberos.principal. dfs.secondary.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal} The server principal used by the Secondary NameNode for web UI SPNEGO authentication when Kerberos security is enabled. Like all other Secondary NameNode settings, it is ignored in an HA setup. If the value is '*', the web server will attempt to login with every principal specified in the keytab file dfs.web.authentication.kerberos.keytab. dfs.web.authentication.kerberos.principal The server principal used by the NameNode for WebHDFS SPNEGO authentication. Required when WebHDFS and security are enabled. In most secure clusters this setting is also used to specify the values for dfs.namenode.kerberos.internal.spnego.principal and dfs.journalnode.kerberos.internal.spnego.principal. dfs.web.authentication.kerberos.keytab The keytab file for the principal corresponding to dfs.web.authentication.kerberos.principal. dfs.namenode.kerberos.principal.pattern * A client-side RegEx that can be configured to control allowed realms to authenticate with (useful in cross-realm env.) dfs.namenode.avoid.read.stale.datanode false Indicate whether or not to avoid reading from "stale" datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Stale datanodes will be moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes. dfs.namenode.avoid.write.stale.datanode false Indicate whether or not to avoid writing to "stale" datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Writes will avoid using stale datanodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads. dfs.namenode.stale.datanode.interval 30000 Default time interval in milliseconds for marking a datanode as "stale", i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as "stale" by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. A stale data node is avoided during lease/block recovery. It can be conditionally avoided for reads (see dfs.namenode.avoid.read.stale.datanode) and for writes (see dfs.namenode.avoid.write.stale.datanode). dfs.namenode.write.stale.datanode.ratio 0.5f When the ratio of number stale datanodes to total datanodes marked is greater than this ratio, stop avoiding writing to stale nodes so as to prevent causing hotspots. dfs.namenode.invalidate.work.pct.per.iteration 0.32f *Note*: Advanced property. Change with caution. This determines the percentage amount of block invalidations (deletes) to do over a single DN heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DN. Value should be a positive, non-zero percentage in float notation (X.Yf), with 1.0f meaning 100%. dfs.namenode.replication.work.multiplier.per.iteration 2 *Note*: Advanced property. Change with caution. This determines the total amount of block transfers to begin in parallel at a DN, for replication, when such a command list is being sent over a DN heartbeat by the NN. The actual number is obtained by multiplying this multiplier with the total number of live nodes in the cluster. The result number is the number of blocks to begin transfers immediately for, per DN heartbeat. This number can be any positive, non-zero integer. nfs.server.port 2049 Specify the port number used by Hadoop NFS. nfs.mountd.port 4242 Specify the port number used by Hadoop mount daemon. nfs.dump.dir /tmp/.hdfs-nfs This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. One needs to make sure the directory has enough space. nfs.rtmax 1048576 This is the maximum size in bytes of a READ request supported by the NFS gateway. If you change this, make sure you also update the nfs mount's rsize(add rsize= # of bytes to the mount directive). nfs.wtmax 1048576 This is the maximum size in bytes of a WRITE request supported by the NFS gateway. If you change this, make sure you also update the nfs mount's wsize(add wsize= # of bytes to the mount directive). nfs.keytab.file *Note*: Advanced property. Change with caution. This is the path to the keytab file for the hdfs-nfs gateway. This is required when the cluster is kerberized. nfs.kerberos.principal *Note*: Advanced property. Change with caution. This is the name of the kerberos principal. This is required when the cluster is kerberized.It must be of this format: nfs-gateway-user/nfs-gateway-host@kerberos-realm nfs.allow.insecure.ports true When set to false, client connections originating from unprivileged ports (those above 1023) will be rejected. This is to ensure that clients connecting to this NFS Gateway must have had root privilege on the machine where they're connecting from. dfs.webhdfs.enabled true Enable WebHDFS (REST API) in Namenodes and Datanodes. hadoop.fuse.connection.timeout 300 The minimum number of seconds that we'll cache libhdfs connection objects in fuse_dfs. Lower values will result in lower memory consumption; higher values may speed up access by avoiding the overhead of creating new connection objects. hadoop.fuse.timer.period 5 The number of seconds between cache expiry checks in fuse_dfs. Lower values will result in fuse_dfs noticing changes to Kerberos ticket caches more quickly. dfs.namenode.metrics.logger.period.seconds 600 This setting controls how frequently the NameNode logs its metrics. The logging configuration must also define one or more appenders for NameNodeMetricsLog for the metrics to be logged. NameNode metrics logging is disabled if this value is set to zero or less than zero. dfs.datanode.metrics.logger.period.seconds 600 This setting controls how frequently the DataNode logs its metrics. The logging configuration must also define one or more appenders for DataNodeMetricsLog for the metrics to be logged. DataNode metrics logging is disabled if this value is set to zero or less than zero. dfs.metrics.percentiles.intervals Comma-delimited set of integers denoting the desired rollover intervals (in seconds) for percentile latency metrics on the Namenode and Datanode. By default, percentile latency metrics are disabled. hadoop.user.group.metrics.percentiles.intervals A comma-separated list of the granularity in seconds for the metrics which describe the 50/75/90/95/99th percentile latency for group resolution in milliseconds. By default, percentile latency metrics are disabled. dfs.encrypt.data.transfer false Whether or not actual block data that is read/written from/to HDFS should be encrypted on the wire. This only needs to be set on the NN and DNs, clients will deduce this automatically. It is possible to override this setting per connection by specifying custom logic via dfs.trustedchannel.resolver.class. dfs.encrypt.data.transfer.algorithm This value may be set to either "3des" or "rc4". If nothing is set, then the configured JCE default on the system is used (usually 3DES.) It is widely believed that 3DES is more cryptographically secure, but RC4 is substantially faster. Note that if AES is supported by both the client and server then this encryption algorithm will only be used to initially transfer keys for AES. (See dfs.encrypt.data.transfer.cipher.suites.) dfs.encrypt.data.transfer.cipher.suites This value may be either undefined or AES/CTR/NoPadding. If defined, then dfs.encrypt.data.transfer uses the specified cipher suite for data encryption. If not defined, then only the algorithm specified in dfs.encrypt.data.transfer.algorithm is used. By default, the property is not defined. dfs.encrypt.data.transfer.cipher.key.bitlength 128 The key bitlength negotiated by dfsclient and datanode for encryption. This value may be set to either 128, 192 or 256. dfs.trustedchannel.resolver.class TrustedChannelResolver is used to determine whether a channel is trusted for plain data transfer. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. The default implementation returns false indicating that the channel is not trusted. dfs.data.transfer.protection A comma-separated list of SASL protection values used for secured connections to the DataNode when reading or writing block data. Possible values are authentication, integrity and privacy. authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled. If dfs.encrypt.data.transfer is set to true, then it supersedes the setting for dfs.data.transfer.protection and enforces that all connections must use a specialized encrypted SASL handshake. This property is ignored for connections to a DataNode listening on a privileged port. In this case, it is assumed that the use of a privileged port establishes sufficient trust. dfs.data.transfer.saslproperties.resolver.class SaslPropertiesResolver used to resolve the QOP used for a connection to the DataNode when reading or writing block data. If not specified, the value of hadoop.security.saslproperties.resolver.class is used as the default value. dfs.datanode.hdfs-blocks-metadata.enabled false Boolean which enables backend datanode-side support for the experimental DistributedFileSystem#getFileVBlockStorageLocations API. dfs.client.file-block-storage-locations.num-threads 10 Number of threads used for making parallel RPCs in DistributedFileSystem#getFileBlockStorageLocations(). dfs.client.file-block-storage-locations.timeout.millis 1000 Timeout (in milliseconds) for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations(). dfs.journalnode.rpc-address 0.0.0.0:8485 The JournalNode RPC server address and port. dfs.journalnode.http-address 0.0.0.0:8480 The address and port the JournalNode HTTP server listens on. If the port is 0 then the server will start on a free port. dfs.journalnode.https-address 0.0.0.0:8481 The address and port the JournalNode HTTPS server listens on. If the port is 0 then the server will start on a free port. dfs.namenode.audit.loggers default List of classes implementing audit loggers that will receive audit events. These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger. The special value "default" can be used to reference the default audit logger, which uses the configured log system. Installing custom audit loggers may affect the performance and stability of the NameNode. Refer to the custom logger's documentation for more details. dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold 10737418240 Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls how much DN volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis. dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction 0.75f Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations. dfs.namenode.edits.noeditlogchannelflush false Specifies whether to flush edit log file channel. When set, expensive FileChannel#force calls are skipped and synchronous disk writes are enabled instead by opening the edit log file with RandomAccessFile("rws") flags. This can significantly improve the performance of edit log writes on the Windows platform. Note that the behavior of the "rws" flags is platform and hardware specific and might not provide the same level of guarantees as FileChannel#force. For example, the write will skip the disk-cache on SAS and SCSI devices while it might not on SATA devices. This is an expert level setting, change with caution. dfs.client.cache.drop.behind.writes Just like dfs.datanode.drop.cache.behind.writes, this setting causes the page cache to be dropped behind HDFS writes, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect. dfs.client.cache.drop.behind.reads Just like dfs.datanode.drop.cache.behind.reads, this setting causes the page cache to be dropped behind HDFS reads, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect. dfs.client.cache.readahead When using remote reads, this setting causes the datanode to read ahead in the block file using posix_fadvise, potentially decreasing I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. When using local reads, this setting determines how much readahead we do in BlockReaderLocal. If the native libraries are not available to the DataNode, this configuration has no effect. dfs.namenode.enable.retrycache true This enables the retry cache on the namenode. Namenode tracks for non-idempotent requests the corresponding response. If a client retries the request, the response from the retry cache is sent. Such operations are tagged with annotation @AtMostOnce in namenode protocols. It is recommended that this flag be set to true. Setting it to false, will result in clients getting failure responses to retried request. This flag must be enabled in HA setup for transparent fail-overs. The entries in the cache have expiration time configurable using dfs.namenode.retrycache.expirytime.millis. dfs.namenode.retrycache.expirytime.millis 600000 The time for which retry cache entries are retained. dfs.namenode.retrycache.heap.percent 0.03f This parameter configures the heap size allocated for retry cache (excluding the response cached). This corresponds to approximately 4096 entries for every 64MB of namenode process java heap size. Assuming retry cache entry expiration time (configured using dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this enables retry cache to support 7 operations per second sustained for 10 minutes. As the heap size is increased, the operation rate linearly increases. dfs.client.mmap.enabled true If this is set to false, the client won't attempt to perform memory-mapped reads. dfs.client.mmap.cache.size 256 When zero-copy reads are used, the DFSClient keeps a cache of recently used memory mapped regions. This parameter controls the maximum number of entries that we will keep in that cache. The larger this number is, the more file descriptors we will potentially use for memory-mapped files. mmaped files also use virtual address space. You may need to increase your ulimit virtual address space limits before increasing the client mmap cache size. Note that you can still do zero-copy reads when this size is set to 0. dfs.client.mmap.cache.timeout.ms 3600000 The minimum length of time that we will keep an mmap entry in the cache between uses. If an entry is in the cache longer than this, and nobody uses it, it will be removed by a background thread. dfs.client.mmap.retry.timeout.ms 300000 The minimum amount of time that we will wait before retrying a failed mmap operation. dfs.client.short.circuit.replica.stale.threshold.ms 1800000 The maximum amount of time that we will consider a short-circuit replica to be valid, if there is no communication from the DataNode. After this time has elapsed, we will re-fetch the short-circuit replica even if it is in the cache. dfs.namenode.path.based.cache.block.map.allocation.percent 0.25 The percentage of the Java heap which we will allocate to the cached blocks map. The cached blocks map is a hash map which uses chained hashing. Smaller maps may be accessed more slowly if the number of cached blocks is large; larger maps will consume more memory. dfs.datanode.max.locked.memory 0 The amount of memory in bytes to use for caching of block replicas in memory on the datanode. The datanode's maximum locked memory soft ulimit (RLIMIT_MEMLOCK) must be set to at least this value, else the datanode will abort on startup. By default, this parameter is set to 0, which disables in-memory caching. If the native libraries are not available to the DataNode, this configuration has no effect. dfs.namenode.list.cache.directives.num.responses 100 This value controls the number of cache directives that the NameNode will send over the wire in response to a listDirectives RPC. dfs.namenode.list.cache.pools.num.responses 100 This value controls the number of cache pools that the NameNode will send over the wire in response to a listPools RPC. dfs.namenode.path.based.cache.refresh.interval.ms 30000 The amount of milliseconds between subsequent path cache rescans. Path cache rescans are when we calculate which blocks should be cached, and on what datanodes. By default, this parameter is set to 30 seconds. dfs.namenode.path.based.cache.retry.interval.ms 30000 When the NameNode needs to uncache something that is cached, or cache something that is not cached, it must direct the DataNodes to do so by sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode heartbeat. This parameter controls how frequently the NameNode will resend these commands. dfs.datanode.fsdatasetcache.max.threads.per.volume 4 The maximum number of threads per volume to use for caching new data on the datanode. These threads consume both I/O and CPU. This can affect normal datanode operations. dfs.cachereport.intervalMsec 10000 Determines cache reporting interval in milliseconds. After this amount of time, the DataNode sends a full report of its cache state to the NameNode. The NameNode uses the cache report to update its map of cached blocks to DataNode locations. This configuration has no effect if in-memory caching has been disabled by setting dfs.datanode.max.locked.memory to 0 (which is the default). If the native libraries are not available to the DataNode, this configuration has no effect. dfs.namenode.edit.log.autoroll.multiplier.threshold 2.0 Determines when an active namenode will roll its own edit log. The actual threshold (in number of edits) is determined by multiplying this value by dfs.namenode.checkpoint.txns. This prevents extremely large edit files from accumulating on the active namenode, which can cause timeouts during namenode startup and pose an administrative hassle. This behavior is intended as a failsafe for when the standby or secondary namenode fail to roll the edit log by the normal checkpoint threshold. dfs.namenode.edit.log.autoroll.check.interval.ms 300000 How often an active namenode will check if it needs to roll its edit log, in milliseconds. dfs.webhdfs.user.provider.user.pattern ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ Valid pattern for user and group names for webhdfs, it must be a valid java regex. dfs.webhdfs.socket.connect-timeout 60s Socket timeout for connecting to WebHDFS servers. This prevents a WebHDFS client from hanging if the server hostname is misconfigured, or the server does not response before the timeout expires. Value is followed by a unit specifier: ns, us, ms, s, m, h, d for nanoseconds, microseconds, milliseconds, seconds, minutes, hours, days respectively. Values should provide units, but milliseconds are assumed. dfs.webhdfs.socket.read-timeout 60s Socket timeout for reading data from WebHDFS servers. This prevents a WebHDFS client from hanging if the server stops sending data. Value is followed by a unit specifier: ns, us, ms, s, m, h, d for nanoseconds, microseconds, milliseconds, seconds, minutes, hours, days respectively. Values should provide units, but milliseconds are assumed. dfs.client.context default The name of the DFSClient context that we should use. Clients that share a context share a socket cache and short-circuit cache, among other things. You should only change this if you don't want to share with another set of threads. dfs.client.read.shortcircuit false This configuration parameter turns on short-circuit local reads. dfs.client.socket.send.buffer.size 0 Socket send buffer size for a write pipeline in DFSClient side. This may affect TCP connection throughput. If it is set to zero or negative value, no buffer size will be set explicitly, thus enable tcp auto-tuning on some system. The default value is 0. dfs.domain.socket.path Optional. This is a path to a UNIX domain socket that will be used for communication between the DataNode and local HDFS clients. If the string "_PORT" is present in this path, it will be replaced by the TCP port of the DataNode. dfs.client.read.shortcircuit.skip.checksum false If this configuration parameter is set, short-circuit local reads will skip checksums. This is normally not recommended, but it may be useful for special setups. You might consider using this if you are doing your own checksumming outside of HDFS. dfs.client.read.shortcircuit.streams.cache.size 256 The DFSClient maintains a cache of recently opened file descriptors. This parameter controls the maximum number of file descriptors in the cache. Setting this higher will use more file descriptors, but potentially provide better performance on workloads involving lots of seeks. dfs.client.read.shortcircuit.streams.cache.expiry.ms 300000 This controls the minimum amount of time file descriptors need to sit in the client cache context before they can be closed for being inactive for too long. dfs.datanode.shared.file.descriptor.paths /dev/shm,/tmp Comma separated paths to the directory on which shared memory segments are created. The client and the DataNode exchange information via this shared memory segment. It tries paths in order until creation of shared memory segment succeeds. dfs.namenode.audit.log.debug.cmdlist A comma separated list of NameNode commands that are written to the HDFS namenode audit log only if the audit log level is debug. dfs.client.use.legacy.blockreader.local false Legacy short-circuit reader implementation based on HDFS-2246 is used if this configuration parameter is true. This is for the platforms other than Linux where the new implementation based on HDFS-347 is not available. dfs.block.local-path-access.user Comma separated list of the users allowd to open block files on legacy short-circuit local read. dfs.client.domain.socket.data.traffic false This control whether we will try to pass normal data traffic over UNIX domain socket rather than over TCP socket on node-local data transfer. This is currently experimental and turned off by default. dfs.namenode.reject-unresolved-dn-topology-mapping false If the value is set to true, then namenode will reject datanode registration if the topology mapping for a datanode is not resolved and NULL is returned (script defined by net.topology.script.file.name fails to execute). Otherwise, datanode will be registered and the default rack will be assigned as the topology path. Topology paths are important for data resiliency, since they define fault domains. Thus it may be unwanted behavior to allow datanode registration with the default rack if the resolving topology failed. dfs.client.slow.io.warning.threshold.ms 30000 The threshold in milliseconds at which we will log a slow io warning in a dfsclient. By default, this parameter is set to 30000 milliseconds (30 seconds). dfs.datanode.slow.io.warning.threshold.ms 300 The threshold in milliseconds at which we will log a slow io warning in a datanode. By default, this parameter is set to 300 milliseconds. dfs.namenode.xattrs.enabled true Whether support for extended attributes is enabled on the NameNode. dfs.namenode.fs-limits.max-xattrs-per-inode 32 Maximum number of extended attributes per inode. dfs.namenode.fs-limits.max-xattr-size 16384 The maximum combined size of the name and value of an extended attribute in bytes. It should be larger than 0, and less than or equal to maximum size hard limit which is 32768. dfs.namenode.lease-recheck-interval-ms 2000 During the release of lease a lock is hold that make any operations on the namenode stuck. In order to not block them during a too long duration we stop releasing lease after this max lock limit. dfs.namenode.max-lock-hold-to-release-lease-ms 25 During the release of lease a lock is hold that make any operations on the namenode stuck. In order to not block them during a too long duration we stop releasing lease after this max lock limit. dfs.namenode.write-lock-reporting-threshold-ms 5000 When a write lock is held on the namenode for a long time, this will be logged as the lock is released. This sets how long the lock must be held for logging to occur. dfs.namenode.read-lock-reporting-threshold-ms 5000 When a read lock is held on the namenode for a long time, this will be logged as the lock is released. This sets how long the lock must be held for logging to occur. dfs.namenode.lock.detailed-metrics.enabled false If true, the namenode will keep track of how long various operations hold the Namesystem lock for and emit this as metrics. These metrics have names of the form FSN(Read|Write)LockNanosOperationName, where OperationName denotes the name of the operation that initiated the lock hold (this will be OTHER for certain uncategorized operations) and they export the hold time values in nanoseconds. dfs.namenode.fslock.fair true If this is true, the FS Namesystem lock will be used in Fair mode, which will help to prevent writer threads from being starved, but can provide lower lock throughput. See java.util.concurrent.locks.ReentrantReadWriteLock for more information on fair/non-fair locks. dfs.namenode.startup.delay.block.deletion.sec 0 The delay in seconds at which we will pause the blocks deletion after Namenode startup. By default it's disabled. In the case a directory has large number of directories and files are deleted, suggested delay is one hour to give the administrator enough time to notice large number of pending deletion blocks and take corrective action. dfs.namenode.list.encryption.zones.num.responses 100 When listing encryption zones, the maximum number of zones that will be returned in a batch. Fetching the list incrementally in batches improves namenode performance. dfs.namenode.edekcacheloader.interval.ms 1000 When KeyProvider is configured, the interval time of warming up edek cache on NN starts up / becomes active. All edeks will be loaded from KMS into provider cache. The edek cache loader will try to warm up the cache until succeed or NN leaves active state. dfs.namenode.edekcacheloader.initial.delay.ms 3000 When KeyProvider is configured, the time delayed until the first attempt to warm up edek cache on NN start up / become active. dfs.namenode.inotify.max.events.per.rpc 1000 Maximum number of events that will be sent to an inotify client in a single RPC response. The default value attempts to amortize away the overhead for this RPC while avoiding huge memory requirements for the client and NameNode (1000 events should consume no more than 1 MB.) dfs.user.home.dir.prefix /user The directory to prepend to user name to get the user's home direcotry. dfs.datanode.cache.revocation.timeout.ms 900000 When the DFSClient reads from a block file which the DataNode is caching, the DFSClient can skip verifying checksums. The DataNode will keep the block file in cache until the client is done. If the client takes an unusually long time, though, the DataNode may need to evict the block file from the cache anyway. This value controls how long the DataNode will wait for the client to release a replica that it is reading without checksums. dfs.datanode.cache.revocation.polling.ms 500 How often the DataNode should poll to see if the clients have stopped using a replica that the DataNode wants to uncache. dfs.datanode.block.id.layout.upgrade.threads 12 The number of threads to use when creating hard links from current to previous blocks during upgrade of a DataNode to block ID-based block layout (see HDFS-6482 for details on the layout). dfs.storage.policy.enabled true Allow users to change the storage policy on files and directories. dfs.namenode.legacy-oiv-image.dir Determines where to save the namespace in the old fsimage format during checkpointing by standby NameNode or SecondaryNameNode. Users can dump the contents of the old format fsimage by oiv_legacy command. If the value is not specified, old format fsimage will not be saved in checkpoint. dfs.namenode.top.enabled true Enable nntop: reporting top users on namenode dfs.namenode.top.window.num.buckets 10 Number of buckets in the rolling window implementation of nntop dfs.namenode.top.num.users 10 Number of top users returned by the top tool dfs.namenode.top.windows.minutes 1,5,25 comma separated list of nntop reporting periods in minutes dfs.webhdfs.ugi.expire.after.access 600000 How long in milliseconds after the last access the cached UGI will expire. With 0, never expire. dfs.namenode.blocks.per.postponedblocks.rescan 10000 Number of blocks to rescan for each iteration of postponedMisreplicatedBlocks. dfs.datanode.block-pinning.enabled false Whether pin blocks on favored DataNode. dfs.client.block.write.locateFollowingBlock.initial.delay.ms 400 The initial delay (unit is ms) for locateFollowingBlock, the delay time will increase exponentially(double) for each retry. dfs.ha.zkfc.nn.http.timeout.ms 20000 The HTTP connection and read timeout value (unit is ms ) when DFS ZKFC tries to get local NN thread dump after local NN becomes SERVICE_NOT_RESPONDING or SERVICE_UNHEALTHY. If it is set to zero, DFS ZKFC won't get local NN thread dump. dfs.namenode.quota.init-threads 4 The number of concurrent threads to be used in quota initialization. The speed of quota initialization also affects the namenode fail-over latency. If the size of name space is big, try increasing this. dfs.datanode.transfer.socket.send.buffer.size 0 Socket send buffer size for DataXceiver (mirroring packets to downstream in pipeline). This may affect TCP connection throughput. If it is set to zero or negative value, no buffer size will be set explicitly, thus enable tcp auto-tuning on some system. The default value is 0. dfs.datanode.transfer.socket.recv.buffer.size 0 Socket receive buffer size for DataXceiver (receiving packets from client during block writing). This may affect TCP connection throughput. If it is set to zero or negative value, no buffer size will be set explicitly, thus enable tcp auto-tuning on some system. The default value is 0. dfs.namenode.upgrade.domain.factor ${dfs.replication} This is valid only when block placement policy is set to BlockPlacementPolicyWithUpgradeDomain. It defines the number of unique upgrade domains any block's replicas should have. When the number of replicas is less or equal to this value, the policy ensures each replica has an unique upgrade domain. When the number of replicas is greater than this value, the policy ensures the number of unique domains is at least this value. dfs.datanode.bp-ready.timeout 20 The maximum wait time for datanode to be ready before failing the received request. Setting this to 0 fails requests right away if the datanode is not yet registered with the namenode. This wait time reduces initial request failures after datanode restart. dfs.webhdfs.rest-csrf.enabled false If true, then enables WebHDFS protection against cross-site request forgery (CSRF). The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. dfs.webhdfs.rest-csrf.custom-header X-XSRF-HEADER The name of a custom header that HTTP requests must send when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. dfs.webhdfs.rest-csrf.methods-to-ignore GET,OPTIONS,HEAD,TRACE A comma-separated list of HTTP methods that do not require HTTP requests to include a custom header when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. dfs.webhdfs.rest-csrf.browser-useragents-regex ^Mozilla.*,^Opera.* A comma-separated list of regular expressions used to match against an HTTP request's User-Agent header when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.reset-csrf.enabled to true. If the incoming User-Agent matches any of these regular expressions, then the request is considered to be sent by a browser, and therefore CSRF prevention is enforced. If the request's User-Agent does not match any of these regular expressions, then the request is considered to be sent by something other than a browser, such as scripted automation. In this case, CSRF is not a potential attack vector, so the prevention is not enforced. This helps achieve backwards-compatibility with existing automation that has not been updated to send the CSRF prevention header. dfs.xframe.enabled true If true, then enables protection against clickjacking by returning X_FRAME_OPTIONS header value set to SAMEORIGIN. Clickjacking protection prevents an attacker from using transparent or opaque layers to trick a user into clicking on a button or link on another page. dfs.xframe.value SAMEORIGIN This configration value allows user to specify the value for the X-FRAME-OPTIONS. The possible values for this field are DENY, SAMEORIGIN and ALLOW-FROM. Any other value will throw an exception when namenode and datanodes are starting up. dfs.http.client.retry.policy.enabled false If "true", enable the retry policy of WebHDFS client. If "false", retry policy is turned off. Enabling the retry policy can be quite useful while using WebHDFS to copy large files between clusters that could timeout, or copy files between HA clusters that could failover during the copy. dfs.http.client.retry.policy.spec 10000,6,60000,10 Specify a policy of multiple linear random retry for WebHDFS client, e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1), ..., the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on. dfs.http.client.failover.max.attempts 15 Specify the max number of failover attempts for WebHDFS client in case of network exception. dfs.http.client.retry.max.attempts 10 Specify the max number of retry attempts for WebHDFS client, if the difference between retried attempts and failovered attempts is larger than the max number of retry attempts, there will be no more retries. dfs.http.client.failover.sleep.base.millis 500 Specify the base amount of time in milliseconds upon which the exponentially increased sleep time between retries or failovers is calculated for WebHDFS client. dfs.http.client.failover.sleep.max.millis 15000 Specify the upper bound of sleep time in milliseconds between retries or failovers for WebHDFS client. dfs.balancer.keytab.enabled false Set to true to enable login using a keytab for Kerberized Hadoop. dfs.balancer.address 0.0.0.0:0 The hostname used for a keytab based Kerberos login. Keytab based login can be enabled with dfs.balancer.keytab.enabled. dfs.balancer.keytab.file The keytab file used by the Balancer to login as its service principal. The principal name is configured with dfs.balancer.kerberos.principal. Keytab based login can be enabled with dfs.balancer.keytab.enabled. dfs.balancer.kerberos.principal The Balancer principal. This is typically set to balancer/_HOST@REALM.TLD. The Balancer will substitute _HOST with its own fully qualified hostname at startup. The _HOST placeholder allows using the same configuration setting on different servers. Keytab based login can be enabled with dfs.balancer.keytab.enabled. dfs.balancer.block-move.timeout 0 Maximum amount of time in milliseconds for a block to move. If this is set greater than 0, Balancer will stop waiting for a block move completion after this time. In typical clusters, a 3 to 5 minute timeout is reasonable. If timeout happens to a large proportion of block moves, this needs to be increased. It could also be that too much work is dispatched and many nodes are constantly exceeding the bandwidth limit as a result. In that case, other balancer parameters might need to be adjusted. It is disabled (0) by default. dfs.balancer.max-no-move-interval 60000 If this specified amount of time has elapsed and no block has been moved out of a source DataNode, on more effort will be made to move blocks out of this DataNode in the current Balancer iteration. dfs.lock.suppress.warning.interval 10s Instrumentation reporting long critical sections will suppress consecutive warnings within this interval. dfs.webhdfs.use.ipc.callq true Enables routing of webhdfs calls through rpc call queue httpfs.buffer.size 4096 The size buffer to be used when creating or opening httpfs filesystem IO stream. dfs.namenode.hosts.provider.classname org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager The class that provides access for host files. org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager is used by default which loads files specified by dfs.hosts and dfs.hosts.exclude. If org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager is used, it will load the JSON file defined in dfs.hosts. To change class name, nn restart is required. "dfsadmin -refreshNodes" only refreshes the configuration files used by the class. sahara-8.0.0/sahara/plugins/vanilla/v2_8_2/resources/oozie-default.xml0000666000175100017510000035122113245514472025667 0ustar zuulzuul00000000000000 oozie.output.compression.codec gz The name of the compression codec to use. The implementation class for the codec needs to be specified through another property oozie.compression.codecs. You can specify a comma separated list of 'Codec_name'='Codec_class' for oozie.compression.codecs where codec class implements the interface org.apache.oozie.compression.CompressionCodec. If oozie.compression.codecs is not specified, gz codec implementation is used by default. oozie.external_monitoring.enable false If the oozie functional metrics needs to be exposed to the metrics-server backend, set it to true If set to true, the following properties has to be specified : oozie.metrics.server.name, oozie.metrics.host, oozie.metrics.prefix, oozie.metrics.report.interval.sec, oozie.metrics.port oozie.external_monitoring.type graphite The name of the server to which we want to send the metrics, would be graphite or ganglia. oozie.external_monitoring.address http://localhost:2020 oozie.external_monitoring.metricPrefix oozie oozie.external_monitoring.reporterIntervalSecs 60 oozie.jmx_monitoring.enable false If the oozie functional metrics needs to be exposed via JMX interface, set it to true. oozie.action.mapreduce.uber.jar.enable false If true, enables the oozie.mapreduce.uber.jar mapreduce workflow configuration property, which is used to specify an uber jar in HDFS. Submitting a workflow with an uber jar requires at least Hadoop 2.2.0 or 1.2.0. If false, workflows which specify the oozie.mapreduce.uber.jar configuration property will fail. oozie.processing.timezone UTC Oozie server timezone. Valid values are UTC and GMT(+/-)####, for example 'GMT+0530' would be India timezone. All dates parsed and genered dates by Oozie Coordinator/Bundle will be done in the specified timezone. The default value of 'UTC' should not be changed under normal circumtances. If for any reason is changed, note that GMT(+/-)#### timezones do not observe DST changes. oozie.base.url http://localhost:8080/oozie Base Oozie URL. oozie.system.id oozie-${user.name} The Oozie system ID. oozie.systemmode NORMAL System mode for Oozie at startup. oozie.delete.runtime.dir.on.shutdown true If the runtime directory should be kept after Oozie shutdowns down. oozie.services org.apache.oozie.service.SchedulerService, org.apache.oozie.service.InstrumentationService, org.apache.oozie.service.MemoryLocksService, org.apache.oozie.service.UUIDService, org.apache.oozie.service.ELService, org.apache.oozie.service.AuthorizationService, org.apache.oozie.service.UserGroupInformationService, org.apache.oozie.service.HadoopAccessorService, org.apache.oozie.service.JobsConcurrencyService, org.apache.oozie.service.URIHandlerService, org.apache.oozie.service.DagXLogInfoService, org.apache.oozie.service.SchemaService, org.apache.oozie.service.LiteWorkflowAppService, org.apache.oozie.service.JPAService, org.apache.oozie.service.StoreService, org.apache.oozie.service.SLAStoreService, org.apache.oozie.service.DBLiteWorkflowStoreService, org.apache.oozie.service.CallbackService, org.apache.oozie.service.ActionService, org.apache.oozie.service.ShareLibService, org.apache.oozie.service.CallableQueueService, org.apache.oozie.service.ActionCheckerService, org.apache.oozie.service.RecoveryService, org.apache.oozie.service.PurgeService, org.apache.oozie.service.CoordinatorEngineService, org.apache.oozie.service.BundleEngineService, org.apache.oozie.service.DagEngineService, org.apache.oozie.service.CoordMaterializeTriggerService, org.apache.oozie.service.StatusTransitService, org.apache.oozie.service.PauseTransitService, org.apache.oozie.service.GroupsService, org.apache.oozie.service.ProxyUserService, org.apache.oozie.service.XLogStreamingService, org.apache.oozie.service.JvmPauseMonitorService, org.apache.oozie.service.SparkConfigurationService All services to be created and managed by Oozie Services singleton. Class names must be separated by commas. oozie.services.ext To add/replace services defined in 'oozie.services' with custom implementations. Class names must be separated by commas. oozie.service.XLogStreamingService.buffer.len 4096 4K buffer for streaming the logs progressively oozie.service.HCatAccessorService.jmsconnections default=java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://localhost:61616;connectionFactoryNames#ConnectionFactory Specify the map of endpoints to JMS configuration properties. In general, endpoint identifies the HCatalog server URL. "default" is used if no endpoint is mentioned in the query. If some JMS property is not defined, the system will use the property defined jndi.properties. jndi.properties files is retrieved from the application classpath. Mapping rules can also be provided for mapping Hcatalog servers to corresponding JMS providers. hcat://${1}.${2}.server.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}:61616 oozie.service.JMSTopicService.topic.name default=${username} Topic options are ${username} or ${jobId} or a fixed string which can be specified as default or for a particular job type. For e.g To have a fixed string topic for workflows, coordinators and bundles, specify in the following comma-separated format: {jobtype1}={some_string1}, {jobtype2}={some_string2} where job type can be WORKFLOW, COORDINATOR or BUNDLE. e.g. Following defines topic for workflow job, workflow action, coordinator job, coordinator action, bundle job and bundle action WORKFLOW=workflow, COORDINATOR=coordinator, BUNDLE=bundle For jobs with no defined topic, default topic will be ${username} oozie.jms.producer.connection.properties java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://localhost:61616;connectionFactoryNames#ConnectionFactory oozie.service.JMSAccessorService.connectioncontext.impl org.apache.oozie.jms.DefaultConnectionContext Specifies the Connection Context implementation oozie.service.ConfigurationService.ignore.system.properties oozie.service.AuthorizationService.security.enabled Specifies "oozie.*" properties to cannot be overriden via Java system properties. Property names must be separted by commas. oozie.service.ConfigurationService.verify.available.properties true Specifies whether the available configurations check is enabled or not. oozie.service.SchedulerService.threads 10 The number of threads to be used by the SchedulerService to run deamon tasks. If maxed out, scheduled daemon tasks will be queued up and delayed until threads become available. oozie.service.AuthorizationService.authorization.enabled false Specifies whether security (user name/admin role) is enabled or not. If disabled any user can manage Oozie system and manage any job. oozie.service.AuthorizationService.default.group.as.acl false Enables old behavior where the User's default group is the job's ACL. oozie.service.InstrumentationService.logging.interval 60 Interval, in seconds, at which instrumentation should be logged by the InstrumentationService. If set to 0 it will not log instrumentation data. oozie.service.PurgeService.older.than 30 Completed workflow jobs older than this value, in days, will be purged by the PurgeService. oozie.service.PurgeService.coord.older.than 7 Completed coordinator jobs older than this value, in days, will be purged by the PurgeService. oozie.service.PurgeService.bundle.older.than 7 Completed bundle jobs older than this value, in days, will be purged by the PurgeService. oozie.service.PurgeService.purge.old.coord.action false Whether to purge completed workflows and their corresponding coordinator actions of long running coordinator jobs if the completed workflow jobs are older than the value specified in oozie.service.PurgeService.older.than. oozie.service.PurgeService.purge.limit 100 Completed Actions purge - limit each purge to this value oozie.service.PurgeService.purge.interval 3600 Interval at which the purge service will run, in seconds. oozie.service.RecoveryService.wf.actions.older.than 120 Age of the actions which are eligible to be queued for recovery, in seconds. oozie.service.RecoveryService.wf.actions.created.time.interval 7 Created time period of the actions which are eligible to be queued for recovery in days. oozie.service.RecoveryService.callable.batch.size 10 This value determines the number of callable which will be batched together to be executed by a single thread. oozie.service.RecoveryService.push.dependency.interval 200 This value determines the delay for push missing dependency command queueing in Recovery Service oozie.service.RecoveryService.interval 60 Interval at which the RecoverService will run, in seconds. oozie.service.RecoveryService.coord.older.than 600 Age of the Coordinator jobs or actions which are eligible to be queued for recovery, in seconds. oozie.service.RecoveryService.bundle.older.than 600 Age of the Bundle jobs which are eligible to be queued for recovery, in seconds. oozie.service.CallableQueueService.queue.size 10000 Max callable queue size oozie.service.CallableQueueService.threads 10 Number of threads used for executing callables oozie.service.CallableQueueService.callable.concurrency 3 Maximum concurrency for a given callable type. Each command is a callable type (submit, start, run, signal, job, jobs, suspend,resume, etc). Each action type is a callable type (Map-Reduce, Pig, SSH, FS, sub-workflow, etc). All commands that use action executors (action-start, action-end, action-kill and action-check) use the action type as the callable type. oozie.service.CallableQueueService.callable.next.eligible true If true, when a callable in the queue has already reached max concurrency, Oozie continuously find next one which has not yet reach max concurrency. oozie.service.CallableQueueService.InterruptMapMaxSize 500 Maximum Size of the Interrupt Map, the interrupt element will not be inserted in the map if exceeded the size. oozie.service.CallableQueueService.InterruptTypes kill,resume,suspend,bundle_kill,bundle_resume,bundle_suspend,coord_kill,coord_change,coord_resume,coord_suspend Getting the types of XCommands that are considered to be of Interrupt type oozie.service.CoordMaterializeTriggerService.lookup.interval 300 Coordinator Job Lookup interval.(in seconds). oozie.service.CoordMaterializeTriggerService.materialization.window 3600 Coordinator Job Lookup command materialized each job for this next "window" duration oozie.service.CoordMaterializeTriggerService.callable.batch.size 10 This value determines the number of callable which will be batched together to be executed by a single thread. oozie.service.CoordMaterializeTriggerService.materialization.system.limit 50 This value determines the number of coordinator jobs to be materialized at a given time. oozie.service.coord.normal.default.timeout 120 Default timeout for a coordinator action input check (in minutes) for normal job. -1 means infinite timeout oozie.service.coord.default.max.timeout 86400 Default maximum timeout for a coordinator action input check (in minutes). 86400= 60days oozie.service.coord.input.check.requeue.interval 60000 Command re-queue interval for coordinator data input check (in millisecond). oozie.service.coord.input.check.requeue.interval.additional.delay 0 This value (in seconds) will be added into oozie.service.coord.input.check.requeue.interval and resulting value will be the requeue interval for the actions which are waiting for a long time without any input. oozie.service.coord.push.check.requeue.interval 600000 Command re-queue interval for push dependencies (in millisecond). oozie.service.coord.default.concurrency 1 Default concurrency for a coordinator job to determine how many maximum action should be executed at the same time. -1 means infinite concurrency. oozie.service.coord.default.throttle 12 Default throttle for a coordinator job to determine how many maximum action should be in WAITING state at the same time. oozie.service.coord.materialization.throttling.factor 0.05 Determine how many maximum actions should be in WAITING state for a single job at any time. The value is calculated by this factor X the total queue size. oozie.service.coord.check.maximum.frequency true When true, Oozie will reject any coordinators with a frequency faster than 5 minutes. It is not recommended to disable this check or submit coordinators with frequencies faster than 5 minutes: doing so can cause unintended behavior and additional system stress. oozie.service.ELService.groups job-submit,workflow,wf-sla-submit,coord-job-submit-freq,coord-job-submit-nofuncs,coord-job-submit-data,coord-job-submit-instances,coord-sla-submit,coord-action-create,coord-action-create-inst,coord-sla-create,coord-action-start,coord-job-wait-timeout,bundle-submit,coord-job-submit-initial-instance List of groups for different ELServices oozie.service.ELService.constants.job-submit EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.functions.job-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.constants.job-submit EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions without having to include all the built in ones. oozie.service.ELService.ext.functions.job-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions without having to include all the built in ones. oozie.service.ELService.constants.workflow KB=org.apache.oozie.util.ELConstantsFunctions#KB, MB=org.apache.oozie.util.ELConstantsFunctions#MB, GB=org.apache.oozie.util.ELConstantsFunctions#GB, TB=org.apache.oozie.util.ELConstantsFunctions#TB, PB=org.apache.oozie.util.ELConstantsFunctions#PB, RECORDS=org.apache.oozie.action.hadoop.HadoopELFunctions#RECORDS, MAP_IN=org.apache.oozie.action.hadoop.HadoopELFunctions#MAP_IN, MAP_OUT=org.apache.oozie.action.hadoop.HadoopELFunctions#MAP_OUT, REDUCE_IN=org.apache.oozie.action.hadoop.HadoopELFunctions#REDUCE_IN, REDUCE_OUT=org.apache.oozie.action.hadoop.HadoopELFunctions#REDUCE_OUT, GROUPS=org.apache.oozie.action.hadoop.HadoopELFunctions#GROUPS EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.workflow EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.workflow firstNotNull=org.apache.oozie.util.ELConstantsFunctions#firstNotNull, concat=org.apache.oozie.util.ELConstantsFunctions#concat, replaceAll=org.apache.oozie.util.ELConstantsFunctions#replaceAll, appendAll=org.apache.oozie.util.ELConstantsFunctions#appendAll, trim=org.apache.oozie.util.ELConstantsFunctions#trim, timestamp=org.apache.oozie.util.ELConstantsFunctions#timestamp, urlEncode=org.apache.oozie.util.ELConstantsFunctions#urlEncode, toJsonStr=org.apache.oozie.util.ELConstantsFunctions#toJsonStr, toPropertiesStr=org.apache.oozie.util.ELConstantsFunctions#toPropertiesStr, toConfigurationStr=org.apache.oozie.util.ELConstantsFunctions#toConfigurationStr, wf:id=org.apache.oozie.DagELFunctions#wf_id, wf:name=org.apache.oozie.DagELFunctions#wf_name, wf:appPath=org.apache.oozie.DagELFunctions#wf_appPath, wf:conf=org.apache.oozie.DagELFunctions#wf_conf, wf:user=org.apache.oozie.DagELFunctions#wf_user, wf:group=org.apache.oozie.DagELFunctions#wf_group, wf:callback=org.apache.oozie.DagELFunctions#wf_callback, wf:transition=org.apache.oozie.DagELFunctions#wf_transition, wf:lastErrorNode=org.apache.oozie.DagELFunctions#wf_lastErrorNode, wf:errorCode=org.apache.oozie.DagELFunctions#wf_errorCode, wf:errorMessage=org.apache.oozie.DagELFunctions#wf_errorMessage, wf:run=org.apache.oozie.DagELFunctions#wf_run, wf:actionData=org.apache.oozie.DagELFunctions#wf_actionData, wf:actionExternalId=org.apache.oozie.DagELFunctions#wf_actionExternalId, wf:actionTrackerUri=org.apache.oozie.DagELFunctions#wf_actionTrackerUri, wf:actionExternalStatus=org.apache.oozie.DagELFunctions#wf_actionExternalStatus, hadoop:counters=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_counters, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf, fs:exists=org.apache.oozie.action.hadoop.FsELFunctions#fs_exists, fs:isDir=org.apache.oozie.action.hadoop.FsELFunctions#fs_isDir, fs:dirSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_dirSize, fs:fileSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_fileSize, fs:blockSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_blockSize, hcat:exists=org.apache.oozie.coord.HCatELFunctions#hcat_exists EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength 100000 The maximum length of the workflow definition in bytes An error will be reported if the length exceeds the given maximum oozie.service.ELService.ext.functions.workflow EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.wf-sla-submit MINUTES=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_MINUTES, HOURS=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_HOURS, DAYS=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_DAYS EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.wf-sla-submit EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.wf-sla-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.wf-sla-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. l oozie.service.ELService.constants.coord-job-submit-freq EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-job-submit-freq EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-job-submit-freq coord:days=org.apache.oozie.coord.CoordELFunctions#ph1_coord_days, coord:months=org.apache.oozie.coord.CoordELFunctions#ph1_coord_months, coord:hours=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hours, coord:minutes=org.apache.oozie.coord.CoordELFunctions#ph1_coord_minutes, coord:endOfDays=org.apache.oozie.coord.CoordELFunctions#ph1_coord_endOfDays, coord:endOfMonths=org.apache.oozie.coord.CoordELFunctions#ph1_coord_endOfMonths, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.functions.coord-job-submit-initial-instance ${oozie.service.ELService.functions.coord-job-submit-nofuncs}, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_dateOffset, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_dateTzOffset EL functions for coord job submit initial instance, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-submit-freq EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-job-wait-timeout EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.constants.coord-job-wait-timeout EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions without having to include all the built in ones. oozie.service.ELService.functions.coord-job-wait-timeout coord:days=org.apache.oozie.coord.CoordELFunctions#ph1_coord_days, coord:months=org.apache.oozie.coord.CoordELFunctions#ph1_coord_months, coord:hours=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hours, coord:minutes=org.apache.oozie.coord.CoordELFunctions#ph1_coord_minutes, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-wait-timeout EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions without having to include all the built in ones. oozie.service.ELService.constants.coord-job-submit-nofuncs MINUTE=org.apache.oozie.coord.CoordELConstants#SUBMIT_MINUTE, HOUR=org.apache.oozie.coord.CoordELConstants#SUBMIT_HOUR, DAY=org.apache.oozie.coord.CoordELConstants#SUBMIT_DAY, MONTH=org.apache.oozie.coord.CoordELConstants#SUBMIT_MONTH, YEAR=org.apache.oozie.coord.CoordELConstants#SUBMIT_YEAR EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-job-submit-nofuncs EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-job-submit-nofuncs coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-submit-nofuncs EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-job-submit-instances EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-job-submit-instances EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-job-submit-instances coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hoursInDay_echo, coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph1_coord_daysInMonth_echo, coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_tzOffset_echo, coord:current=org.apache.oozie.coord.CoordELFunctions#ph1_coord_current_echo, coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_currentRange_echo, coord:offset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_offset_echo, coord:latest=org.apache.oozie.coord.CoordELFunctions#ph1_coord_latest_echo, coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_latestRange_echo, coord:future=org.apache.oozie.coord.CoordELFunctions#ph1_coord_future_echo, coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_futureRange_echo, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, coord:epochTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_epochTime_echo, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph1_coord_absolute_echo, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateTzOffset_echo EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-submit-instances EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-job-submit-data EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-job-submit-data EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-job-submit-data coord:dataIn=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataIn_echo, coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataOut_echo, coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_wrap, coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actualTime_echo_wrap, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateTzOffset_echo, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, coord:epochTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_epochTime_echo, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actionId_echo, coord:name=org.apache.oozie.coord.CoordELFunctions#ph1_coord_name_echo, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:databaseIn=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseIn_echo, coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseOut_echo, coord:tableIn=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableIn_echo, coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableOut_echo, coord:dataInPartitionFilter=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionFilter_echo, coord:dataInPartitionMin=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionMin_echo, coord:dataInPartitionMax=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionMax_echo, coord:dataInPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitions_echo, coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitions_echo, coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitionValue_echo, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-submit-data EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-sla-submit MINUTES=org.apache.oozie.coord.CoordELConstants#SUBMIT_MINUTES, HOURS=org.apache.oozie.coord.CoordELConstants#SUBMIT_HOURS, DAYS=org.apache.oozie.coord.CoordELConstants#SUBMIT_DAYS EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-sla-submit EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.bundle-submit bundle:conf=org.apache.oozie.bundle.BundleELFunctions#bundle_conf oozie.service.ELService.functions.coord-sla-submit coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataOut_echo, coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed, coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actualTime_echo_wrap, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateTzOffset_echo, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, coord:epochTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_epochTime_echo, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actionId_echo, coord:name=org.apache.oozie.coord.CoordELFunctions#ph1_coord_name_echo, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseOut_echo, coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableOut_echo, coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitions_echo, coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitionValue_echo, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-sla-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-action-create EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-action-create EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-action-create coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph2_coord_hoursInDay, coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph2_coord_daysInMonth, coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_tzOffset, coord:current=org.apache.oozie.coord.CoordELFunctions#ph2_coord_current, coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_currentRange, coord:offset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_offset, coord:latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latestRange_echo, coord:future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_futureRange_echo, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph2_coord_actionId, coord:name=org.apache.oozie.coord.CoordELFunctions#ph2_coord_name, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, coord:epochTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_epochTime, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_echo, coord:absoluteRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_range, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-action-create EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-action-create-inst EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-action-create-inst EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-action-create-inst coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph2_coord_hoursInDay, coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph2_coord_daysInMonth, coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_tzOffset, coord:current=org.apache.oozie.coord.CoordELFunctions#ph2_coord_current_echo, coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_currentRange_echo, coord:offset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_offset_echo, coord:latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latestRange_echo, coord:future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_futureRange_echo, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, coord:epochTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_epochTime, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_echo, coord:absoluteRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_range, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_dateOffset, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_dateTzOffset EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-action-create-inst EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-sla-create EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-sla-create MINUTES=org.apache.oozie.coord.CoordELConstants#SUBMIT_MINUTES, HOURS=org.apache.oozie.coord.CoordELConstants#SUBMIT_HOURS, DAYS=org.apache.oozie.coord.CoordELConstants#SUBMIT_DAYS EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-sla-create coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dataOut, coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_nominalTime, coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_actualTime, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_dateOffset, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_dateTzOffset, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, coord:epochTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_epochTime, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph2_coord_actionId, coord:name=org.apache.oozie.coord.CoordELFunctions#ph2_coord_name, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseOut, coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_tableOut, coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitions, coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitionValue, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-sla-create EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-action-start EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-action-start EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-action-start coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph3_coord_hoursInDay, coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph3_coord_daysInMonth, coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_tzOffset, coord:latest=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latest, coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latestRange, coord:future=org.apache.oozie.coord.CoordELFunctions#ph3_coord_future, coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph3_coord_futureRange, coord:dataIn=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dataIn, coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dataOut, coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime, coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_actualTime, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateOffset, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateTzOffset, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime, coord:epochTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_epochTime, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph3_coord_actionId, coord:name=org.apache.oozie.coord.CoordELFunctions#ph3_coord_name, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:databaseIn=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseIn, coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseOut, coord:tableIn=org.apache.oozie.coord.HCatELFunctions#ph3_coord_tableIn, coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_tableOut, coord:dataInPartitionFilter=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionFilter, coord:dataInPartitionMin=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionMin, coord:dataInPartitionMax=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionMax, coord:dataInPartitions=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitions, coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitions, coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitionValue, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-action-start EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.latest-el.use-current-time false Determine whether to use the current time to determine the latest dependency or the action creation time. This is for backward compatibility with older oozie behaviour. oozie.service.UUIDService.generator counter random : generated UUIDs will be random strings. counter: generated UUIDs generated will be a counter postfixed with the system startup time. oozie.service.DBLiteWorkflowStoreService.status.metrics.collection.interval 5 Workflow Status metrics collection interval in minutes. oozie.service.DBLiteWorkflowStoreService.status.metrics.window 3600 Workflow Status metrics collection window in seconds. Workflow status will be instrumented for the window. oozie.db.schema.name oozie Oozie DataBase Name oozie.service.JPAService.create.db.schema false Creates Oozie DB. If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP. If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. oozie.service.JPAService.validate.db.connection true Validates DB connections from the DB connection pool. If the 'oozie.service.JPAService.create.db.schema' property is set to true, this property is ignored. oozie.service.JPAService.validate.db.connection.eviction.interval 300000 Validates DB connections from the DB connection pool. When validate db connection 'TestWhileIdle' is true, the number of milliseconds to sleep between runs of the idle object evictor thread. oozie.service.JPAService.validate.db.connection.eviction.num 10 Validates DB connections from the DB connection pool. When validate db connection 'TestWhileIdle' is true, the number of objects to examine during each run of the idle object evictor thread. oozie.service.JPAService.connection.data.source org.apache.commons.dbcp.BasicDataSource DataSource to be used for connection pooling. oozie.service.JPAService.connection.properties DataSource connection properties. oozie.service.JPAService.jdbc.driver org.apache.derby.jdbc.EmbeddedDriver JDBC driver class. oozie.service.JPAService.jdbc.url jdbc:derby:${oozie.data.dir}/${oozie.db.schema.name}-db;create=true JDBC URL. oozie.service.JPAService.jdbc.username sa DB user name. oozie.service.JPAService.jdbc.password DB user password. IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value, if empty Configuration assumes it is NULL. IMPORTANT: if the StoreServicePasswordService is active, it will reset this value with the value given in the console. oozie.service.JPAService.pool.max.active.conn 10 Max number of connections. oozie.service.JPAService.openjpa.BrokerImpl non-finalizing The default OpenJPAEntityManager implementation automatically closes itself during instance finalization. This guards against accidental resource leaks that may occur if a developer fails to explicitly close EntityManagers when finished with them, but it also incurs a scalability bottleneck, since the JVM must perform synchronization during instance creation, and since the finalizer thread will have more instances to monitor. To avoid this overhead, set the openjpa.BrokerImpl configuration property to non-finalizing. To use default implementation set it to empty space. oozie.service.SchemaService.wf.schemas oozie-workflow-0.1.xsd,oozie-workflow-0.2.xsd,oozie-workflow-0.2.5.xsd,oozie-workflow-0.3.xsd,oozie-workflow-0.4.xsd, oozie-workflow-0.4.5.xsd,oozie-workflow-0.5.xsd, shell-action-0.1.xsd,shell-action-0.2.xsd,shell-action-0.3.xsd, email-action-0.1.xsd,email-action-0.2.xsd, hive-action-0.2.xsd,hive-action-0.3.xsd,hive-action-0.4.xsd,hive-action-0.5.xsd,hive-action-0.6.xsd, sqoop-action-0.2.xsd,sqoop-action-0.3.xsd,sqoop-action-0.4.xsd, ssh-action-0.1.xsd,ssh-action-0.2.xsd, distcp-action-0.1.xsd,distcp-action-0.2.xsd, oozie-sla-0.1.xsd,oozie-sla-0.2.xsd, hive2-action-0.1.xsd, hive2-action-0.2.xsd, spark-action-0.1.xsd,spark-action-0.2.xsd List of schemas for workflows (separated by commas). oozie.service.SchemaService.wf.ext.schemas List of additional schemas for workflows (separated by commas). oozie.service.SchemaService.coord.schemas oozie-coordinator-0.1.xsd,oozie-coordinator-0.2.xsd,oozie-coordinator-0.3.xsd,oozie-coordinator-0.4.xsd, oozie-coordinator-0.5.xsd,oozie-sla-0.1.xsd,oozie-sla-0.2.xsd List of schemas for coordinators (separated by commas). oozie.service.SchemaService.coord.ext.schemas List of additional schemas for coordinators (separated by commas). oozie.service.SchemaService.bundle.schemas oozie-bundle-0.1.xsd,oozie-bundle-0.2.xsd List of schemas for bundles (separated by commas). oozie.service.SchemaService.bundle.ext.schemas List of additional schemas for bundles (separated by commas). oozie.service.SchemaService.sla.schemas gms-oozie-sla-0.1.xsd,oozie-sla-0.2.xsd List of schemas for semantic validation for GMS SLA (separated by commas). oozie.service.SchemaService.sla.ext.schemas List of additional schemas for semantic validation for GMS SLA (separated by commas). oozie.service.CallbackService.base.url ${oozie.base.url}/callback Base callback URL used by ActionExecutors. oozie.service.CallbackService.early.requeue.max.retries 5 If Oozie receives a callback too early (while the action is in PREP state), it will requeue the command this many times to give the action time to transition to RUNNING. oozie.servlet.CallbackServlet.max.data.len 2048 Max size in characters for the action completion data output. oozie.external.stats.max.size -1 Max size in bytes for action stats. -1 means infinite value. oozie.JobCommand.job.console.url ${oozie.base.url}?job= Base console URL for a workflow job. oozie.service.ActionService.executor.classes org.apache.oozie.action.decision.DecisionActionExecutor, org.apache.oozie.action.hadoop.JavaActionExecutor, org.apache.oozie.action.hadoop.FsActionExecutor, org.apache.oozie.action.hadoop.MapReduceActionExecutor, org.apache.oozie.action.hadoop.PigActionExecutor, org.apache.oozie.action.hadoop.HiveActionExecutor, org.apache.oozie.action.hadoop.ShellActionExecutor, org.apache.oozie.action.hadoop.SqoopActionExecutor, org.apache.oozie.action.hadoop.DistcpActionExecutor, org.apache.oozie.action.hadoop.Hive2ActionExecutor, org.apache.oozie.action.ssh.SshActionExecutor, org.apache.oozie.action.oozie.SubWorkflowActionExecutor, org.apache.oozie.action.email.EmailActionExecutor, org.apache.oozie.action.hadoop.SparkActionExecutor List of ActionExecutors classes (separated by commas). Only action types with associated executors can be used in workflows. oozie.service.ActionService.executor.ext.classes List of ActionExecutors extension classes (separated by commas). Only action types with associated executors can be used in workflows. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ActionCheckerService.action.check.interval 60 The frequency at which the ActionCheckService will run. oozie.service.ActionCheckerService.action.check.delay 600 The time, in seconds, between an ActionCheck for the same action. oozie.service.ActionCheckerService.callable.batch.size 10 This value determines the number of actions which will be batched together to be executed by a single thread. oozie.service.StatusTransitService.statusTransit.interval 60 The frequency in seconds at which the StatusTransitService will run. oozie.service.StatusTransitService.backward.support.for.coord.status false true, if coordinator job submits using 'uri:oozie:coordinator:0.1' and wants to keep Oozie 2.x status transit. if set true, 1. SUCCEEDED state in coordinator job means materialization done. 2. No DONEWITHERROR state in coordinator job 3. No PAUSED or PREPPAUSED state in coordinator job 4. PREPSUSPENDED becomes SUSPENDED in coordinator job oozie.service.StatusTransitService.backward.support.for.states.without.error true true, if you want to keep Oozie 3.2 status transit. Change it to false for Oozie 4.x releases. if set true, No states like RUNNINGWITHERROR, SUSPENDEDWITHERROR and PAUSEDWITHERROR for coordinator and bundle oozie.service.PauseTransitService.PauseTransit.interval 60 The frequency in seconds at which the PauseTransitService will run. oozie.action.max.output.data 2048 Max size in characters for output data. oozie.action.fs.glob.max 50000 Maximum number of globbed files. oozie.action.launcher.am.restart.kill.childjobs true Multiple instances of launcher jobs can happen due to RM non-work preserving recovery on RM restart, AM recovery due to crashes or AM network connectivity loss. This could also lead to orphaned child jobs of the old AM attempts leading to conflicting runs. This kills child jobs of previous attempts using YARN application tags. oozie.action.launcher.mapreduce.job.ubertask.enable true Enables Uber Mode for the launcher job in YARN/Hadoop 2 (no effect in Hadoop 1) for all action types by default. This can be overridden on a per-action-type basis by setting oozie.action.#action-type#.launcher.mapreduce.job.ubertask.enable in oozie-site.xml (where #action-type# is the action type; for example, "pig"). And that can be overridden on a per-action basis by setting oozie.launcher.mapreduce.job.ubertask.enable in an action's configuration section in a workflow. In summary, the priority is this: 1. action's configuration section in a workflow 2. oozie.action.#action-type#.launcher.mapreduce.job.ubertask.enable in oozie-site 3. oozie.action.launcher.mapreduce.job.ubertask.enable in oozie-site oozie.action.shell.launcher.mapreduce.job.ubertask.enable false The Shell action may have issues with the $PATH environment when using Uber Mode, and so Uber Mode is disabled by default for it. See oozie.action.launcher.mapreduce.job.ubertask.enable oozie.action.spark.setup.hadoop.conf.dir false Oozie action.xml (oozie.action.conf.xml) contains all the hadoop configuration and user provided configurations. This property will allow users to copy Oozie action.xml as hadoop *-site configurations files. The advantage is, user need not to manage these files into spark sharelib. If user wants to manage the hadoop configurations themselves, it should should disable it. oozie.action.shell.setup.hadoop.conf.dir false The Shell action is commonly used to run programs that rely on HADOOP_CONF_DIR (e.g. hive, beeline, sqoop, etc). With YARN, HADOO_CONF_DIR is set to the NodeManager's copies of Hadoop's *-site.xml files, which can be problematic because (a) they are for meant for the NM, not necessarily clients, and (b) they won't have any of the configs that Oozie, or the user through Oozie, sets. When this property is set to true, The Shell action will prepare the *-site.xml files based on the correct config and set HADOOP_CONF_DIR to point to it. Setting it to false will make Oozie leave HADOOP_CONF_DIR alone. This can also be set at the Action level by putting it in the Shell Action's configuration section, which also has priorty. That all said, it's recommended to use the appropriate action type when possible. oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties true Toggle to control if a log4j.properties file should be written into the configuration directory prepared when oozie.action.shell.setup.hadoop.conf.dir is enabled. This is used to control logging behavior of log4j using commands run within the shell action script, and to ensure logging does not impact output data capture if leaked to stdout. Content of the written file is determined by the value of oozie.action.shell.setup.hadoop.conf.dir.log4j.content. oozie.action.shell.setup.hadoop.conf.dir.log4j.content log4j.rootLogger=${hadoop.root.logger} hadoop.root.logger=INFO,console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n The value to write into a log4j.properties file under the config directory created when oozie.action.shell.setup.hadoop.conf.dir and oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties properties are both enabled. The values must be properly newline separated and in format expected by Log4J. Trailing and preceding whitespaces will be trimmed when reading this property. This is used to control logging behavior of log4j using commands run within the shell action script. oozie.action.launcher.yarn.timeline-service.enabled false Enables/disables getting delegation tokens for ATS for the launcher job in YARN/Hadoop 2.6 (no effect in Hadoop 1) for all action types by default if tez-site.xml is present in distributed cache. This can be overridden on a per-action basis by setting oozie.launcher.yarn.timeline-service.enabled in an action's configuration section in a workflow. oozie.action.rootlogger.log.level INFO Logging level for root logger oozie.action.retries.max 3 The number of retries for executing an action in case of failure oozie.action.retry.interval 10 The interval between retries of an action in case of failure oozie.action.retry.policy periodic Retry policy of an action in case of failure. Possible values are periodic/exponential oozie.action.ssh.delete.remote.tmp.dir true If set to true, it will delete temporary directory at the end of execution of ssh action. oozie.action.ssh.http.command curl Command to use for callback to oozie, normally is 'curl' or 'wget'. The command must available in PATH environment variable of the USER@HOST box shell. oozie.action.ssh.http.command.post.options --data-binary @#stdout --request POST --header "content-type:text/plain" The callback command POST options. Used when the ouptut of the ssh action is captured. oozie.action.ssh.allow.user.at.host true Specifies whether the user specified by the ssh action is allowed or is to be replaced by the Job user oozie.action.subworkflow.max.depth 50 The maximum depth for subworkflows. For example, if set to 3, then a workflow can start subwf1, which can start subwf2, which can start subwf3; but if subwf3 tries to start subwf4, then the action will fail. This is helpful in preventing errant workflows from starting infintely recursive subworkflows. oozie.service.HadoopAccessorService.kerberos.enabled false Indicates if Oozie is configured to use Kerberos. local.realm LOCALHOST Kerberos Realm used by Oozie and Hadoop. Using 'local.realm' to be aligned with Hadoop configuration oozie.service.HadoopAccessorService.keytab.file ${user.home}/oozie.keytab Location of the Oozie user keytab file. oozie.service.HadoopAccessorService.kerberos.principal ${user.name}/localhost@${local.realm} Kerberos principal for Oozie service. oozie.service.HadoopAccessorService.jobTracker.whitelist Whitelisted job tracker for Oozie service. oozie.service.HadoopAccessorService.nameNode.whitelist Whitelisted job tracker for Oozie service. oozie.service.HadoopAccessorService.hadoop.configurations *=hadoop-conf Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of the Hadoop service (JobTracker, YARN, HDFS). The wildcard '*' configuration is used when there is no exact match for an authority. The HADOOP_CONF_DIR contains the relevant Hadoop *-site.xml files. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute (i.e. to point to Hadoop client conf/ directories in the local filesystem. oozie.service.HadoopAccessorService.action.configurations *=action-conf Comma separated AUTHORITY=ACTION_CONF_DIR, where AUTHORITY is the HOST:PORT of the Hadoop MapReduce service (JobTracker, YARN). The wildcard '*' configuration is used when there is no exact match for an authority. The ACTION_CONF_DIR may contain ACTION.xml files where ACTION is the action type ('java', 'map-reduce', 'pig', 'hive', 'sqoop', etc.). If the ACTION.xml file exists, its properties will be used as defaults properties for the action. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute (i.e. to point to Hadoop client conf/ directories in the local filesystem. oozie.service.HadoopAccessorService.action.configurations.load.default.resources true true means that default and site xml files of hadoop (core-default, core-site, hdfs-default, hdfs-site, mapred-default, mapred-site, yarn-default, yarn-site) are parsed into actionConf on Oozie server. false means that site xml files are not loaded on server, instead loaded on launcher node. This is only done for pig and hive actions which handle loading those files automatically from the classpath on launcher task. It defaults to true. oozie.credentials.credentialclasses A list of credential class mapping for CredentialsProvider oozie.credentials.skip false This determines if Oozie should skip getting credentials from the credential providers. This can be overwritten at a job-level or action-level. oozie.actions.main.classnames distcp=org.apache.hadoop.tools.DistCp A list of class name mapping for Action classes oozie.service.WorkflowAppService.system.libpath /user/${user.name}/share/lib System library path to use for workflow applications. This path is added to workflow application if their job properties sets the property 'oozie.use.system.libpath' to true. oozie.command.default.lock.timeout 5000 Default timeout (in milliseconds) for commands for acquiring an exclusive lock on an entity. oozie.command.default.requeue.delay 10000 Default time (in milliseconds) for commands that are requeued for delayed execution. oozie.service.LiteWorkflowStoreService.user.retry.max 3 Automatic retry max count for workflow action is 3 in default. oozie.service.LiteWorkflowStoreService.user.retry.inteval 10 Automatic retry interval for workflow action is in minutes and the default value is 10 minutes. oozie.service.LiteWorkflowStoreService.user.retry.policy periodic Automatic retry policy for workflow action. Possible values are periodic or exponential, periodic being the default. oozie.service.LiteWorkflowStoreService.user.retry.error.code JA008,JA009,JA017,JA018,JA019,FS009,FS008,FS014 Automatic retry interval for workflow action is handled for these specified error code: FS009, FS008 is file exists error when using chmod in fs action. FS014 is permission error in fs action JA018 is output directory exists error in workflow map-reduce action. JA019 is error while executing distcp action. JA017 is job not exists error in action executor. JA008 is FileNotFoundException in action executor. JA009 is IOException in action executor. ALL is the any kind of error in action executor. oozie.service.LiteWorkflowStoreService.user.retry.error.code.ext Automatic retry interval for workflow action is handled for these specified extra error code: ALL is the any kind of error in action executor. oozie.service.LiteWorkflowStoreService.node.def.version _oozie_inst_v_2 NodeDef default version, _oozie_inst_v_0, _oozie_inst_v_1 or _oozie_inst_v_2 oozie.authentication.type simple Defines authentication used for Oozie HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# oozie.server.authentication.type ${oozie.authentication.type} Defines authentication used for Oozie server communicating to other Oozie server over HTTP(s). Supported values are: simple | kerberos | #AUTHENTICATOR_CLASSNAME# oozie.authentication.token.validity 36000 Indicates how long (in seconds) an authentication token is valid before it has to be renewed. oozie.authentication.cookie.domain The domain to use for the HTTP cookie that stores the authentication token. In order to authentiation to work correctly across multiple hosts the domain must be correctly set. oozie.authentication.simple.anonymous.allowed true Indicates if anonymous requests are allowed when using 'simple' authentication. oozie.authentication.kerberos.principal HTTP/localhost@${local.realm} Indicates the Kerberos principal to be used for HTTP endpoint. The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification. oozie.authentication.kerberos.keytab ${oozie.service.HadoopAccessorService.keytab.file} Location of the keytab file with the credentials for the principal. Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop. oozie.authentication.kerberos.name.rules DEFAULT The kerberos names rules is to resolve kerberos principal names, refer to Hadoop's KerberosName for more details. oozie.coord.execution.none.tolerance 1 Default time tolerance in minutes after action nominal time for an action to be skipped when execution order is "NONE" oozie.coord.actions.default.length 1000 Default number of coordinator actions to be retrieved by the info command oozie.validate.ForkJoin true If true, fork and join should be validated at wf submission time. oozie.workflow.parallel.fork.action.start true Determines how Oozie processes starting of forked actions. If true, forked actions and their job submissions are done in parallel which is best for performance. If false, they are submitted sequentially. oozie.coord.action.get.all.attributes false Setting to true is not recommended as coord job/action info will bring all columns of the action in memory. Set it true only if backward compatibility for action/job info is required. oozie.service.HadoopAccessorService.supported.filesystems hdfs,hftp,webhdfs Enlist the different filesystems supported for federation. If wildcard "*" is specified, then ALL file schemes will be allowed. oozie.service.URIHandlerService.uri.handlers org.apache.oozie.dependency.FSURIHandler Enlist the different uri handlers supported for data availability checks. oozie.notification.url.connection.timeout 10000 Defines the timeout, in milliseconds, for Oozie HTTP notification callbacks. Oozie does HTTP notifications for workflow jobs which set the 'oozie.wf.action.notification.url', 'oozie.wf.worklfow.notification.url' and/or 'oozie.coord.action.notification.url' properties in their job.properties. Refer to section '5 Oozie Notifications' in the Workflow specification for details. oozie.hadoop-2.0.2-alpha.workaround.for.distributed.cache false Due to a bug in Hadoop 2.0.2-alpha, MAPREDUCE-4820, launcher jobs fail to set the distributed cache for the action job because the local JARs are implicitly included triggering a duplicate check. This flag removes the distributed cache files for the action as they'll be included from the local JARs of the JobClient (MRApps) submitting the action job from the launcher. oozie.service.EventHandlerService.filter.app.types workflow_job, coordinator_action The app-types among workflow/coordinator/bundle job/action for which for which events system is enabled. oozie.service.EventHandlerService.event.queue org.apache.oozie.event.MemoryEventQueue The implementation for EventQueue in use by the EventHandlerService. oozie.service.EventHandlerService.event.listeners org.apache.oozie.jms.JMSJobEventListener oozie.service.EventHandlerService.queue.size 10000 Maximum number of events to be contained in the event queue. oozie.service.EventHandlerService.worker.interval 30 The default interval (seconds) at which the worker threads will be scheduled to run and process events. oozie.service.EventHandlerService.batch.size 10 The batch size for batched draining per thread from the event queue. oozie.service.EventHandlerService.worker.threads 3 Number of worker threads to be scheduled to run and process events. oozie.sla.service.SLAService.capacity 5000 Maximum number of sla records to be contained in the memory structure. oozie.sla.service.SLAService.alert.events END_MISS Default types of SLA events for being alerted of. oozie.sla.service.SLAService.calculator.impl org.apache.oozie.sla.SLACalculatorMemory The implementation for SLACalculator in use by the SLAService. oozie.sla.service.SLAService.job.event.latency 90000 Time in milliseconds to account of latency of getting the job status event to compare against and decide sla miss/met oozie.sla.service.SLAService.check.interval 30 Time interval, in seconds, at which SLA Worker will be scheduled to run oozie.sla.disable.alerts.older.than 48 Time threshold, in HOURS, for disabling SLA alerting for jobs whose nominal time is older than this. oozie.zookeeper.connection.string localhost:2181 Comma-separated values of host:port pairs of the ZooKeeper servers. oozie.zookeeper.namespace oozie The namespace to use. All of the Oozie Servers that are planning on talking to each other should have the same namespace. oozie.zookeeper.connection.timeout 180 Default ZK connection timeout (in sec). oozie.zookeeper.session.timeout 300 Default ZK session timeout (in sec). If connection is lost even after retry, then Oozie server will shutdown itself if oozie.zookeeper.server.shutdown.ontimeout is true. oozie.zookeeper.max.retries 10 Maximum number of times to retry. oozie.zookeeper.server.shutdown.ontimeout true If true, Oozie server will shutdown itself on ZK connection timeout. oozie.http.hostname localhost Oozie server host name. oozie.http.port 11000 Oozie server port. oozie.https.port 11443 Oozie ssl server port. oozie.instance.id ${oozie.http.hostname} Each Oozie server should have its own unique instance id. The default is system property =${OOZIE_HTTP_HOSTNAME}= (i.e. the hostname). oozie.service.ShareLibService.mapping.file Sharelib mapping files contains list of key=value, where key will be the sharelib name for the action and value is a comma separated list of DFS directories or jar files. Example. oozie.pig_10=hdfs:///share/lib/pig/pig-0.10.1/lib/ oozie.pig=hdfs:///share/lib/pig/pig-0.11.1/lib/ oozie.distcp=hdfs:///share/lib/hadoop-2.2.0/share/hadoop/tools/lib/hadoop-distcp-2.2.0.jar oozie.service.ShareLibService.fail.fast.on.startup false Fails server starup if sharelib initilzation fails. oozie.service.ShareLibService.purge.interval 1 How often, in days, Oozie should check for old ShareLibs and LauncherLibs to purge from HDFS. oozie.service.ShareLibService.temp.sharelib.retention.days 7 ShareLib retention time in days. oozie.action.ship.launcher.jar false Specifies whether launcher jar is shipped or not. oozie.action.jobinfo.enable false JobInfo will contain information of bundle, coordinator, workflow and actions. If enabled, hadoop job will have property(oozie.job.info) which value is multiple key/value pair separated by ",". This information can be used for analytics like how many oozie jobs are submitted for a particular period, what is the total number of failed pig jobs, etc from mapreduce job history logs and configuration. User can also add custom workflow property to jobinfo by adding property which prefix with "oozie.job.info." Eg. oozie.job.info="bundle.id=,bundle.name=,coord.name=,coord.nominal.time=,coord.name=,wf.id=, wf.name=,action.name=,action.type=,launcher=true" oozie.service.XLogStreamingService.max.log.scan.duration -1 Max log scan duration in hours. If log scan request end_date - start_date > value, then exception is thrown to reduce the scan duration. -1 indicate no limit. oozie.service.XLogStreamingService.actionlist.max.log.scan.duration -1 Max log scan duration in hours for coordinator job when list of actions are specified. If log streaming request end_date - start_date > value, then exception is thrown to reduce the scan duration. -1 indicate no limit. This setting is separate from max.log.scan.duration as we want to allow higher durations when actions are specified. oozie.service.JvmPauseMonitorService.warn-threshold.ms 10000 The JvmPauseMonitorService runs a thread that repeatedly tries to detect when the JVM pauses, which could indicate that the JVM or host machine is overloaded or other problems. This thread sleeps for 500ms; if it sleeps for significantly longer, then there is likely a problem. This property specifies the threadshold for when Oozie should log a WARN level message; there is also a counter named "jvm.pause.warn-threshold". oozie.service.JvmPauseMonitorService.info-threshold.ms 1000 The JvmPauseMonitorService runs a thread that repeatedly tries to detect when the JVM pauses, which could indicate that the JVM or host machine is overloaded or other problems. This thread sleeps for 500ms; if it sleeps for significantly longer, then there is likely a problem. This property specifies the threadshold for when Oozie should log an INFO level message; there is also a counter named "jvm.pause.info-threshold". oozie.service.ZKLocksService.locks.reaper.threshold 300 The frequency at which the ChildReaper will run. Duration should be in sec. Default is 5 min. oozie.service.ZKLocksService.locks.reaper.threads 2 Number of fixed threads used by ChildReaper to delete empty locks. oozie.service.AbandonedCoordCheckerService.check.interval 1440 Interval, in minutes, at which AbandonedCoordCheckerService should run. oozie.service.AbandonedCoordCheckerService.check.delay 60 Delay, in minutes, at which AbandonedCoordCheckerService should run. oozie.service.AbandonedCoordCheckerService.failure.limit 25 Failure limit. A job is considered to be abandoned/faulty if total number of actions in failed/timedout/suspended >= "Failure limit" and there are no succeeded action. oozie.service.AbandonedCoordCheckerService.kill.jobs false If true, AbandonedCoordCheckerService will kill abandoned coords. oozie.service.AbandonedCoordCheckerService.job.older.than 2880 In minutes, job will be considered as abandoned/faulty if job is older than this value. oozie.notification.proxy System level proxy setting for job notifications. oozie.wf.rerun.disablechild false By setting this option, workflow rerun will be disabled if parent workflow or coordinator exist and it will only rerun through parent. oozie.use.system.libpath false Default value of oozie.use.system.libpath. If user haven't specified =oozie.use.system.libpath= in the job.properties and this value is true and Oozie will include sharelib jars for workflow. oozie.service.PauseTransitService.callable.batch.size 10 This value determines the number of callable which will be batched together to be executed by a single thread. oozie.configuration.substitute.depth 20 This value determines the depth of substitution in configurations. If set -1, No limitation on substitution. oozie.service.SparkConfigurationService.spark.configurations *=spark-conf Comma separated AUTHORITY=SPARK_CONF_DIR, where AUTHORITY is the HOST:PORT of the ResourceManager of a YARN cluster. The wildcard '*' configuration is used when there is no exact match for an authority. The SPARK_CONF_DIR contains the relevant spark-defaults.conf properties file. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute. This is only used when the Spark master is set to either "yarn-client" or "yarn-cluster". oozie.service.SparkConfigurationService.spark.configurations.ignore.spark.yarn.jar true If true, Oozie will ignore the "spark.yarn.jar" property from any Spark configurations specified in oozie.service.SparkConfigurationService.spark.configurations. If false, Oozie will not ignore it. It is recommended to leave this as true because it can interfere with the jars in the Spark sharelib. oozie.email.attachment.enabled true This value determines whether to support email attachment of a file on HDFS. Set it false if there is any security concern. oozie.email.smtp.host localhost The host where the email action may find the SMTP server. oozie.email.smtp.port 25 The port to connect to for the SMTP server, for email actions. oozie.email.smtp.auth false Boolean property that toggles if authentication is to be done or not when using email actions. oozie.email.smtp.username If authentication is enabled for email actions, the username to login as (to the SMTP server). oozie.email.smtp.password If authentication is enabled for email actions, the password to login with (to the SMTP server). oozie.email.from.address oozie@localhost The from address to be used for mailing all emails done via the email action. oozie.email.smtp.socket.timeout.ms 10000 The timeout to apply over all SMTP server socket operations done during the email action. oozie.actions.default.name-node The default value to use for the <name-node> element in applicable action types. This value will be used when neither the action itself nor the global section specifies a <name-node>. As expected, it should be of the form "hdfs://HOST:PORT". oozie.actions.default.job-tracker The default value to use for the <job-tracker> element in applicable action types. This value will be used when neither the action itself nor the global section specifies a <job-tracker>. As expected, it should be of the form "HOST:PORT". sahara-8.0.0/sahara/plugins/vanilla/hadoop2/0000775000175100017510000000000013245515027020710 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/hadoop2/config.py0000666000175100017510000004301413245514472022536 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from oslo_config import cfg from oslo_log import log as logging import six from sahara import context from sahara.i18n import _ from sahara.plugins import utils from sahara.plugins.vanilla.hadoop2 import config_helper as c_helper from sahara.plugins.vanilla.hadoop2 import oozie_helper as o_helper from sahara.plugins.vanilla.hadoop2 import utils as u from sahara.plugins.vanilla import utils as vu from sahara.service.castellan import utils as key_manager from sahara.swift import swift_helper as swift from sahara.topology import topology_helper as th from sahara.utils import cluster_progress_ops as cpo from sahara.utils import configs as s_cfg from sahara.utils import files as f from sahara.utils import proxy from sahara.utils import xmlutils as x CONF = cfg.CONF LOG = logging.getLogger(__name__) HADOOP_CONF_DIR = '/opt/hadoop/etc/hadoop' OOZIE_CONF_DIR = '/opt/oozie/conf' HIVE_CONF_DIR = '/opt/hive/conf' HADOOP_USER = 'hadoop' HADOOP_GROUP = 'hadoop' PORTS_MAP = { "namenode": [50070, 9000], "secondarynamenode": [50090], "resourcemanager": [8088, 8032], "historyserver": [19888], "datanode": [50010, 50075, 50020], "nodemanager": [8042], "oozie": [11000], "hiveserver": [9999, 10000], "spark history server": [18080], "zookeeper": [2181, 2888, 3888] } def configure_cluster(pctx, cluster): LOG.debug("Configuring cluster") if (CONF.use_identity_api_v3 and CONF.use_domain_for_proxy_users and vu.get_hiveserver(cluster) and c_helper.is_swift_enabled(pctx, cluster)): cluster = proxy.create_proxy_user_for_cluster(cluster) instances = utils.get_instances(cluster) configure_instances(pctx, instances) configure_topology_data(pctx, cluster) configure_zookeeper(cluster) configure_spark(cluster) def configure_zookeeper(cluster, instances=None): zk_servers = vu.get_zk_servers(cluster) if zk_servers: zk_conf = c_helper.generate_zk_basic_config(cluster) zk_conf += _form_zk_servers_to_quorum(cluster, instances) _push_zk_configs_to_nodes(cluster, zk_conf, instances) def _form_zk_servers_to_quorum(cluster, to_delete_instances=None): quorum = [] instances = map(vu.get_instance_hostname, vu.get_zk_servers(cluster)) if to_delete_instances: delete_instances = map(vu.get_instance_hostname, to_delete_instances) reserve_instances = list(set(instances) - set(delete_instances)) # keep the original order of instances reserve_instances.sort(key=instances.index) else: reserve_instances = instances for index, instance in enumerate(reserve_instances): quorum.append("server.%s=%s:2888:3888" % (index, instance)) return '\n'.join(quorum) def _push_zk_configs_to_nodes(cluster, zk_conf, to_delete_instances=None): instances = vu.get_zk_servers(cluster) if to_delete_instances: for instance in to_delete_instances: if instance in instances: instances.remove(instance) for index, instance in enumerate(instances): with instance.remote() as r: r.write_file_to('/opt/zookeeper/conf/zoo.cfg', zk_conf, run_as_root=True) r.execute_command( 'sudo su - -c "echo %s > /var/zookeeper/myid" hadoop' % index) def configure_spark(cluster): spark_servers = vu.get_spark_history_server(cluster) if spark_servers: extra = _extract_spark_configs_to_extra(cluster) _push_spark_configs_to_node(cluster, extra) def _push_spark_configs_to_node(cluster, extra): spark_master = vu.get_spark_history_server(cluster) if spark_master: _push_spark_configs_to_existing_node(spark_master, cluster, extra) _push_cleanup_job(spark_master, extra) with spark_master.remote() as r: r.execute_command('sudo su - -c "mkdir /tmp/spark-events" hadoop') def _push_spark_configs_to_existing_node(spark_master, cluster, extra): sp_home = c_helper.get_spark_home(cluster) files = { os.path.join(sp_home, 'conf/spark-env.sh'): extra['sp_master'], os.path.join( sp_home, 'conf/spark-defaults.conf'): extra['sp_defaults'] } with spark_master.remote() as r: r.write_files_to(files, run_as_root=True) def _push_cleanup_job(sp_master, extra): with sp_master.remote() as r: if extra['job_cleanup']['valid']: r.write_file_to('/opt/hadoop/tmp-cleanup.sh', extra['job_cleanup']['script'], run_as_root=True) r.execute_command("sudo chmod 755 /opt/hadoop/tmp-cleanup.sh") cmd = 'sudo sh -c \'echo "%s" > /etc/cron.d/spark-cleanup\'' r.execute_command(cmd % extra['job_cleanup']['cron']) else: r.execute_command("sudo rm -f /opt/hadoop/tmp-cleanup.sh") r.execute_command("sudo rm -f /etc/cron.d/spark-cleanup") def _extract_spark_configs_to_extra(cluster): sp_master = utils.get_instance(cluster, "spark history server") extra = dict() config_master = '' if sp_master is not None: config_master = c_helper.generate_spark_env_configs(cluster) # Any node that might be used to run spark-submit will need # these libs for swift integration config_defaults = c_helper.generate_spark_executor_classpath(cluster) extra['job_cleanup'] = c_helper.generate_job_cleanup_config(cluster) extra['sp_master'] = config_master extra['sp_defaults'] = config_defaults return extra def configure_instances(pctx, instances): if len(instances) == 0: return cpo.add_provisioning_step( instances[0].cluster_id, _("Configure instances"), len(instances)) for instance in instances: with context.set_current_instance_id(instance.instance_id): _configure_instance(pctx, instance) @cpo.event_wrapper(True) def _configure_instance(pctx, instance): _provisioning_configs(pctx, instance) _post_configuration(pctx, instance) def _provisioning_configs(pctx, instance): xmls, env = _generate_configs(pctx, instance) _push_xml_configs(instance, xmls) _push_env_configs(instance, env) def _generate_configs(pctx, instance): hadoop_xml_confs = _get_hadoop_configs(pctx, instance) user_xml_confs, user_env_confs = _get_user_configs( pctx, instance.node_group) xml_confs = s_cfg.merge_configs(user_xml_confs, hadoop_xml_confs) env_confs = s_cfg.merge_configs(pctx['env_confs'], user_env_confs) return xml_confs, env_confs def _get_hadoop_configs(pctx, instance): cluster = instance.node_group.cluster nn_hostname = vu.get_instance_hostname(vu.get_namenode(cluster)) dirs = _get_hadoop_dirs(instance) confs = { 'Hadoop': { 'fs.defaultFS': 'hdfs://%s:9000' % nn_hostname }, 'HDFS': { 'dfs.namenode.name.dir': ','.join(dirs['hadoop_name_dirs']), 'dfs.datanode.data.dir': ','.join(dirs['hadoop_data_dirs']), 'dfs.hosts': '%s/dn-include' % HADOOP_CONF_DIR, 'dfs.hosts.exclude': '%s/dn-exclude' % HADOOP_CONF_DIR } } res_hostname = vu.get_instance_hostname(vu.get_resourcemanager(cluster)) if res_hostname: confs['YARN'] = { 'yarn.nodemanager.aux-services': 'mapreduce_shuffle', 'yarn.resourcemanager.hostname': '%s' % res_hostname, 'yarn.resourcemanager.nodes.include-path': '%s/nm-include' % ( HADOOP_CONF_DIR), 'yarn.resourcemanager.nodes.exclude-path': '%s/nm-exclude' % ( HADOOP_CONF_DIR) } confs['MapReduce'] = { 'mapreduce.framework.name': 'yarn' } hs_hostname = vu.get_instance_hostname(vu.get_historyserver(cluster)) if hs_hostname: confs['MapReduce']['mapreduce.jobhistory.address'] = ( "%s:10020" % hs_hostname) oozie = vu.get_oozie(cluster) if oozie: hadoop_cfg = { 'hadoop.proxyuser.hadoop.hosts': '*', 'hadoop.proxyuser.hadoop.groups': 'hadoop' } confs['Hadoop'].update(hadoop_cfg) oozie_cfg = o_helper.get_oozie_required_xml_configs(HADOOP_CONF_DIR) if c_helper.is_mysql_enabled(pctx, cluster): oozie_cfg.update(o_helper.get_oozie_mysql_configs(cluster)) confs['JobFlow'] = oozie_cfg if c_helper.is_swift_enabled(pctx, cluster): swift_configs = {} for config in swift.get_swift_configs(): swift_configs[config['name']] = config['value'] confs['Hadoop'].update(swift_configs) if c_helper.is_data_locality_enabled(pctx, cluster): confs['Hadoop'].update(th.TOPOLOGY_CONFIG) confs['Hadoop'].update({"topology.script.file.name": HADOOP_CONF_DIR + "/topology.sh"}) hive_hostname = vu.get_instance_hostname(vu.get_hiveserver(cluster)) if hive_hostname: hive_pass = u.get_hive_password(cluster) hive_cfg = { 'hive.warehouse.subdir.inherit.perms': True, 'javax.jdo.option.ConnectionURL': 'jdbc:derby:;databaseName=/opt/hive/metastore_db;create=true' } if c_helper.is_mysql_enabled(pctx, cluster): hive_cfg.update({ 'javax.jdo.option.ConnectionURL': 'jdbc:mysql://%s/metastore' % hive_hostname, 'javax.jdo.option.ConnectionDriverName': 'com.mysql.jdbc.Driver', 'javax.jdo.option.ConnectionUserName': 'hive', 'javax.jdo.option.ConnectionPassword': hive_pass, 'datanucleus.autoCreateSchema': 'false', 'datanucleus.fixedDatastore': 'true', 'hive.metastore.uris': 'thrift://%s:9083' % hive_hostname, }) proxy_configs = cluster.cluster_configs.get('proxy_configs') if proxy_configs and c_helper.is_swift_enabled(pctx, cluster): hive_cfg.update({ swift.HADOOP_SWIFT_USERNAME: proxy_configs['proxy_username'], swift.HADOOP_SWIFT_PASSWORD: key_manager.get_secret( proxy_configs['proxy_password']), swift.HADOOP_SWIFT_TRUST_ID: proxy_configs['proxy_trust_id'], swift.HADOOP_SWIFT_DOMAIN_NAME: CONF.proxy_user_domain_name }) confs['Hive'] = hive_cfg return confs def _get_user_configs(pctx, node_group): ng_xml_confs, ng_env_confs = _separate_configs(node_group.node_configs, pctx['env_confs']) cl_xml_confs, cl_env_confs = _separate_configs( node_group.cluster.cluster_configs, pctx['env_confs']) xml_confs = s_cfg.merge_configs(cl_xml_confs, ng_xml_confs) env_confs = s_cfg.merge_configs(cl_env_confs, ng_env_confs) return xml_confs, env_confs def _separate_configs(configs, all_env_configs): xml_configs = {} env_configs = {} for service, params in six.iteritems(configs): for param, value in six.iteritems(params): if all_env_configs.get(service, {}).get(param): if not env_configs.get(service): env_configs[service] = {} env_configs[service][param] = value else: if not xml_configs.get(service): xml_configs[service] = {} xml_configs[service][param] = value return xml_configs, env_configs def _generate_xml(configs): xml_confs = {} for service, confs in six.iteritems(configs): xml_confs[service] = x.create_hadoop_xml(confs) return xml_confs def _push_env_configs(instance, configs): nn_heap = configs['HDFS']['NameNode Heap Size'] snn_heap = configs['HDFS']['SecondaryNameNode Heap Size'] dn_heap = configs['HDFS']['DataNode Heap Size'] rm_heap = configs['YARN']['ResourceManager Heap Size'] nm_heap = configs['YARN']['NodeManager Heap Size'] hs_heap = configs['MapReduce']['JobHistoryServer Heap Size'] with instance.remote() as r: r.replace_remote_string( '%s/hadoop-env.sh' % HADOOP_CONF_DIR, 'export HADOOP_NAMENODE_OPTS=.*', 'export HADOOP_NAMENODE_OPTS="-Xmx%dm"' % nn_heap) r.replace_remote_string( '%s/hadoop-env.sh' % HADOOP_CONF_DIR, 'export HADOOP_SECONDARYNAMENODE_OPTS=.*', 'export HADOOP_SECONDARYNAMENODE_OPTS="-Xmx%dm"' % snn_heap) r.replace_remote_string( '%s/hadoop-env.sh' % HADOOP_CONF_DIR, 'export HADOOP_DATANODE_OPTS=.*', 'export HADOOP_DATANODE_OPTS="-Xmx%dm"' % dn_heap) r.replace_remote_string( '%s/yarn-env.sh' % HADOOP_CONF_DIR, '\\#export YARN_RESOURCEMANAGER_HEAPSIZE=.*', 'export YARN_RESOURCEMANAGER_HEAPSIZE=%d' % rm_heap) r.replace_remote_string( '%s/yarn-env.sh' % HADOOP_CONF_DIR, '\\#export YARN_NODEMANAGER_HEAPSIZE=.*', 'export YARN_NODEMANAGER_HEAPSIZE=%d' % nm_heap) r.replace_remote_string( '%s/mapred-env.sh' % HADOOP_CONF_DIR, 'export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=.*', 'export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=%d' % hs_heap) def _push_xml_configs(instance, configs): xmls = _generate_xml(configs) service_to_conf_map = { 'Hadoop': '%s/core-site.xml' % HADOOP_CONF_DIR, 'HDFS': '%s/hdfs-site.xml' % HADOOP_CONF_DIR, 'YARN': '%s/yarn-site.xml' % HADOOP_CONF_DIR, 'MapReduce': '%s/mapred-site.xml' % HADOOP_CONF_DIR, 'JobFlow': '%s/oozie-site.xml' % OOZIE_CONF_DIR, 'Hive': '%s/hive-site.xml' % HIVE_CONF_DIR } xml_confs = {} for service, confs in six.iteritems(xmls): if service not in service_to_conf_map.keys(): continue xml_confs[service_to_conf_map[service]] = confs _push_configs_to_instance(instance, xml_confs) def _push_configs_to_instance(instance, configs): LOG.debug("Push configs to instance {instance}".format( instance=instance.instance_name)) with instance.remote() as r: for fl, data in six.iteritems(configs): r.write_file_to(fl, data, run_as_root=True) def _post_configuration(pctx, instance): dirs = _get_hadoop_dirs(instance) args = { 'hadoop_user': HADOOP_USER, 'hadoop_group': HADOOP_GROUP, 'hadoop_conf_dir': HADOOP_CONF_DIR, 'oozie_conf_dir': OOZIE_CONF_DIR, 'hadoop_name_dirs': " ".join(dirs['hadoop_name_dirs']), 'hadoop_data_dirs': " ".join(dirs['hadoop_data_dirs']), 'hadoop_log_dir': dirs['hadoop_log_dir'], 'hadoop_secure_dn_log_dir': dirs['hadoop_secure_dn_log_dir'], 'yarn_log_dir': dirs['yarn_log_dir'] } post_conf_script = f.get_file_text( 'plugins/vanilla/hadoop2/resources/post_conf.template') post_conf_script = post_conf_script.format(**args) with instance.remote() as r: r.write_file_to('/tmp/post_conf.sh', post_conf_script) r.execute_command('chmod +x /tmp/post_conf.sh') r.execute_command('sudo /tmp/post_conf.sh') if c_helper.is_data_locality_enabled(pctx, instance.cluster): t_script = HADOOP_CONF_DIR + '/topology.sh' r.write_file_to(t_script, f.get_file_text( 'plugins/vanilla/hadoop2/resources/topology.sh'), run_as_root=True) r.execute_command('chmod +x ' + t_script, run_as_root=True) def _get_hadoop_dirs(instance): dirs = {} storage_paths = instance.storage_paths() dirs['hadoop_name_dirs'] = _make_hadoop_paths( storage_paths, '/hdfs/namenode') dirs['hadoop_data_dirs'] = _make_hadoop_paths( storage_paths, '/hdfs/datanode') dirs['hadoop_log_dir'] = _make_hadoop_paths( storage_paths, '/hadoop/logs')[0] dirs['hadoop_secure_dn_log_dir'] = _make_hadoop_paths( storage_paths, '/hadoop/logs/secure')[0] dirs['yarn_log_dir'] = _make_hadoop_paths( storage_paths, '/yarn/logs')[0] return dirs def _make_hadoop_paths(paths, hadoop_dir): return [path + hadoop_dir for path in paths] @cpo.event_wrapper( True, step=_("Configure topology data"), param=('cluster', 1)) def configure_topology_data(pctx, cluster): if c_helper.is_data_locality_enabled(pctx, cluster): LOG.warning("Node group awareness is not implemented in YARN yet " "so enable_hypervisor_awareness set to False explicitly") tpl_map = th.generate_topology_map(cluster, is_node_awareness=False) topology_data = "\n".join( [k + " " + v for k, v in tpl_map.items()]) + "\n" for ng in cluster.node_groups: for i in ng.instances: i.remote().write_file_to(HADOOP_CONF_DIR + "/topology.data", topology_data, run_as_root=True) def get_open_ports(node_group): ports = [] for key in PORTS_MAP: if key in node_group.node_processes: ports += PORTS_MAP[key] return ports sahara-8.0.0/sahara/plugins/vanilla/hadoop2/validation.py0000666000175100017510000001453113245514472023425 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ from sahara.plugins import exceptions as ex from sahara.plugins import utils as u from sahara.plugins.vanilla.hadoop2 import config_helper as cu from sahara.plugins.vanilla import utils as vu from sahara.utils import general as gu def validate_cluster_creating(pctx, cluster): nn_count = _get_inst_count(cluster, 'namenode') if nn_count != 1: raise ex.InvalidComponentCountException('namenode', 1, nn_count) snn_count = _get_inst_count(cluster, 'secondarynamenode') if snn_count > 1: raise ex.InvalidComponentCountException('secondarynamenode', _('0 or 1'), snn_count) rm_count = _get_inst_count(cluster, 'resourcemanager') if rm_count > 1: raise ex.InvalidComponentCountException('resourcemanager', _('0 or 1'), rm_count) hs_count = _get_inst_count(cluster, 'historyserver') if hs_count > 1: raise ex.InvalidComponentCountException('historyserver', _('0 or 1'), hs_count) nm_count = _get_inst_count(cluster, 'nodemanager') if rm_count == 0: if nm_count > 0: raise ex.RequiredServiceMissingException('resourcemanager', required_by='nodemanager') oo_count = _get_inst_count(cluster, 'oozie') dn_count = _get_inst_count(cluster, 'datanode') if oo_count > 1: raise ex.InvalidComponentCountException('oozie', _('0 or 1'), oo_count) if oo_count == 1: if dn_count < 1: raise ex.RequiredServiceMissingException('datanode', required_by='oozie') if nm_count < 1: raise ex.RequiredServiceMissingException('nodemanager', required_by='oozie') if hs_count != 1: raise ex.RequiredServiceMissingException('historyserver', required_by='oozie') spark_hist_count = _get_inst_count(cluster, 'spark history server') if spark_hist_count > 1: raise ex.InvalidComponentCountException('spark history server', _('0 or 1'), spark_hist_count) rep_factor = cu.get_config_value(pctx, 'HDFS', 'dfs.replication', cluster) if dn_count < rep_factor: raise ex.InvalidComponentCountException( 'datanode', rep_factor, dn_count, _('Number of datanodes must be ' 'not less than ' 'dfs.replication.')) hive_count = _get_inst_count(cluster, 'hiveserver') if hive_count > 1: raise ex.InvalidComponentCountException('hive', _('0 or 1'), hive_count) zk_count = _get_inst_count(cluster, 'zookeeper') if zk_count > 0 and (zk_count % 2) != 1: raise ex.InvalidComponentCountException( 'zookeeper', _('odd'), zk_count, _('Number of zookeeper nodes' 'should be in odd.')) def validate_additional_ng_scaling(cluster, additional): rm = vu.get_resourcemanager(cluster) scalable_processes = _get_scalable_processes() for ng_id in additional: ng = gu.get_by_id(cluster.node_groups, ng_id) if not set(ng.node_processes).issubset(scalable_processes): msg = _("Vanilla plugin cannot scale nodegroup with processes: %s") raise ex.NodeGroupCannotBeScaled(ng.name, msg % ' '.join(ng.node_processes)) if not rm and 'nodemanager' in ng.node_processes: msg = _("Vanilla plugin cannot scale node group with processes " "which have no master-processes run in cluster") raise ex.NodeGroupCannotBeScaled(ng.name, msg) def validate_existing_ng_scaling(pctx, cluster, existing): scalable_processes = _get_scalable_processes() dn_to_delete = 0 for ng in cluster.node_groups: if ng.id in existing: if ng.count > existing[ng.id] and "datanode" in ng.node_processes: dn_to_delete += ng.count - existing[ng.id] if not set(ng.node_processes).issubset(scalable_processes): msg = _("Vanilla plugin cannot scale nodegroup " "with processes: %s") raise ex.NodeGroupCannotBeScaled( ng.name, msg % ' '.join(ng.node_processes)) dn_amount = len(vu.get_datanodes(cluster)) rep_factor = cu.get_config_value(pctx, 'HDFS', 'dfs.replication', cluster) if dn_to_delete > 0 and dn_amount - dn_to_delete < rep_factor: msg = _("Vanilla plugin cannot shrink cluster because it would be " "not enough nodes for replicas (replication factor is %s)") raise ex.ClusterCannotBeScaled( cluster.name, msg % rep_factor) def validate_zookeeper_node_count(zk_ng, existing, additional): zk_amount = 0 for ng in zk_ng: if ng.id in existing: zk_amount += existing[ng.id] else: zk_amount += ng.count for ng_id in additional: ng = gu.get_by_id(zk_ng, ng_id) if "zookeeper" in ng.node_processes: zk_amount += ng.count if (zk_amount % 2) != 1: msg = _("Vanilla plugin cannot scale cluster because it must keep" " zookeeper service in odd.") raise ex.ClusterCannotBeScaled(zk_ng[0].cluster.name, msg) def _get_scalable_processes(): return ['datanode', 'nodemanager', 'zookeeper'] def _get_inst_count(cluster, process): return sum([ng.count for ng in u.get_node_groups(cluster, process)]) sahara-8.0.0/sahara/plugins/vanilla/hadoop2/starting_scripts.py0000666000175100017510000000444013245514472024673 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import utils from sahara.plugins.vanilla.hadoop2 import run_scripts as run from sahara.plugins.vanilla import utils as vu from sahara.utils import cluster_progress_ops as cpo def start_namenode(cluster): nn = vu.get_namenode(cluster) _start_namenode(nn) @cpo.event_wrapper( True, step=utils.start_process_event_message('NameNode')) def _start_namenode(nn): run.format_namenode(nn) run.start_hadoop_process(nn, 'namenode') def start_secondarynamenode(cluster): snn = vu.get_secondarynamenode(cluster) if snn: _start_secondarynamenode(snn) @cpo.event_wrapper( True, step=utils.start_process_event_message("SecondaryNameNodes")) def _start_secondarynamenode(snn): run.start_hadoop_process(snn, 'secondarynamenode') def start_resourcemanager(cluster): rm = vu.get_resourcemanager(cluster) if rm: _start_resourcemanager(rm) @cpo.event_wrapper( True, step=utils.start_process_event_message('ResourceManager')) def _start_resourcemanager(snn): run.start_yarn_process(snn, 'resourcemanager') def start_historyserver(cluster): hs = vu.get_historyserver(cluster) if hs: run.start_historyserver(hs) def start_oozie(pctx, cluster): oo = vu.get_oozie(cluster) if oo: run.start_oozie_process(pctx, oo) def start_hiveserver(pctx, cluster): hiveserver = vu.get_hiveserver(cluster) if hiveserver: run.start_hiveserver_process(pctx, hiveserver) def start_spark(cluster): spark = vu.get_spark_history_server(cluster) if spark: run.start_spark_history_server(spark) def start_zookeeper(cluster): zk_servers = vu.get_zk_servers(cluster) if zk_servers: run.start_zk_server(zk_servers) sahara-8.0.0/sahara/plugins/vanilla/hadoop2/keypairs.py0000666000175100017510000000627013245514472023123 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from castellan.common.objects import passphrase from castellan import key_manager from oslo_log import log as logging from sahara import conductor from sahara import context from sahara.utils import cluster as utils from sahara.utils import crypto cond = conductor.API LOG = logging.getLogger(__name__) def _provision_key(instance, keypair): def append_to(remote, file, *args, **kwargs): kwargs['run_as_root'] = True path = "/home/hadoop/.ssh/%s" % file remote.append_to_file(path, *args, **kwargs) public, private = keypair['public'], keypair['private'] folder = '/home/hadoop/.ssh' with context.set_current_instance_id(instance_id=instance.instance_id): with instance.remote() as r: r.execute_command('sudo mkdir -p %s' % folder) append_to(r, 'authorized_keys', public) append_to(r, 'id_rsa', private) append_to(r, 'id_rsa.pub', public) r.execute_command('sudo chown -R hadoop %s' % folder) r.execute_command("sudo chmod 600 %s/id_rsa" % folder) LOG.debug("Passwordless ssh enabled") def _get_secret(secret): key = key_manager.API().get(context.current(), secret) return key.get_encoded() def _store_secret(secret): key = passphrase.Passphrase(secret) password = key_manager.API().store(context.current(), key) return password def _remove_secret(secret): key_manager.API().delete(context.current(), secret) def provision_keypairs(cluster, instances=None): extra = cluster.extra.to_dict() if cluster.extra else {} # use same keypair for scaling keypair = extra.get('vanilla_keypair') if not instances: instances = utils.get_instances(cluster) else: # scaling if not keypair: # cluster created before mitaka, skipping provisioning return if not keypair: private, public = crypto.generate_key_pair() keypair = {'public': public, 'private': private} extra['vanilla_keypair'] = keypair extra['vanilla_keypair']['private'] = _store_secret( keypair['private']) cond.cluster_update(context.ctx(), cluster, {'extra': extra}) else: keypair['private'] = _get_secret(keypair['private']) with context.ThreadGroup() as tg: for instance in instances: tg.spawn( 'provision-key-%s' % instance.instance_name, _provision_key, instance, keypair) def drop_key(cluster): extra = cluster.extra.to_dict() if cluster.extra else {} keypair = extra.get('vanilla_keypair') if keypair: _remove_secret(keypair['private']) sahara-8.0.0/sahara/plugins/vanilla/hadoop2/recommendations_utils.py0000666000175100017510000000331413245514472025677 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import recommendations_utils as ru def recommend_configs(cluster, plugin_configs, scaling): yarn_configs = [ 'yarn.nodemanager.resource.memory-mb', 'yarn.scheduler.minimum-allocation-mb', 'yarn.scheduler.maximum-allocation-mb', 'yarn.nodemanager.vmem-check-enabled', ] mapred_configs = [ 'yarn.app.mapreduce.am.resource.mb', 'yarn.app.mapreduce.am.command-opts', 'mapreduce.map.memory.mb', 'mapreduce.reduce.memory.mb', 'mapreduce.map.java.opts', 'mapreduce.reduce.java.opts', 'mapreduce.task.io.sort.mb', ] configs_to_configure = { 'cluster_configs': { 'dfs.replication': ('HDFS', 'dfs.replication') }, 'node_configs': { } } for mapr in mapred_configs: configs_to_configure['node_configs'][mapr] = ("MapReduce", mapr) for yarn in yarn_configs: configs_to_configure['node_configs'][yarn] = ('YARN', yarn) provider = ru.HadoopAutoConfigsProvider( configs_to_configure, plugin_configs, cluster, scaling) provider.apply_recommended_configs() sahara-8.0.0/sahara/plugins/vanilla/hadoop2/config_helper.py0000666000175100017510000003027713245514472024104 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import provisioning as p from sahara.plugins import utils from sahara.utils import files as f from sahara.utils import types CONF = cfg.CONF CONF.import_opt("enable_data_locality", "sahara.topology.topology_helper") HIDDEN_CONFS = [ 'dfs.hosts', 'dfs.hosts.exclude', 'dfs.namenode.data.dir', 'dfs.namenode.name.dir', 'fs.default.name', 'fs.defaultFS', 'fs.swift.impl', 'hadoop.proxyuser.hadoop.groups', 'hadoop.proxyuser.hadoop.hosts', 'mapreduce.framework.name', 'mapreduce.jobhistory.address', 'mapreduce.jobhistory.done.dir', 'mapreduce.jobhistory.intermediate-done-dir', 'mapreduce.jobhistory.webapp.address', 'yarn.nodemanager.aux-services', 'yarn.resourcemanager.address', 'yarn.resourcemanager.admin.address', 'yarn.resourcemanager.hostname', 'yarn.resourcemanager.nodes.exclude-path', 'yarn.resourcemanager.nodes.include-path', 'yarn.resourcemanager.resource-tracker.address', 'yarn.resourcemanager.scheduler.address', 'yarn.resourcemanager.webapp.address' ] CLUSTER_WIDE_CONFS = [ 'dfs.blocksize', 'dfs.namenode.replication.min', 'dfs.permissions.enabled', 'dfs.replication', 'dfs.replication.max', 'io.compression.codecs', 'io.file.buffer.size', 'mapreduce.job.counters.max', 'mapreduce.map.output.compress.codec', 'mapreduce.output.fileoutputformat.compress.codec', 'mapreduce.output.fileoutputformat.compress.type', 'mapredude.map.output.compress', 'mapredude.output.fileoutputformat.compress' ] PRIORITY_1_CONFS = [ 'dfs.datanode.du.reserved', 'dfs.datanode.failed.volumes.tolerated', 'dfs.datanode.handler.count', 'dfs.datanode.max.transfer.threads', 'dfs.namenode.handler.count', 'mapred.child.java.opts', 'mapred.jobtracker.maxtasks.per.job', 'mapreduce.jobtracker.handler.count', 'mapreduce.map.java.opts', 'mapreduce.reduce.java.opts', 'mapreduce.task.io.sort.mb', 'mapreduce.tasktracker.map.tasks.maximum', 'mapreduce.tasktracker.reduce.tasks.maximum', 'yarn.nodemanager.resource.cpu-vcores', 'yarn.nodemanager.resource.memory-mb', 'yarn.scheduler.maximum-allocation-mb', 'yarn.scheduler.maximum-allocation-vcores', 'yarn.scheduler.minimum-allocation-mb', 'yarn.scheduler.minimum-allocation-vcores' ] _default_executor_classpath = ":".join( ['/opt/hadoop/share/hadoop/tools/lib/hadoop-openstack-2.7.1.jar']) SPARK_CONFS = { 'Spark': { "OPTIONS": [ { 'name': 'Executor extra classpath', 'description': 'Value for spark.executor.extraClassPath' ' in spark-defaults.conf' ' (default: %s)' % _default_executor_classpath, 'default': '%s' % _default_executor_classpath, 'priority': 2, }, { 'name': 'Spark home', 'description': 'The location of the spark installation' ' (default: /opt/spark)', 'default': '/opt/spark', 'priority': 2, }, { 'name': 'Minimum cleanup seconds', 'description': 'Job data will never be purged before this' ' amount of time elapses (default: 86400 = 1 day)', 'default': '86400', 'priority': 2, }, { 'name': 'Maximum cleanup seconds', 'description': 'Job data will always be purged after this' ' amount of time elapses (default: 1209600 = 14 days)', 'default': '1209600', 'priority': 2, }, { 'name': 'Minimum cleanup megabytes', 'description': 'No job data will be purged unless the total' ' job data exceeds this size (default: 4096 = 4GB)', 'default': '4096', 'priority': 2, }, ] } } ZOOKEEPER_CONFS = { "ZooKeeper": { "OPTIONS": [ { 'name': 'tickTime', 'description': 'The number of milliseconds of each tick', 'default': 2000, 'priority': 2, }, { 'name': 'initLimit', 'description': 'The number of ticks that the initial' ' synchronization phase can take', 'default': 10, 'priority': 2, }, { 'name': 'syncLimit', 'description': 'The number of ticks that can pass between' ' sending a request and getting an acknowledgement', 'default': 5, 'priority': 2, }, ] } } # for now we have not so many cluster-wide configs # lets consider all of them having high priority PRIORITY_1_CONFS += CLUSTER_WIDE_CONFS def init_xml_configs(xml_confs): configs = [] for service, config_lists in six.iteritems(xml_confs): for config_list in config_lists: for config in config_list: if config['name'] not in HIDDEN_CONFS: cfg = p.Config(config['name'], service, "node", is_optional=True, config_type="string", default_value=str(config['value']), description=config['description']) if cfg.default_value in ["true", "false"]: cfg.config_type = "bool" cfg.default_value = (cfg.default_value == 'true') elif types.is_int(cfg.default_value): cfg.config_type = "int" cfg.default_value = int(cfg.default_value) if config['name'] in CLUSTER_WIDE_CONFS: cfg.scope = 'cluster' if config['name'] in PRIORITY_1_CONFS: cfg.priority = 1 configs.append(cfg) return configs ENABLE_SWIFT = p.Config('Enable Swift', 'general', 'cluster', config_type="bool", priority=1, default_value=True, is_optional=False) ENABLE_MYSQL = p.Config('Enable MySQL', 'general', 'cluster', config_type="bool", priority=1, default_value=True, is_optional=True) ENABLE_DATA_LOCALITY = p.Config('Enable Data Locality', 'general', 'cluster', config_type="bool", priority=1, default_value=True, is_optional=True) DATANODES_DECOMMISSIONING_TIMEOUT = p.Config( 'DataNodes decommissioning timeout', 'general', 'cluster', config_type='int', priority=1, default_value=3600 * 4, is_optional=True, description='Timeout for datanode decommissioning operation' ' during scaling, in seconds') NODEMANAGERS_DECOMMISSIONING_TIMEOUT = p.Config( 'NodeManagers decommissioning timeout', 'general', 'cluster', config_type='int', priority=1, default_value=300, is_optional=True, description='Timeout for NodeManager decommissioning operation' ' during scaling, in seconds') DATANODES_STARTUP_TIMEOUT = p.Config( 'DataNodes startup timeout', 'general', 'cluster', config_type='int', priority=1, default_value=10800, is_optional=True, description='Timeout for DataNodes startup, in seconds') def init_env_configs(env_confs): configs = [] for service, config_items in six.iteritems(env_confs): for name, value in six.iteritems(config_items): configs.append(p.Config(name, service, "node", default_value=value, priority=1, config_type="int")) return configs def _init_general_configs(): configs = [ENABLE_SWIFT, ENABLE_MYSQL, DATANODES_STARTUP_TIMEOUT, DATANODES_DECOMMISSIONING_TIMEOUT, NODEMANAGERS_DECOMMISSIONING_TIMEOUT] if CONF.enable_data_locality: configs.append(ENABLE_DATA_LOCALITY) return configs PLUGIN_GENERAL_CONFIGS = _init_general_configs() def get_config_value(pctx, service, name, cluster=None): if cluster: for ng in cluster.node_groups: cl_param = ng.configuration().get(service, {}).get(name) if cl_param is not None: return cl_param for c in pctx['all_confs']: if c.applicable_target == service and c.name == name: return c.default_value raise ex.NotFoundException( {"name": name, "service": service}, _("Unable to get parameter '%(name)s' from service %(service)s")) def is_swift_enabled(pctx, cluster): return get_config_value(pctx, ENABLE_SWIFT.applicable_target, ENABLE_SWIFT.name, cluster) def is_mysql_enabled(pctx, cluster): return get_config_value( pctx, ENABLE_MYSQL.applicable_target, ENABLE_MYSQL.name, cluster) def is_data_locality_enabled(pctx, cluster): if not CONF.enable_data_locality: return False return get_config_value(pctx, ENABLE_DATA_LOCALITY.applicable_target, ENABLE_DATA_LOCALITY.name, cluster) def _get_spark_opt_default(opt_name): for opt in SPARK_CONFS["Spark"]["OPTIONS"]: if opt_name == opt["name"]: return opt["default"] return None def generate_spark_env_configs(cluster): configs = [] # point to the hadoop conf dir so that Spark can read things # like the swift configuration without having to copy core-site # to /opt/spark/conf HADOOP_CONF_DIR = '/opt/hadoop/etc/hadoop' configs.append('HADOOP_CONF_DIR=' + HADOOP_CONF_DIR) # Hadoop and YARN configs there are in one folder configs.append('YARN_CONF_DIR=' + HADOOP_CONF_DIR) return '\n'.join(configs) def generate_spark_executor_classpath(cluster): cp = utils.get_config_value_or_default( "Spark", "Executor extra classpath", cluster) if cp: return "spark.executor.extraClassPath " + cp return "\n" def generate_job_cleanup_config(cluster): args = { 'minimum_cleanup_megabytes': utils.get_config_value_or_default( "Spark", "Minimum cleanup megabytes", cluster), 'minimum_cleanup_seconds': utils.get_config_value_or_default( "Spark", "Minimum cleanup seconds", cluster), 'maximum_cleanup_seconds': utils.get_config_value_or_default( "Spark", "Maximum cleanup seconds", cluster) } job_conf = {'valid': (args['maximum_cleanup_seconds'] > 0 and (args['minimum_cleanup_megabytes'] > 0 and args['minimum_cleanup_seconds'] > 0))} if job_conf['valid']: job_conf['cron'] = f.get_file_text( 'plugins/vanilla/hadoop2/resources/spark-cleanup.cron'), job_cleanup_script = f.get_file_text( 'plugins/vanilla/hadoop2/resources/tmp-cleanup.sh.template') job_conf['script'] = job_cleanup_script.format(**args) return job_conf def get_spark_home(cluster): return utils.get_config_value_or_default("Spark", "Spark home", cluster) def generate_zk_basic_config(cluster): args = { 'ticktime': utils.get_config_value_or_default( "ZooKeeper", "tickTime", cluster), 'initlimit': utils.get_config_value_or_default( "ZooKeeper", "initLimit", cluster), 'synclimit': utils.get_config_value_or_default( "ZooKeeper", "syncLimit", cluster) } zoo_cfg = f.get_file_text( 'plugins/vanilla/hadoop2/resources/zoo_sample.cfg') return zoo_cfg.format(**args) sahara-8.0.0/sahara/plugins/vanilla/hadoop2/run_scripts.py0000666000175100017510000002516313245514472023651 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from oslo_log import log as logging from sahara import context from sahara.i18n import _ from sahara.plugins import utils as pu from sahara.plugins.vanilla.hadoop2 import config_helper as c_helper from sahara.plugins.vanilla.hadoop2 import oozie_helper from sahara.plugins.vanilla.hadoop2 import utils as u from sahara.plugins.vanilla import utils as vu from sahara.utils import cluster_progress_ops as cpo from sahara.utils import edp from sahara.utils import files from sahara.utils import poll_utils LOG = logging.getLogger(__name__) def start_dn_nm_processes(instances): filternames = ['datanode', 'nodemanager'] instances = pu.instances_with_services(instances, filternames) if len(instances) == 0: return cpo.add_provisioning_step( instances[0].cluster_id, pu.start_process_event_message("DataNodes, NodeManagers"), len(instances)) with context.ThreadGroup() as tg: for instance in instances: with context.set_current_instance_id(instance.instance_id): processes = set(instance.node_group.node_processes) processes = processes.intersection(filternames) tg.spawn('vanilla-start-processes-%s' % instance.instance_name, _start_processes, instance, list(processes)) @cpo.event_wrapper(True) def _start_processes(instance, processes): with instance.remote() as r: if 'datanode' in processes: r.execute_command( 'sudo su - -c "hadoop-daemon.sh start datanode" hadoop') if 'nodemanager' in processes: r.execute_command( 'sudo su - -c "yarn-daemon.sh start nodemanager" hadoop') def start_hadoop_process(instance, process): instance.remote().execute_command( 'sudo su - -c "hadoop-daemon.sh start %s" hadoop' % process) def start_yarn_process(instance, process): instance.remote().execute_command( 'sudo su - -c "yarn-daemon.sh start %s" hadoop' % process) @cpo.event_wrapper(True, step=pu.start_process_event_message("HistoryServer")) def start_historyserver(instance): instance.remote().execute_command( 'sudo su - -c "mr-jobhistory-daemon.sh start historyserver" hadoop') @cpo.event_wrapper(True, step=pu.start_process_event_message("Oozie")) def start_oozie_process(pctx, instance): with context.set_current_instance_id(instance.instance_id): with instance.remote() as r: if c_helper.is_mysql_enabled(pctx, instance.cluster): _start_mysql(r) LOG.debug("Creating Oozie DB Schema") sql_script = files.get_file_text( 'plugins/vanilla/hadoop2/resources/create_oozie_db.sql') password = oozie_helper.get_oozie_mysql_configs( instance.cluster)[ 'oozie.service.JPAService.jdbc.password'] sql_script = sql_script.replace("password", password) script_location = "create_oozie_db.sql" r.write_file_to(script_location, sql_script) r.execute_command('mysql -u root < %(script_location)s && ' 'rm %(script_location)s' % {"script_location": script_location}) _oozie_share_lib(r) _start_oozie(r) @cpo.event_wrapper( True, step=pu.start_process_event_message("Spark History Server")) def start_spark_history_server(master): sp_home = c_helper.get_spark_home(master.cluster) with context.set_current_instance_id(master.instance_id): with master.remote() as r: r.execute_command('sudo su - -c "bash %s" hadoop' % os.path.join( sp_home, 'sbin/start-history-server.sh')) def start_zk_server(instances): cpo.add_provisioning_step( instances[0].cluster_id, pu.start_process_event_message("ZooKeeper"), len(instances)) with context.ThreadGroup() as tg: for instance in instances: with context.set_current_instance_id(instance.instance_id): tg.spawn('ZK-start-processes-%s' % instance.instance_name, _start_zk_processes, instance, 'start') def refresh_zk_servers(cluster, to_delete_instances=None): instances = vu.get_zk_servers(cluster) if to_delete_instances: for instance in to_delete_instances: if instance in instances: instances.remove(instance) cpo.add_provisioning_step( cluster.id, pu.start_process_event_message("ZooKeeper"), len(instances)) with context.ThreadGroup() as tg: for instance in instances: with context.set_current_instance_id(instance.instance_id): tg.spawn('ZK-restart-processes-%s' % instance.instance_name, _start_zk_processes, instance, 'restart') @cpo.event_wrapper(True) def _start_zk_processes(instance, operation): with instance.remote() as r: r.execute_command( 'sudo su - -c "bash /opt/zookeeper/bin/zkServer.sh %s"' ' hadoop' % operation) def format_namenode(instance): instance.remote().execute_command( 'sudo su - -c "hdfs namenode -format" hadoop') @cpo.event_wrapper( True, step=pu.start_process_event_message("Oozie"), param=('cluster', 0)) def refresh_hadoop_nodes(cluster): nn = vu.get_namenode(cluster) nn.remote().execute_command( 'sudo su - -c "hdfs dfsadmin -refreshNodes" hadoop') @cpo.event_wrapper( True, step=_("Refresh %s nodes") % "YARN", param=('cluster', 0)) def refresh_yarn_nodes(cluster): rm = vu.get_resourcemanager(cluster) rm.remote().execute_command( 'sudo su - -c "yarn rmadmin -refreshNodes" hadoop') def _oozie_share_lib(remote): LOG.debug("Sharing Oozie libs") # remote.execute_command('sudo su - -c "/opt/oozie/bin/oozie-setup.sh ' # 'sharelib create -fs hdfs://%s:8020" hadoop' # % nn_hostname) # TODO(alazarev) return 'oozie-setup.sh sharelib create' back # when #1262023 is resolved remote.execute_command( 'sudo su - -c "mkdir /tmp/oozielib && ' 'tar zxf /opt/oozie/oozie-sharelib-*.tar.gz -C ' '/tmp/oozielib && ' 'hadoop fs -mkdir /user && ' 'hadoop fs -mkdir /user/hadoop && ' 'hadoop fs -put /tmp/oozielib/share /user/hadoop/ && ' 'hadoop fs -put /opt/oozie/libtools/commons-httpclient-3.1.jar ' '/user/hadoop/share/lib/oozie/ &&' 'rm -rf /tmp/oozielib" hadoop') LOG.debug("Creating sqlfile for Oozie") remote.execute_command('sudo su - -c "/opt/oozie/bin/ooziedb.sh ' 'create -sqlfile oozie.sql ' '-run Validate DB Connection" hadoop') def _start_mysql(remote): LOG.debug("Starting mysql") remote.execute_command('/opt/start-mysql.sh') def _start_oozie(remote): remote.execute_command( 'sudo su - -c "/opt/oozie/bin/oozied.sh start" hadoop') @cpo.event_wrapper( True, step=_("Await %s start up") % "DataNodes", param=('cluster', 0)) def await_datanodes(cluster): datanodes_count = len(vu.get_datanodes(cluster)) if datanodes_count < 1: return l_message = _("Waiting on %s datanodes to start up") % datanodes_count with vu.get_namenode(cluster).remote() as r: poll_utils.plugin_option_poll( cluster, _check_datanodes_count, c_helper.DATANODES_STARTUP_TIMEOUT, l_message, 1, { 'remote': r, 'count': datanodes_count}) def _check_datanodes_count(remote, count): if count < 1: return True LOG.debug("Checking datanode count") exit_code, stdout = remote.execute_command( 'sudo su -lc "hdfs dfsadmin -report" hadoop | ' 'grep \'Live datanodes\|Datanodes available:\' | ' 'grep -o \'[0-9]\+\' | head -n 1') LOG.debug("Datanode count='{count}'".format(count=stdout.rstrip())) return exit_code == 0 and stdout and int(stdout) == count def _hive_create_warehouse_dir(remote): LOG.debug("Creating Hive warehouse dir") remote.execute_command("sudo su - -c 'hadoop fs -mkdir -p " "/user/hive/warehouse' hadoop") def _hive_copy_shared_conf(remote, dest): LOG.debug("Copying shared Hive conf") dirname, filename = os.path.split(dest) remote.execute_command( "sudo su - -c 'hadoop fs -mkdir -p %s && " "hadoop fs -put /opt/hive/conf/hive-site.xml " "%s' hadoop" % (dirname, dest)) def _hive_create_db(remote): LOG.debug("Creating Hive metastore db") remote.execute_command("mysql -u root < /tmp/create_hive_db.sql") def _hive_metastore_start(remote): LOG.debug("Starting Hive Metastore Server") remote.execute_command("sudo su - -c 'nohup /opt/hive/bin/hive" " --service metastore > /dev/null &' hadoop") @cpo.event_wrapper(True, step=pu.start_process_event_message("HiveServer")) def start_hiveserver_process(pctx, instance): with context.set_current_instance_id(instance.instance_id): with instance.remote() as r: _hive_create_warehouse_dir(r) _hive_copy_shared_conf( r, edp.get_hive_shared_conf_path('hadoop')) if c_helper.is_mysql_enabled(pctx, instance.cluster): oozie = vu.get_oozie(instance.node_group.cluster) if not oozie or instance.hostname() != oozie.hostname(): _start_mysql(r) version = instance.cluster.hadoop_version sql_script = files.get_file_text( 'plugins/vanilla/v{}/resources/create_hive_db.sql'.format( version.replace('.', '_')) ) sql_script = sql_script.replace( '{{password}}', u.get_hive_password(instance.cluster)) r.write_file_to('/tmp/create_hive_db.sql', sql_script) _hive_create_db(r) _hive_metastore_start(r) LOG.info("Hive Metastore server at {host} has been " "started".format(host=instance.hostname())) sahara-8.0.0/sahara/plugins/vanilla/hadoop2/oozie_helper.py0000666000175100017510000000420113245514472023750 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.vanilla.hadoop2 import utils as u def get_oozie_required_xml_configs(hadoop_conf_dir): """Following configs differ from default configs in oozie-default.xml.""" return { 'oozie.service.ActionService.executor.ext.classes': 'org.apache.oozie.action.email.EmailActionExecutor,' 'org.apache.oozie.action.hadoop.HiveActionExecutor,' 'org.apache.oozie.action.hadoop.ShellActionExecutor,' 'org.apache.oozie.action.hadoop.SqoopActionExecutor,' 'org.apache.oozie.action.hadoop.DistcpActionExecutor', 'oozie.service.SchemaService.wf.ext.schemas': 'shell-action-0.1.xsd,shell-action-0.2.xsd,shell-action-0.3.xsd,' 'email-action-0.1.xsd,hive-action-0.2.xsd,hive-action-0.3.xsd,' 'hive-action-0.4.xsd,hive-action-0.5.xsd,sqoop-action-0.2.xsd,' 'sqoop-action-0.3.xsd,sqoop-action-0.4.xsd,ssh-action-0.1.xsd,' 'ssh-action-0.2.xsd,distcp-action-0.1.xsd,distcp-action-0.2.xsd,' 'oozie-sla-0.1.xsd,oozie-sla-0.2.xsd', 'oozie.service.JPAService.create.db.schema': 'false', 'oozie.service.HadoopAccessorService.hadoop.configurations': '*=%s' % ( hadoop_conf_dir) } def get_oozie_mysql_configs(cluster): return { 'oozie.service.JPAService.jdbc.driver': 'com.mysql.jdbc.Driver', 'oozie.service.JPAService.jdbc.url': 'jdbc:mysql://localhost:3306/oozie', 'oozie.service.JPAService.jdbc.username': 'oozie', 'oozie.service.JPAService.jdbc.password': u.get_oozie_password( cluster) } sahara-8.0.0/sahara/plugins/vanilla/hadoop2/__init__.py0000666000175100017510000000000013245514472023014 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/hadoop2/utils.py0000666000175100017510000000552513245514472022436 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re from oslo_log import log as logging from sahara import conductor as cond from sahara import context from sahara.plugins.vanilla import utils as u from sahara.service.castellan import utils as castellan conductor = cond.API LOG = logging.getLogger(__name__) def get_datanodes_status(cluster): statuses = {} namenode = u.get_namenode(cluster) status_regexp = r'^Hostname: (.*)\nDecommission Status : (.*)$' matcher = re.compile(status_regexp, re.MULTILINE) dfs_report = namenode.remote().execute_command( 'sudo su - -c "hdfs dfsadmin -report" hadoop')[1] for host, status in matcher.findall(dfs_report): statuses[host] = status.lower() return statuses def get_nodemanagers_status(cluster): statuses = {} resourcemanager = u.get_resourcemanager(cluster) status_regexp = r'^(\S+):\d+\s+(\w+)' matcher = re.compile(status_regexp, re.MULTILINE) yarn_report = resourcemanager.remote().execute_command( 'sudo su - -c "yarn node -all -list" hadoop')[1] for host, status in matcher.findall(yarn_report): statuses[host] = status.lower() return statuses def get_oozie_password(cluster): cluster = conductor.cluster_get(context.ctx(), cluster) extra = cluster.extra.to_dict() if 'oozie_pass_id' not in extra: extra['oozie_pass_id'] = u.generate_random_password() conductor.cluster_update(context.ctx(), cluster, {'extra': extra}) return castellan.get_secret(extra['oozie_pass_id']) def delete_oozie_password(cluster): extra = cluster.extra.to_dict() if 'oozie_pass_id' in extra: castellan.delete_secret(extra['oozie_pass_id']) else: LOG.warning("Cluster hasn't Oozie password") def get_hive_password(cluster): cluster = conductor.cluster_get(context.ctx(), cluster) extra = cluster.extra.to_dict() if 'hive_pass_id' not in extra: extra['hive_pass_id'] = u.generate_random_password() conductor.cluster_update(context.ctx(), cluster, {'extra': extra}) return castellan.get_secret(extra['hive_pass_id']) def delete_hive_password(cluster): extra = cluster.extra.to_dict() if 'hive_pass_id' in extra: castellan.delete_secret(extra['hive_pass_id']) else: LOG.warning("Cluster hasn't hive password") sahara-8.0.0/sahara/plugins/vanilla/hadoop2/scaling.py0000666000175100017510000001300313245514472022704 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.i18n import _ from sahara.plugins import utils as u from sahara.plugins.vanilla.hadoop2 import config from sahara.plugins.vanilla.hadoop2 import config_helper as c_helper from sahara.plugins.vanilla.hadoop2 import run_scripts as run from sahara.plugins.vanilla.hadoop2 import utils as pu from sahara.plugins.vanilla import utils as vu from sahara.swift import swift_helper from sahara.utils import cluster_progress_ops as cpo from sahara.utils import poll_utils HADOOP_CONF_DIR = config.HADOOP_CONF_DIR def scale_cluster(pctx, cluster, instances): config.configure_instances(pctx, instances) _update_include_files(cluster) run.refresh_hadoop_nodes(cluster) rm = vu.get_resourcemanager(cluster) if rm: run.refresh_yarn_nodes(cluster) config.configure_topology_data(pctx, cluster) run.start_dn_nm_processes(instances) swift_helper.install_ssl_certs(instances) config.configure_zookeeper(cluster) run.refresh_zk_servers(cluster) def _get_instances_with_service(instances, service): return [instance for instance in instances if service in instance.node_group.node_processes] @cpo.event_wrapper(True, step=_("Update include files"), param=('cluster', 0)) def _update_include_files(cluster, dec_instances=None): dec_instances = dec_instances or [] dec_instances_ids = [instance.id for instance in dec_instances] instances = u.get_instances(cluster) inst_filter = lambda inst: inst.id not in dec_instances_ids datanodes = filter(inst_filter, vu.get_datanodes(cluster)) nodemanagers = filter(inst_filter, vu.get_nodemanagers(cluster)) dn_hosts = u.generate_fqdn_host_names(datanodes) nm_hosts = u.generate_fqdn_host_names(nodemanagers) for instance in instances: with instance.remote() as r: r.execute_command( 'sudo su - -c "echo \'%s\' > %s/dn-include" hadoop' % ( dn_hosts, HADOOP_CONF_DIR)) r.execute_command( 'sudo su - -c "echo \'%s\' > %s/nm-include" hadoop' % ( nm_hosts, HADOOP_CONF_DIR)) def decommission_nodes(pctx, cluster, instances): datanodes = _get_instances_with_service(instances, 'datanode') nodemanagers = _get_instances_with_service(instances, 'nodemanager') _update_exclude_files(cluster, instances) run.refresh_hadoop_nodes(cluster) rm = vu.get_resourcemanager(cluster) if rm: run.refresh_yarn_nodes(cluster) _check_nodemanagers_decommission(cluster, nodemanagers) _check_datanodes_decommission(cluster, datanodes) _update_include_files(cluster, instances) _clear_exclude_files(cluster) run.refresh_hadoop_nodes(cluster) config.configure_topology_data(pctx, cluster) config.configure_zookeeper(cluster, instances) # TODO(shuyingya):should invent a way to lastly restart the leader node run.refresh_zk_servers(cluster, instances) def _update_exclude_files(cluster, instances): datanodes = _get_instances_with_service(instances, 'datanode') nodemanagers = _get_instances_with_service(instances, 'nodemanager') dn_hosts = u.generate_fqdn_host_names(datanodes) nm_hosts = u.generate_fqdn_host_names(nodemanagers) for instance in u.get_instances(cluster): with instance.remote() as r: r.execute_command( 'sudo su - -c "echo \'%s\' > %s/dn-exclude" hadoop' % ( dn_hosts, HADOOP_CONF_DIR)) r.execute_command( 'sudo su - -c "echo \'%s\' > %s/nm-exclude" hadoop' % ( nm_hosts, HADOOP_CONF_DIR)) def _clear_exclude_files(cluster): for instance in u.get_instances(cluster): with instance.remote() as r: r.execute_command( 'sudo su - -c "echo > %s/dn-exclude" hadoop' % HADOOP_CONF_DIR) r.execute_command( 'sudo su - -c "echo > %s/nm-exclude" hadoop' % HADOOP_CONF_DIR) def is_decommissioned(cluster, check_func, instances): statuses = check_func(cluster) for instance in instances: if statuses[instance.fqdn()] != 'decommissioned': return False return True def _check_decommission(cluster, instances, check_func, option): poll_utils.plugin_option_poll( cluster, is_decommissioned, option, _("Wait for decommissioning"), 5, {'cluster': cluster, 'check_func': check_func, 'instances': instances}) @cpo.event_wrapper( True, step=_("Decommission %s") % "NodeManagers", param=('cluster', 0)) def _check_nodemanagers_decommission(cluster, instances): _check_decommission(cluster, instances, pu.get_nodemanagers_status, c_helper.NODEMANAGERS_DECOMMISSIONING_TIMEOUT) @cpo.event_wrapper( True, step=_("Decommission %s") % "DataNodes", param=('cluster', 0)) def _check_datanodes_decommission(cluster, instances): _check_decommission(cluster, instances, pu.get_datanodes_status, c_helper.DATANODES_DECOMMISSIONING_TIMEOUT) sahara-8.0.0/sahara/plugins/vanilla/hadoop2/edp_engine.py0000666000175100017510000000170013245514472023362 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.vanilla import edp_engine from sahara.service.edp import hdfs_helper class EdpOozieEngine(edp_engine.EdpOozieEngine): def create_hdfs_dir(self, remote, dir_name): hdfs_helper.create_dir_hadoop2(remote, dir_name, self.get_hdfs_user()) def get_resource_manager_uri(self, cluster): return cluster['info']['YARN']['ResourceManager'] sahara-8.0.0/sahara/plugins/vanilla/hadoop2/resources/0000775000175100017510000000000013245515027022722 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/hadoop2/resources/zoo_sample.cfg0000666000175100017510000000166013245514472025563 0ustar zuulzuul00000000000000# The number of milliseconds of each tick tickTime={ticktime} # The number of ticks that the initial # synchronization phase can take initLimit={initlimit} # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit={synclimit} # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/var/zookeeper # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 sahara-8.0.0/sahara/plugins/vanilla/hadoop2/resources/topology.sh0000777000175100017510000000062513245514472025145 0ustar zuulzuul00000000000000#!/bin/bash HADOOP_CONF=/opt/hadoop/etc/hadoop while [ $# -gt 0 ] ; do nodeArg=$1 exec< ${HADOOP_CONF}/topology.data result="" while read line ; do ar=( $line ) if [ "${ar[0]}" = "$nodeArg" ] ; then result="${ar[1]}" fi done shift if [ -z "$result" ] ; then echo -n "/default/rack " else echo -n "$result " fi done sahara-8.0.0/sahara/plugins/vanilla/hadoop2/resources/spark-cleanup.cron0000666000175100017510000000013613245514472026357 0ustar zuulzuul00000000000000# Cleans up old Spark job directories once per hour. 0 * * * * root /etc/hadoop/tmp-cleanup.shsahara-8.0.0/sahara/plugins/vanilla/hadoop2/resources/post_conf.template0000666000175100017510000000222313245514472026455 0ustar zuulzuul00000000000000#!/bin/bash # change permission to config hadoop_configs=('core-site.xml' 'hdfs-site.xml' 'mapred-site.xml' 'yarn-site.xml') for conf in "${{hadoop_configs[@]}}" do chown -R {hadoop_group}:{hadoop_user} {hadoop_conf_dir}/$conf done chown -R {hadoop_group}:{hadoop_user} {oozie_conf_dir}/oozie-site.xml # create dirs for hdfs and mapreduce service dirs=({hadoop_name_dirs} {hadoop_data_dirs} {hadoop_log_dir} {hadoop_secure_dn_log_dir} {yarn_log_dir}) for dir in "${{dirs[@]}}" do mkdir -p $dir chown -R {hadoop_group}:{hadoop_user} $dir done # change hadoop log dir sed -i "s,\#export HADOOP_LOG_DIR=.*,export HADOOP_LOG_DIR={hadoop_log_dir}," {hadoop_conf_dir}/hadoop-env.sh sed -i "s,export HADOOP_SECURE_DN_LOG_DIR=.*,export HADOOP_SECURE_DN_LOG_DIR={hadoop_secure_dn_log_dir}," {hadoop_conf_dir}/hadoop-env.sh # change yarn log dir sed -i "s,YARN_LOG_DIR=.*,YARN_LOG_DIR={yarn_log_dir}," {hadoop_conf_dir}/yarn-env.sh # prepare scaling files sc_all_files=('dn-include' 'nm-include' 'dn-exclude' 'nm-exclude') for file in "${{sc_all_files[@]}}" do touch {hadoop_conf_dir}/$file chown {hadoop_group}:{hadoop_user} {hadoop_conf_dir}/$file done sahara-8.0.0/sahara/plugins/vanilla/hadoop2/resources/tmp-cleanup.sh.template0000666000175100017510000000230513245514472027322 0ustar zuulzuul00000000000000#!/bin/sh MINIMUM_CLEANUP_MEGABYTES={minimum_cleanup_megabytes} MINIMUM_CLEANUP_SECONDS={minimum_cleanup_seconds} MAXIMUM_CLEANUP_SECONDS={maximum_cleanup_seconds} CURRENT_TIMESTAMP=`date +%s` POSSIBLE_CLEANUP_THRESHOLD=$(($CURRENT_TIMESTAMP - $MINIMUM_CLEANUP_SECONDS)) DEFINITE_CLEANUP_THRESHOLD=$(($CURRENT_TIMESTAMP - $MAXIMUM_CLEANUP_SECONDS)) unset MAY_DELETE unset WILL_DELETE if [ ! -d /tmp/spark-edp ] then exit 0 fi cd /tmp/spark-edp for JOB in $(find . -maxdepth 1 -mindepth 1 -type d -printf '%f\n') do for EXECUTION in $(find $JOB -maxdepth 1 -mindepth 1 -type d -printf '%f\n') do TIMESTAMP=`stat $JOB/$EXECUTION --printf '%Y'` if [[ $TIMESTAMP -lt $DEFINITE_CLEANUP_THRESHOLD ]] then WILL_DELETE="$WILL_DELETE $JOB/$EXECUTION" else if [[ $TIMESTAMP -lt $POSSIBLE_CLEANUP_THRESHOLD ]] then MAY_DELETE="$MAY_DELETE $JOB/$EXECUTION" fi fi done done for EXECUTION in $WILL_DELETE do rm -Rf $EXECUTION done for EXECUTION in $(ls $MAY_DELETE -trd) do if [[ `du -s -BM | grep -o '[0-9]\+'` -le $MINIMUM_CLEANUP_MEGABYTES ]]; then break fi rm -Rf $EXECUTION done sahara-8.0.0/sahara/plugins/vanilla/hadoop2/resources/create_oozie_db.sql0000666000175100017510000000031013245514472026557 0ustar zuulzuul00000000000000create database oozie; grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'password'; grant all privileges on oozie.* to 'oozie'@'%' identified by 'password'; flush privileges; exit sahara-8.0.0/sahara/plugins/vanilla/plugin.py0000666000175100017510000001005113245514472021226 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara.i18n import _ from sahara.plugins import provisioning as p from sahara.plugins.vanilla import versionfactory as vhf class VanillaProvider(p.ProvisioningPluginBase): def __init__(self): self.version_factory = vhf.VersionFactory.get_instance() def get_description(self): return _('The Apache Vanilla plugin provides the ability to launch ' 'upstream Vanilla Apache Hadoop cluster without any ' 'management consoles. It can also deploy the Oozie ' 'component.') def _get_version_handler(self, hadoop_version): return self.version_factory.get_version_handler(hadoop_version) def get_node_processes(self, hadoop_version): return self._get_version_handler(hadoop_version).get_node_processes() def get_labels(self): default = {'enabled': {'status': True}, 'stable': {'status': True}} result = {'plugin_labels': copy.deepcopy(default)} result['version_labels'] = { version: copy.deepcopy(default) for version in self.get_versions() } return result def get_versions(self): return self.version_factory.get_versions() def get_title(self): return "Vanilla Apache Hadoop" def get_configs(self, hadoop_version): return self._get_version_handler(hadoop_version).get_plugin_configs() def configure_cluster(self, cluster): return self._get_version_handler( cluster.hadoop_version).configure_cluster(cluster) def start_cluster(self, cluster): return self._get_version_handler( cluster.hadoop_version).start_cluster(cluster) def validate(self, cluster): return self._get_version_handler( cluster.hadoop_version).validate(cluster) def scale_cluster(self, cluster, instances): return self._get_version_handler( cluster.hadoop_version).scale_cluster(cluster, instances) def decommission_nodes(self, cluster, instances): return self._get_version_handler( cluster.hadoop_version).decommission_nodes(cluster, instances) def validate_scaling(self, cluster, existing, additional): return self._get_version_handler( cluster.hadoop_version).validate_scaling(cluster, existing, additional) def get_edp_engine(self, cluster, job_type): return self._get_version_handler( cluster.hadoop_version).get_edp_engine(cluster, job_type) def get_edp_job_types(self, versions=None): res = {} for vers in self.version_factory.get_versions(): if not versions or vers in versions: vh = self.version_factory.get_version_handler(vers) res[vers] = vh.get_edp_job_types() return res def get_edp_config_hints(self, job_type, version): version_handler = ( self.version_factory.get_version_handler(version)) return version_handler.get_edp_config_hints(job_type) def get_open_ports(self, node_group): return self._get_version_handler( node_group.cluster.hadoop_version).get_open_ports(node_group) def on_terminate_cluster(self, cluster): return self._get_version_handler( cluster.hadoop_version).on_terminate_cluster(cluster) def recommend_configs(self, cluster, scaling=False): return self._get_version_handler( cluster.hadoop_version).recommend_configs(cluster, scaling) sahara-8.0.0/sahara/plugins/vanilla/confighints_helper.py0000666000175100017510000000275213245514472023613 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp.oozie.workflow_creator import workflow_factory from sahara.utils import xmlutils def get_possible_hive_config_from(file_name): '''Return the possible configs, args, params for a Hive job.''' config = { 'configs': xmlutils.load_hadoop_xml_defaults(file_name), 'params': {} } return config def get_possible_mapreduce_config_from(file_name): '''Return the possible configs, args, params for a MapReduce job.''' config = { 'configs': get_possible_pig_config_from(file_name).get('configs') } config['configs'] += workflow_factory.get_possible_mapreduce_configs() return config def get_possible_pig_config_from(file_name): '''Return the possible configs, args, params for a Pig job.''' config = { 'configs': xmlutils.load_hadoop_xml_defaults(file_name), 'args': [], 'params': {} } return config sahara-8.0.0/sahara/plugins/vanilla/__init__.py0000666000175100017510000000000013245514472021460 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/utils.py0000666000175100017510000000334113245514472021074 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils from sahara.plugins import utils as u from sahara.service.castellan import utils as castellan def get_namenode(cluster): return u.get_instance(cluster, "namenode") def get_resourcemanager(cluster): return u.get_instance(cluster, 'resourcemanager') def get_nodemanagers(cluster): return u.get_instances(cluster, 'nodemanager') def get_oozie(cluster): return u.get_instance(cluster, "oozie") def get_spark_history_server(cluster): return u.get_instance(cluster, "spark history server") def get_hiveserver(cluster): return u.get_instance(cluster, "hiveserver") def get_datanodes(cluster): return u.get_instances(cluster, 'datanode') def get_secondarynamenode(cluster): return u.get_instance(cluster, 'secondarynamenode') def get_historyserver(cluster): return u.get_instance(cluster, 'historyserver') def get_instance_hostname(instance): return instance.hostname() if instance else None def get_zk_servers(cluster): return u.get_instances(cluster, 'zookeeper') def generate_random_password(): password = uuidutils.generate_uuid() return castellan.store_secret(password) sahara-8.0.0/sahara/plugins/vanilla/abstractversionhandler.py0000666000175100017510000000323513245514472024505 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import six @six.add_metaclass(abc.ABCMeta) class AbstractVersionHandler(object): @abc.abstractmethod def get_node_processes(self): return @abc.abstractmethod def get_plugin_configs(self): return @abc.abstractmethod def configure_cluster(self, cluster): return @abc.abstractmethod def start_cluster(self, cluster): return @abc.abstractmethod def validate(self, cluster): return @abc.abstractmethod def scale_cluster(self, cluster, instances): return @abc.abstractmethod def decommission_nodes(self, cluster, instances): return @abc.abstractmethod def validate_scaling(self, cluster, existing, additional): return @abc.abstractmethod def get_edp_engine(self, cluster, job_type): return def get_edp_job_types(self): return [] def get_edp_config_hints(self, job_type): return {} @abc.abstractmethod def get_open_ports(self, node_group): return def on_terminate_cluster(self, cluster): pass sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/0000775000175100017510000000000013245515027020351 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/versionhandler.py0000666000175100017510000001374413245514472023764 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from sahara import conductor from sahara import context from sahara.plugins import utils from sahara.plugins.vanilla import abstractversionhandler as avm from sahara.plugins.vanilla.hadoop2 import config as c from sahara.plugins.vanilla.hadoop2 import keypairs from sahara.plugins.vanilla.hadoop2 import recommendations_utils as ru from sahara.plugins.vanilla.hadoop2 import run_scripts as run from sahara.plugins.vanilla.hadoop2 import scaling as sc from sahara.plugins.vanilla.hadoop2 import starting_scripts as s_scripts from sahara.plugins.vanilla.hadoop2 import utils as u from sahara.plugins.vanilla.hadoop2 import validation as vl from sahara.plugins.vanilla import utils as vu from sahara.plugins.vanilla.v2_7_1 import config_helper as c_helper from sahara.plugins.vanilla.v2_7_1 import edp_engine from sahara.swift import swift_helper from sahara.utils import cluster as cluster_utils conductor = conductor.API CONF = cfg.CONF class VersionHandler(avm.AbstractVersionHandler): def __init__(self): self.pctx = { 'env_confs': c_helper.get_env_configs(), 'all_confs': c_helper.get_plugin_configs() } def get_plugin_configs(self): return self.pctx['all_confs'] def get_node_processes(self): return { "Hadoop": [], "MapReduce": ["historyserver"], "HDFS": ["namenode", "datanode", "secondarynamenode"], "YARN": ["resourcemanager", "nodemanager"], "JobFlow": ["oozie"], "Hive": ["hiveserver"], "Spark": ["spark history server"], "ZooKeeper": ["zookeeper"] } def validate(self, cluster): vl.validate_cluster_creating(self.pctx, cluster) def update_infra(self, cluster): pass def configure_cluster(self, cluster): c.configure_cluster(self.pctx, cluster) def start_cluster(self, cluster): keypairs.provision_keypairs(cluster) s_scripts.start_namenode(cluster) s_scripts.start_secondarynamenode(cluster) s_scripts.start_resourcemanager(cluster) run.start_dn_nm_processes(utils.get_instances(cluster)) run.await_datanodes(cluster) s_scripts.start_historyserver(cluster) s_scripts.start_oozie(self.pctx, cluster) s_scripts.start_hiveserver(self.pctx, cluster) s_scripts.start_zookeeper(cluster) swift_helper.install_ssl_certs(cluster_utils.get_instances(cluster)) self._set_cluster_info(cluster) s_scripts.start_spark(cluster) def decommission_nodes(self, cluster, instances): sc.decommission_nodes(self.pctx, cluster, instances) def validate_scaling(self, cluster, existing, additional): vl.validate_additional_ng_scaling(cluster, additional) vl.validate_existing_ng_scaling(self.pctx, cluster, existing) zk_ng = utils.get_node_groups(cluster, "zookeeper") if zk_ng: vl.validate_zookeeper_node_count(zk_ng, existing, additional) def scale_cluster(self, cluster, instances): keypairs.provision_keypairs(cluster, instances) sc.scale_cluster(self.pctx, cluster, instances) def _set_cluster_info(self, cluster): nn = vu.get_namenode(cluster) rm = vu.get_resourcemanager(cluster) hs = vu.get_historyserver(cluster) oo = vu.get_oozie(cluster) sp = vu.get_spark_history_server(cluster) info = {} if rm: info['YARN'] = { 'Web UI': 'http://%s:%s' % (rm.get_ip_or_dns_name(), '8088'), 'ResourceManager': 'http://%s:%s' % ( rm.get_ip_or_dns_name(), '8032') } if nn: info['HDFS'] = { 'Web UI': 'http://%s:%s' % (nn.get_ip_or_dns_name(), '50070'), 'NameNode': 'hdfs://%s:%s' % (nn.hostname(), '9000') } if oo: info['JobFlow'] = { 'Oozie': 'http://%s:%s' % (oo.get_ip_or_dns_name(), '11000') } if hs: info['MapReduce JobHistory Server'] = { 'Web UI': 'http://%s:%s' % (hs.get_ip_or_dns_name(), '19888') } if sp: info['Apache Spark'] = { 'Spark UI': 'http://%s:%s' % (sp.management_ip, '4040'), 'Spark History Server UI': 'http://%s:%s' % (sp.management_ip, '18080') } ctx = context.ctx() conductor.cluster_update(ctx, cluster, {'info': info}) def get_edp_engine(self, cluster, job_type): if job_type in edp_engine.EdpOozieEngine.get_supported_job_types(): return edp_engine.EdpOozieEngine(cluster) if job_type in edp_engine.EdpSparkEngine.get_supported_job_types(): return edp_engine.EdpSparkEngine(cluster) return None def get_edp_job_types(self): return (edp_engine.EdpOozieEngine.get_supported_job_types() + edp_engine.EdpSparkEngine.get_supported_job_types()) def get_edp_config_hints(self, job_type): return edp_engine.EdpOozieEngine.get_possible_job_config(job_type) def on_terminate_cluster(self, cluster): u.delete_oozie_password(cluster) keypairs.drop_key(cluster) def get_open_ports(self, node_group): return c.get_open_ports(node_group) def recommend_configs(self, cluster, scaling): ru.recommend_configs(cluster, self.get_plugin_configs(), scaling) sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/config_helper.py0000666000175100017510000000734613245514472023546 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six from sahara.plugins import provisioning as p from sahara.plugins.vanilla.hadoop2 import config_helper as c_helper from sahara.utils import xmlutils as x CONF = cfg.CONF CONF.import_opt("enable_data_locality", "sahara.topology.topology_helper") CORE_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_7_1/resources/core-default.xml') HDFS_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_7_1/resources/hdfs-default.xml') MAPRED_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_7_1/resources/mapred-default.xml') YARN_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_7_1/resources/yarn-default.xml') OOZIE_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_7_1/resources/oozie-default.xml') HIVE_DEFAULT = x.load_hadoop_xml_defaults( 'plugins/vanilla/v2_7_1/resources/hive-default.xml') XML_CONFS = { "Hadoop": [CORE_DEFAULT], "HDFS": [HDFS_DEFAULT], "YARN": [YARN_DEFAULT], "MapReduce": [MAPRED_DEFAULT], "JobFlow": [OOZIE_DEFAULT], "Hive": [HIVE_DEFAULT] } ENV_CONFS = { "YARN": { 'ResourceManager Heap Size': 1024, 'NodeManager Heap Size': 1024 }, "HDFS": { 'NameNode Heap Size': 1024, 'SecondaryNameNode Heap Size': 1024, 'DataNode Heap Size': 1024 }, "MapReduce": { 'JobHistoryServer Heap Size': 1024 }, "JobFlow": { 'Oozie Heap Size': 1024 } } # Initialise plugin Hadoop configurations PLUGIN_XML_CONFIGS = c_helper.init_xml_configs(XML_CONFS) PLUGIN_ENV_CONFIGS = c_helper.init_env_configs(ENV_CONFS) def _init_all_configs(): configs = [] configs.extend(PLUGIN_XML_CONFIGS) configs.extend(PLUGIN_ENV_CONFIGS) configs.extend(c_helper.PLUGIN_GENERAL_CONFIGS) configs.extend(_get_spark_configs()) configs.extend(_get_zookeeper_configs()) return configs def _get_spark_configs(): spark_configs = [] for service, config_items in six.iteritems(c_helper.SPARK_CONFS): for item in config_items['OPTIONS']: cfg = p.Config(name=item["name"], description=item["description"], default_value=item["default"], applicable_target=service, scope="cluster", is_optional=True, priority=item["priority"]) spark_configs.append(cfg) return spark_configs def _get_zookeeper_configs(): zk_configs = [] for service, config_items in six.iteritems(c_helper.ZOOKEEPER_CONFS): for item in config_items['OPTIONS']: cfg = p.Config(name=item["name"], description=item["description"], default_value=item["default"], applicable_target=service, scope="cluster", is_optional=True, priority=item["priority"]) zk_configs.append(cfg) return zk_configs PLUGIN_CONFIGS = _init_all_configs() def get_plugin_configs(): return PLUGIN_CONFIGS def get_xml_configs(): return PLUGIN_XML_CONFIGS def get_env_configs(): return ENV_CONFS sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/__init__.py0000666000175100017510000000000013245514472022455 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/edp_engine.py0000666000175100017510000000663413245514472023036 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import utils as plugin_utils from sahara.plugins.vanilla import confighints_helper as ch_helper from sahara.plugins.vanilla.hadoop2 import edp_engine from sahara.plugins.vanilla import utils as v_utils from sahara.service.edp.spark import engine as edp_spark_engine from sahara.utils import edp class EdpOozieEngine(edp_engine.EdpOozieEngine): @staticmethod def get_possible_job_config(job_type): if edp.compare_job_type(job_type, edp.JOB_TYPE_HIVE): return {'job_config': ch_helper.get_possible_hive_config_from( 'plugins/vanilla/v2_7_1/resources/hive-default.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_MAPREDUCE_STREAMING): return {'job_config': ch_helper.get_possible_mapreduce_config_from( 'plugins/vanilla/v2_7_1/resources/mapred-default.xml')} if edp.compare_job_type(job_type, edp.JOB_TYPE_PIG): return {'job_config': ch_helper.get_possible_pig_config_from( 'plugins/vanilla/v2_7_1/resources/mapred-default.xml')} return edp_engine.EdpOozieEngine.get_possible_job_config(job_type) class EdpSparkEngine(edp_spark_engine.SparkJobEngine): edp_base_version = "2.7.1" def __init__(self, cluster): super(EdpSparkEngine, self).__init__(cluster) self.master = plugin_utils.get_instance(cluster, "spark history server") self.plugin_params["spark-user"] = "sudo -u hadoop " self.plugin_params["spark-submit"] = os.path.join( plugin_utils.get_config_value_or_default( "Spark", "Spark home", self.cluster), "bin/spark-submit") self.plugin_params["deploy-mode"] = "cluster" self.plugin_params["master"] = "yarn" driver_cp = plugin_utils.get_config_value_or_default( "Spark", "Executor extra classpath", self.cluster) self.plugin_params["driver-class-path"] = driver_cp @staticmethod def edp_supported(version): return version >= EdpSparkEngine.edp_base_version @staticmethod def job_type_supported(job_type): return (job_type in edp_spark_engine.SparkJobEngine.get_supported_job_types()) def validate_job_execution(self, cluster, job, data): if (not self.edp_supported(cluster.hadoop_version) or not v_utils.get_spark_history_server(cluster)): raise ex.InvalidDataException( _('Spark {base} or higher required to run {type} jobs').format( base=EdpSparkEngine.edp_base_version, type=job.type)) super(EdpSparkEngine, self).validate_job_execution(cluster, job, data) sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/0000775000175100017510000000000013245515027022363 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/hive-default.xml0000666000175100017510000022237513245514472025502 0ustar zuulzuul00000000000000 mapred.reduce.tasks -1 The default number of reduce tasks per job. Typically set to a prime close to the number of available hosts. Ignored when mapred.job.tracker is "local". Hadoop set this to 1 by default, whereas hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers. hive.exec.reducers.bytes.per.reducer 1000000000 size per reducer.The default is 1G, i.e if the input size is 10G, it will use 10 reducers. hive.exec.reducers.max 999 max number of reducers will be used. If the one specified in the configuration parameter mapred.reduce.tasks is negative, hive will use this one as the max number of reducers when automatically determine number of reducers. hive.cli.print.header false Whether to print the names of the columns in query output. hive.cli.print.current.db false Whether to include the current database in the hive prompt. hive.cli.prompt hive Command line prompt configuration value. Other hiveconf can be used in this configuration value. Variable substitution will only be invoked at the hive cli startup. hive.cli.pretty.output.num.cols -1 The number of columns to use when formatting output generated by the DESCRIBE PRETTY table_name command. If the value of this property is -1, then hive will use the auto-detected terminal width. hive.exec.scratchdir /tmp/hive-${user.name} Scratch space for Hive jobs hive.exec.local.scratchdir /tmp/${user.name} Local scratch space for Hive jobs hive.test.mode false whether hive is running in test mode. If yes, it turns on sampling and prefixes the output tablename hive.test.mode.prefix test_ if hive is running in test mode, prefixes the output table by this string hive.test.mode.samplefreq 32 if hive is running in test mode and table is not bucketed, sampling frequency hive.test.mode.nosamplelist if hive is running in test mode, dont sample the above comma seperated list of tables hive.metastore.uris Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore. javax.jdo.option.ConnectionURL jdbc:derby:;databaseName=metastore_db;create=true JDBC connect string for a JDBC metastore javax.jdo.option.ConnectionDriverName org.apache.derby.jdbc.EmbeddedDriver Driver class name for a JDBC metastore javax.jdo.PersistenceManagerFactoryClass org.datanucleus.jdo.JDOPersistenceManagerFactory class implementing the jdo persistence javax.jdo.option.DetachAllOnCommit true detaches all objects from session so that they can be used after transaction is committed javax.jdo.option.NonTransactionalRead true reads outside of transactions javax.jdo.option.ConnectionUserName APP username to use against metastore database javax.jdo.option.ConnectionPassword mine password to use against metastore database javax.jdo.option.Multithreaded true Set this to true if multiple threads access metastore through JDO concurrently. datanucleus.connectionPoolingType DBCP Uses a DBCP connection pool for JDBC metastore datanucleus.validateTables false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.validateColumns false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.validateConstraints false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.storeManagerType rdbms metadata store type datanucleus.autoCreateSchema true creates necessary schema on a startup if one doesn't exist. set this to false, after creating it once datanucleus.autoStartMechanismMode checked throw exception if metadata tables are incorrect datanucleus.transactionIsolation read-committed Default transaction isolation level for identity generation. datanucleus.cache.level2 false Use a level 2 cache. Turn this off if metadata is changed independently of hive metastore server datanucleus.cache.level2.type SOFT SOFT=soft reference based cache, WEAK=weak reference based cache. datanucleus.identifierFactory datanucleus Name of the identifier factory to use when generating table/column names etc. 'datanucleus' is used for backward compatibility datanucleus.plugin.pluginRegistryBundleCheck LOG Defines what happens when plugin bundles are found and are duplicated [EXCEPTION|LOG|NONE] hive.metastore.warehouse.dir /user/hive/warehouse location of default database for the warehouse hive.metastore.execute.setugi false In unsecure mode, setting this property to true will cause the metastore to execute DFS operations using the client's reported user and group permissions. Note that this property must be set on both the client and server sides. Further note that its best effort. If client sets its to true and server sets it to false, client setting will be ignored. hive.metastore.event.listeners list of comma seperated listeners for metastore events. hive.metastore.partition.inherit.table.properties list of comma seperated keys occurring in table properties which will get inherited to newly created partitions. * implies all the keys will get inherited. hive.metadata.export.location When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, it is the location to which the metadata will be exported. The default is an empty string, which results in the metadata being exported to the current user's home directory on HDFS. hive.metadata.move.exported.metadata.to.trash When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, this setting determines if the metadata that is exported will subsequently be moved to the user's trash directory alongside the dropped table data. This ensures that the metadata will be cleaned up along with the dropped table data. hive.metastore.partition.name.whitelist.pattern Partition names will be checked against this regex pattern and rejected if not matched. hive.metastore.end.function.listeners list of comma separated listeners for the end of metastore functions. hive.metastore.event.expiry.duration 0 Duration after which events expire from events table (in seconds) hive.metastore.event.clean.freq 0 Frequency at which timer task runs to purge expired events in metastore(in seconds). hive.metastore.connect.retries 5 Number of retries while opening a connection to metastore hive.metastore.failure.retries 3 Number of retries upon failure of Thrift metastore calls hive.metastore.client.connect.retry.delay 1 Number of seconds for the client to wait between consecutive connection attempts hive.metastore.client.socket.timeout 20 MetaStore Client socket timeout in seconds hive.metastore.rawstore.impl org.apache.hadoop.hive.metastore.ObjectStore Name of the class that implements org.apache.hadoop.hive.metastore.rawstore interface. This class is used to store and retrieval of raw metadata objects such as table, database hive.metastore.batch.retrieve.max 300 Maximum number of objects (tables/partitions) can be retrieved from metastore in one batch. The higher the number, the less the number of round trips is needed to the Hive metastore server, but it may also cause higher memory requirement at the client side. hive.metastore.batch.retrieve.table.partition.max 1000 Maximum number of table partitions that metastore internally retrieves in one batch. hive.default.fileformat TextFile Default file format for CREATE TABLE statement. Options are TextFile and SequenceFile. Users can explicitly say CREATE TABLE ... STORED AS <TEXTFILE|SEQUENCEFILE> to override hive.fileformat.check true Whether to check file format or not when loading data files hive.map.aggr true Whether to use map-side aggregation in Hive Group By queries hive.groupby.skewindata false Whether there is skew in data to optimize group by queries hive.optimize.multigroupby.common.distincts true Whether to optimize a multi-groupby query with the same distinct. Consider a query like: from src insert overwrite table dest1 select col1, count(distinct colx) group by col1 insert overwrite table dest2 select col2, count(distinct colx) group by col2; With this parameter set to true, first we spray by the distinct value (colx), and then perform the 2 groups bys. This makes sense if map-side aggregation is turned off. However, with maps-side aggregation, it might be useful in some cases to treat the 2 inserts independently, thereby performing the query above in 2MR jobs instead of 3 (due to spraying by distinct key first). If this parameter is turned off, we dont consider the fact that the distinct key is the same across different MR jobs. hive.groupby.mapaggr.checkinterval 100000 Number of rows after which size of the grouping keys/aggregation classes is performed hive.mapred.local.mem 0 For local mode, memory of the mappers/reducers hive.mapjoin.followby.map.aggr.hash.percentmemory 0.3 Portion of total memory to be used by map-side grup aggregation hash table, when this group by is followed by map join hive.map.aggr.hash.force.flush.memory.threshold 0.9 The max memory to be used by map-side grup aggregation hash table, if the memory usage is higher than this number, force to flush data hive.map.aggr.hash.percentmemory 0.5 Portion of total memory to be used by map-side grup aggregation hash table hive.map.aggr.hash.min.reduction 0.5 Hash aggregation will be turned off if the ratio between hash table size and input rows is bigger than this number. Set to 1 to make sure hash aggregation is never turned off. hive.optimize.cp true Whether to enable column pruner hive.optimize.index.filter false Whether to enable automatic use of indexes hive.optimize.index.groupby false Whether to enable optimization of group-by queries using Aggregate indexes. hive.optimize.ppd true Whether to enable predicate pushdown hive.optimize.ppd.storage true Whether to push predicates down into storage handlers. Ignored when hive.optimize.ppd is false. hive.ppd.recognizetransivity true Whether to transitively replicate predicate filters over equijoin conditions. hive.optimize.groupby true Whether to enable the bucketed group by from bucketed partitions/tables. hive.optimize.skewjoin.compiletime false Whether to create a separate plan for skewed keys for the tables in the join. This is based on the skewed keys stored in the metadata. At compile time, the plan is broken into different joins: one for the skewed keys, and the other for the remaining keys. And then, a union is performed for the 2 joins generated above. So unless the same skewed key is present in both the joined tables, the join for the skewed key will be performed as a map-side join. The main difference between this paramater and hive.optimize.skewjoin is that this parameter uses the skew information stored in the metastore to optimize the plan at compile time itself. If there is no skew information in the metadata, this parameter will not have any affect. Both hive.optimize.skewjoin.compiletime and hive.optimize.skewjoin should be set to true. Ideally, hive.optimize.skewjoin should be renamed as hive.optimize.skewjoin.runtime, but not doing so for backward compatibility. If the skew information is correctly stored in the metadata, hive.optimize.skewjoin.compiletime would change the query plan to take care of it, and hive.optimize.skewjoin will be a no-op. hive.optimize.union.remove false Whether to remove the union and push the operators between union and the filesink above union. This avoids an extra scan of the output by union. This is independently useful for union queries, and specially useful when hive.optimize.skewjoin.compiletime is set to true, since an extra union is inserted. The merge is triggered if either of hive.merge.mapfiles or hive.merge.mapredfiles is set to true. If the user has set hive.merge.mapfiles to true and hive.merge.mapredfiles to false, the idea was the number of reducers are few, so the number of files anyway are small. However, with this optimization, we are increasing the number of files possibly by a big margin. So, we merge aggresively. hive.mapred.supports.subdirectories false Whether the version of hadoop which is running supports sub-directories for tables/partitions. Many hive optimizations can be applied if the hadoop version supports sub-directories for tables/partitions. It was added by MAPREDUCE-1501 hive.multigroupby.singlemr false Whether to optimize multi group by query to generate single M/R job plan. If the multi group by query has common group by keys, it will be optimized to generate single M/R job. hive.map.groupby.sorted false If the bucketing/sorting properties of the table exactly match the grouping key, whether to perform the group by in the mapper by using BucketizedHiveInputFormat. The only downside to this is that it limits the number of mappers to the number of files. hive.map.groupby.sorted.testmode false If the bucketing/sorting properties of the table exactly match the grouping key, whether to perform the group by in the mapper by using BucketizedHiveInputFormat. If the test mode is set, the plan is not converted, but a query property is set to denote the same. hive.new.job.grouping.set.cardinality 30 Whether a new map-reduce job should be launched for grouping sets/rollups/cubes. For a query like: select a, b, c, count(1) from T group by a, b, c with rollup; 4 rows are created per row: (a, b, c), (a, b, null), (a, null, null), (null, null, null). This can lead to explosion across map-reduce boundary if the cardinality of T is very high, and map-side aggregation does not do a very good job. This parameter decides if hive should add an additional map-reduce job. If the grouping set cardinality (4 in the example above), is more than this value, a new MR job is added under the assumption that the orginal group by will reduce the data size. hive.join.emit.interval 1000 How many rows in the right-most join operand Hive should buffer before emitting the join result. hive.join.cache.size 25000 How many rows in the joining tables (except the streaming table) should be cached in memory. hive.mapjoin.bucket.cache.size 100 How many values in each keys in the map-joined table should be cached in memory. hive.mapjoin.cache.numrows 25000 How many rows should be cached by jdbm for map join. hive.optimize.skewjoin false Whether to enable skew join optimization. The algorithm is as follows: At runtime, detect the keys with a large skew. Instead of processing those keys, store them temporarily in a hdfs directory. In a follow-up map-reduce job, process those skewed keys. The same key need not be skewed for all the tables, and so, the follow-up map-reduce job (for the skewed keys) would be much faster, since it would be a map-join. hive.skewjoin.key 100000 Determine if we get a skew key in join. If we see more than the specified number of rows with the same key in join operator, we think the key as a skew join key. hive.skewjoin.mapjoin.map.tasks 10000 Determine the number of map task used in the follow up map join job for a skew join. It should be used together with hive.skewjoin.mapjoin.min.split to perform a fine grained control. hive.skewjoin.mapjoin.min.split 33554432 Determine the number of map task at most used in the follow up map join job for a skew join by specifying the minimum split size. It should be used together with hive.skewjoin.mapjoin.map.tasks to perform a fine grained control. hive.mapred.mode nonstrict The mode in which the hive operations are being performed. In strict mode, some risky queries are not allowed to run. They include: Cartesian Product. No partition being picked up for a query. Comparing bigints and strings. Comparing bigints and doubles. Orderby without limit. hive.enforce.bucketmapjoin false If the user asked for bucketed map-side join, and it cannot be performed, should the query fail or not ? For eg, if the buckets in the tables being joined are not a multiple of each other, bucketed map-side join cannot be performed, and the query will fail if hive.enforce.bucketmapjoin is set to true. hive.exec.script.maxerrsize 100000 Maximum number of bytes a script is allowed to emit to standard error (per map-reduce task). This prevents runaway scripts from filling logs partitions to capacity hive.exec.script.allow.partial.consumption false When enabled, this option allows a user script to exit successfully without consuming all the data from the standard input. hive.script.operator.id.env.var HIVE_SCRIPT_OPERATOR_ID Name of the environment variable that holds the unique script operator ID in the user's transform function (the custom mapper/reducer that the user has specified in the query) hive.script.operator.truncate.env false Truncate each environment variable for external script in scripts operator to 20KB (to fit system limits) hive.exec.compress.output false This controls whether the final outputs of a query (to a local/hdfs file or a hive table) is compressed. The compression codec and other options are determined from hadoop config variables mapred.output.compress* hive.exec.compress.intermediate false This controls whether intermediate files produced by hive between multiple map-reduce jobs are compressed. The compression codec and other options are determined from hadoop config variables mapred.output.compress* hive.exec.parallel false Whether to execute jobs in parallel hive.exec.parallel.thread.number 8 How many jobs at most can be executed in parallel hive.exec.rowoffset false Whether to provide the row offset virtual column hive.task.progress false Whether Hive should periodically update task progress counters during execution. Enabling this allows task progress to be monitored more closely in the job tracker, but may impose a performance penalty. This flag is automatically set to true for jobs with hive.exec.dynamic.partition set to true. hive.hwi.war.file lib/hive-hwi-0.11.0.war This sets the path to the HWI war file, relative to ${HIVE_HOME}. hive.hwi.listen.host 0.0.0.0 This is the host address the Hive Web Interface will listen on hive.hwi.listen.port 9999 This is the port the Hive Web Interface will listen on hive.exec.pre.hooks Comma-separated list of pre-execution hooks to be invoked for each statement. A pre-execution hook is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.exec.post.hooks Comma-separated list of post-execution hooks to be invoked for each statement. A post-execution hook is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.exec.failure.hooks Comma-separated list of on-failure hooks to be invoked for each statement. An on-failure hook is specified as the name of Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.metastore.init.hooks A comma separated list of hooks to be invoked at the beginning of HMSHandler initialization. Aninit hook is specified as the name of Java class which extends org.apache.hadoop.hive.metastore.MetaStoreInitListener. hive.client.stats.publishers Comma-separated list of statistics publishers to be invoked on counters on each job. A client stats publisher is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.stats.ClientStatsPublisher interface. hive.client.stats.counters Subset of counters that should be of interest for hive.client.stats.publishers (when one wants to limit their publishing). Non-display names should be used hive.merge.mapfiles true Merge small files at the end of a map-only job hive.merge.mapredfiles false Merge small files at the end of a map-reduce job hive.heartbeat.interval 1000 Send a heartbeat after this interval - used by mapjoin and filter operators hive.merge.size.per.task 256000000 Size of merged files at the end of the job hive.merge.smallfiles.avgsize 16000000 When the average output file size of a job is less than this number, Hive will start an additional map-reduce job to merge the output files into bigger files. This is only done for map-only jobs if hive.merge.mapfiles is true, and for map-reduce jobs if hive.merge.mapredfiles is true. hive.mapjoin.smalltable.filesize 25000000 The threshold for the input file size of the small tables; if the file size is smaller than this threshold, it will try to convert the common join into map join hive.ignore.mapjoin.hint true Ignore the mapjoin hint hive.mapjoin.localtask.max.memory.usage 0.90 This number means how much memory the local task can take to hold the key/value into in-memory hash table; If the local task's memory usage is more than this number, the local task will be abort by themself. It means the data of small table is too large to be hold in the memory. hive.mapjoin.followby.gby.localtask.max.memory.usage 0.55 This number means how much memory the local task can take to hold the key/value into in-memory hash table when this map join followed by a group by; If the local task's memory usage is more than this number, the local task will be abort by themself. It means the data of small table is too large to be hold in the memory. hive.mapjoin.check.memory.rows 100000 The number means after how many rows processed it needs to check the memory usage hive.auto.convert.join false Whether Hive enable the optimization about converting common join into mapjoin based on the input file size hive.auto.convert.join.noconditionaltask true Whether Hive enable the optimization about converting common join into mapjoin based on the input file size. If this paramater is on, and the sum of size for n-1 of the tables/partitions for a n-way join is smaller than the specified size, the join is directly converted to a mapjoin (there is no conditional task). hive.auto.convert.join.noconditionaltask.size 10000000 If hive.auto.convert.join.noconditionaltask is off, this parameter does not take affect. However, if it is on, and the sum of size for n-1 of the tables/partitions for a n-way join is smaller than this size, the join is directly converted to a mapjoin(there is no conditional task). The default is 10MB hive.optimize.mapjoin.mapreduce false If hive.auto.convert.join is off, this parameter does not take affect. If it is on, and if there are map-join jobs followed by a map-reduce job (for e.g a group by), each map-only job is merged with the following map-reduce job. hive.script.auto.progress false Whether Hive Tranform/Map/Reduce Clause should automatically send progress information to TaskTracker to avoid the task getting killed because of inactivity. Hive sends progress information when the script is outputting to stderr. This option removes the need of periodically producing stderr messages, but users should be cautious because this may prevent infinite loops in the scripts to be killed by TaskTracker. hive.script.serde org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe The default serde for trasmitting input data to and reading output data from the user scripts. hive.binary.record.max.length 1000 Read from a binary stream and treat each hive.binary.record.max.length bytes as a record. The last record before the end of stream can have less than hive.binary.record.max.length bytes hive.script.recordreader org.apache.hadoop.hive.ql.exec.TextRecordReader The default record reader for reading data from the user scripts. hive.script.recordwriter org.apache.hadoop.hive.ql.exec.TextRecordWriter The default record writer for writing data to the user scripts. hive.input.format org.apache.hadoop.hive.ql.io.CombineHiveInputFormat The default input format. Set this to HiveInputFormat if you encounter problems with CombineHiveInputFormat. hive.udtf.auto.progress false Whether Hive should automatically send progress information to TaskTracker when using UDTF's to prevent the task getting killed because of inactivity. Users should be cautious because this may prevent TaskTracker from killing tasks with infinte loops. hive.mapred.reduce.tasks.speculative.execution true Whether speculative execution for reducers should be turned on. hive.exec.counters.pull.interval 1000 The interval with which to poll the JobTracker for the counters the running job. The smaller it is the more load there will be on the jobtracker, the higher it is the less granular the caught will be. hive.querylog.location /tmp/${user.name} Location of Hive run time structured log file hive.querylog.enable.plan.progress true Whether to log the plan's progress every time a job's progress is checked. These logs are written to the location specified by hive.querylog.location hive.querylog.plan.progress.interval 60000 The interval to wait between logging the plan's progress in milliseconds. If there is a whole number percentage change in the progress of the mappers or the reducers, the progress is logged regardless of this value. The actual interval will be the ceiling of (this value divided by the value of hive.exec.counters.pull.interval) multiplied by the value of hive.exec.counters.pull.interval I.e. if it is not divide evenly by the value of hive.exec.counters.pull.interval it will be logged less frequently than specified. This only has an effect if hive.querylog.enable.plan.progress is set to true. hive.enforce.bucketing false Whether bucketing is enforced. If true, while inserting into the table, bucketing is enforced. hive.enforce.sorting false Whether sorting is enforced. If true, while inserting into the table, sorting is enforced. hive.optimize.bucketingsorting true If hive.enforce.bucketing or hive.enforce.sorting is true, dont create a reducer for enforcing bucketing/sorting for queries of the form: insert overwrite table T2 select * from T1; where T1 and T2 are bucketed/sorted by the same keys into the same number of buckets. hive.enforce.sortmergebucketmapjoin false If the user asked for sort-merge bucketed map-side join, and it cannot be performed, should the query fail or not ? hive.auto.convert.sortmerge.join false Will the join be automatically converted to a sort-merge join, if the joined tables pass the criteria for sort-merge join. hive.auto.convert.sortmerge.join.bigtable.selection.policy org.apache.hadoop.hive.ql.optimizer.AvgPartitionSizeBasedBigTableSelectorForAutoSMJ The policy to choose the big table for automatic conversion to sort-merge join. By default, the table with the largest partitions is assigned the big table. All policies are: . based on position of the table - the leftmost table is selected org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSMJ. . based on total size (all the partitions selected in the query) of the table org.apache.hadoop.hive.ql.optimizer.TableSizeBasedBigTableSelectorForAutoSMJ. . based on average size (all the partitions selected in the query) of the table org.apache.hadoop.hive.ql.optimizer.AvgPartitionSizeBasedBigTableSelectorForAutoSMJ. New policies can be added in future. hive.metastore.ds.connection.url.hook Name of the hook to use for retriving the JDO connection URL. If empty, the value in javax.jdo.option.ConnectionURL is used hive.metastore.ds.retry.attempts 1 The number of times to retry a metastore call if there were a connection error hive.metastore.ds.retry.interval 1000 The number of miliseconds between metastore retry attempts hive.metastore.server.min.threads 200 Minimum number of worker threads in the Thrift server's pool. hive.metastore.server.max.threads 100000 Maximum number of worker threads in the Thrift server's pool. hive.metastore.server.tcp.keepalive true Whether to enable TCP keepalive for the metastore server. Keepalive will prevent accumulation of half-open connections. hive.metastore.sasl.enabled false If true, the metastore thrift interface will be secured with SASL. Clients must authenticate with Kerberos. hive.metastore.thrift.framed.transport.enabled false If true, the metastore thrift interface will use TFramedTransport. When false (default) a standard TTransport is used. hive.metastore.kerberos.keytab.file The path to the Kerberos Keytab file containing the metastore thrift server's service principal. hive.metastore.kerberos.principal hive-metastore/_HOST@EXAMPLE.COM The service principal for the metastore thrift server. The special string _HOST will be replaced automatically with the correct host name. hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.MemoryTokenStore The delegation token store implementation. Set to org.apache.hadoop.hive.thrift.ZooKeeperTokenStore for load-balanced cluster. hive.cluster.delegation.token.store.zookeeper.connectString localhost:2181 The ZooKeeper token store connect string. hive.cluster.delegation.token.store.zookeeper.znode /hive/cluster/delegation The root path for token store data. hive.cluster.delegation.token.store.zookeeper.acl sasl:hive/host1@EXAMPLE.COM:cdrwa,sasl:hive/host2@EXAMPLE.COM:cdrwa ACL for token store entries. List comma separated all server principals for the cluster. hive.metastore.cache.pinobjtypes Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order List of comma separated metastore object types that should be pinned in the cache hive.optimize.reducededuplication true Remove extra map-reduce jobs if the data is already clustered by the same key which needs to be used again. This should always be set to true. Since it is a new feature, it has been made configurable. hive.optimize.reducededuplication.min.reducer 4 Reduce deduplication merges two RSs by moving key/parts/reducer-num of the child RS to parent RS. That means if reducer-num of the child RS is fixed (order by or forced bucketing) and small, it can make very slow, single MR. The optimization will be disabled if number of reducers is less than specified value. hive.exec.dynamic.partition true Whether or not to allow dynamic partitions in DML/DDL. hive.exec.dynamic.partition.mode strict In strict mode, the user must specify at least one static partition in case the user accidentally overwrites all partitions. hive.exec.max.dynamic.partitions 1000 Maximum number of dynamic partitions allowed to be created in total. hive.exec.max.dynamic.partitions.pernode 100 Maximum number of dynamic partitions allowed to be created in each mapper/reducer node. hive.exec.max.created.files 100000 Maximum number of HDFS files created by all mappers/reducers in a MapReduce job. hive.exec.default.partition.name __HIVE_DEFAULT_PARTITION__ The default partition name in case the dynamic partition column value is null/empty string or anyother values that cannot be escaped. This value must not contain any special character used in HDFS URI (e.g., ':', '%', '/' etc). The user has to be aware that the dynamic partition value should not contain this value to avoid confusions. hive.stats.dbclass jdbc:derby The default database that stores temporary hive statistics. hive.stats.autogather true A flag to gather statistics automatically during the INSERT OVERWRITE command. hive.stats.jdbcdriver org.apache.derby.jdbc.EmbeddedDriver The JDBC driver for the database that stores temporary hive statistics. hive.stats.dbconnectionstring jdbc:derby:;databaseName=TempStatsStore;create=true The default connection string for the database that stores temporary hive statistics. hive.stats.default.publisher The Java class (implementing the StatsPublisher interface) that is used by default if hive.stats.dbclass is not JDBC or HBase. hive.stats.default.aggregator The Java class (implementing the StatsAggregator interface) that is used by default if hive.stats.dbclass is not JDBC or HBase. hive.stats.jdbc.timeout 30 Timeout value (number of seconds) used by JDBC connection and statements. hive.stats.retries.max 0 Maximum number of retries when stats publisher/aggregator got an exception updating intermediate database. Default is no tries on failures. hive.stats.retries.wait 3000 The base waiting window (in milliseconds) before the next retry. The actual wait time is calculated by baseWindow * failues baseWindow * (failure 1) * (random number between [0.0,1.0]). hive.stats.reliable false Whether queries will fail because stats cannot be collected completely accurately. If this is set to true, reading/writing from/into a partition may fail becuase the stats could not be computed accurately. hive.stats.collect.tablekeys false Whether join and group by keys on tables are derived and maintained in the QueryPlan. This is useful to identify how tables are accessed and to determine if they should be bucketed. hive.stats.collect.scancols false Whether column accesses are tracked in the QueryPlan. This is useful to identify how tables are accessed and to determine if there are wasted columns that can be trimmed. hive.stats.ndv.error 20.0 Standard error expressed in percentage. Provides a tradeoff between accuracy and compute cost.A lower value for error indicates higher accuracy and a higher compute cost. hive.stats.key.prefix.max.length 200 Determines if when the prefix of the key used for intermediate stats collection exceeds a certain length, a hash of the key is used instead. If the value < 0 then hashing is never used, if the value >= 0 then hashing is used only when the key prefixes length exceeds that value. The key prefix is defined as everything preceding the task ID in the key. hive.support.concurrency false Whether hive supports concurrency or not. A zookeeper instance must be up and running for the default hive lock manager to support read-write locks. hive.lock.numretries 100 The number of times you want to try to get all the locks hive.unlock.numretries 10 The number of times you want to retry to do one unlock hive.lock.sleep.between.retries 60 The sleep time (in seconds) between various retries hive.zookeeper.quorum The list of zookeeper servers to talk to. This is only needed for read/write locks. hive.zookeeper.client.port 2181 The port of zookeeper servers to talk to. This is only needed for read/write locks. hive.zookeeper.session.timeout 600000 Zookeeper client's session timeout. The client is disconnected, and as a result, all locks released, if a heartbeat is not sent in the timeout. hive.zookeeper.namespace hive_zookeeper_namespace The parent node under which all zookeeper nodes are created. hive.zookeeper.clean.extra.nodes false Clean extra nodes at the end of the session. fs.har.impl org.apache.hadoop.hive.shims.HiveHarFileSystem The implementation for accessing Hadoop Archives. Note that this won't be applicable to Hadoop vers less than 0.20 hive.archive.enabled false Whether archiving operations are permitted hive.fetch.output.serde org.apache.hadoop.hive.serde2.DelimitedJSONSerDe The serde used by FetchTask to serialize the fetch output. hive.exec.mode.local.auto false Let hive determine whether to run in local mode automatically hive.exec.drop.ignorenonexistent true Do not report an error if DROP TABLE/VIEW specifies a non-existent table/view hive.exec.show.job.failure.debug.info true If a job fails, whether to provide a link in the CLI to the task with the most failures, along with debugging hints if applicable. hive.auto.progress.timeout 0 How long to run autoprogressor for the script/UDTF operators (in seconds). Set to 0 for forever. hive.hbase.wal.enabled true Whether writes to HBase should be forced to the write-ahead log. Disabling this improves HBase write performance at the risk of lost writes in case of a crash. hive.table.parameters.default Default property values for newly created tables hive.entity.separator @ Separator used to construct names of tables and partitions. For example, dbname@tablename@partitionname hive.ddl.createtablelike.properties.whitelist Table Properties to copy over when executing a Create Table Like. hive.variable.substitute true This enables substitution using syntax like ${var} ${system:var} and ${env:var}. hive.variable.substitute.depth 40 The maximum replacements the substitution engine will do. hive.conf.validation true Eables type checking for registered hive configurations hive.security.authorization.enabled false enable or disable the hive client authorization hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider the hive client authorization manager class name. The user defined authorization class should implement interface org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider. hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.DefaultHiveMetastoreAuthorizationProvider authorization manager class name to be used in the metastore for authorization. The user defined authorization class should implement interface org.apache.hadoop.hive.ql.security.authorization.HiveMetastoreAuthorizationProvider. hive.security.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator hive client authenticator manager class name. The user defined authenticator should implement interface org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider. hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator authenticator manager class name to be used in the metastore for authentication. The user defined authenticator should implement interface org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider. hive.security.authorization.createtable.user.grants the privileges automatically granted to some users whenever a table gets created. An example like "userX,userY:select;userZ:create" will grant select privilege to userX and userY, and grant create privilege to userZ whenever a new table created. hive.security.authorization.createtable.group.grants the privileges automatically granted to some groups whenever a table gets created. An example like "groupX,groupY:select;groupZ:create" will grant select privilege to groupX and groupY, and grant create privilege to groupZ whenever a new table created. hive.security.authorization.createtable.role.grants the privileges automatically granted to some roles whenever a table gets created. An example like "roleX,roleY:select;roleZ:create" will grant select privilege to roleX and roleY, and grant create privilege to roleZ whenever a new table created. hive.security.authorization.createtable.owner.grants the privileges automatically granted to the owner whenever a table gets created. An example like "select,drop" will grant select and drop privilege to the owner of the table hive.metastore.authorization.storage.checks false Should the metastore do authorization checks against the underlying storage for operations like drop-partition (disallow the drop-partition if the user in question doesn't have permissions to delete the corresponding directory on the storage). hive.error.on.empty.partition false Whether to throw an excpetion if dynamic partition insert generates empty results. hive.index.compact.file.ignore.hdfs false True the hdfs location stored in the index file will be igbored at runtime. If the data got moved or the name of the cluster got changed, the index data should still be usable. hive.optimize.index.filter.compact.minsize 5368709120 Minimum size (in bytes) of the inputs on which a compact index is automatically used. hive.optimize.index.filter.compact.maxsize -1 Maximum size (in bytes) of the inputs on which a compact index is automatically used. A negative number is equivalent to infinity. hive.index.compact.query.max.size 10737418240 The maximum number of bytes that a query using the compact index can read. Negative value is equivalent to infinity. hive.index.compact.query.max.entries 10000000 The maximum number of index entries to read during a query that uses the compact index. Negative value is equivalent to infinity. hive.index.compact.binary.search true Whether or not to use a binary search to find the entries in an index table that match the filter, where possible hive.exim.uri.scheme.whitelist hdfs,pfile A comma separated list of acceptable URI schemes for import and export. hive.lock.mapred.only.operation false This param is to control whether or not only do lock on queries that need to execute at least one mapred job. hive.limit.row.max.size 100000 When trying a smaller subset of data for simple LIMIT, how much size we need to guarantee each row to have at least. hive.limit.optimize.limit.file 10 When trying a smaller subset of data for simple LIMIT, maximum number of files we can sample. hive.limit.optimize.enable false Whether to enable to optimization to trying a smaller subset of data for simple LIMIT first. hive.limit.optimize.fetch.max 50000 Maximum number of rows allowed for a smaller subset of data for simple LIMIT, if it is a fetch query. Insert queries are not restricted by this limit. hive.rework.mapredwork false should rework the mapred work or not. This is first introduced by SymlinkTextInputFormat to replace symlink files with real paths at compile time. hive.exec.concatenate.check.index true If this sets to true, hive will throw error when doing 'alter table tbl_name [partSpec] concatenate' on a table/partition that has indexes on it. The reason the user want to set this to true is because it can help user to avoid handling all index drop, recreation, rebuild work. This is very helpful for tables with thousands of partitions. hive.sample.seednumber 0 A number used to percentage sampling. By changing this number, user will change the subsets of data sampled. hive.io.exception.handlers A list of io exception handler class names. This is used to construct a list exception handlers to handle exceptions thrown by record readers hive.autogen.columnalias.prefix.label _c String used as a prefix when auto generating column alias. By default the prefix label will be appended with a column position number to form the column alias. Auto generation would happen if an aggregate function is used in a select clause without an explicit alias. hive.autogen.columnalias.prefix.includefuncname false Whether to include function name in the column alias auto generated by hive. hive.exec.perf.logger org.apache.hadoop.hive.ql.log.PerfLogger The class responsible logging client side performance metrics. Must be a subclass of org.apache.hadoop.hive.ql.log.PerfLogger hive.start.cleanup.scratchdir false To cleanup the hive scratchdir while starting the hive server hive.output.file.extension String used as a file extension for output files. If not set, defaults to the codec extension for text files (e.g. ".gz"), or no extension otherwise. hive.insert.into.multilevel.dirs false Where to insert into multilevel directories like "insert directory '/HIVEFT25686/chinna/' from table" hive.warehouse.subdir.inherit.perms false Set this to true if the the table directories should inherit the permission of the warehouse or database directory instead of being created with the permissions derived from dfs umask hive.exec.job.debug.capture.stacktraces true Whether or not stack traces parsed from the task logs of a sampled failed task for each failed job should be stored in the SessionState hive.exec.driver.run.hooks A comma separated list of hooks which implement HiveDriverRunHook and will be run at the beginning and end of Driver.run, these will be run in the order specified hive.ddl.output.format text The data format to use for DDL output. One of "text" (for human readable text) or "json" (for a json object). hive.transform.escape.input false This adds an option to escape special chars (newlines, carriage returns and tabs) when they are passed to the user script. This is useful if the hive tables can contain data that contains special characters. hive.exec.rcfile.use.explicit.header true If this is set the header for RC Files will simply be RCF. If this is not set the header will be that borrowed from sequence files, e.g. SEQ- followed by the input and output RC File formats. hive.multi.insert.move.tasks.share.dependencies false If this is set all move tasks for tables/partitions (not directories) at the end of a multi-insert query will only begin once the dependencies for all these move tasks have been met. Advantages: If concurrency is enabled, the locks will only be released once the query has finished, so with this config enabled, the time when the table/partition is generated will be much closer to when the lock on it is released. Disadvantages: If concurrency is not enabled, with this disabled, the tables/partitions which are produced by this query and finish earlier will be available for querying much earlier. Since the locks are only released once the query finishes, this does not apply if concurrency is enabled. hive.fetch.task.conversion minimal Some select queries can be converted to single FETCH task minimizing latency. Currently the query should be single sourced not having any subquery and should not have any aggregations or distincts (which incurrs RS), lateral views and joins. 1. minimal : SELECT STAR, FILTER on partition columns, LIMIT only 2. more : SELECT, FILTER, LIMIT only (TABLESAMPLE, virtual columns) hive.hmshandler.retry.attempts 1 The number of times to retry a HMSHandler call if there were a connection error hive.hmshandler.retry.interval 1000 The number of miliseconds between HMSHandler retry attempts hive.server.read.socket.timeout 10 Timeout for the HiveServer to close the connection if no response from the client in N seconds, defaults to 10 seconds. hive.server.tcp.keepalive true Whether to enable TCP keepalive for the Hive server. Keepalive will prevent accumulation of half-open connections. hive.decode.partition.name false Whether to show the unquoted partition names in query results. hive.log4j.file Hive log4j configuration file. If the property is not set, then logging will be initialized using hive-log4j.properties found on the classpath. If the property is set, the value must be a valid URI (java.net.URI, e.g. "file:///tmp/my-logging.properties"), which you can then extract a URL from and pass to PropertyConfigurator.configure(URL). hive.exec.log4j.file Hive log4j configuration file for execution mode(sub command). If the property is not set, then logging will be initialized using hive-exec-log4j.properties found on the classpath. If the property is set, the value must be a valid URI (java.net.URI, e.g. "file:///tmp/my-logging.properties"), which you can then extract a URL from and pass to PropertyConfigurator.configure(URL). hive.exec.infer.bucket.sort false If this is set, when writing partitions, the metadata will include the bucketing/sorting properties with which the data was written if any (this will not overwrite the metadata inherited from the table if the table is bucketed/sorted) hive.exec.infer.bucket.sort.num.buckets.power.two false If this is set, when setting the number of reducers for the map reduce task which writes the final output files, it will choose a number which is a power of two, unless the user specifies the number of reducers to use using mapred.reduce.tasks. The number of reducers may be set to a power of two, only to be followed by a merge task meaning preventing anything from being inferred. With hive.exec.infer.bucket.sort set to true: Advantages: If this is not set, the number of buckets for partitions will seem arbitrary, which means that the number of mappers used for optimized joins, for example, will be very low. With this set, since the number of buckets used for any partition is a power of two, the number of mappers used for optimized joins will be the least number of buckets used by any partition being joined. Disadvantages: This may mean a much larger or much smaller number of reducers being used in the final map reduce job, e.g. if a job was originally going to take 257 reducers, it will now take 512 reducers, similarly if the max number of reducers is 511, and a job was going to use this many, it will now use 256 reducers. hive.groupby.orderby.position.alias false Whether to enable using Column Position Alias in Group By or Order By hive.server2.thrift.min.worker.threads 5 Minimum number of Thrift worker threads hive.server2.thrift.max.worker.threads 100 Maximum number of Thrift worker threads hive.server2.thrift.port 10000 Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT hive.server2.thrift.bind.host localhost Bind host on which to run the HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_BIND_HOST hive.server2.authentication NONE Client authentication types. NONE: no authentication check LDAP: LDAP/AD based authentication KERBEROS: Kerberos/GSSAPI authentication CUSTOM: Custom authentication provider (Use with property hive.server2.custom.authentication.class) hive.server2.custom.authentication.class Custom authentication class. Used when property 'hive.server2.authentication' is set to 'CUSTOM'. Provided class must be a proper implementation of the interface org.apache.hive.service.auth.PasswdAuthenticationProvider. HiveServer2 will call its Authenticate(user, passed) method to authenticate requests. The implementation may optionally extend the Hadoop's org.apache.hadoop.conf.Configured class to grab Hive's Configuration object. >hive.server2.authentication.kerberos.principal Kerberos server principal >hive.server2.authentication.kerberos.keytab Kerberos keytab file for server principal hive.server2.authentication.ldap.url LDAP connection URL hive.server2.authentication.ldap.baseDN LDAP base DN hive.server2.enable.doAs true Setting this property to true will have hive server2 execute hive operations as the user making the calls to it. sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/create_hive_db.sql0000666000175100017510000000066213245514472026040 0ustar zuulzuul00000000000000CREATE DATABASE metastore; USE metastore; SOURCE /opt/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql; CREATE USER 'hive'@'localhost' IDENTIFIED BY '{{password}}'; REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hive'@'localhost'; GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'localhost' IDENTIFIED BY '{{password}}'; GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%' IDENTIFIED BY '{{password}}'; FLUSH PRIVILEGES; exit sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/core-default.xml0000666000175100017510000016735313245514472025503 0ustar zuulzuul00000000000000 hadoop.common.configuration.version 0.23.0 version of this configuration file hadoop.tmp.dir /tmp/hadoop-${user.name} A base for other temporary directories. io.native.lib.available true Controls whether to use native libraries for bz2 and zlib compression codecs or not. The property does not control any other native libraries. hadoop.http.filter.initializers org.apache.hadoop.http.lib.StaticUserWebFilter A comma separated list of class names. Each class in the list must extend org.apache.hadoop.http.FilterInitializer. The corresponding Filter will be initialized. Then, the Filter will be applied to all user facing jsp and servlet web pages. The ordering of the list defines the ordering of the filters. hadoop.security.authorization false Is service-level authorization enabled? hadoop.security.instrumentation.requires.admin false Indicates if administrator ACLs are required to access instrumentation servlets (JMX, METRICS, CONF, STACKS). hadoop.security.authentication simple Possible values are simple (no authentication), and kerberos hadoop.security.group.mapping org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback Class for user to group mapping (get groups for a given user) for ACL. The default implementation, org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, will determine if the Java Native Interface (JNI) is available. If JNI is available the implementation will use the API within hadoop to resolve a list of groups for a user. If JNI is not available then the shell implementation, ShellBasedUnixGroupsMapping, is used. This implementation shells out to the Linux/Unix environment with the bash -c groups command to resolve a list of groups for a user. hadoop.security.groups.cache.secs 300 This is the config controlling the validity of the entries in the cache containing the user->group mapping. When this duration has expired, then the implementation of the group mapping provider is invoked to get the groups of the user and then cached back. hadoop.security.groups.negative-cache.secs 30 Expiration time for entries in the the negative user-to-group mapping caching, in seconds. This is useful when invalid users are retrying frequently. It is suggested to set a small value for this expiration, since a transient error in group lookup could temporarily lock out a legitimate user. Set this to zero or negative value to disable negative user-to-group caching. hadoop.security.groups.cache.warn.after.ms 5000 If looking up a single user to group takes longer than this amount of milliseconds, we will log a warning message. hadoop.security.group.mapping.ldap.url The URL of the LDAP server to use for resolving user groups when using the LdapGroupsMapping user to group mapping. hadoop.security.group.mapping.ldap.ssl false Whether or not to use SSL when connecting to the LDAP server. hadoop.security.group.mapping.ldap.ssl.keystore File path to the SSL keystore that contains the SSL certificate required by the LDAP server. hadoop.security.group.mapping.ldap.ssl.keystore.password.file The path to a file containing the password of the LDAP SSL keystore. IMPORTANT: This file should be readable only by the Unix user running the daemons. hadoop.security.group.mapping.ldap.bind.user The distinguished name of the user to bind as when connecting to the LDAP server. This may be left blank if the LDAP server supports anonymous binds. hadoop.security.group.mapping.ldap.bind.password.file The path to a file containing the password of the bind user. IMPORTANT: This file should be readable only by the Unix user running the daemons. hadoop.security.group.mapping.ldap.base The search base for the LDAP connection. This is a distinguished name, and will typically be the root of the LDAP directory. hadoop.security.group.mapping.ldap.search.filter.user (&(objectClass=user)(sAMAccountName={0})) An additional filter to use when searching for LDAP users. The default will usually be appropriate for Active Directory installations. If connecting to an LDAP server with a non-AD schema, this should be replaced with (&(objectClass=inetOrgPerson)(uid={0}). {0} is a special string used to denote where the username fits into the filter. hadoop.security.group.mapping.ldap.search.filter.group (objectClass=group) An additional filter to use when searching for LDAP groups. This should be changed when resolving groups against a non-Active Directory installation. posixGroups are currently not a supported group class. hadoop.security.group.mapping.ldap.search.attr.member member The attribute of the group object that identifies the users that are members of the group. The default will usually be appropriate for any LDAP installation. hadoop.security.group.mapping.ldap.search.attr.group.name cn The attribute of the group object that identifies the group name. The default will usually be appropriate for all LDAP systems. hadoop.security.group.mapping.ldap.directory.search.timeout 10000 The attribute applied to the LDAP SearchControl properties to set a maximum time limit when searching and awaiting a result. Set to 0 if infinite wait period is desired. Default is 10 seconds. Units in milliseconds. hadoop.security.service.user.name.key For those cases where the same RPC protocol is implemented by multiple servers, this configuration is required for specifying the principal name to use for the service when the client wishes to make an RPC call. hadoop.security.uid.cache.secs 14400 This is the config controlling the validity of the entries in the cache containing the userId to userName and groupId to groupName used by NativeIO getFstat(). hadoop.rpc.protection authentication A comma-separated list of protection values for secured sasl connections. Possible values are authentication, integrity and privacy. authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled. hadoop.security.saslproperties.resolver.class can be used to override the hadoop.rpc.protection for a connection at the server side. hadoop.security.saslproperties.resolver.class SaslPropertiesResolver used to resolve the QOP used for a connection. If not specified, the full set of values specified in hadoop.rpc.protection is used while determining the QOP used for the connection. If a class is specified, then the QOP values returned by the class will be used while determining the QOP used for the connection. hadoop.work.around.non.threadsafe.getpwuid false Some operating systems or authentication modules are known to have broken implementations of getpwuid_r and getpwgid_r, such that these calls are not thread-safe. Symptoms of this problem include JVM crashes with a stack trace inside these functions. If your system exhibits this issue, enable this configuration parameter to include a lock around the calls as a workaround. An incomplete list of some systems known to have this issue is available at http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations hadoop.kerberos.kinit.command kinit Used to periodically renew Kerberos credentials when provided to Hadoop. The default setting assumes that kinit is in the PATH of users running the Hadoop client. Change this to the absolute path to kinit if this is not the case. hadoop.security.auth_to_local Maps kerberos principals to local user names io.file.buffer.size 4096 The size of buffer for use in sequence files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. io.bytes.per.checksum 512 The number of bytes per checksum. Must not be larger than io.file.buffer.size. io.skip.checksum.errors false If true, when a checksum error is encountered while reading a sequence file, entries are skipped, instead of throwing an exception. io.compression.codecs A comma-separated list of the compression codec classes that can be used for compression/decompression. In addition to any classes specified with this property (which take precedence), codec classes on the classpath are discovered using a Java ServiceLoader. io.compression.codec.bzip2.library system-native The native-code library to be used for compression and decompression by the bzip2 codec. This library could be specified either by by name or the full pathname. In the former case, the library is located by the dynamic linker, usually searching the directories specified in the environment variable LD_LIBRARY_PATH. The value of "system-native" indicates that the default system library should be used. To indicate that the algorithm should operate entirely in Java, specify "java-builtin". io.serializations org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization A list of serialization classes that can be used for obtaining serializers and deserializers. io.seqfile.local.dir ${hadoop.tmp.dir}/io/local The local directory where sequence file stores intermediate data files during merge. May be a comma-separated list of directories on different devices in order to spread disk i/o. Directories that do not exist are ignored. io.map.index.skip 0 Number of index entries to skip between each entry. Zero by default. Setting this to values larger than zero can facilitate opening large MapFiles using less memory. io.map.index.interval 128 MapFile consist of two files - data file (tuples) and index file (keys). For every io.map.index.interval records written in the data file, an entry (record-key, data-file-position) is written in the index file. This is to allow for doing binary search later within the index file to look up records by their keys and get their closest positions in the data file. fs.defaultFS file:/// The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem. fs.default.name file:/// Deprecated. Use (fs.defaultFS) property instead fs.trash.interval 0 Number of minutes after which the checkpoint gets deleted. If zero, the trash feature is disabled. This option may be configured both on the server and the client. If trash is disabled server side then the client side configuration is checked. If trash is enabled on the server side then the value configured on the server is used and the client configuration value is ignored. fs.trash.checkpoint.interval 0 Number of minutes between trash checkpoints. Should be smaller or equal to fs.trash.interval. If zero, the value is set to the value of fs.trash.interval. Every time the checkpointer runs it creates a new checkpoint out of current and removes checkpoints created more than fs.trash.interval minutes ago. fs.AbstractFileSystem.file.impl org.apache.hadoop.fs.local.LocalFs The AbstractFileSystem for file: uris. fs.AbstractFileSystem.har.impl org.apache.hadoop.fs.HarFs The AbstractFileSystem for har: uris. fs.AbstractFileSystem.hdfs.impl org.apache.hadoop.fs.Hdfs The FileSystem for hdfs: uris. fs.AbstractFileSystem.viewfs.impl org.apache.hadoop.fs.viewfs.ViewFs The AbstractFileSystem for view file system for viewfs: uris (ie client side mount table:). fs.AbstractFileSystem.ftp.impl org.apache.hadoop.fs.ftp.FtpFs The FileSystem for Ftp: uris. fs.ftp.host 0.0.0.0 FTP filesystem connects to this server fs.ftp.host.port 21 FTP filesystem connects to fs.ftp.host on this port fs.df.interval 60000 Disk usage statistics refresh interval in msec. fs.du.interval 600000 File space usage statistics refresh interval in msec. fs.s3.block.size 67108864 Block size to use when writing files to S3. fs.s3.buffer.dir ${hadoop.tmp.dir}/s3 Determines where on the local filesystem the S3 filesystem should store files before sending them to S3 (or after retrieving them from S3). fs.s3.maxRetries 4 The maximum number of retries for reading or writing files to S3, before we signal failure to the application. fs.s3.sleepTimeSeconds 10 The number of seconds to sleep between each S3 retry. fs.swift.impl org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem The implementation class of the OpenStack Swift Filesystem fs.automatic.close true By default, FileSystem instances are automatically closed at program exit using a JVM shutdown hook. Setting this property to false disables this behavior. This is an advanced option that should only be used by server applications requiring a more carefully orchestrated shutdown sequence. fs.s3n.block.size 67108864 Block size to use when reading files using the native S3 filesystem (s3n: URIs). fs.s3n.multipart.uploads.enabled false Setting this property to true enables multiple uploads to native S3 filesystem. When uploading a file, it is split into blocks if the size is larger than fs.s3n.multipart.uploads.block.size. fs.s3n.multipart.uploads.block.size 67108864 The block size for multipart uploads to native S3 filesystem. Default size is 64MB. fs.s3n.multipart.copy.block.size 5368709120 The block size for multipart copy in native S3 filesystem. Default size is 5GB. fs.s3n.server-side-encryption-algorithm Specify a server-side encryption algorithm for S3. The default is NULL, and the only other currently allowable value is AES256. fs.s3a.awsAccessKeyId AWS access key ID. Omit for Role-based authentication. fs.s3a.awsSecretAccessKey AWS secret key. Omit for Role-based authentication. fs.s3a.connection.maximum 15 Controls the maximum number of simultaneous connections to S3. fs.s3a.connection.ssl.enabled true Enables or disables SSL connections to S3. fs.s3a.endpoint AWS S3 endpoint to connect to. An up-to-date list is provided in the AWS Documentation: regions and endpoints. Without this property, the standard region (s3.amazonaws.com) is assumed. fs.s3a.proxy.host Hostname of the (optional) proxy server for S3 connections. fs.s3a.proxy.port Proxy server port. If this property is not set but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with the value of fs.s3a.connection.ssl.enabled). fs.s3a.proxy.username Username for authenticating with proxy server. fs.s3a.proxy.password Password for authenticating with proxy server. fs.s3a.proxy.domain Domain for authenticating with proxy server. fs.s3a.proxy.workstation Workstation for authenticating with proxy server. fs.s3a.attempts.maximum 10 How many times we should retry commands on transient errors. fs.s3a.connection.establish.timeout 5000 Socket connection setup timeout in milliseconds. fs.s3a.connection.timeout 50000 Socket connection timeout in milliseconds. fs.s3a.paging.maximum 5000 How many keys to request from S3 when doing directory listings at a time. fs.s3a.threads.max 256 Maximum number of concurrent active (part)uploads, which each use a thread from the threadpool. fs.s3a.threads.core 15 Number of core threads in the threadpool. fs.s3a.threads.keepalivetime 60 Number of seconds a thread can be idle before being terminated. fs.s3a.max.total.tasks 1000 Number of (part)uploads allowed to the queue before blocking additional uploads. fs.s3a.multipart.size 104857600 How big (in bytes) to split upload or copy operations up into. fs.s3a.multipart.threshold 2147483647 Threshold before uploads or copies use parallel multipart operations. fs.s3a.acl.default Set a canned ACL for newly created and copied objects. Value may be private, public-read, public-read-write, authenticated-read, log-delivery-write, bucket-owner-read, or bucket-owner-full-control. fs.s3a.multipart.purge false True if you want to purge existing multipart uploads that may not have been completed/aborted correctly fs.s3a.multipart.purge.age 86400 Minimum age in seconds of multipart uploads to purge fs.s3a.buffer.dir ${hadoop.tmp.dir}/s3a Comma separated list of directories that will be used to buffer file uploads to. fs.s3a.fast.upload false Upload directly from memory instead of buffering to disk first. Memory usage and parallelism can be controlled as up to fs.s3a.multipart.size memory is consumed for each (part)upload actively uploading (fs.s3a.threads.max) or queueing (fs.s3a.max.total.tasks) fs.s3a.fast.buffer.size 1048576 Size of initial memory buffer in bytes allocated for an upload. No effect if fs.s3a.fast.upload is false. fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem The implementation class of the S3A Filesystem io.seqfile.compress.blocksize 1000000 The minimum block size for compression in block compressed SequenceFiles. io.seqfile.lazydecompress true Should values of block-compressed SequenceFiles be decompressed only when necessary. io.seqfile.sorter.recordlimit 1000000 The limit on number of records to be kept in memory in a spill in SequenceFiles.Sorter io.mapfile.bloom.size 1048576 The size of BloomFilter-s used in BloomMapFile. Each time this many keys is appended the next BloomFilter will be created (inside a DynamicBloomFilter). Larger values minimize the number of filters, which slightly increases the performance, but may waste too much space if the total number of keys is usually much smaller than this number. io.mapfile.bloom.error.rate 0.005 The rate of false positives in BloomFilter-s used in BloomMapFile. As this value decreases, the size of BloomFilter-s increases exponentially. This value is the probability of encountering false positives (default is 0.5%). hadoop.util.hash.type murmur The default implementation of Hash. Currently this can take one of the two values: 'murmur' to select MurmurHash and 'jenkins' to select JenkinsHash. ipc.client.idlethreshold 4000 Defines the threshold number of connections after which connections will be inspected for idleness. ipc.client.kill.max 10 Defines the maximum number of clients to disconnect in one go. ipc.client.connection.maxidletime 10000 The maximum time in msec after which a client will bring down the connection to the server. ipc.client.connect.max.retries 10 Indicates the number of retries a client will make to establish a server connection. ipc.client.connect.retry.interval 1000 Indicates the number of milliseconds a client will wait for before retrying to establish a server connection. ipc.client.connect.timeout 20000 Indicates the number of milliseconds a client will wait for the socket to establish a server connection. ipc.client.connect.max.retries.on.timeouts 45 Indicates the number of retries a client will make on socket timeout to establish a server connection. ipc.server.listen.queue.size 128 Indicates the length of the listen queue for servers accepting client connections. hadoop.security.impersonation.provider.class A class which implements ImpersonationProvider interface, used to authorize whether one user can impersonate a specific user. If not specified, the DefaultImpersonationProvider will be used. If a class is specified, then that class will be used to determine the impersonation capability. hadoop.rpc.socket.factory.class.default org.apache.hadoop.net.StandardSocketFactory Default SocketFactory to use. This parameter is expected to be formatted as "package.FactoryClassName". hadoop.rpc.socket.factory.class.ClientProtocol SocketFactory to use to connect to a DFS. If null or empty, use hadoop.rpc.socket.class.default. This socket factory is also used by DFSClient to create sockets to DataNodes. hadoop.socks.server Address (host:port) of the SOCKS server to be used by the SocksSocketFactory. net.topology.node.switch.mapping.impl org.apache.hadoop.net.ScriptBasedMapping The default implementation of the DNSToSwitchMapping. It invokes a script specified in net.topology.script.file.name to resolve node names. If the value for net.topology.script.file.name is not set, the default value of DEFAULT_RACK is returned for all node names. net.topology.impl org.apache.hadoop.net.NetworkTopology The default implementation of NetworkTopology which is classic three layer one. net.topology.script.file.name The script name that should be invoked to resolve DNS names to NetworkTopology names. Example: the script would take host.foo.bar as an argument, and return /rack1 as the output. net.topology.script.number.args 100 The max number of args that the script configured with net.topology.script.file.name should be run with. Each arg is an IP address. net.topology.table.file.name The file name for a topology file, which is used when the net.topology.node.switch.mapping.impl property is set to org.apache.hadoop.net.TableMapping. The file format is a two column text file, with columns separated by whitespace. The first column is a DNS or IP address and the second column specifies the rack where the address maps. If no entry corresponding to a host in the cluster is found, then /default-rack is assumed. file.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. file.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than file.stream-buffer-size file.client-write-packet-size 65536 Packet size for clients to write file.blocksize 67108864 Block size file.replication 1 Replication factor s3.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. s3.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than s3.stream-buffer-size s3.client-write-packet-size 65536 Packet size for clients to write s3.blocksize 67108864 Block size s3.replication 3 Replication factor s3native.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. s3native.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than s3native.stream-buffer-size s3native.client-write-packet-size 65536 Packet size for clients to write s3native.blocksize 67108864 Block size s3native.replication 3 Replication factor ftp.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. ftp.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than ftp.stream-buffer-size ftp.client-write-packet-size 65536 Packet size for clients to write ftp.blocksize 67108864 Block size ftp.replication 3 Replication factor tfile.io.chunk.size 1048576 Value chunk size in bytes. Default to 1MB. Values of the length less than the chunk size is guaranteed to have known value length in read time (See also TFile.Reader.Scanner.Entry.isValueLengthKnown()). tfile.fs.output.buffer.size 262144 Buffer size used for FSDataOutputStream in bytes. tfile.fs.input.buffer.size 262144 Buffer size used for FSDataInputStream in bytes. hadoop.http.authentication.type simple Defines authentication used for Oozie HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# hadoop.http.authentication.token.validity 36000 Indicates how long (in seconds) an authentication token is valid before it has to be renewed. hadoop.http.authentication.signature.secret.file ${user.home}/hadoop-http-auth-signature-secret The signature secret for signing the authentication tokens. The same secret should be used for JT/NN/DN/TT configurations. hadoop.http.authentication.cookie.domain The domain to use for the HTTP cookie that stores the authentication token. In order to authentiation to work correctly across all Hadoop nodes web-consoles the domain must be correctly set. IMPORTANT: when using IP addresses, browsers ignore cookies with domain settings. For this setting to work properly all nodes in the cluster must be configured to generate URLs with hostname.domain names on it. hadoop.http.authentication.simple.anonymous.allowed true Indicates if anonymous requests are allowed when using 'simple' authentication. hadoop.http.authentication.kerberos.principal HTTP/_HOST@LOCALHOST Indicates the Kerberos principal to be used for HTTP endpoint. The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification. hadoop.http.authentication.kerberos.keytab ${user.home}/hadoop.keytab Location of the keytab file with the credentials for the principal. Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop. dfs.ha.fencing.methods List of fencing methods to use for service fencing. May contain builtin methods (eg shell and sshfence) or user-defined method. dfs.ha.fencing.ssh.connect-timeout 30000 SSH connection timeout, in milliseconds, to use with the builtin sshfence fencer. dfs.ha.fencing.ssh.private-key-files The SSH private key files to use with the builtin sshfence fencer. The user name to filter as, on static web filters while rendering content. An example use is the HDFS web UI (user to be used for browsing files). hadoop.http.staticuser.user dr.who ha.zookeeper.quorum A list of ZooKeeper server addresses, separated by commas, that are to be used by the ZKFailoverController in automatic failover. ha.zookeeper.session-timeout.ms 5000 The session timeout to use when the ZKFC connects to ZooKeeper. Setting this value to a lower value implies that server crashes will be detected more quickly, but risks triggering failover too aggressively in the case of a transient error or network blip. ha.zookeeper.parent-znode /hadoop-ha The ZooKeeper znode under which the ZK failover controller stores its information. Note that the nameservice ID is automatically appended to this znode, so it is not normally necessary to configure this, even in a federated environment. ha.zookeeper.acl world:anyone:rwcda A comma-separated list of ZooKeeper ACLs to apply to the znodes used by automatic failover. These ACLs are specified in the same format as used by the ZooKeeper CLI. If the ACL itself contains secrets, you may instead specify a path to a file, prefixed with the '@' symbol, and the value of this configuration will be loaded from within. ha.zookeeper.auth A comma-separated list of ZooKeeper authentications to add when connecting to ZooKeeper. These are specified in the same format as used by the "addauth" command in the ZK CLI. It is important that the authentications specified here are sufficient to access znodes with the ACL specified in ha.zookeeper.acl. If the auths contain secrets, you may instead specify a path to a file, prefixed with the '@' symbol, and the value of this configuration will be loaded from within. hadoop.ssl.keystores.factory.class org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory The keystores factory to use for retrieving certificates. hadoop.ssl.require.client.cert false Whether client certificates are required hadoop.ssl.hostname.verifier DEFAULT The hostname verifier to provide for HttpsURLConnections. Valid values are: DEFAULT, STRICT, STRICT_I6, DEFAULT_AND_LOCALHOST and ALLOW_ALL hadoop.ssl.server.conf ssl-server.xml Resource file from which ssl server keystore information will be extracted. This file is looked up in the classpath, typically it should be in Hadoop conf/ directory. hadoop.ssl.client.conf ssl-client.xml Resource file from which ssl client keystore information will be extracted This file is looked up in the classpath, typically it should be in Hadoop conf/ directory. hadoop.ssl.enabled false Deprecated. Use dfs.http.policy and yarn.http.policy instead. hadoop.ssl.enabled.protocols TLSv1 Protocols supported by the ssl. hadoop.jetty.logs.serve.aliases true Enable/Disable aliases serving from jetty fs.permissions.umask-mode 022 The umask used when creating files and directories. Can be in octal or in symbolic. Examples are: "022" (octal for u=rwx,g=r-x,o=r-x in symbolic), or "u=rwx,g=rwx,o=" (symbolic for 007 in octal). ha.health-monitor.connect-retry-interval.ms 1000 How often to retry connecting to the service. ha.health-monitor.check-interval.ms 1000 How often to check the service. ha.health-monitor.sleep-after-disconnect.ms 1000 How long to sleep after an unexpected RPC error. ha.health-monitor.rpc-timeout.ms 45000 Timeout for the actual monitorHealth() calls. ha.failover-controller.new-active.rpc-timeout.ms 60000 Timeout that the FC waits for the new active to become active ha.failover-controller.graceful-fence.rpc-timeout.ms 5000 Timeout that the FC waits for the old active to go to standby ha.failover-controller.graceful-fence.connection.retries 1 FC connection retries for graceful fencing ha.failover-controller.cli-check.rpc-timeout.ms 20000 Timeout that the CLI (manual) FC waits for monitorHealth, getServiceState ipc.client.fallback-to-simple-auth-allowed false When a client is configured to attempt a secure connection, but attempts to connect to an insecure server, that server may instruct the client to switch to SASL SIMPLE (unsecure) authentication. This setting controls whether or not the client will accept this instruction from the server. When false (the default), the client will not allow the fallback to SIMPLE authentication, and will abort the connection. fs.client.resolve.remote.symlinks true Whether to resolve symlinks when accessing a remote Hadoop filesystem. Setting this to false causes an exception to be thrown upon encountering a symlink. This setting does not apply to local filesystems, which automatically resolve local symlinks. nfs.exports.allowed.hosts * rw By default, the export can be mounted by any client. The value string contains machine name and access privilege, separated by whitespace characters. The machine name format can be a single host, a Java regular expression, or an IPv4 address. The access privilege uses rw or ro to specify read/write or read-only access of the machines to exports. If the access privilege is not provided, the default is read-only. Entries are separated by ";". For example: "192.168.0.0/22 rw ; host.*\.example\.com ; host1.test.org ro;". Only the NFS gateway needs to restart after this property is updated. hadoop.user.group.static.mapping.overrides dr.who=; Static mapping of user to groups. This will override the groups if available in the system for the specified user. In otherwords, groups look-up will not happen for these users, instead groups mapped in this configuration will be used. Mapping should be in this format. user1=group1,group2;user2=;user3=group2; Default, "dr.who=;" will consider "dr.who" as user without groups. rpc.metrics.quantile.enable false Setting this property to true and rpc.metrics.percentiles.intervals to a comma-separated list of the granularity in seconds, the 50/75/90/95/99th percentile latency for rpc queue/processing time in milliseconds are added to rpc metrics. rpc.metrics.percentiles.intervals A comma-separated list of the granularity in seconds for the metrics which describe the 50/75/90/95/99th percentile latency for rpc queue/processing time. The metrics are outputted if rpc.metrics.quantile.enable is set to true. hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE The prefix for a given crypto codec, contains a comma-separated list of implementation classes for a given crypto codec (eg EXAMPLECIPHERSUITE). The first implementation will be used if available, others are fallbacks. hadoop.security.crypto.codec.classes.aes.ctr.nopadding org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,org.apache.hadoop.crypto.JceAesCtrCryptoCodec Comma-separated list of crypto codec implementations for AES/CTR/NoPadding. The first implementation will be used if available, others are fallbacks. hadoop.security.crypto.cipher.suite AES/CTR/NoPadding Cipher suite for crypto codec. hadoop.security.crypto.jce.provider The JCE provider name used in CryptoCodec. hadoop.security.crypto.buffer.size 8192 The buffer size used by CryptoInputStream and CryptoOutputStream. hadoop.security.java.secure.random.algorithm SHA1PRNG The java secure random algorithm. hadoop.security.secure.random.impl Implementation of secure random. hadoop.security.random.device.file.path /dev/urandom OS security random device file path. fs.har.impl.disable.cache true Don't cache 'har' filesystem instances. hadoop.security.kms.client.authentication.retry-count 1 Number of time to retry connecting to KMS on authentication failure hadoop.security.kms.client.encrypted.key.cache.size 500 Size of the EncryptedKeyVersion cache Queue for each key hadoop.security.kms.client.encrypted.key.cache.low-watermark 0.3f If size of the EncryptedKeyVersion cache Queue falls below the low watermark, this cache queue will be scheduled for a refill hadoop.security.kms.client.encrypted.key.cache.num.refill.threads 2 Number of threads to use for refilling depleted EncryptedKeyVersion cache Queues hadoop.security.kms.client.encrypted.key.cache.expiry 43200000 Cache expiry time for a Key, after which the cache Queue for this key will be dropped. Default = 12hrs hadoop.htrace.spanreceiver.classes A comma separated list of the fully-qualified class name of classes implementing SpanReceiver. The tracing system works by collecting information in structs called 'Spans'. It is up to you to choose how you want to receive this information by implementing the SpanReceiver interface. ipc.server.max.connections 0 The maximum number of concurrent connections a server is allowed to accept. If this limit is exceeded, incoming connections will first fill the listen queue and then may go to an OS-specific listen overflow queue. The client may fail or timeout, but the server can avoid running out of file descriptors using this feature. 0 means no limit. Is the registry enabled in the YARN Resource Manager? If true, the YARN RM will, as needed. create the user and system paths, and purge service records when containers, application attempts and applications complete. If false, the paths must be created by other means, and no automatic cleanup of service records will take place. hadoop.registry.rm.enabled false The root zookeeper node for the registry hadoop.registry.zk.root /registry Zookeeper session timeout in milliseconds hadoop.registry.zk.session.timeout.ms 60000 Zookeeper connection timeout in milliseconds hadoop.registry.zk.connection.timeout.ms 15000 Zookeeper connection retry count before failing hadoop.registry.zk.retry.times 5 hadoop.registry.zk.retry.interval.ms 1000 Zookeeper retry limit in milliseconds, during exponential backoff. This places a limit even if the retry times and interval limit, combined with the backoff policy, result in a long retry period hadoop.registry.zk.retry.ceiling.ms 60000 List of hostname:port pairs defining the zookeeper quorum binding for the registry hadoop.registry.zk.quorum localhost:2181 Key to set if the registry is secure. Turning it on changes the permissions policy from "open access" to restrictions on kerberos with the option of a user adding one or more auth key pairs down their own tree. hadoop.registry.secure false A comma separated list of Zookeeper ACL identifiers with system access to the registry in a secure cluster. These are given full access to all entries. If there is an "@" at the end of a SASL entry it instructs the registry client to append the default kerberos domain. hadoop.registry.system.acls sasl:yarn@, sasl:mapred@, sasl:hdfs@ The kerberos realm: used to set the realm of system principals which do not declare their realm, and any other accounts that need the value. If empty, the default realm of the running process is used. If neither are known and the realm is needed, then the registry service/client will fail. hadoop.registry.kerberos.realm Key to define the JAAS context. Used in secure mode hadoop.registry.jaas.context Client sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/README.rst0000666000175100017510000000272013245514472024060 0ustar zuulzuul00000000000000Apache Hadoop Configurations for Sahara ======================================= This directory contains default XML configuration files: * core-default.xml * hdfs-default.xml * mapred-default.xml * yarn-default.xml * oozie-default.xml * hive-default.xml These files are applied for Sahara's plugin of Apache Hadoop version 2.7.1 Files were taken from here: * `core-default.xml `_ * `hdfs-default.xml `_ * `yarn-default.xml `_ * `mapred-default.xml `_ * `oozie-default.xml `_ * `hive-default.xml `_ XML configs are used to expose default Hadoop configurations to the users through Sahara's REST API. It allows users to override some config values which will be pushed to the provisioned VMs running Hadoop services as part of appropriate xml config. sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/yarn-default.xml0000666000175100017510000017324613245514472025522 0ustar zuulzuul00000000000000 Factory to create client IPC classes. yarn.ipc.client.factory.class Factory to create server IPC classes. yarn.ipc.server.factory.class Factory to create serializeable records. yarn.ipc.record.factory.class RPC class implementation yarn.ipc.rpc.class org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC The hostname of the RM. yarn.resourcemanager.hostname 0.0.0.0 The address of the applications manager interface in the RM. yarn.resourcemanager.address ${yarn.resourcemanager.hostname}:8032 The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This is most useful for making RM listen to all interfaces by setting to 0.0.0.0. yarn.resourcemanager.bind-host The number of threads used to handle applications manager requests. yarn.resourcemanager.client.thread-count 50 Number of threads used to launch/cleanup AM. yarn.resourcemanager.amlauncher.thread-count 50 Retry times to connect with NM. yarn.resourcemanager.nodemanager-connect-retries 10 The expiry interval for application master reporting. yarn.am.liveness-monitor.expiry-interval-ms 600000 The Kerberos principal for the resource manager. yarn.resourcemanager.principal The address of the scheduler interface. yarn.resourcemanager.scheduler.address ${yarn.resourcemanager.hostname}:8030 Number of threads to handle scheduler interface. yarn.resourcemanager.scheduler.client.thread-count 50 This configures the HTTP endpoint for Yarn Daemons.The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https yarn.http.policy HTTP_ONLY The http address of the RM web application. yarn.resourcemanager.webapp.address ${yarn.resourcemanager.hostname}:8088 The https adddress of the RM web application. yarn.resourcemanager.webapp.https.address ${yarn.resourcemanager.hostname}:8090 yarn.resourcemanager.resource-tracker.address ${yarn.resourcemanager.hostname}:8031 Are acls enabled. yarn.acl.enable false ACL of who can be admin of the YARN cluster. yarn.admin.acl * The address of the RM admin interface. yarn.resourcemanager.admin.address ${yarn.resourcemanager.hostname}:8033 Number of threads used to handle RM admin interface. yarn.resourcemanager.admin.client.thread-count 1 Maximum time to wait to establish connection to ResourceManager. yarn.resourcemanager.connect.max-wait.ms 900000 How often to try connecting to the ResourceManager. yarn.resourcemanager.connect.retry-interval.ms 30000 The maximum number of application attempts. It's a global setting for all application masters. Each application master can specify its individual maximum number of application attempts via the API, but the individual number cannot be more than the global upper bound. If it is, the resourcemanager will override it. The default number is set to 2, to allow at least one retry for AM. yarn.resourcemanager.am.max-attempts 2 How often to check that containers are still alive. yarn.resourcemanager.container.liveness-monitor.interval-ms 600000 The keytab for the resource manager. yarn.resourcemanager.keytab /etc/krb5.keytab Flag to enable override of the default kerberos authentication filter with the RM authentication filter to allow authentication using delegation tokens(fallback to kerberos if the tokens are missing). Only applicable when the http authentication type is kerberos. yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled true How long to wait until a node manager is considered dead. yarn.nm.liveness-monitor.expiry-interval-ms 600000 Path to file with nodes to include. yarn.resourcemanager.nodes.include-path Path to file with nodes to exclude. yarn.resourcemanager.nodes.exclude-path Number of threads to handle resource tracker calls. yarn.resourcemanager.resource-tracker.client.thread-count 50 The class to use as the resource scheduler. yarn.resourcemanager.scheduler.class org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this will throw a InvalidResourceRequestException. yarn.scheduler.minimum-allocation-mb 1024 The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this will throw a InvalidResourceRequestException. yarn.scheduler.maximum-allocation-mb 8192 The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this will throw a InvalidResourceRequestException. yarn.scheduler.minimum-allocation-vcores 1 The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this will throw a InvalidResourceRequestException. yarn.scheduler.maximum-allocation-vcores 32 Enable RM to recover state after starting. If true, then yarn.resourcemanager.store.class must be specified. yarn.resourcemanager.recovery.enabled false Enable RM work preserving recovery. This configuration is private to YARN for experimenting the feature. yarn.resourcemanager.work-preserving-recovery.enabled true Set the amount of time RM waits before allocating new containers on work-preserving-recovery. Such wait period gives RM a chance to settle down resyncing with NMs in the cluster on recovery, before assigning new containers to applications. yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms 10000 The class to use as the persistent store. If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore is used, the store is implicitly fenced; meaning a single ResourceManager is able to use the store at any point in time. More details on this implicit fencing, along with setting up appropriate ACLs is discussed under yarn.resourcemanager.zk-state-store.root-node.acl. yarn.resourcemanager.store.class org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore The maximum number of completed applications RM state store keeps, less than or equals to ${yarn.resourcemanager.max-completed-applications}. By default, it equals to ${yarn.resourcemanager.max-completed-applications}. This ensures that the applications kept in the state store are consistent with the applications remembered in RM memory. Any values larger than ${yarn.resourcemanager.max-completed-applications} will be reset to ${yarn.resourcemanager.max-completed-applications}. Note that this value impacts the RM recovery performance.Typically, a smaller value indicates better performance on RM recovery. yarn.resourcemanager.state-store.max-completed-applications ${yarn.resourcemanager.max-completed-applications} Host:Port of the ZooKeeper server to be used by the RM. This must be supplied when using the ZooKeeper based implementation of the RM state store and/or embedded automatic failover in a HA setting. yarn.resourcemanager.zk-address Number of times RM tries to connect to ZooKeeper. yarn.resourcemanager.zk-num-retries 1000 Retry interval in milliseconds when connecting to ZooKeeper. When HA is enabled, the value here is NOT used. It is generated automatically from yarn.resourcemanager.zk-timeout-ms and yarn.resourcemanager.zk-num-retries. yarn.resourcemanager.zk-retry-interval-ms 1000 Full path of the ZooKeeper znode where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore as the value for yarn.resourcemanager.store.class yarn.resourcemanager.zk-state-store.parent-path /rmstore ZooKeeper session timeout in milliseconds. Session expiration is managed by the ZooKeeper cluster itself, not by the client. This value is used by the cluster to determine when the client's session expires. Expirations happens when the cluster does not hear from the client within the specified session timeout period (i.e. no heartbeat). yarn.resourcemanager.zk-timeout-ms 10000 ACL's to be used for ZooKeeper znodes. yarn.resourcemanager.zk-acl world:anyone:rwcda ACLs to be used for the root znode when using ZKRMStateStore in a HA scenario for fencing. ZKRMStateStore supports implicit fencing to allow a single ResourceManager write-access to the store. For fencing, the ResourceManagers in the cluster share read-write-admin privileges on the root node, but the Active ResourceManager claims exclusive create-delete permissions. By default, when this property is not set, we use the ACLs from yarn.resourcemanager.zk-acl for shared admin access and rm-address:random-number for username-based exclusive create-delete access. This property allows users to set ACLs of their choice instead of using the default mechanism. For fencing to work, the ACLs should be carefully set differently on each ResourceManger such that all the ResourceManagers have shared admin access and the Active ResourceManger takes over (exclusively) the create-delete access. yarn.resourcemanager.zk-state-store.root-node.acl Specify the auths to be used for the ACL's specified in both the yarn.resourcemanager.zk-acl and yarn.resourcemanager.zk-state-store.root-node.acl properties. This takes a comma-separated list of authentication mechanisms, each of the form 'scheme:auth' (the same syntax used for the 'addAuth' command in the ZK CLI). yarn.resourcemanager.zk-auth URI pointing to the location of the FileSystem path where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore as the value for yarn.resourcemanager.store.class yarn.resourcemanager.fs.state-store.uri ${hadoop.tmp.dir}/yarn/system/rmstore hdfs client retry policy specification. hdfs client retry is always enabled. Specified in pairs of sleep-time and number-of-retries and (t0, n0), (t1, n1), ..., the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on. yarn.resourcemanager.fs.state-store.retry-policy-spec 2000, 500 the number of retries to recover from IOException in FileSystemRMStateStore. yarn.resourcemanager.fs.state-store.num-retries 0 Retry interval in milliseconds in FileSystemRMStateStore. yarn.resourcemanager.fs.state-store.retry-interval-ms 1000 Local path where the RM state will be stored when using org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore as the value for yarn.resourcemanager.store.class yarn.resourcemanager.leveldb-state-store.path ${hadoop.tmp.dir}/yarn/system/rmstore Enable RM high-availability. When enabled, (1) The RM starts in the Standby mode by default, and transitions to the Active mode when prompted to. (2) The nodes in the RM ensemble are listed in yarn.resourcemanager.ha.rm-ids (3) The id of each RM either comes from yarn.resourcemanager.ha.id if yarn.resourcemanager.ha.id is explicitly specified or can be figured out by matching yarn.resourcemanager.address.{id} with local address (4) The actual physical addresses come from the configs of the pattern - {rpc-config}.{id} yarn.resourcemanager.ha.enabled false Enable automatic failover. By default, it is enabled only when HA is enabled yarn.resourcemanager.ha.automatic-failover.enabled true Enable embedded automatic failover. By default, it is enabled only when HA is enabled. The embedded elector relies on the RM state store to handle fencing, and is primarily intended to be used in conjunction with ZKRMStateStore. yarn.resourcemanager.ha.automatic-failover.embedded true The base znode path to use for storing leader information, when using ZooKeeper based leader election. yarn.resourcemanager.ha.automatic-failover.zk-base-path /yarn-leader-election Name of the cluster. In a HA setting, this is used to ensure the RM participates in leader election for this cluster and ensures it does not affect other clusters yarn.resourcemanager.cluster-id The list of RM nodes in the cluster when HA is enabled. See description of yarn.resourcemanager.ha .enabled for full details on how this is used. yarn.resourcemanager.ha.rm-ids The id (string) of the current RM. When HA is enabled, this is an optional config. The id of current RM can be set by explicitly specifying yarn.resourcemanager.ha.id or figured out by matching yarn.resourcemanager.address.{id} with local address See description of yarn.resourcemanager.ha.enabled for full details on how this is used. yarn.resourcemanager.ha.id When HA is enabled, the class to be used by Clients, AMs and NMs to failover to the Active RM. It should extend org.apache.hadoop.yarn.client.RMFailoverProxyProvider yarn.client.failover-proxy-provider org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider When HA is enabled, the max number of times FailoverProxyProvider should attempt failover. When set, this overrides the yarn.resourcemanager.connect.max-wait.ms. When not set, this is inferred from yarn.resourcemanager.connect.max-wait.ms. yarn.client.failover-max-attempts When HA is enabled, the sleep base (in milliseconds) to be used for calculating the exponential delay between failovers. When set, this overrides the yarn.resourcemanager.connect.* settings. When not set, yarn.resourcemanager.connect.retry-interval.ms is used instead. yarn.client.failover-sleep-base-ms When HA is enabled, the maximum sleep time (in milliseconds) between failovers. When set, this overrides the yarn.resourcemanager.connect.* settings. When not set, yarn.resourcemanager.connect.retry-interval.ms is used instead. yarn.client.failover-sleep-max-ms When HA is enabled, the number of retries per attempt to connect to a ResourceManager. In other words, it is the ipc.client.connect.max.retries to be used during failover attempts yarn.client.failover-retries 0 When HA is enabled, the number of retries per attempt to connect to a ResourceManager on socket timeouts. In other words, it is the ipc.client.connect.max.retries.on.timeouts to be used during failover attempts yarn.client.failover-retries-on-socket-timeouts 0 The maximum number of completed applications RM keeps. yarn.resourcemanager.max-completed-applications 10000 Interval at which the delayed token removal thread runs yarn.resourcemanager.delayed.delegation-token.removal-interval-ms 30000 If true, ResourceManager will have proxy-user privileges. Use case: In a secure cluster, YARN requires the user hdfs delegation-tokens to do localization and log-aggregation on behalf of the user. If this is set to true, ResourceManager is able to request new hdfs delegation tokens on behalf of the user. This is needed by long-running-service, because the hdfs tokens will eventually expire and YARN requires new valid tokens to do localization and log-aggregation. Note that to enable this use case, the corresponding HDFS NameNode has to configure ResourceManager as the proxy-user so that ResourceManager can itself ask for new tokens on behalf of the user when tokens are past their max-life-time. yarn.resourcemanager.proxy-user-privileges.enabled false Interval for the roll over for the master key used to generate application tokens yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs 86400 Interval for the roll over for the master key used to generate container tokens. It is expected to be much greater than yarn.nm.liveness-monitor.expiry-interval-ms and yarn.resourcemanager.rm.container-allocation.expiry-interval-ms. Otherwise the behavior is undefined. yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs 86400 The heart-beat interval in milliseconds for every NodeManager in the cluster. yarn.resourcemanager.nodemanagers.heartbeat-interval-ms 1000 The minimum allowed version of a connecting nodemanager. The valid values are NONE (no version checking), EqualToRM (the nodemanager's version is equal to or greater than the RM version), or a Version String. yarn.resourcemanager.nodemanager.minimum.version NONE Enable a set of periodic monitors (specified in yarn.resourcemanager.scheduler.monitor.policies) that affect the scheduler. yarn.resourcemanager.scheduler.monitor.enable false The list of SchedulingEditPolicy classes that interact with the scheduler. A particular module may be incompatible with the scheduler, other policies, or a configuration of either. yarn.resourcemanager.scheduler.monitor.policies org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy The class to use as the configuration provider. If org.apache.hadoop.yarn.LocalConfigurationProvider is used, the local configuration will be loaded. If org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider is used, the configuration which will be loaded should be uploaded to remote File system first. yarn.resourcemanager.configuration.provider-class org.apache.hadoop.yarn.LocalConfigurationProvider The setting that controls whether yarn system metrics is published on the timeline server or not by RM. yarn.resourcemanager.system-metrics-publisher.enabled false Number of worker threads that send the yarn system metrics data. yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size 10 The hostname of the NM. yarn.nodemanager.hostname 0.0.0.0 The address of the container manager in the NM. yarn.nodemanager.address ${yarn.nodemanager.hostname}:0 The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is most useful for making NM listen to all interfaces by setting to 0.0.0.0. yarn.nodemanager.bind-host Environment variables that should be forwarded from the NodeManager's environment to the container's. yarn.nodemanager.admin-env MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX Environment variables that containers may override rather than use NodeManager's default. yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME who will execute(launch) the containers. yarn.nodemanager.container-executor.class org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor Number of threads container manager uses. yarn.nodemanager.container-manager.thread-count 20 Number of threads used in cleanup. yarn.nodemanager.delete.thread-count 4 Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory. To diagnose Yarn application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications' log directories is configurable with the yarn.nodemanager.log-dirs property (see also below). yarn.nodemanager.delete.debug-delay-sec 0 Keytab for NM. yarn.nodemanager.keytab /etc/krb5.keytab List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this. yarn.nodemanager.local-dirs ${hadoop.tmp.dir}/nm-local-dir It limits the maximum number of files which will be localized in a single local directory. If the limit is reached then sub-directories will be created and new files will be localized in them. If it is set to a value less than or equal to 36 [which are sub-directories (0-9 and then a-z)] then NodeManager will fail to start. For example; [for public cache] if this is configured with a value of 40 ( 4 files + 36 sub-directories) and the local-dir is "/tmp/local-dir1" then it will allow 4 files to be created directly inside "/tmp/local-dir1/filecache". For files that are localized further it will create a sub-directory "0" inside "/tmp/local-dir1/filecache" and will localize files inside it until it becomes full. If a file is removed from a sub-directory that is marked full, then that sub-directory will be used back again to localize files. yarn.nodemanager.local-cache.max-files-per-directory 8192 Address where the localizer IPC is. yarn.nodemanager.localizer.address ${yarn.nodemanager.hostname}:8040 Interval in between cache cleanups. yarn.nodemanager.localizer.cache.cleanup.interval-ms 600000 Target size of localizer cache in MB, per nodemanager. It is a target retention size that only includes resources with PUBLIC and PRIVATE visibility and excludes resources with APPLICATION visibility yarn.nodemanager.localizer.cache.target-size-mb 10240 Number of threads to handle localization requests. yarn.nodemanager.localizer.client.thread-count 5 Number of threads to use for localization fetching. yarn.nodemanager.localizer.fetch.thread-count 4 Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container. yarn.nodemanager.log-dirs ${yarn.log.dir}/userlogs Whether to enable log aggregation. Log aggregation collects each container's logs and moves these logs onto a file-system, for e.g. HDFS, after the application completes. Users can configure the "yarn.nodemanager.remote-app-log-dir" and "yarn.nodemanager.remote-app-log-dir-suffix" properties to determine where these logs are moved to. Users can access the logs via the Application Timeline Server. yarn.log-aggregation-enable false How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node. yarn.log-aggregation.retain-seconds -1 How long to wait between aggregated log retention checks. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful set this too small and you will spam the name node. yarn.log-aggregation.retain-check-interval-seconds -1 Time in seconds to retain user logs. Only applicable if log aggregation is disabled yarn.nodemanager.log.retain-seconds 10800 Where to aggregate logs to. yarn.nodemanager.remote-app-log-dir /tmp/logs The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam} yarn.nodemanager.remote-app-log-dir-suffix logs Amount of physical memory, in MB, that can be allocated for containers. yarn.nodemanager.resource.memory-mb 8192 Whether physical memory limits will be enforced for containers. yarn.nodemanager.pmem-check-enabled true Whether virtual memory limits will be enforced for containers. yarn.nodemanager.vmem-check-enabled true Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio. yarn.nodemanager.vmem-pmem-ratio 2.1 Number of vcores that can be allocated for containers. This is used by the RM scheduler when allocating resources for containers. This is not used to limit the number of physical cores used by YARN containers. yarn.nodemanager.resource.cpu-vcores 8 Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU. yarn.nodemanager.resource.percentage-physical-cpu-limit 100 NM Webapp address. yarn.nodemanager.webapp.address ${yarn.nodemanager.hostname}:8042 How often to monitor containers. yarn.nodemanager.container-monitor.interval-ms 3000 Class that calculates containers current resource utilization. yarn.nodemanager.container-monitor.resource-calculator.class Frequency of running node health script. yarn.nodemanager.health-checker.interval-ms 600000 Script time out period. yarn.nodemanager.health-checker.script.timeout-ms 1200000 The health check script to run. yarn.nodemanager.health-checker.script.path The arguments to pass to the health check script. yarn.nodemanager.health-checker.script.opts Frequency of running disk health checker code. yarn.nodemanager.disk-health-checker.interval-ms 120000 The minimum fraction of number of disks to be healthy for the nodemanager to launch new containers. This correspond to both yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there are less number of healthy local-dirs (or log-dirs) available, then new containers will not be launched on this node. yarn.nodemanager.disk-health-checker.min-healthy-disks 0.25 The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the nodemanager will check for full disk. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage 90.0 The minimum space that must be available on a disk for it to be used. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb 0 The path to the Linux container executor. yarn.nodemanager.linux-container-executor.path The class which should help the LCE handle resources. yarn.nodemanager.linux-container-executor.resources-handler.class org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler The cgroups hierarchy under which to place YARN proccesses (cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured), then this cgroups hierarchy must already exist and be writable by the NodeManager user, otherwise the NodeManager may fail. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. yarn.nodemanager.linux-container-executor.cgroups.hierarchy /hadoop-yarn Whether the LCE should attempt to mount cgroups if not found. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. yarn.nodemanager.linux-container-executor.cgroups.mount false Where the LCE should attempt to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux distribution in use. This path must exist before the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and yarn.nodemanager.linux-container-executor.cgroups.mount is true. yarn.nodemanager.linux-container-executor.cgroups.mount-path This determines which of the two modes that LCE should use on a non-secure cluster. If this value is set to true, then all containers will be launched as the user specified in yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If this value is set to false, then containers will run as the user who submitted the application. yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users true The UNIX user that containers will run as when Linux-container-executor is used in nonsecure mode (a use case for this is using cgroups) if the yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users is set to true. yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user nobody The allowed pattern for UNIX user names enforced by Linux-container-executor when used in nonsecure mode (use case for this is using cgroups). The default value is taken from /usr/sbin/adduser yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern ^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$ This flag determines whether apps should run with strict resource limits or be allowed to consume spare resources if they need them. For example, turning the flag on will restrict apps to use only their share of CPU, even if the node has spare CPU cycles. The default value is false i.e. use available resources. Please note that turning this flag on may reduce job throughput on the cluster. yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage false This flag determines whether memory limit will be set for the Windows Job Object of the containers launched by the default container executor. yarn.nodemanager.windows-container.memory-limit.enabled false This flag determines whether CPU limit will be set for the Windows Job Object of the containers launched by the default container executor. yarn.nodemanager.windows-container.cpu-limit.enabled false T-file compression types used to compress aggregated logs. yarn.nodemanager.log-aggregation.compression-type none The kerberos principal for the node manager. yarn.nodemanager.principal A comma separated list of services where service name should only contain a-zA-Z0-9_ and can not start with numbers yarn.nodemanager.aux-services No. of ms to wait between sending a SIGTERM and SIGKILL to a container yarn.nodemanager.sleep-delay-before-sigkill.ms 250 Max time to wait for a process to come up when trying to cleanup a container yarn.nodemanager.process-kill-wait.ms 2000 The minimum allowed version of a resourcemanager that a nodemanager will connect to. The valid values are NONE (no version checking), EqualToNM (the resourcemanager's version is equal to or greater than the NM version), or a Version String. yarn.nodemanager.resourcemanager.minimum.version NONE Max number of threads in NMClientAsync to process container management events yarn.client.nodemanager-client-async.thread-pool-max-size 500 Max time to wait to establish a connection to NM yarn.client.nodemanager-connect.max-wait-ms 180000 Time interval between each attempt to connect to NM yarn.client.nodemanager-connect.retry-interval-ms 10000 Maximum number of proxy connections to cache for node managers. If set to a value greater than zero then the cache is enabled and the NMClient and MRAppMaster will cache the specified number of node manager proxies. There will be at max one proxy per node manager. Ex. configuring it to a value of 5 will make sure that client will at max have 5 proxies cached with 5 different node managers. These connections for these proxies will be timed out if idle for more than the system wide idle timeout period. Note that this could cause issues on large clusters as many connections could linger simultaneously and lead to a large number of connection threads. The token used for authentication will be used only at connection creation time. If a new token is received then the earlier connection should be closed in order to use the new token. This and (yarn.client.nodemanager-client-async.thread-pool-max-size) are related and should be in sync (no need for them to be equal). If the value of this property is zero then the connection cache is disabled and connections will use a zero idle timeout to prevent too many connection threads on large clusters. yarn.client.max-cached-nodemanagers-proxies 0 Enable the node manager to recover after starting yarn.nodemanager.recovery.enabled false The local filesystem directory in which the node manager will store state when recovery is enabled. yarn.nodemanager.recovery.dir ${hadoop.tmp.dir}/yarn-nm-recovery yarn.nodemanager.docker-container-executor.exec-name /usr/bin/docker Name or path to the Docker client. yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler mapreduce.job.jar mapreduce.job.hdfs-servers ${fs.defaultFS} The kerberos principal for the proxy, if the proxy is not running as part of the RM. yarn.web-proxy.principal Keytab for WebAppProxy, if the proxy is not running as part of the RM. yarn.web-proxy.keytab The address for the web proxy as HOST:PORT, if this is not given then the proxy will run as part of the RM yarn.web-proxy.address CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries. When this value is empty, the following default CLASSPATH for YARN applications would be used. For Linux: $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, $HADOOP_YARN_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/* For Windows: %HADOOP_CONF_DIR%, %HADOOP_COMMON_HOME%/share/hadoop/common/*, %HADOOP_COMMON_HOME%/share/hadoop/common/lib/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/lib/* yarn.application.classpath Indicate to clients whether timeline service is enabled or not. If enabled, clients will put entities and events to the timeline server. yarn.timeline-service.enabled false The hostname of the timeline service web application. yarn.timeline-service.hostname 0.0.0.0 This is default address for the timeline server to start the RPC server. yarn.timeline-service.address ${yarn.timeline-service.hostname}:10200 The http address of the timeline service web application. yarn.timeline-service.webapp.address ${yarn.timeline-service.hostname}:8188 The https address of the timeline service web application. yarn.timeline-service.webapp.https.address ${yarn.timeline-service.hostname}:8190 The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively. This is most useful for making the service listen to all interfaces by setting to 0.0.0.0. yarn.timeline-service.bind-host Store class name for timeline store. yarn.timeline-service.store-class org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore Enable age off of timeline store data. yarn.timeline-service.ttl-enable true Time to live for timeline store data in milliseconds. yarn.timeline-service.ttl-ms 604800000 Store file name for leveldb timeline store. yarn.timeline-service.leveldb-timeline-store.path ${hadoop.tmp.dir}/yarn/timeline Length of time to wait between deletion cycles of leveldb timeline store in milliseconds. yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms 300000 Size of read cache for uncompressed blocks for leveldb timeline store in bytes. yarn.timeline-service.leveldb-timeline-store.read-cache-size 104857600 Size of cache for recently read entity start times for leveldb timeline store in number of entities. yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size 10000 Size of cache for recently written entity start times for leveldb timeline store in number of entities. yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size 10000 Handler thread count to serve the client RPC requests. yarn.timeline-service.handler-thread-count 10 yarn.timeline-service.http-authentication.type simple Defines authentication used for the timeline server HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# yarn.timeline-service.http-authentication.simple.anonymous.allowed true Indicates if anonymous requests are allowed by the timeline server when using 'simple' authentication. The Kerberos principal for the timeline server. yarn.timeline-service.principal The Kerberos keytab for the timeline server. yarn.timeline-service.keytab /etc/krb5.keytab Default maximum number of retires for timeline servive client and value -1 means no limit. yarn.timeline-service.client.max-retries 30 Client policy for whether timeline operations are non-fatal yarn.timeline-service.client.best-effort false Default retry time interval for timeline servive client. yarn.timeline-service.client.retry-interval-ms 1000 Enable timeline server to recover state after starting. If true, then yarn.timeline-service.state-store-class must be specified. yarn.timeline-service.recovery.enabled false Store class name for timeline state store. yarn.timeline-service.state-store-class org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore Store file name for leveldb state store. yarn.timeline-service.leveldb-state-store.path ${hadoop.tmp.dir}/yarn/timeline Whether the shared cache is enabled yarn.sharedcache.enabled false The root directory for the shared cache yarn.sharedcache.root-dir /sharedcache The level of nested directories before getting to the checksum directories. It must be non-negative. yarn.sharedcache.nested-level 3 The implementation to be used for the SCM store yarn.sharedcache.store.class org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore The implementation to be used for the SCM app-checker yarn.sharedcache.app-checker.class org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker A resource in the in-memory store is considered stale if the time since the last reference exceeds the staleness period. This value is specified in minutes. yarn.sharedcache.store.in-memory.staleness-period-mins 10080 Initial delay before the in-memory store runs its first check to remove dead initial applications. Specified in minutes. yarn.sharedcache.store.in-memory.initial-delay-mins 10 The frequency at which the in-memory store checks to remove dead initial applications. Specified in minutes. yarn.sharedcache.store.in-memory.check-period-mins 720 The address of the admin interface in the SCM (shared cache manager) yarn.sharedcache.admin.address 0.0.0.0:8047 The number of threads used to handle SCM admin interface (1 by default) yarn.sharedcache.admin.thread-count 1 The address of the web application in the SCM (shared cache manager) yarn.sharedcache.webapp.address 0.0.0.0:8788 The frequency at which a cleaner task runs. Specified in minutes. yarn.sharedcache.cleaner.period-mins 1440 Initial delay before the first cleaner task is scheduled. Specified in minutes. yarn.sharedcache.cleaner.initial-delay-mins 10 The time to sleep between processing each shared cache resource. Specified in milliseconds. yarn.sharedcache.cleaner.resource-sleep-ms 0 The address of the node manager interface in the SCM (shared cache manager) yarn.sharedcache.uploader.server.address 0.0.0.0:8046 The number of threads used to handle shared cache manager requests from the node manager (50 by default) yarn.sharedcache.uploader.server.thread-count 50 The address of the client interface in the SCM (shared cache manager) yarn.sharedcache.client-server.address 0.0.0.0:8045 The number of threads used to handle shared cache manager requests from clients (50 by default) yarn.sharedcache.client-server.thread-count 50 The algorithm used to compute checksums of files (SHA-256 by default) yarn.sharedcache.checksum.algo.impl org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl The replication factor for the node manager uploader for the shared cache (10 by default) yarn.sharedcache.nm.uploader.replication.factor 10 The number of threads used to upload files from a node manager instance (20 by default) yarn.sharedcache.nm.uploader.thread-count 20 The interval that the yarn client library uses to poll the completion status of the asynchronous API of application client protocol. yarn.client.application-client-protocol.poll-interval-ms 200 RSS usage of a process computed via /proc/pid/stat is not very accurate as it includes shared pages of a process. /proc/pid/smaps provides useful information like Private_Dirty, Private_Clean, Shared_Dirty, Shared_Clean which can be used for computing more accurate RSS. When this flag is enabled, RSS is computed as Min(Shared_Dirty, Pss) + Private_Clean + Private_Dirty. It excludes read-only shared mappings in RSS computation. yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled false Defines how often NMs wake up to upload log files. The default value is -1. By default, the logs will be uploaded when the application is finished. By setting this configure, logs can be uploaded periodically when the application is running. The minimum rolling-interval-seconds can be set is 3600. yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds -1 sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/mapred-default.xml0000666000175100017510000022436013245514472026013 0ustar zuulzuul00000000000000 mapreduce.jobtracker.jobhistory.location If job tracker is static the history files are stored in this single well known place. If No value is set here, by default, it is in the local file system at ${hadoop.log.dir}/history. mapreduce.jobtracker.jobhistory.task.numberprogresssplits 12 Every task attempt progresses from 0.0 to 1.0 [unless it fails or is killed]. We record, for each task attempt, certain statistics over each twelfth of the progress range. You can change the number of intervals we divide the entire range of progress into by setting this property. Higher values give more precision to the recorded data, but costs more memory in the job tracker at runtime. Each increment in this attribute costs 16 bytes per running task. mapreduce.job.userhistorylocation User can specify a location to store the history files of a particular job. If nothing is specified, the logs are stored in output directory. The files are stored in "_logs/history/" in the directory. User can stop logging by giving the value "none". mapreduce.jobtracker.jobhistory.completed.location The completed job history files are stored at this single well known location. If nothing is specified, the files are stored at ${mapreduce.jobtracker.jobhistory.location}/done. mapreduce.job.committer.setup.cleanup.needed true true, if job needs job-setup and job-cleanup. false, otherwise mapreduce.task.io.sort.factor 10 The number of streams to merge at once while sorting files. This determines the number of open file handles. mapreduce.task.io.sort.mb 100 The total amount of buffer memory to use while sorting files, in megabytes. By default, gives each merge stream 1MB, which should minimize seeks. mapreduce.map.sort.spill.percent 0.80 The soft limit in the serialization buffer. Once reached, a thread will begin to spill the contents to disk in the background. Note that collection will not block if this threshold is exceeded while a spill is already in progress, so spills may be larger than this threshold when it is set to less than .5 mapreduce.jobtracker.address local The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. mapreduce.local.clientfactory.class.name org.apache.hadoop.mapred.LocalClientFactory This the client factory that is responsible for creating local job runner client mapreduce.jobtracker.http.address 0.0.0.0:50030 The job tracker http server address and port the server will listen on. If the port is 0 then the server will start on a free port. mapreduce.jobtracker.handler.count 10 The number of server threads for the JobTracker. This should be roughly 4% of the number of tasktracker nodes. mapreduce.tasktracker.report.address 127.0.0.1:0 The interface and port that task tracker server listens on. Since it is only connected to by the tasks, it uses the local interface. EXPERT ONLY. Should only be changed if your host does not have the loopback interface. mapreduce.cluster.local.dir ${hadoop.tmp.dir}/mapred/local The local directory where MapReduce stores intermediate data files. May be a comma-separated list of directories on different devices in order to spread disk i/o. Directories that do not exist are ignored. mapreduce.jobtracker.system.dir ${hadoop.tmp.dir}/mapred/system The directory where MapReduce stores control files. mapreduce.jobtracker.staging.root.dir ${hadoop.tmp.dir}/mapred/staging The root of the staging area for users' job files In practice, this should be the directory where users' home directories are located (usually /user) mapreduce.cluster.temp.dir ${hadoop.tmp.dir}/mapred/temp A shared directory for temporary files. mapreduce.tasktracker.local.dir.minspacestart 0 If the space in mapreduce.cluster.local.dir drops under this, do not ask for more tasks. Value in bytes. mapreduce.tasktracker.local.dir.minspacekill 0 If the space in mapreduce.cluster.local.dir drops under this, do not ask more tasks until all the current ones have finished and cleaned up. Also, to save the rest of the tasks we have running, kill one of them, to clean up some space. Start with the reduce tasks, then go with the ones that have finished the least. Value in bytes. mapreduce.jobtracker.expire.trackers.interval 600000 Expert: The time-interval, in miliseconds, after which a tasktracker is declared 'lost' if it doesn't send heartbeats. mapreduce.tasktracker.instrumentation org.apache.hadoop.mapred.TaskTrackerMetricsInst Expert: The instrumentation class to associate with each TaskTracker. mapreduce.tasktracker.resourcecalculatorplugin Name of the class whose instance will be used to query resource information on the tasktracker. The class must be an instance of org.apache.hadoop.util.ResourceCalculatorPlugin. If the value is null, the tasktracker attempts to use a class appropriate to the platform. Currently, the only platform supported is Linux. mapreduce.tasktracker.taskmemorymanager.monitoringinterval 5000 The interval, in milliseconds, for which the tasktracker waits between two cycles of monitoring its tasks' memory usage. Used only if tasks' memory management is enabled via mapred.tasktracker.tasks.maxmemory. mapreduce.tasktracker.tasks.sleeptimebeforesigkill 5000 The time, in milliseconds, the tasktracker waits for sending a SIGKILL to a task, after it has been sent a SIGTERM. This is currently not used on WINDOWS where tasks are just sent a SIGTERM. mapreduce.job.maps 2 The default number of map tasks per job. Ignored when mapreduce.jobtracker.address is "local". mapreduce.job.reduces 1 The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapreduce.jobtracker.address is "local". mapreduce.jobtracker.restart.recover false "true" to enable (job) recovery upon restart, "false" to start afresh mapreduce.jobtracker.jobhistory.block.size 3145728 The block size of the job history file. Since the job recovery uses job history, its important to dump job history to disk as soon as possible. Note that this is an expert level parameter. The default value is set to 3 MB. mapreduce.jobtracker.taskscheduler org.apache.hadoop.mapred.JobQueueTaskScheduler The class responsible for scheduling the tasks. mapreduce.job.running.map.limit 0 The maximum number of simultaneous map tasks per job. There is no limit if this value is 0 or negative. mapreduce.job.running.reduce.limit 0 The maximum number of simultaneous reduce tasks per job. There is no limit if this value is 0 or negative. mapreduce.job.reducer.preempt.delay.sec 0 The threshold in terms of seconds after which an unsatisfied mapper request triggers reducer preemption to free space. Default 0 implies that the reduces should be preempted immediately after allocation if there is currently no room for newly allocated mappers. mapreduce.job.max.split.locations 10 The max number of block locations to store for each split for locality calculation. mapreduce.job.split.metainfo.maxsize 10000000 The maximum permissible size of the split metainfo file. The JobTracker won't attempt to read split metainfo files bigger than the configured value. No limits if set to -1. mapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob The maximum number of running tasks for a job before it gets preempted. No limits if undefined. mapreduce.map.maxattempts 4 Expert: The maximum number of attempts per map task. In other words, framework will try to execute a map task these many number of times before giving up on it. mapreduce.reduce.maxattempts 4 Expert: The maximum number of attempts per reduce task. In other words, framework will try to execute a reduce task these many number of times before giving up on it. mapreduce.reduce.shuffle.fetch.retry.enabled ${yarn.nodemanager.recovery.enabled} Set to enable fetch retry during host restart. mapreduce.reduce.shuffle.fetch.retry.interval-ms 1000 Time of interval that fetcher retry to fetch again when some non-fatal failure happens because of some events like NM restart. mapreduce.reduce.shuffle.fetch.retry.timeout-ms 30000 Timeout value for fetcher to retry to fetch again when some non-fatal failure happens because of some events like NM restart. mapreduce.reduce.shuffle.retry-delay.max.ms 60000 The maximum number of ms the reducer will delay before retrying to download map data. mapreduce.reduce.shuffle.parallelcopies 5 The default number of parallel transfers run by reduce during the copy(shuffle) phase. mapreduce.reduce.shuffle.connect.timeout 180000 Expert: The maximum amount of time (in milli seconds) reduce task spends in trying to connect to a tasktracker for getting map output. mapreduce.reduce.shuffle.read.timeout 180000 Expert: The maximum amount of time (in milli seconds) reduce task waits for map output data to be available for reading after obtaining connection. mapreduce.shuffle.connection-keep-alive.enable false set to true to support keep-alive connections. mapreduce.shuffle.connection-keep-alive.timeout 5 The number of seconds a shuffle client attempts to retain http connection. Refer "Keep-Alive: timeout=" header in Http specification mapreduce.task.timeout 600000 The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string. A value of 0 disables the timeout. mapreduce.tasktracker.map.tasks.maximum 2 The maximum number of map tasks that will be run simultaneously by a task tracker. mapreduce.tasktracker.reduce.tasks.maximum 2 The maximum number of reduce tasks that will be run simultaneously by a task tracker. mapreduce.map.memory.mb 1024 The amount of memory to request from the scheduler for each map task. mapreduce.map.cpu.vcores 1 The number of virtual cores to request from the scheduler for each map task. mapreduce.reduce.memory.mb 1024 The amount of memory to request from the scheduler for each reduce task. mapreduce.reduce.cpu.vcores 1 The number of virtual cores to request from the scheduler for each reduce task. mapreduce.jobtracker.retiredjobs.cache.size 1000 The number of retired job status to keep in the cache. mapreduce.tasktracker.outofband.heartbeat false Expert: Set this to true to let the tasktracker send an out-of-band heartbeat on task-completion for better latency. mapreduce.jobtracker.jobhistory.lru.cache.size 5 The number of job history files loaded in memory. The jobs are loaded when they are first accessed. The cache is cleared based on LRU. mapreduce.jobtracker.instrumentation org.apache.hadoop.mapred.JobTrackerMetricsInst Expert: The instrumentation class to associate with each JobTracker. mapred.child.java.opts -Xmx200m Java opts for the task processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. mapred.child.env User added environment variables for the task processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit nodemanager's B env variable on Unix. 3) B=%B%;c This is inherit nodemanager's B env variable on Windows. mapreduce.admin.user.env Expert: Additional execution environment entries for map and reduce task processes. This is not an additive property. You must preserve the original value if you want your map and reduce tasks to have access to native libraries (compression, etc). When this value is empty, the command to set execution envrionment will be OS dependent: For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native. For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin. mapreduce.map.log.level INFO The logging level for the map task. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting here could be overridden if "mapreduce.job.log4j-properties-file" is set. mapreduce.reduce.log.level INFO The logging level for the reduce task. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting here could be overridden if "mapreduce.job.log4j-properties-file" is set. mapreduce.map.cpu.vcores 1 The number of virtual cores required for each map task. mapreduce.reduce.cpu.vcores 1 The number of virtual cores required for each reduce task. mapreduce.reduce.merge.inmem.threshold 1000 The threshold, in terms of the number of files for the in-memory merge process. When we accumulate threshold number of files we initiate the in-memory merge and spill to disk. A value of 0 or less than 0 indicates we want to DON'T have any threshold and instead depend only on the ramfs's memory consumption to trigger the merge. mapreduce.reduce.shuffle.merge.percent 0.66 The usage threshold at which an in-memory merge will be initiated, expressed as a percentage of the total memory allocated to storing in-memory map outputs, as defined by mapreduce.reduce.shuffle.input.buffer.percent. mapreduce.reduce.shuffle.input.buffer.percent 0.70 The percentage of memory to be allocated from the maximum heap size to storing map outputs during the shuffle. mapreduce.reduce.input.buffer.percent 0.0 The percentage of memory- relative to the maximum heap size- to retain map outputs during the reduce. When the shuffle is concluded, any remaining map outputs in memory must consume less than this threshold before the reduce can begin. mapreduce.reduce.shuffle.memory.limit.percent 0.25 Expert: Maximum percentage of the in-memory limit that a single shuffle can consume mapreduce.shuffle.ssl.enabled false Whether to use SSL for for the Shuffle HTTP endpoints. mapreduce.shuffle.ssl.file.buffer.size 65536 Buffer size for reading spills from file when using SSL. mapreduce.shuffle.max.connections 0 Max allowed connections for the shuffle. Set to 0 (zero) to indicate no limit on the number of connections. mapreduce.shuffle.max.threads 0 Max allowed threads for serving shuffle connections. Set to zero to indicate the default of 2 times the number of available processors (as reported by Runtime.availableProcessors()). Netty is used to serve requests, so a thread is not needed for each connection. mapreduce.shuffle.transferTo.allowed This option can enable/disable using nio transferTo method in the shuffle phase. NIO transferTo does not perform well on windows in the shuffle phase. Thus, with this configuration property it is possible to disable it, in which case custom transfer method will be used. Recommended value is false when running Hadoop on Windows. For Linux, it is recommended to set it to true. If nothing is set then the default value is false for Windows, and true for Linux. mapreduce.shuffle.transfer.buffer.size 131072 This property is used only if mapreduce.shuffle.transferTo.allowed is set to false. In that case, this property defines the size of the buffer used in the buffer copy code for the shuffle phase. The size of this buffer determines the size of the IO requests. mapreduce.reduce.markreset.buffer.percent 0.0 The percentage of memory -relative to the maximum heap size- to be used for caching values when using the mark-reset functionality. mapreduce.map.speculative true If true, then multiple instances of some map tasks may be executed in parallel. mapreduce.reduce.speculative true If true, then multiple instances of some reduce tasks may be executed in parallel. mapreduce.job.speculative.speculative-cap-running-tasks 0.1 The max percent (0-1) of running tasks that can be speculatively re-executed at any time. mapreduce.job.speculative.speculative-cap-total-tasks 0.01 The max percent (0-1) of all tasks that can be speculatively re-executed at any time. mapreduce.job.speculative.minimum-allowed-tasks 10 The minimum allowed tasks that can be speculatively re-executed at any time. mapreduce.job.speculative.retry-after-no-speculate 1000 The waiting time(ms) to do next round of speculation if there is no task speculated in this round. mapreduce.job.speculative.retry-after-speculate 15000 The waiting time(ms) to do next round of speculation if there are tasks speculated in this round. mapreduce.job.map.output.collector.class org.apache.hadoop.mapred.MapTask$MapOutputBuffer The MapOutputCollector implementation(s) to use. This may be a comma-separated list of class names, in which case the map task will try to initialize each of the collectors in turn. The first to successfully initialize will be used. mapreduce.job.speculative.slowtaskthreshold 1.0 The number of standard deviations by which a task's ave progress-rates must be lower than the average of all running tasks' for the task to be considered too slow. mapreduce.job.jvm.numtasks 1 How many tasks to run per jvm. If set to -1, there is no limit. mapreduce.job.ubertask.enable false Whether to enable the small-jobs "ubertask" optimization, which runs "sufficiently small" jobs sequentially within a single JVM. "Small" is defined by the following maxmaps, maxreduces, and maxbytes settings. Note that configurations for application masters also affect the "Small" definition - yarn.app.mapreduce.am.resource.mb must be larger than both mapreduce.map.memory.mb and mapreduce.reduce.memory.mb, and yarn.app.mapreduce.am.resource.cpu-vcores must be larger than both mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores to enable ubertask. Users may override this value. mapreduce.job.ubertask.maxmaps 9 Threshold for number of maps, beyond which job is considered too big for the ubertasking optimization. Users may override this value, but only downward. mapreduce.job.ubertask.maxreduces 1 Threshold for number of reduces, beyond which job is considered too big for the ubertasking optimization. CURRENTLY THE CODE CANNOT SUPPORT MORE THAN ONE REDUCE and will ignore larger values. (Zero is a valid max, however.) Users may override this value, but only downward. mapreduce.job.ubertask.maxbytes Threshold for number of input bytes, beyond which job is considered too big for the ubertasking optimization. If no value is specified, dfs.block.size is used as a default. Be sure to specify a default value in mapred-site.xml if the underlying filesystem is not HDFS. Users may override this value, but only downward. mapreduce.job.emit-timeline-data false Specifies if the Application Master should emit timeline data to the timeline server. Individual jobs can override this value. mapreduce.input.fileinputformat.split.minsize 0 The minimum size chunk that map input should be split into. Note that some file formats may have minimum split sizes that take priority over this setting. mapreduce.input.fileinputformat.list-status.num-threads 1 The number of threads to use to list and fetch block locations for the specified input paths. Note: multiple threads should not be used if a custom non thread-safe path filter is used. mapreduce.jobtracker.maxtasks.perjob -1 The maximum number of tasks for a single job. A value of -1 indicates that there is no maximum. mapreduce.input.lineinputformat.linespermap 1 When using NLineInputFormat, the number of lines of input data to include in each split. mapreduce.client.submit.file.replication 10 The replication level for submitted job files. This should be around the square root of the number of nodes. mapreduce.tasktracker.dns.interface default The name of the Network Interface from which a task tracker should report its IP address. mapreduce.tasktracker.dns.nameserver default The host name or IP address of the name server (DNS) which a TaskTracker should use to determine the host name used by the JobTracker for communication and display purposes. mapreduce.tasktracker.http.threads 40 The number of worker threads that for the http server. This is used for map output fetching mapreduce.tasktracker.http.address 0.0.0.0:50060 The task tracker http server address and port. If the port is 0 then the server will start on a free port. mapreduce.task.files.preserve.failedtasks false Should the files for failed tasks be kept. This should only be used on jobs that are failing, because the storage is never reclaimed. It also prevents the map outputs from being erased from the reduce directory as they are consumed. mapreduce.output.fileoutputformat.compress false Should the job outputs be compressed? mapreduce.output.fileoutputformat.compress.type RECORD If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK. mapreduce.output.fileoutputformat.compress.codec org.apache.hadoop.io.compress.DefaultCodec If the job outputs are compressed, how should they be compressed? mapreduce.map.output.compress false Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression. mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.DefaultCodec If the map outputs are compressed, how should they be compressed? map.sort.class org.apache.hadoop.util.QuickSort The default sort class for sorting keys. mapreduce.task.userlog.limit.kb 0 The maximum size of user-logs of each task in KB. 0 disables the cap. yarn.app.mapreduce.am.container.log.limit.kb 0 The maximum size of the MRAppMaster attempt container logs in KB. 0 disables the cap. yarn.app.mapreduce.task.container.log.backups 0 Number of backup files for task logs when using ContainerRollingLogAppender (CRLA). See org.apache.log4j.RollingFileAppender.maxBackupIndex. By default, ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA is enabled for tasks when both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.task.container.log.backups are greater than zero. yarn.app.mapreduce.am.container.log.backups 0 Number of backup files for the ApplicationMaster logs when using ContainerRollingLogAppender (CRLA). See org.apache.log4j.RollingFileAppender.maxBackupIndex. By default, ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA is enabled for the ApplicationMaster when both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.am.container.log.backups are greater than zero. yarn.app.mapreduce.shuffle.log.separate true If enabled ('true') logging generated by the client-side shuffle classes in a reducer will be written in a dedicated log file 'syslog.shuffle' instead of 'syslog'. yarn.app.mapreduce.shuffle.log.limit.kb 0 Maximum size of the syslog.shuffle file in kilobytes (0 for no limit). yarn.app.mapreduce.shuffle.log.backups 0 If yarn.app.mapreduce.shuffle.log.limit.kb and yarn.app.mapreduce.shuffle.log.backups are greater than zero then a ContainerRollngLogAppender is used instead of ContainerLogAppender for syslog.shuffle. See org.apache.log4j.RollingFileAppender.maxBackupIndex mapreduce.job.userlog.retain.hours 24 The maximum time, in hours, for which the user-logs are to be retained after the job completion. mapreduce.jobtracker.hosts.filename Names a file that contains the list of nodes that may connect to the jobtracker. If the value is empty, all hosts are permitted. mapreduce.jobtracker.hosts.exclude.filename Names a file that contains the list of hosts that should be excluded by the jobtracker. If the value is empty, no hosts are excluded. mapreduce.jobtracker.heartbeats.in.second 100 Expert: Approximate number of heart-beats that could arrive at JobTracker in a second. Assuming each RPC can be processed in 10msec, the default value is made 100 RPCs in a second. mapreduce.jobtracker.tasktracker.maxblacklists 4 The number of blacklists for a taskTracker by various jobs after which the task tracker could be blacklisted across all jobs. The tracker will be given a tasks later (after a day). The tracker will become a healthy tracker after a restart. mapreduce.job.maxtaskfailures.per.tracker 3 The number of task-failures on a tasktracker of a given job after which new tasks of that job aren't assigned to it. It MUST be less than mapreduce.map.maxattempts and mapreduce.reduce.maxattempts otherwise the failed task will never be tried on a different node. mapreduce.client.output.filter FAILED The filter for controlling the output of the task's userlogs sent to the console of the JobClient. The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and ALL. mapreduce.client.completion.pollinterval 5000 The interval (in milliseconds) between which the JobClient polls the JobTracker for updates about job status. You may want to set this to a lower value to make tests run faster on a single node system. Adjusting this value in production may lead to unwanted client-server traffic. mapreduce.client.progressmonitor.pollinterval 1000 The interval (in milliseconds) between which the JobClient reports status to the console and checks for job completion. You may want to set this to a lower value to make tests run faster on a single node system. Adjusting this value in production may lead to unwanted client-server traffic. mapreduce.jobtracker.persist.jobstatus.active true Indicates if persistency of job status information is active or not. mapreduce.jobtracker.persist.jobstatus.hours 1 The number of hours job status information is persisted in DFS. The job status information will be available after it drops of the memory queue and between jobtracker restarts. With a zero value the job status information is not persisted at all in DFS. mapreduce.jobtracker.persist.jobstatus.dir /jobtracker/jobsInfo The directory where the job status information is persisted in a file system to be available after it drops of the memory queue and between jobtracker restarts. mapreduce.task.profile false To set whether the system should collect profiler information for some of the tasks in this job? The information is stored in the user log directory. The value is "true" if task profiling is enabled. mapreduce.task.profile.maps 0-2 To set the ranges of map tasks to profile. mapreduce.task.profile has to be set to true for the value to be accounted. mapreduce.task.profile.reduces 0-2 To set the ranges of reduce tasks to profile. mapreduce.task.profile has to be set to true for the value to be accounted. mapreduce.task.profile.params -agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s JVM profiler parameters used to profile map and reduce task attempts. This string may contain a single format specifier %s that will be replaced by the path to profile.out in the task attempt log directory. To specify different profiling options for map tasks and reduce tasks, more specific parameters mapreduce.task.profile.map.params and mapreduce.task.profile.reduce.params should be used. mapreduce.task.profile.map.params ${mapreduce.task.profile.params} Map-task-specific JVM profiler parameters. See mapreduce.task.profile.params mapreduce.task.profile.reduce.params ${mapreduce.task.profile.params} Reduce-task-specific JVM profiler parameters. See mapreduce.task.profile.params mapreduce.task.skip.start.attempts 2 The number of Task attempts AFTER which skip mode will be kicked off. When skip mode is kicked off, the tasks reports the range of records which it will process next, to the TaskTracker. So that on failures, TT knows which ones are possibly the bad records. On further executions, those are skipped. mapreduce.map.skip.proc.count.autoincr true The flag which if set to true, SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented by MapRunner after invoking the map function. This value must be set to false for applications which process the records asynchronously or buffer the input records. For example streaming. In such cases applications should increment this counter on their own. mapreduce.reduce.skip.proc.count.autoincr true The flag which if set to true, SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented by framework after invoking the reduce function. This value must be set to false for applications which process the records asynchronously or buffer the input records. For example streaming. In such cases applications should increment this counter on their own. mapreduce.job.skip.outdir If no value is specified here, the skipped records are written to the output directory at _logs/skip. User can stop writing skipped records by giving the value "none". mapreduce.map.skip.maxrecords 0 The number of acceptable skip records surrounding the bad record PER bad record in mapper. The number includes the bad record as well. To turn the feature of detection/skipping of bad records off, set the value to 0. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to indicate that framework need not try to narrow down. Whatever records(depends on application) get skipped are acceptable. mapreduce.reduce.skip.maxgroups 0 The number of acceptable skip groups surrounding the bad group PER bad group in reducer. The number includes the bad group as well. To turn the feature of detection/skipping of bad groups off, set the value to 0. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to indicate that framework need not try to narrow down. Whatever groups(depends on application) get skipped are acceptable. mapreduce.ifile.readahead true Configuration key to enable/disable IFile readahead. mapreduce.ifile.readahead.bytes 4194304 Configuration key to set the IFile readahead length in bytes. mapreduce.jobtracker.taskcache.levels 2 This is the max level of the task cache. For example, if the level is 2, the tasks cached are at the host level and at the rack level. mapreduce.job.queuename default Queue to which a job is submitted. This must match one of the queues defined in mapred-queues.xml for the system. Also, the ACL setup for the queue must allow the current user to submit a job to the queue. Before specifying a queue, ensure that the system is configured with the queue, and access is allowed for submitting jobs to the queue. mapreduce.job.tags Tags for the job that will be passed to YARN at submission time. Queries to YARN for applications can filter on these tags. mapreduce.cluster.acls.enabled false Specifies whether ACLs should be checked for authorization of users for doing various queue and job level operations. ACLs are disabled by default. If enabled, access control checks are made by JobTracker and TaskTracker when requests are made by users for queue operations like submit job to a queue and kill a job in the queue and job operations like viewing the job-details (See mapreduce.job.acl-view-job) or for modifying the job (See mapreduce.job.acl-modify-job) using Map/Reduce APIs, RPCs or via the console and web user interfaces. For enabling this flag(mapreduce.cluster.acls.enabled), this is to be set to true in mapred-site.xml on JobTracker node and on all TaskTracker nodes. mapreduce.job.acl-modify-job Job specific access-control list for 'modifying' the job. It is only used if authorization is enabled in Map/Reduce by setting the configuration property mapreduce.cluster.acls.enabled to true. This specifies the list of users and/or groups who can do modification operations on the job. For specifying a list of users and groups the format to use is "user1,user2 group1,group". If set to '*', it allows all users/groups to modify this job. If set to ' '(i.e. space), it allows none. This configuration is used to guard all the modifications with respect to this job and takes care of all the following operations: o killing this job o killing a task of this job, failing a task of this job o setting the priority of this job Each of these operations are also protected by the per-queue level ACL "acl-administer-jobs" configured via mapred-queues.xml. So a caller should have the authorization to satisfy either the queue-level ACL or the job-level ACL. Irrespective of this ACL configuration, (a) job-owner, (b) the user who started the cluster, (c) members of an admin configured supergroup configured via mapreduce.cluster.permissions.supergroup and (d) queue administrators of the queue to which this job was submitted to configured via acl-administer-jobs for the specific queue in mapred-queues.xml can do all the modification operations on a job. By default, nobody else besides job-owner, the user who started the cluster, members of supergroup and queue administrators can perform modification operations on a job. mapreduce.job.acl-view-job Job specific access-control list for 'viewing' the job. It is only used if authorization is enabled in Map/Reduce by setting the configuration property mapreduce.cluster.acls.enabled to true. This specifies the list of users and/or groups who can view private details about the job. For specifying a list of users and groups the format to use is "user1,user2 group1,group". If set to '*', it allows all users/groups to modify this job. If set to ' '(i.e. space), it allows none. This configuration is used to guard some of the job-views and at present only protects APIs that can return possibly sensitive information of the job-owner like o job-level counters o task-level counters o tasks' diagnostic information o task-logs displayed on the TaskTracker web-UI and o job.xml showed by the JobTracker's web-UI Every other piece of information of jobs is still accessible by any other user, for e.g., JobStatus, JobProfile, list of jobs in the queue, etc. Irrespective of this ACL configuration, (a) job-owner, (b) the user who started the cluster, (c) members of an admin configured supergroup configured via mapreduce.cluster.permissions.supergroup and (d) queue administrators of the queue to which this job was submitted to configured via acl-administer-jobs for the specific queue in mapred-queues.xml can do all the view operations on a job. By default, nobody else besides job-owner, the user who started the cluster, memebers of supergroup and queue administrators can perform view operations on a job. mapreduce.tasktracker.indexcache.mb 10 The maximum memory that a task tracker allows for the index cache that is used when serving map outputs to reducers. mapreduce.job.token.tracking.ids.enabled false Whether to write tracking ids of tokens to job-conf. When true, the configuration property "mapreduce.job.token.tracking.ids" is set to the token-tracking-ids of the job mapreduce.job.token.tracking.ids When mapreduce.job.token.tracking.ids.enabled is set to true, this is set by the framework to the token-tracking-ids used by the job. mapreduce.task.merge.progress.records 10000 The number of records to process during merge before sending a progress notification to the TaskTracker. mapreduce.task.combine.progress.records 10000 The number of records to process during combine output collection before sending a progress notification. mapreduce.job.reduce.slowstart.completedmaps 0.05 Fraction of the number of maps in the job which should be complete before reduces are scheduled for the job. mapreduce.job.complete.cancel.delegation.tokens true if false - do not unregister/cancel delegation tokens from renewal, because same tokens may be used by spawned jobs mapreduce.tasktracker.taskcontroller org.apache.hadoop.mapred.DefaultTaskController TaskController which is used to launch and manage task execution mapreduce.tasktracker.group Expert: Group to which TaskTracker belongs. If LinuxTaskController is configured via mapreduce.tasktracker.taskcontroller, the group owner of the task-controller binary should be same as this group. mapreduce.shuffle.port 13562 Default port that the ShuffleHandler will run on. ShuffleHandler is a service run at the NodeManager to facilitate transfers of intermediate Map outputs to requesting Reducers. mapreduce.job.reduce.shuffle.consumer.plugin.class org.apache.hadoop.mapreduce.task.reduce.Shuffle Name of the class whose instance will be used to send shuffle requests by reducetasks of this job. The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin. mapreduce.tasktracker.healthchecker.script.path Absolute path to the script which is periodicallyrun by the node health monitoring service to determine if the node is healthy or not. If the value of this key is empty or the file does not exist in the location configured here, the node health monitoring service is not started. mapreduce.tasktracker.healthchecker.interval 60000 Frequency of the node health script to be run, in milliseconds mapreduce.tasktracker.healthchecker.script.timeout 600000 Time after node health script should be killed if unresponsive and considered that the script has failed. mapreduce.tasktracker.healthchecker.script.args List of arguments which are to be passed to node health script when it is being launched comma seperated. mapreduce.job.counters.limit 120 Limit on the number of user counters allowed per job. mapreduce.framework.name local The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn. yarn.app.mapreduce.am.staging-dir /tmp/hadoop-yarn/staging The staging dir used while submitting jobs. mapreduce.am.max-attempts 2 The maximum number of application attempts. It is a application-specific setting. It should not be larger than the global number set by resourcemanager. Otherwise, it will be override. The default number is set to 2, to allow at least one retry for AM. mapreduce.job.end-notification.url Indicates url which will be called on completion of job to inform end status of job. User can give at most 2 variables with URI : $jobId and $jobStatus. If they are present in URI, then they will be replaced by their respective values. mapreduce.job.end-notification.retry.attempts 0 The number of times the submitter of the job wants to retry job end notification if it fails. This is capped by mapreduce.job.end-notification.max.attempts mapreduce.job.end-notification.retry.interval 1000 The number of milliseconds the submitter of the job wants to wait before job end notification is retried if it fails. This is capped by mapreduce.job.end-notification.max.retry.interval mapreduce.job.end-notification.max.attempts 5 true The maximum number of times a URL will be read for providing job end notification. Cluster administrators can set this to limit how long after end of a job, the Application Master waits before exiting. Must be marked as final to prevent users from overriding this. mapreduce.job.log4j-properties-file Used to override the default settings of log4j in container-log4j.properties for NodeManager. Like container-log4j.properties, it requires certain framework appenders properly defined in this overriden file. The file on the path will be added to distributed cache and classpath. If no-scheme is given in the path, it defaults to point to a log4j file on the local FS. mapreduce.job.end-notification.max.retry.interval 5000 true The maximum amount of time (in milliseconds) to wait before retrying job end notification. Cluster administrators can set this to limit how long the Application Master waits before exiting. Must be marked as final to prevent users from overriding this. yarn.app.mapreduce.am.env User added environment variables for the MR App Master processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit tasktracker's B env variable. yarn.app.mapreduce.am.admin.user.env Environment variables for the MR App Master processes for admin purposes. These values are set first and can be overridden by the user env (yarn.app.mapreduce.am.env) Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit app master's B env variable. yarn.app.mapreduce.am.command-opts -Xmx1024m Java opts for the MR App Master processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. yarn.app.mapreduce.am.admin-command-opts Java opts for the MR App Master processes for admin purposes. It will appears before the opts set by yarn.app.mapreduce.am.command-opts and thus its options can be overridden user. Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. yarn.app.mapreduce.am.job.task.listener.thread-count 30 The number of threads used to handle RPC calls in the MR AppMaster from remote tasks yarn.app.mapreduce.am.job.client.port-range Range of ports that the MapReduce AM can use when binding. Leave blank if you want all possible ports. For example 50000-50050,50100-50200 yarn.app.mapreduce.am.job.committer.cancel-timeout 60000 The amount of time in milliseconds to wait for the output committer to cancel an operation if the job is killed yarn.app.mapreduce.am.job.committer.commit-window 10000 Defines a time window in milliseconds for output commit operations. If contact with the RM has occurred within this window then commits are allowed, otherwise the AM will not allow output commits until contact with the RM has been re-established. mapreduce.fileoutputcommitter.algorithm.version 1 The file output committer algorithm version valid algorithm version number: 1 or 2 default to 1, which is the original algorithm In algorithm version 1, 1. commitTask will rename directory $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/ to $joboutput/_temporary/$appAttemptID/$taskID/ 2. recoverTask will also do a rename $joboutput/_temporary/$appAttemptID/$taskID/ to $joboutput/_temporary/($appAttemptID + 1)/$taskID/ 3. commitJob will merge every task output file in $joboutput/_temporary/$appAttemptID/$taskID/ to $joboutput/, then it will delete $joboutput/_temporary/ and write $joboutput/_SUCCESS It has a performance regression, which is discussed in MAPREDUCE-4815. If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. the commit is single-threaded and waits until all tasks have completed before commencing. algorithm version 2 will change the behavior of commitTask, recoverTask, and commitJob. 1. commitTask will rename all files in $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/ to $joboutput/ 2. recoverTask actually doesn't require to do anything, but for upgrade from version 1 to version 2 case, it will check if there are any files in $joboutput/_temporary/($appAttemptID - 1)/$taskID/ and rename them to $joboutput/ 3. commitJob can simply delete $joboutput/_temporary and write $joboutput/_SUCCESS This algorithm will reduce the output commit time for large jobs by having the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms 1000 The interval in ms at which the MR AppMaster should send heartbeats to the ResourceManager yarn.app.mapreduce.client-am.ipc.max-retries 3 The number of client retries to the AM - before reconnecting to the RM to fetch Application Status. yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts 3 The number of client retries on socket timeouts to the AM - before reconnecting to the RM to fetch Application Status. yarn.app.mapreduce.client.max-retries 3 The number of client retries to the RM/HS before throwing exception. This is a layer above the ipc. yarn.app.mapreduce.am.resource.mb 1536 The amount of memory the MR AppMaster needs. yarn.app.mapreduce.am.resource.cpu-vcores 1 The number of virtual CPU cores the MR AppMaster needs. yarn.app.mapreduce.am.hard-kill-timeout-ms 10000 Number of milliseconds to wait before the job client kills the application. yarn.app.mapreduce.client.job.max-retries 0 The number of retries the client will make for getJob and dependent calls. The default is 0 as this is generally only needed for non-HDFS DFS where additional, high level retries are required to avoid spurious failures during the getJob call. 30 is a good value for WASB yarn.app.mapreduce.client.job.retry-interval 2000 The delay between getJob retries in ms for retries configured with yarn.app.mapreduce.client.job.max-retries. CLASSPATH for MR applications. A comma-separated list of CLASSPATH entries. If mapreduce.application.framework is set then this must specify the appropriate classpath for that archive, and the name of the archive must be present in the classpath. If mapreduce.app-submission.cross-platform is false, platform-specific environment vairable expansion syntax would be used to construct the default CLASSPATH entries. For Linux: $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*. For Windows: %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*, %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*. If mapreduce.app-submission.cross-platform is true, platform-agnostic default CLASSPATH for MR applications would be used: {{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*, {{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/* Parameter expansion marker will be replaced by NodeManager on container launch based on the underlying OS accordingly. mapreduce.application.classpath If enabled, user can submit an application cross-platform i.e. submit an application from a Windows client to a Linux/Unix server or vice versa. mapreduce.app-submission.cross-platform false Path to the MapReduce framework archive. If set, the framework archive will automatically be distributed along with the job, and this path would normally reside in a public location in an HDFS filesystem. As with distributed cache files, this can be a URL with a fragment specifying the alias to use for the archive name. For example, hdfs:/mapred/framework/hadoop-mapreduce-2.1.1.tar.gz#mrframework would alias the localized archive as "mrframework". Note that mapreduce.application.classpath must include the appropriate classpath for the specified framework. The base name of the archive, or alias of the archive if an alias is used, must appear in the specified classpath. mapreduce.application.framework.path mapreduce.job.classloader false Whether to use a separate (isolated) classloader for user classes in the task JVM. mapreduce.job.classloader.system.classes Used to override the default definition of the system classes for the job classloader. The system classes are a comma-separated list of patterns that indicate whether to load a class from the system classpath, instead from the user-supplied JARs, when mapreduce.job.classloader is enabled. A positive pattern is defined as: 1. A single class name 'C' that matches 'C' and transitively all nested classes 'C$*' defined in C; 2. A package name ending with a '.' (e.g., "com.example.") that matches all classes from that package. A negative pattern is defined by a '-' in front of a positive pattern (e.g., "-com.example."). A class is considered a system class if and only if it matches one of the positive patterns and none of the negative ones. More formally: A class is a member of the inclusion set I if it matches one of the positive patterns. A class is a member of the exclusion set E if it matches one of the negative patterns. The set of system classes S = I \ E. mapreduce.jobhistory.address 0.0.0.0:10020 MapReduce JobHistory Server IPC host:port mapreduce.jobhistory.webapp.address 0.0.0.0:19888 MapReduce JobHistory Server Web UI host:port mapreduce.jobhistory.keytab Location of the kerberos keytab file for the MapReduce JobHistory Server. /etc/security/keytab/jhs.service.keytab mapreduce.jobhistory.principal Kerberos principal name for the MapReduce JobHistory Server. jhs/_HOST@REALM.TLD mapreduce.jobhistory.intermediate-done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate mapreduce.jobhistory.done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done mapreduce.jobhistory.cleaner.enable true mapreduce.jobhistory.cleaner.interval-ms 86400000 How often the job history cleaner checks for files to delete, in milliseconds. Defaults to 86400000 (one day). Files are only deleted if they are older than mapreduce.jobhistory.max-age-ms. mapreduce.jobhistory.max-age-ms 604800000 Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week). mapreduce.jobhistory.client.thread-count 10 The number of threads to handle client API requests mapreduce.jobhistory.datestring.cache.size 200000 Size of the date string cache. Effects the number of directories which will be scanned to find a job. mapreduce.jobhistory.joblist.cache.size 20000 Size of the job list cache mapreduce.jobhistory.loadedjobs.cache.size 5 Size of the loaded job cache mapreduce.jobhistory.move.interval-ms 180000 Scan for history files to more from intermediate done dir to done dir at this frequency. mapreduce.jobhistory.move.thread-count 3 The number of threads used to move files. mapreduce.jobhistory.store.class The HistoryStorage class to use to cache history data. mapreduce.jobhistory.minicluster.fixed.ports false Whether to use fixed ports with the minicluster mapreduce.jobhistory.admin.address 0.0.0.0:10033 The address of the History server admin interface. mapreduce.jobhistory.admin.acl * ACL of who can be admin of the History server. mapreduce.jobhistory.recovery.enable false Enable the history server to store server state and recover server state upon startup. If enabled then mapreduce.jobhistory.recovery.store.class must be specified. mapreduce.jobhistory.recovery.store.class org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService The HistoryServerStateStoreService class to store history server state for recovery. mapreduce.jobhistory.recovery.store.fs.uri ${hadoop.tmp.dir}/mapred/history/recoverystore The URI where history server state will be stored if HistoryServerFileSystemStateStoreService is configured as the recovery storage class. mapreduce.jobhistory.recovery.store.leveldb.path ${hadoop.tmp.dir}/mapred/history/recoverystore The URI where history server state will be stored if HistoryServerLeveldbSystemStateStoreService is configured as the recovery storage class. mapreduce.jobhistory.http.policy HTTP_ONLY This configures the HTTP endpoint for JobHistoryServer web UI. The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size 10 The initial size of thread pool to launch containers in the app master. sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/hdfs-default.xml0000666000175100017510000023710113245514472025464 0ustar zuulzuul00000000000000 hadoop.hdfs.configuration.version 1 version of this configuration file dfs.namenode.rpc-address RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port. dfs.namenode.rpc-bind-host The actual address the RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.rpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0. dfs.namenode.servicerpc-address RPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port. If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default. dfs.namenode.servicerpc-bind-host The actual address the service RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.servicerpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0. dfs.namenode.secondary.http-address 0.0.0.0:50090 The secondary namenode http server address and port. dfs.namenode.secondary.https-address 0.0.0.0:50091 The secondary namenode HTTPS server address and port. dfs.datanode.address 0.0.0.0:50010 The datanode server address and port for data transfer. dfs.datanode.http.address 0.0.0.0:50075 The datanode http server address and port. dfs.datanode.ipc.address 0.0.0.0:50020 The datanode ipc server address and port. dfs.datanode.handler.count 10 The number of server threads for the datanode. dfs.namenode.http-address 0.0.0.0:50070 The address and the base port where the dfs namenode web ui will listen on. dfs.namenode.http-bind-host The actual adress the HTTP server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.http-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node HTTP server listen on all interfaces by setting it to 0.0.0.0. dfs.namenode.heartbeat.recheck-interval 300000 This time decides the interval to check for expired datanodes. With this value and dfs.heartbeat.interval, the interval of deciding the datanode is stale or not is also calculated. The unit of this configuration is millisecond. dfs.http.policy HTTP_ONLY Decide if HTTPS(SSL) is supported on HDFS This configures the HTTP endpoint for HDFS daemons: The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https - HTTP_AND_HTTPS : Service is provided both on http and https dfs.client.https.need-auth false Whether SSL client certificate authentication is required dfs.client.cached.conn.retry 3 The number of times the HDFS client will pull a socket from the cache. Once this number is exceeded, the client will try to create a new socket. dfs.https.server.keystore.resource ssl-server.xml Resource file from which ssl server keystore information will be extracted dfs.client.https.keystore.resource ssl-client.xml Resource file from which ssl client keystore information will be extracted dfs.datanode.https.address 0.0.0.0:50475 The datanode secure http server address and port. dfs.namenode.https-address 0.0.0.0:50470 The namenode secure http server address and port. dfs.namenode.https-bind-host The actual adress the HTTPS server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.https-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node HTTPS server listen on all interfaces by setting it to 0.0.0.0. dfs.datanode.dns.interface default The name of the Network Interface from which a data node should report its IP address. dfs.datanode.dns.nameserver default The host name or IP address of the name server (DNS) which a DataNode should use to determine the host name used by the NameNode for communication and display purposes. dfs.namenode.backup.address 0.0.0.0:50100 The backup node server address and port. If the port is 0 then the server will start on a free port. dfs.namenode.backup.http-address 0.0.0.0:50105 The backup node http server address and port. If the port is 0 then the server will start on a free port. dfs.namenode.replication.considerLoad true Decide if chooseTarget considers the target's load or not dfs.default.chunk.view.size 32768 The number of bytes to view for a file on the browser. dfs.datanode.du.reserved 0 Reserved space in bytes per volume. Always leave this much space free for non dfs use. dfs.namenode.name.dir file://${hadoop.tmp.dir}/dfs/name Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. dfs.namenode.name.dir.restore false Set to true to enable NameNode to attempt recovering a previously failed dfs.namenode.name.dir. When enabled, a recovery of any failed directory is attempted during checkpoint. dfs.namenode.fs-limits.max-component-length 255 Defines the maximum number of bytes in UTF-8 encoding in each component of a path. A value of 0 will disable the check. dfs.namenode.fs-limits.max-directory-items 1048576 Defines the maximum number of items that a directory may contain. Cannot set the property to a value less than 1 or more than 6400000. dfs.namenode.fs-limits.min-block-size 1048576 Minimum block size in bytes, enforced by the Namenode at create time. This prevents the accidental creation of files with tiny block sizes (and thus many blocks), which can degrade performance. dfs.namenode.fs-limits.max-blocks-per-file 1048576 Maximum number of blocks per file, enforced by the Namenode on write. This prevents the creation of extremely large files which can degrade performance. dfs.namenode.edits.dir ${dfs.namenode.name.dir} Determines where on the local filesystem the DFS name node should store the transaction (edits) file. If this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. Default value is same as dfs.namenode.name.dir dfs.namenode.shared.edits.dir A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster. dfs.namenode.edits.journal-plugin.qjournal org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager dfs.permissions.enabled true If "true", enable permission checking in HDFS. If "false", permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories. dfs.permissions.superusergroup supergroup The name of the group of super-users. dfs.namenode.acls.enabled false Set to true to enable support for HDFS ACLs (Access Control Lists). By default, ACLs are disabled. When ACLs are disabled, the NameNode rejects all RPCs related to setting or getting ACLs. dfs.namenode.lazypersist.file.scrub.interval.sec 300 The NameNode periodically scans the namespace for LazyPersist files with missing blocks and unlinks them from the namespace. This configuration key controls the interval between successive scans. Set it to a negative value to disable this behavior. dfs.block.access.token.enable false If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanodes. dfs.block.access.key.update.interval 600 Interval in minutes at which namenode updates its access keys. dfs.block.access.token.lifetime 600 The lifetime of access tokens in minutes. dfs.datanode.data.dir file://${hadoop.tmp.dir}/dfs/data Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. The directories should be tagged with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS storage policies. The default storage type will be DISK if the directory does not have a storage type tagged explicitly. Directories that do not exist will be created if local filesystem permission allows. dfs.datanode.data.dir.perm 700 Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic. dfs.replication 3 Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. dfs.replication.max 512 Maximal block replication. dfs.namenode.replication.min 1 Minimal block replication. dfs.blocksize 134217728 The default block size for new files, in bytes. You can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.), Or provide complete size in bytes (such as 134217728 for 128 MB). dfs.client.block.write.retries 3 The number of retries for writing blocks to the data nodes, before we signal failure to the application. dfs.client.block.write.replace-datanode-on-failure.enable true If there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. The feature is to add new datanodes to the pipeline. This is a site-wide property to enable/disable the feature. When the cluster size is extremely small, e.g. 3 nodes or less, cluster administrators may want to set the policy to NEVER in the default configuration file or disable this feature. Otherwise, users may experience an unusually high rate of pipeline failures since it is impossible to find new datanodes for replacement. See also dfs.client.block.write.replace-datanode-on-failure.policy dfs.client.block.write.replace-datanode-on-failure.policy DEFAULT This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. ALWAYS: always add a new datanode when an existing datanode is removed. NEVER: never add a new datanode. DEFAULT: Let r be the replication number. Let n be the number of existing datanodes. Add a new datanode only if r is greater than or equal to 3 and either (1) floor(r/2) is greater than or equal to n; or (2) r is greater than n and the block is hflushed/appended. dfs.client.block.write.replace-datanode-on-failure.best-effort false This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. Best effort means that the client will try to replace a failed datanode in write pipeline (provided that the policy is satisfied), however, it continues the write operation in case that the datanode replacement also fails. Suppose the datanode replacement fails. false: An exception should be thrown so that the write will fail. true : The write should be resumed with the remaining datandoes. Note that setting this property to true allows writing to a pipeline with a smaller number of datanodes. As a result, it increases the probability of data loss. dfs.blockreport.intervalMsec 21600000 Determines block reporting interval in milliseconds. dfs.blockreport.initialDelay 0 Delay for first block report in seconds. dfs.blockreport.split.threshold 1000000 If the number of blocks on the DataNode is below this threshold then it will send block reports for all Storage Directories in a single message. If the number of blocks exceeds this threshold then the DataNode will send block reports for each Storage Directory in separate messages. Set to zero to always split. dfs.datanode.directoryscan.interval 21600 Interval in seconds for Datanode to scan data directories and reconcile the difference between blocks in memory and on the disk. dfs.datanode.directoryscan.threads 1 How many threads should the threadpool used to compile reports for volumes in parallel have. dfs.heartbeat.interval 3 Determines datanode heartbeat interval in seconds. dfs.namenode.handler.count 10 The number of server threads for the namenode. dfs.namenode.safemode.threshold-pct 0.999f Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.namenode.replication.min. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent. dfs.namenode.safemode.min.datanodes 0 Specifies the number of datanodes that must be considered alive before the name node exits safemode. Values less than or equal to 0 mean not to take the number of live datanodes into account when deciding whether to remain in safe mode during startup. Values greater than the number of datanodes in the cluster will make safe mode permanent. dfs.namenode.safemode.extension 30000 Determines extension of safe mode in milliseconds after the threshold level is reached. dfs.namenode.resource.check.interval 5000 The interval in milliseconds at which the NameNode resource checker runs. The checker calculates the number of the NameNode storage volumes whose available spaces are more than dfs.namenode.resource.du.reserved, and enters safemode if the number becomes lower than the minimum value specified by dfs.namenode.resource.checked.volumes.minimum. dfs.namenode.resource.du.reserved 104857600 The amount of space to reserve/require for a NameNode storage directory in bytes. The default is 100MB. dfs.namenode.resource.checked.volumes A list of local directories for the NameNode resource checker to check in addition to the local edits directories. dfs.namenode.resource.checked.volumes.minimum 1 The minimum number of redundant NameNode storage volumes required. dfs.datanode.balance.bandwidthPerSec 1048576 Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. dfs.hosts Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted. dfs.hosts.exclude Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded. dfs.namenode.max.objects 0 The maximum number of files, directories and blocks dfs supports. A value of zero indicates no limit to the number of objects that dfs supports. dfs.namenode.datanode.registration.ip-hostname-check true If true (the default), then the namenode requires that a connecting datanode's address must be resolved to a hostname. If necessary, a reverse DNS lookup is performed. All attempts to register a datanode from an unresolvable address are rejected. It is recommended that this setting be left on to prevent accidental registration of datanodes listed by hostname in the excludes file during a DNS outage. Only set this to false in environments where there is no infrastructure to support reverse DNS lookup. dfs.namenode.decommission.interval 30 Namenode periodicity in seconds to check if decommission is complete. dfs.namenode.decommission.blocks.per.interval 500000 The approximate number of blocks to process per decommission interval, as defined in dfs.namenode.decommission.interval. dfs.namenode.decommission.max.concurrent.tracked.nodes 100 The maximum number of decommission-in-progress datanodes nodes that will be tracked at one time by the namenode. Tracking a decommission-in-progress datanode consumes additional NN memory proportional to the number of blocks on the datnode. Having a conservative limit reduces the potential impact of decomissioning a large number of nodes at once. A value of 0 means no limit will be enforced. dfs.namenode.replication.interval 3 The periodicity in seconds with which the namenode computes replication work for datanodes. dfs.namenode.accesstime.precision 3600000 The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS. dfs.datanode.plugins Comma-separated list of datanode plug-ins to be activated. dfs.namenode.plugins Comma-separated list of namenode plug-ins to be activated. dfs.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. dfs.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than dfs.stream-buffer-size dfs.client-write-packet-size 65536 Packet size for clients to write dfs.client.write.exclude.nodes.cache.expiry.interval.millis 600000 The maximum period to keep a DN in the excluded nodes list at a client. After this period, in milliseconds, the previously excluded node(s) will be removed automatically from the cache and will be considered good for block allocations again. Useful to lower or raise in situations where you keep a file open for very long periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance restarts. Defaults to 10 minutes. dfs.namenode.checkpoint.dir file://${hadoop.tmp.dir}/dfs/namesecondary Determines where on the local filesystem the DFS secondary name node should store the temporary images to merge. If this is a comma-delimited list of directories then the image is replicated in all of the directories for redundancy. dfs.namenode.checkpoint.edits.dir ${dfs.namenode.checkpoint.dir} Determines where on the local filesystem the DFS secondary name node should store the temporary edits to merge. If this is a comma-delimited list of directories then the edits is replicated in all of the directories for redundancy. Default value is same as dfs.namenode.checkpoint.dir dfs.namenode.checkpoint.period 3600 The number of seconds between two periodic checkpoints. dfs.namenode.checkpoint.txns 1000000 The Secondary NameNode or CheckpointNode will create a checkpoint of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless of whether 'dfs.namenode.checkpoint.period' has expired. dfs.namenode.checkpoint.check.period 60 The SecondaryNameNode and CheckpointNode will poll the NameNode every 'dfs.namenode.checkpoint.check.period' seconds to query the number of uncheckpointed transactions. dfs.namenode.checkpoint.max-retries 3 The SecondaryNameNode retries failed checkpointing. If the failure occurs while loading fsimage or replaying edits, the number of retries is limited by this variable. dfs.namenode.num.checkpoints.retained 2 The number of image checkpoint files (fsimage_*) that will be retained by the NameNode and Secondary NameNode in their storage directories. All edit logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained checkpoint will also be retained. dfs.namenode.num.extra.edits.retained 1000000 The number of extra transactions which should be retained beyond what is minimally necessary for a NN restart. It does not translate directly to file's age, or the number of files kept, but to the number of transactions (here "edits" means transactions). One edit file may contain several transactions (edits). During checkpoint, NameNode will identify the total number of edits to retain as extra by checking the latest checkpoint transaction value, subtracted by the value of this property. Then, it scans edits files to identify the older ones that don't include the computed range of retained transactions that are to be kept around, and purges them subsequently. The retainment can be useful for audit purposes or for an HA setup where a remote Standby Node may have been offline for some time and need to have a longer backlog of retained edits in order to start again. Typically each edit is on the order of a few hundred bytes, so the default of 1 million edits should be on the order of hundreds of MBs or low GBs. NOTE: Fewer extra edits may be retained than value specified for this setting if doing so would mean that more segments would be retained than the number configured by dfs.namenode.max.extra.edits.segments.retained. dfs.namenode.max.extra.edits.segments.retained 10000 The maximum number of extra edit log segments which should be retained beyond what is minimally necessary for a NN restart. When used in conjunction with dfs.namenode.num.extra.edits.retained, this configuration property serves to cap the number of extra edits files to a reasonable value. dfs.namenode.delegation.key.update-interval 86400000 The update interval for master key for delegation tokens in the namenode in milliseconds. dfs.namenode.delegation.token.max-lifetime 604800000 The maximum lifetime in milliseconds for which a delegation token is valid. dfs.namenode.delegation.token.renew-interval 86400000 The renewal interval for delegation token in milliseconds. dfs.datanode.failed.volumes.tolerated 0 The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown. dfs.image.compress false Should the dfs image be compressed? dfs.image.compression.codec org.apache.hadoop.io.compress.DefaultCodec If the dfs image is compressed, how should they be compressed? This has to be a codec defined in io.compression.codecs. dfs.image.transfer.timeout 60000 Socket timeout for image transfer in milliseconds. This timeout and the related dfs.image.transfer.bandwidthPerSec parameter should be configured such that normal image transfer can complete successfully. This timeout prevents client hangs when the sender fails during image transfer. This is socket timeout during image transfer. dfs.image.transfer.bandwidthPerSec 0 Maximum bandwidth used for image transfer in bytes per second. This can help keep normal namenode operations responsive during checkpointing. The maximum bandwidth and timeout in dfs.image.transfer.timeout should be set such that normal image transfers can complete successfully. A default value of 0 indicates that throttling is disabled. dfs.image.transfer.chunksize 65536 Chunksize in bytes to upload the checkpoint. Chunked streaming is used to avoid internal buffering of contents of image file of huge size. dfs.namenode.support.allow.format true Does HDFS namenode allow itself to be formatted? You may consider setting this to false for any production cluster, to avoid any possibility of formatting a running DFS. dfs.datanode.max.transfer.threads 4096 Specifies the maximum number of threads to use for transferring data in and out of the DN. dfs.datanode.scan.period.hours 504 If this is positive, the DataNode will not scan any individual block more than once in the specified scan period. If this is negative, the block scanner is disabled. If this is set to zero, then the default value of 504 hours or 3 weeks is used. Prior versions of HDFS incorrectly documented that setting this key to zero will disable the block scanner. dfs.block.scanner.volume.bytes.per.second 1048576 If this is 0, the DataNode's block scanner will be disabled. If this is positive, this is the number of bytes per second that the DataNode's block scanner will try to scan from each volume. dfs.datanode.readahead.bytes 4194304 While reading block files, if the Hadoop native libraries are available, the datanode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader's position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the datanode will attempt to read ahead. This feature may be disabled by configuring this property to 0. If the native libraries are not available, this configuration has no effect. dfs.datanode.drop.cache.behind.reads false In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This behavior is automatically disabled for workloads which read only short sections of a block (e.g HBase random-IO workloads). This may improve performance for some workloads by freeing buffer cache space usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect. dfs.datanode.drop.cache.behind.writes false In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache space usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect. dfs.datanode.sync.behind.writes false If this configuration is enabled, the datanode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. If the Hadoop native libraries are not available, this configuration has no effect. dfs.client.failover.max.attempts 15 Expert only. The number of client failover attempts that should be made before the failover is considered failed. dfs.client.failover.sleep.base.millis 500 Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the base value used in the failover calculation. The first failover will retry immediately. The 2nd failover attempt will delay at least dfs.client.failover.sleep.base.millis milliseconds. And so on. dfs.client.failover.sleep.max.millis 15000 Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the maximum value to wait between failovers. Specifically, the time between two failover attempts will not exceed +/- 50% of dfs.client.failover.sleep.max.millis milliseconds. dfs.client.failover.connection.retries 0 Expert only. Indicates the number of retries a failover IPC client will make to establish a server connection. dfs.client.failover.connection.retries.on.timeouts 0 Expert only. The number of retry attempts a failover IPC client will make on socket timeout when establishing a server connection. dfs.client.datanode-restart.timeout 30 Expert only. The time to wait, in seconds, from reception of an datanode shutdown notification for quick restart, until declaring the datanode dead and invoking the normal recovery mechanisms. The notification is sent by a datanode when it is being shutdown using the shutdownDatanode admin command with the upgrade option. dfs.nameservices Comma-separated list of nameservices. dfs.nameservice.id The ID of this nameservice. If the nameservice ID is not configured or more than one nameservice is configured for dfs.nameservices it is determined automatically by matching the local node's address with the configured address. dfs.internal.nameservices Comma-separated list of nameservices that belong to this cluster. Datanode will report to all the nameservices in this list. By default this is set to the value of dfs.nameservices. dfs.ha.namenodes.EXAMPLENAMESERVICE The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). dfs.ha.namenode.id The ID of this namenode. If the namenode ID is not configured it is determined automatically by matching the local node's address with the configured address. dfs.ha.log-roll.period 120 How often, in seconds, the StandbyNode should ask the active to roll edit logs. Since the StandbyNode only reads from finalized log segments, the StandbyNode will only be as up-to-date as how often the logs are rolled. Note that failover triggers a log roll so the StandbyNode will be up to date before it becomes active. dfs.ha.tail-edits.period 60 How often, in seconds, the StandbyNode should check for new finalized log segments in the shared edits log. dfs.ha.automatic-failover.enabled false Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. dfs.client.use.datanode.hostname false Whether clients should use datanode hostnames when connecting to datanodes. dfs.datanode.use.datanode.hostname false Whether datanodes should use datanode hostnames when connecting to other datanodes for data transfer. dfs.client.local.interfaces A comma separated list of network interface names to use for data transfer between the client and datanodes. When creating a connection to read from or write to a datanode, the client chooses one of the specified interfaces at random and binds its socket to the IP of that interface. Individual names may be specified as either an interface name (eg "eth0"), a subinterface name (eg "eth0:0"), or an IP address (which may be specified using CIDR notation to match a range of IPs). dfs.datanode.shared.file.descriptor.paths /dev/shm,/tmp A comma-separated list of paths to use when creating file descriptors that will be shared between the DataNode and the DFSClient. Typically we use /dev/shm, so that the file descriptors will not be written to disk. Systems that don't have /dev/shm will fall back to /tmp by default. dfs.short.circuit.shared.memory.watcher.interrupt.check.ms 60000 The length of time in milliseconds that the short-circuit shared memory watcher will go between checking for java interruptions sent from other threads. This is provided mainly for unit tests. dfs.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal} dfs.secondary.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal} dfs.namenode.kerberos.principal.pattern * A client-side RegEx that can be configured to control allowed realms to authenticate with (useful in cross-realm env.) dfs.namenode.avoid.read.stale.datanode false Indicate whether or not to avoid reading from "stale" datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Stale datanodes will be moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes. dfs.namenode.avoid.write.stale.datanode false Indicate whether or not to avoid writing to "stale" datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Writes will avoid using stale datanodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads. dfs.namenode.stale.datanode.interval 30000 Default time interval for marking a datanode as "stale", i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as "stale" by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. A stale data node is avoided during lease/block recovery. It can be conditionally avoided for reads (see dfs.namenode.avoid.read.stale.datanode) and for writes (see dfs.namenode.avoid.write.stale.datanode). dfs.namenode.write.stale.datanode.ratio 0.5f When the ratio of number stale datanodes to total datanodes marked is greater than this ratio, stop avoiding writing to stale nodes so as to prevent causing hotspots. dfs.namenode.invalidate.work.pct.per.iteration 0.32f *Note*: Advanced property. Change with caution. This determines the percentage amount of block invalidations (deletes) to do over a single DN heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DN. Value should be a positive, non-zero percentage in float notation (X.Yf), with 1.0f meaning 100%. dfs.namenode.replication.work.multiplier.per.iteration 2 *Note*: Advanced property. Change with caution. This determines the total amount of block transfers to begin in parallel at a DN, for replication, when such a command list is being sent over a DN heartbeat by the NN. The actual number is obtained by multiplying this multiplier with the total number of live nodes in the cluster. The result number is the number of blocks to begin transfers immediately for, per DN heartbeat. This number can be any positive, non-zero integer. nfs.server.port 2049 Specify the port number used by Hadoop NFS. nfs.mountd.port 4242 Specify the port number used by Hadoop mount daemon. nfs.dump.dir /tmp/.hdfs-nfs This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. One needs to make sure the directory has enough space. nfs.rtmax 1048576 This is the maximum size in bytes of a READ request supported by the NFS gateway. If you change this, make sure you also update the nfs mount's rsize(add rsize= # of bytes to the mount directive). nfs.wtmax 1048576 This is the maximum size in bytes of a WRITE request supported by the NFS gateway. If you change this, make sure you also update the nfs mount's wsize(add wsize= # of bytes to the mount directive). nfs.keytab.file *Note*: Advanced property. Change with caution. This is the path to the keytab file for the hdfs-nfs gateway. This is required when the cluster is kerberized. nfs.kerberos.principal *Note*: Advanced property. Change with caution. This is the name of the kerberos principal. This is required when the cluster is kerberized.It must be of this format: nfs-gateway-user/nfs-gateway-host@kerberos-realm nfs.allow.insecure.ports true When set to false, client connections originating from unprivileged ports (those above 1023) will be rejected. This is to ensure that clients connecting to this NFS Gateway must have had root privilege on the machine where they're connecting from. dfs.webhdfs.enabled true Enable WebHDFS (REST API) in Namenodes and Datanodes. hadoop.fuse.connection.timeout 300 The minimum number of seconds that we'll cache libhdfs connection objects in fuse_dfs. Lower values will result in lower memory consumption; higher values may speed up access by avoiding the overhead of creating new connection objects. hadoop.fuse.timer.period 5 The number of seconds between cache expiry checks in fuse_dfs. Lower values will result in fuse_dfs noticing changes to Kerberos ticket caches more quickly. dfs.metrics.percentiles.intervals Comma-delimited set of integers denoting the desired rollover intervals (in seconds) for percentile latency metrics on the Namenode and Datanode. By default, percentile latency metrics are disabled. hadoop.user.group.metrics.percentiles.intervals A comma-separated list of the granularity in seconds for the metrics which describe the 50/75/90/95/99th percentile latency for group resolution in milliseconds. By default, percentile latency metrics are disabled. dfs.encrypt.data.transfer false Whether or not actual block data that is read/written from/to HDFS should be encrypted on the wire. This only needs to be set on the NN and DNs, clients will deduce this automatically. It is possible to override this setting per connection by specifying custom logic via dfs.trustedchannel.resolver.class. dfs.encrypt.data.transfer.algorithm This value may be set to either "3des" or "rc4". If nothing is set, then the configured JCE default on the system is used (usually 3DES.) It is widely believed that 3DES is more cryptographically secure, but RC4 is substantially faster. Note that if AES is supported by both the client and server then this encryption algorithm will only be used to initially transfer keys for AES. (See dfs.encrypt.data.transfer.cipher.suites.) dfs.encrypt.data.transfer.cipher.suites This value may be either undefined or AES/CTR/NoPadding. If defined, then dfs.encrypt.data.transfer uses the specified cipher suite for data encryption. If not defined, then only the algorithm specified in dfs.encrypt.data.transfer.algorithm is used. By default, the property is not defined. dfs.encrypt.data.transfer.cipher.key.bitlength 128 The key bitlength negotiated by dfsclient and datanode for encryption. This value may be set to either 128, 192 or 256. dfs.trustedchannel.resolver.class TrustedChannelResolver is used to determine whether a channel is trusted for plain data transfer. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. The default implementation returns false indicating that the channel is not trusted. dfs.data.transfer.protection A comma-separated list of SASL protection values used for secured connections to the DataNode when reading or writing block data. Possible values are authentication, integrity and privacy. authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled. If dfs.encrypt.data.transfer is set to true, then it supersedes the setting for dfs.data.transfer.protection and enforces that all connections must use a specialized encrypted SASL handshake. This property is ignored for connections to a DataNode listening on a privileged port. In this case, it is assumed that the use of a privileged port establishes sufficient trust. dfs.data.transfer.saslproperties.resolver.class SaslPropertiesResolver used to resolve the QOP used for a connection to the DataNode when reading or writing block data. If not specified, the value of hadoop.security.saslproperties.resolver.class is used as the default value. dfs.datanode.hdfs-blocks-metadata.enabled false Boolean which enables backend datanode-side support for the experimental DistributedFileSystem#getFileVBlockStorageLocations API. dfs.client.file-block-storage-locations.num-threads 10 Number of threads used for making parallel RPCs in DistributedFileSystem#getFileBlockStorageLocations(). dfs.client.file-block-storage-locations.timeout.millis 1000 Timeout (in milliseconds) for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations(). dfs.journalnode.rpc-address 0.0.0.0:8485 The JournalNode RPC server address and port. dfs.journalnode.http-address 0.0.0.0:8480 The address and port the JournalNode HTTP server listens on. If the port is 0 then the server will start on a free port. dfs.journalnode.https-address 0.0.0.0:8481 The address and port the JournalNode HTTPS server listens on. If the port is 0 then the server will start on a free port. dfs.namenode.audit.loggers default List of classes implementing audit loggers that will receive audit events. These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger. The special value "default" can be used to reference the default audit logger, which uses the configured log system. Installing custom audit loggers may affect the performance and stability of the NameNode. Refer to the custom logger's documentation for more details. dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold 10737418240 Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls how much DN volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis. dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction 0.75f Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations. dfs.namenode.edits.noeditlogchannelflush false Specifies whether to flush edit log file channel. When set, expensive FileChannel#force calls are skipped and synchronous disk writes are enabled instead by opening the edit log file with RandomAccessFile("rws") flags. This can significantly improve the performance of edit log writes on the Windows platform. Note that the behavior of the "rws" flags is platform and hardware specific and might not provide the same level of guarantees as FileChannel#force. For example, the write will skip the disk-cache on SAS and SCSI devices while it might not on SATA devices. This is an expert level setting, change with caution. dfs.client.cache.drop.behind.writes Just like dfs.datanode.drop.cache.behind.writes, this setting causes the page cache to be dropped behind HDFS writes, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect. dfs.client.cache.drop.behind.reads Just like dfs.datanode.drop.cache.behind.reads, this setting causes the page cache to be dropped behind HDFS reads, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect. dfs.client.cache.readahead When using remote reads, this setting causes the datanode to read ahead in the block file using posix_fadvise, potentially decreasing I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. When using local reads, this setting determines how much readahead we do in BlockReaderLocal. If the native libraries are not available to the DataNode, this configuration has no effect. dfs.namenode.enable.retrycache true This enables the retry cache on the namenode. Namenode tracks for non-idempotent requests the corresponding response. If a client retries the request, the response from the retry cache is sent. Such operations are tagged with annotation @AtMostOnce in namenode protocols. It is recommended that this flag be set to true. Setting it to false, will result in clients getting failure responses to retried request. This flag must be enabled in HA setup for transparent fail-overs. The entries in the cache have expiration time configurable using dfs.namenode.retrycache.expirytime.millis. dfs.namenode.retrycache.expirytime.millis 600000 The time for which retry cache entries are retained. dfs.namenode.retrycache.heap.percent 0.03f This parameter configures the heap size allocated for retry cache (excluding the response cached). This corresponds to approximately 4096 entries for every 64MB of namenode process java heap size. Assuming retry cache entry expiration time (configured using dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this enables retry cache to support 7 operations per second sustained for 10 minutes. As the heap size is increased, the operation rate linearly increases. dfs.client.mmap.enabled true If this is set to false, the client won't attempt to perform memory-mapped reads. dfs.client.mmap.cache.size 256 When zero-copy reads are used, the DFSClient keeps a cache of recently used memory mapped regions. This parameter controls the maximum number of entries that we will keep in that cache. The larger this number is, the more file descriptors we will potentially use for memory-mapped files. mmaped files also use virtual address space. You may need to increase your ulimit virtual address space limits before increasing the client mmap cache size. Note that you can still do zero-copy reads when this size is set to 0. dfs.client.mmap.cache.timeout.ms 3600000 The minimum length of time that we will keep an mmap entry in the cache between uses. If an entry is in the cache longer than this, and nobody uses it, it will be removed by a background thread. dfs.client.mmap.retry.timeout.ms 300000 The minimum amount of time that we will wait before retrying a failed mmap operation. dfs.client.short.circuit.replica.stale.threshold.ms 1800000 The maximum amount of time that we will consider a short-circuit replica to be valid, if there is no communication from the DataNode. After this time has elapsed, we will re-fetch the short-circuit replica even if it is in the cache. dfs.namenode.path.based.cache.block.map.allocation.percent 0.25 The percentage of the Java heap which we will allocate to the cached blocks map. The cached blocks map is a hash map which uses chained hashing. Smaller maps may be accessed more slowly if the number of cached blocks is large; larger maps will consume more memory. dfs.datanode.max.locked.memory 0 The amount of memory in bytes to use for caching of block replicas in memory on the datanode. The datanode's maximum locked memory soft ulimit (RLIMIT_MEMLOCK) must be set to at least this value, else the datanode will abort on startup. By default, this parameter is set to 0, which disables in-memory caching. If the native libraries are not available to the DataNode, this configuration has no effect. dfs.namenode.list.cache.directives.num.responses 100 This value controls the number of cache directives that the NameNode will send over the wire in response to a listDirectives RPC. dfs.namenode.list.cache.pools.num.responses 100 This value controls the number of cache pools that the NameNode will send over the wire in response to a listPools RPC. dfs.namenode.path.based.cache.refresh.interval.ms 30000 The amount of milliseconds between subsequent path cache rescans. Path cache rescans are when we calculate which blocks should be cached, and on what datanodes. By default, this parameter is set to 30 seconds. dfs.namenode.path.based.cache.retry.interval.ms 30000 When the NameNode needs to uncache something that is cached, or cache something that is not cached, it must direct the DataNodes to do so by sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode heartbeat. This parameter controls how frequently the NameNode will resend these commands. dfs.datanode.fsdatasetcache.max.threads.per.volume 4 The maximum number of threads per volume to use for caching new data on the datanode. These threads consume both I/O and CPU. This can affect normal datanode operations. dfs.cachereport.intervalMsec 10000 Determines cache reporting interval in milliseconds. After this amount of time, the DataNode sends a full report of its cache state to the NameNode. The NameNode uses the cache report to update its map of cached blocks to DataNode locations. This configuration has no effect if in-memory caching has been disabled by setting dfs.datanode.max.locked.memory to 0 (which is the default). If the native libraries are not available to the DataNode, this configuration has no effect. dfs.namenode.edit.log.autoroll.multiplier.threshold 2.0 Determines when an active namenode will roll its own edit log. The actual threshold (in number of edits) is determined by multiplying this value by dfs.namenode.checkpoint.txns. This prevents extremely large edit files from accumulating on the active namenode, which can cause timeouts during namenode startup and pose an administrative hassle. This behavior is intended as a failsafe for when the standby or secondary namenode fail to roll the edit log by the normal checkpoint threshold. dfs.namenode.edit.log.autoroll.check.interval.ms 300000 How often an active namenode will check if it needs to roll its edit log, in milliseconds. dfs.webhdfs.user.provider.user.pattern ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ Valid pattern for user and group names for webhdfs, it must be a valid java regex. dfs.client.context default The name of the DFSClient context that we should use. Clients that share a context share a socket cache and short-circuit cache, among other things. You should only change this if you don't want to share with another set of threads. dfs.client.read.shortcircuit false This configuration parameter turns on short-circuit local reads. dfs.domain.socket.path Optional. This is a path to a UNIX domain socket that will be used for communication between the DataNode and local HDFS clients. If the string "_PORT" is present in this path, it will be replaced by the TCP port of the DataNode. dfs.client.read.shortcircuit.skip.checksum false If this configuration parameter is set, short-circuit local reads will skip checksums. This is normally not recommended, but it may be useful for special setups. You might consider using this if you are doing your own checksumming outside of HDFS. dfs.client.read.shortcircuit.streams.cache.size 256 The DFSClient maintains a cache of recently opened file descriptors. This parameter controls the size of that cache. Setting this higher will use more file descriptors, but potentially provide better performance on workloads involving lots of seeks. dfs.client.read.shortcircuit.streams.cache.expiry.ms 300000 This controls the minimum amount of time file descriptors need to sit in the client cache context before they can be closed for being inactive for too long. dfs.datanode.shared.file.descriptor.paths /dev/shm,/tmp Comma separated paths to the directory on which shared memory segments are created. The client and the DataNode exchange information via this shared memory segment. It tries paths in order until creation of shared memory segment succeeds. dfs.client.use.legacy.blockreader.local false Legacy short-circuit reader implementation based on HDFS-2246 is used if this configuration parameter is true. This is for the platforms other than Linux where the new implementation based on HDFS-347 is not available. dfs.block.local-path-access.user Comma separated list of the users allowd to open block files on legacy short-circuit local read. dfs.client.domain.socket.data.traffic false This control whether we will try to pass normal data traffic over UNIX domain socket rather than over TCP socket on node-local data transfer. This is currently experimental and turned off by default. dfs.namenode.reject-unresolved-dn-topology-mapping false If the value is set to true, then namenode will reject datanode registration if the topology mapping for a datanode is not resolved and NULL is returned (script defined by net.topology.script.file.name fails to execute). Otherwise, datanode will be registered and the default rack will be assigned as the topology path. Topology paths are important for data resiliency, since they define fault domains. Thus it may be unwanted behavior to allow datanode registration with the default rack if the resolving topology failed. dfs.client.slow.io.warning.threshold.ms 30000 The threshold in milliseconds at which we will log a slow io warning in a dfsclient. By default, this parameter is set to 30000 milliseconds (30 seconds). dfs.datanode.slow.io.warning.threshold.ms 300 The threshold in milliseconds at which we will log a slow io warning in a datanode. By default, this parameter is set to 300 milliseconds. dfs.namenode.xattrs.enabled true Whether support for extended attributes is enabled on the NameNode. dfs.namenode.fs-limits.max-xattrs-per-inode 32 Maximum number of extended attributes per inode. dfs.namenode.fs-limits.max-xattr-size 16384 The maximum combined size of the name and value of an extended attribute in bytes. dfs.namenode.startup.delay.block.deletion.sec 0 The delay in seconds at which we will pause the blocks deletion after Namenode startup. By default it's disabled. In the case a directory has large number of directories and files are deleted, suggested delay is one hour to give the administrator enough time to notice large number of pending deletion blocks and take corrective action. dfs.namenode.list.encryption.zones.num.responses 100 When listing encryption zones, the maximum number of zones that will be returned in a batch. Fetching the list incrementally in batches improves namenode performance. dfs.namenode.inotify.max.events.per.rpc 1000 Maximum number of events that will be sent to an inotify client in a single RPC response. The default value attempts to amortize away the overhead for this RPC while avoiding huge memory requirements for the client and NameNode (1000 events should consume no more than 1 MB.) dfs.user.home.dir.prefix /user The directory to prepend to user name to get the user's home direcotry. dfs.datanode.cache.revocation.timeout.ms 900000 When the DFSClient reads from a block file which the DataNode is caching, the DFSClient can skip verifying checksums. The DataNode will keep the block file in cache until the client is done. If the client takes an unusually long time, though, the DataNode may need to evict the block file from the cache anyway. This value controls how long the DataNode will wait for the client to release a replica that it is reading without checksums. dfs.datanode.cache.revocation.polling.ms 500 How often the DataNode should poll to see if the clients have stopped using a replica that the DataNode wants to uncache. dfs.datanode.block.id.layout.upgrade.threads 12 The number of threads to use when creating hard links from current to previous blocks during upgrade of a DataNode to block ID-based block layout (see HDFS-6482 for details on the layout). dfs.encryption.key.provider.uri The KeyProvider to use when interacting with encryption keys used when reading and writing to an encryption zone. dfs.storage.policy.enabled true Allow users to change the storage policy on files and directories. dfs.namenode.legacy-oiv-image.dir Determines where to save the namespace in the old fsimage format during checkpointing by standby NameNode or SecondaryNameNode. Users can dump the contents of the old format fsimage by oiv_legacy command. If the value is not specified, old format fsimage will not be saved in checkpoint. dfs.namenode.top.enabled true Enable nntop: reporting top users on namenode dfs.namenode.top.window.num.buckets 10 Number of buckets in the rolling window implementation of nntop dfs.namenode.top.num.users 10 Number of top users returned by the top tool dfs.namenode.top.windows.minutes 1,5,25 comma separated list of nntop reporting periods in minutes dfs.namenode.blocks.per.postponedblocks.rescan 10000 Number of blocks to rescan for each iteration of postponedMisreplicatedBlocks. dfs.datanode.block-pinning.enabled false Whether pin blocks on favored DataNode. sahara-8.0.0/sahara/plugins/vanilla/v2_7_1/resources/oozie-default.xml0000666000175100017510000031700213245514472025664 0ustar zuulzuul00000000000000 oozie.output.compression.codec gz The name of the compression codec to use. The implementation class for the codec needs to be specified through another property oozie.compression.codecs. You can specify a comma separated list of 'Codec_name'='Codec_class' for oozie.compression.codecs where codec class implements the interface org.apache.oozie.compression.CompressionCodec. If oozie.compression.codecs is not specified, gz codec implementation is used by default. oozie.action.mapreduce.uber.jar.enable false If true, enables the oozie.mapreduce.uber.jar mapreduce workflow configuration property, which is used to specify an uber jar in HDFS. Submitting a workflow with an uber jar requires at least Hadoop 2.2.0 or 1.2.0. If false, workflows which specify the oozie.mapreduce.uber.jar configuration property will fail. oozie.processing.timezone UTC Oozie server timezone. Valid values are UTC and GMT(+/-)####, for example 'GMT+0530' would be India timezone. All dates parsed and genered dates by Oozie Coordinator/Bundle will be done in the specified timezone. The default value of 'UTC' should not be changed under normal circumtances. If for any reason is changed, note that GMT(+/-)#### timezones do not observe DST changes. oozie.base.url http://localhost:8080/oozie Base Oozie URL. oozie.system.id oozie-${user.name} The Oozie system ID. oozie.systemmode NORMAL System mode for Oozie at startup. oozie.delete.runtime.dir.on.shutdown true If the runtime directory should be kept after Oozie shutdowns down. oozie.services org.apache.oozie.service.SchedulerService, org.apache.oozie.service.InstrumentationService, org.apache.oozie.service.MemoryLocksService, org.apache.oozie.service.UUIDService, org.apache.oozie.service.ELService, org.apache.oozie.service.AuthorizationService, org.apache.oozie.service.UserGroupInformationService, org.apache.oozie.service.HadoopAccessorService, org.apache.oozie.service.JobsConcurrencyService, org.apache.oozie.service.URIHandlerService, org.apache.oozie.service.DagXLogInfoService, org.apache.oozie.service.SchemaService, org.apache.oozie.service.LiteWorkflowAppService, org.apache.oozie.service.JPAService, org.apache.oozie.service.StoreService, org.apache.oozie.service.SLAStoreService, org.apache.oozie.service.DBLiteWorkflowStoreService, org.apache.oozie.service.CallbackService, org.apache.oozie.service.ActionService, org.apache.oozie.service.ShareLibService, org.apache.oozie.service.CallableQueueService, org.apache.oozie.service.ActionCheckerService, org.apache.oozie.service.RecoveryService, org.apache.oozie.service.PurgeService, org.apache.oozie.service.CoordinatorEngineService, org.apache.oozie.service.BundleEngineService, org.apache.oozie.service.DagEngineService, org.apache.oozie.service.CoordMaterializeTriggerService, org.apache.oozie.service.StatusTransitService, org.apache.oozie.service.PauseTransitService, org.apache.oozie.service.GroupsService, org.apache.oozie.service.ProxyUserService, org.apache.oozie.service.XLogStreamingService, org.apache.oozie.service.JvmPauseMonitorService, org.apache.oozie.service.SparkConfigurationService All services to be created and managed by Oozie Services singleton. Class names must be separated by commas. oozie.services.ext To add/replace services defined in 'oozie.services' with custom implementations. Class names must be separated by commas. oozie.service.XLogStreamingService.buffer.len 4096 4K buffer for streaming the logs progressively oozie.service.HCatAccessorService.jmsconnections default=java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://localhost:61616;connectionFactoryNames#ConnectionFactory Specify the map of endpoints to JMS configuration properties. In general, endpoint identifies the HCatalog server URL. "default" is used if no endpoint is mentioned in the query. If some JMS property is not defined, the system will use the property defined jndi.properties. jndi.properties files is retrieved from the application classpath. Mapping rules can also be provided for mapping Hcatalog servers to corresponding JMS providers. hcat://${1}.${2}.server.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}:61616 oozie.service.JMSTopicService.topic.name default=${username} Topic options are ${username} or ${jobId} or a fixed string which can be specified as default or for a particular job type. For e.g To have a fixed string topic for workflows, coordinators and bundles, specify in the following comma-separated format: {jobtype1}={some_string1}, {jobtype2}={some_string2} where job type can be WORKFLOW, COORDINATOR or BUNDLE. e.g. Following defines topic for workflow job, workflow action, coordinator job, coordinator action, bundle job and bundle action WORKFLOW=workflow, COORDINATOR=coordinator, BUNDLE=bundle For jobs with no defined topic, default topic will be ${username} oozie.jms.producer.connection.properties java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://localhost:61616;connectionFactoryNames#ConnectionFactory oozie.service.JMSAccessorService.connectioncontext.impl org.apache.oozie.jms.DefaultConnectionContext Specifies the Connection Context implementation oozie.service.ConfigurationService.ignore.system.properties oozie.service.AuthorizationService.security.enabled Specifies "oozie.*" properties to cannot be overriden via Java system properties. Property names must be separted by commas. oozie.service.ConfigurationService.verify.available.properties true Specifies whether the available configurations check is enabled or not. oozie.service.SchedulerService.threads 10 The number of threads to be used by the SchedulerService to run deamon tasks. If maxed out, scheduled daemon tasks will be queued up and delayed until threads become available. oozie.service.AuthorizationService.authorization.enabled false Specifies whether security (user name/admin role) is enabled or not. If disabled any user can manage Oozie system and manage any job. oozie.service.AuthorizationService.default.group.as.acl false Enables old behavior where the User's default group is the job's ACL. oozie.service.InstrumentationService.logging.interval 60 Interval, in seconds, at which instrumentation should be logged by the InstrumentationService. If set to 0 it will not log instrumentation data. oozie.service.PurgeService.older.than 30 Completed workflow jobs older than this value, in days, will be purged by the PurgeService. oozie.service.PurgeService.coord.older.than 7 Completed coordinator jobs older than this value, in days, will be purged by the PurgeService. oozie.service.PurgeService.bundle.older.than 7 Completed bundle jobs older than this value, in days, will be purged by the PurgeService. oozie.service.PurgeService.purge.old.coord.action false Whether to purge completed workflows and their corresponding coordinator actions of long running coordinator jobs if the completed workflow jobs are older than the value specified in oozie.service.PurgeService.older.than. oozie.service.PurgeService.purge.limit 100 Completed Actions purge - limit each purge to this value oozie.service.PurgeService.purge.interval 3600 Interval at which the purge service will run, in seconds. oozie.service.RecoveryService.wf.actions.older.than 120 Age of the actions which are eligible to be queued for recovery, in seconds. oozie.service.RecoveryService.wf.actions.created.time.interval 7 Created time period of the actions which are eligible to be queued for recovery in days. oozie.service.RecoveryService.callable.batch.size 10 This value determines the number of callable which will be batched together to be executed by a single thread. oozie.service.RecoveryService.push.dependency.interval 200 This value determines the delay for push missing dependency command queueing in Recovery Service oozie.service.RecoveryService.interval 60 Interval at which the RecoverService will run, in seconds. oozie.service.RecoveryService.coord.older.than 600 Age of the Coordinator jobs or actions which are eligible to be queued for recovery, in seconds. oozie.service.RecoveryService.bundle.older.than 600 Age of the Bundle jobs which are eligible to be queued for recovery, in seconds. oozie.service.CallableQueueService.queue.size 10000 Max callable queue size oozie.service.CallableQueueService.threads 10 Number of threads used for executing callables oozie.service.CallableQueueService.callable.concurrency 3 Maximum concurrency for a given callable type. Each command is a callable type (submit, start, run, signal, job, jobs, suspend,resume, etc). Each action type is a callable type (Map-Reduce, Pig, SSH, FS, sub-workflow, etc). All commands that use action executors (action-start, action-end, action-kill and action-check) use the action type as the callable type. oozie.service.CallableQueueService.callable.next.eligible true If true, when a callable in the queue has already reached max concurrency, Oozie continuously find next one which has not yet reach max concurrency. oozie.service.CallableQueueService.InterruptMapMaxSize 500 Maximum Size of the Interrupt Map, the interrupt element will not be inserted in the map if exceeded the size. oozie.service.CallableQueueService.InterruptTypes kill,resume,suspend,bundle_kill,bundle_resume,bundle_suspend,coord_kill,coord_change,coord_resume,coord_suspend Getting the types of XCommands that are considered to be of Interrupt type oozie.service.CoordMaterializeTriggerService.lookup.interval 300 Coordinator Job Lookup interval.(in seconds). oozie.service.CoordMaterializeTriggerService.materialization.window 3600 Coordinator Job Lookup command materialized each job for this next "window" duration oozie.service.CoordMaterializeTriggerService.callable.batch.size 10 This value determines the number of callable which will be batched together to be executed by a single thread. oozie.service.CoordMaterializeTriggerService.materialization.system.limit 50 This value determines the number of coordinator jobs to be materialized at a given time. oozie.service.coord.normal.default.timeout 120 Default timeout for a coordinator action input check (in minutes) for normal job. -1 means infinite timeout oozie.service.coord.default.max.timeout 86400 Default maximum timeout for a coordinator action input check (in minutes). 86400= 60days oozie.service.coord.input.check.requeue.interval 60000 Command re-queue interval for coordinator data input check (in millisecond). oozie.service.coord.push.check.requeue.interval 600000 Command re-queue interval for push dependencies (in millisecond). oozie.service.coord.default.concurrency 1 Default concurrency for a coordinator job to determine how many maximum action should be executed at the same time. -1 means infinite concurrency. oozie.service.coord.default.throttle 12 Default throttle for a coordinator job to determine how many maximum action should be in WAITING state at the same time. oozie.service.coord.materialization.throttling.factor 0.05 Determine how many maximum actions should be in WAITING state for a single job at any time. The value is calculated by this factor X the total queue size. oozie.service.coord.check.maximum.frequency true When true, Oozie will reject any coordinators with a frequency faster than 5 minutes. It is not recommended to disable this check or submit coordinators with frequencies faster than 5 minutes: doing so can cause unintended behavior and additional system stress. oozie.service.ELService.groups job-submit,workflow,wf-sla-submit,coord-job-submit-freq,coord-job-submit-nofuncs,coord-job-submit-data,coord-job-submit-instances,coord-sla-submit,coord-action-create,coord-action-create-inst,coord-sla-create,coord-action-start,coord-job-wait-timeout List of groups for different ELServices oozie.service.ELService.constants.job-submit EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.functions.job-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.constants.job-submit EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions without having to include all the built in ones. oozie.service.ELService.ext.functions.job-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions without having to include all the built in ones. oozie.service.ELService.constants.workflow KB=org.apache.oozie.util.ELConstantsFunctions#KB, MB=org.apache.oozie.util.ELConstantsFunctions#MB, GB=org.apache.oozie.util.ELConstantsFunctions#GB, TB=org.apache.oozie.util.ELConstantsFunctions#TB, PB=org.apache.oozie.util.ELConstantsFunctions#PB, RECORDS=org.apache.oozie.action.hadoop.HadoopELFunctions#RECORDS, MAP_IN=org.apache.oozie.action.hadoop.HadoopELFunctions#MAP_IN, MAP_OUT=org.apache.oozie.action.hadoop.HadoopELFunctions#MAP_OUT, REDUCE_IN=org.apache.oozie.action.hadoop.HadoopELFunctions#REDUCE_IN, REDUCE_OUT=org.apache.oozie.action.hadoop.HadoopELFunctions#REDUCE_OUT, GROUPS=org.apache.oozie.action.hadoop.HadoopELFunctions#GROUPS EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.workflow EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.workflow firstNotNull=org.apache.oozie.util.ELConstantsFunctions#firstNotNull, concat=org.apache.oozie.util.ELConstantsFunctions#concat, replaceAll=org.apache.oozie.util.ELConstantsFunctions#replaceAll, appendAll=org.apache.oozie.util.ELConstantsFunctions#appendAll, trim=org.apache.oozie.util.ELConstantsFunctions#trim, timestamp=org.apache.oozie.util.ELConstantsFunctions#timestamp, urlEncode=org.apache.oozie.util.ELConstantsFunctions#urlEncode, toJsonStr=org.apache.oozie.util.ELConstantsFunctions#toJsonStr, toPropertiesStr=org.apache.oozie.util.ELConstantsFunctions#toPropertiesStr, toConfigurationStr=org.apache.oozie.util.ELConstantsFunctions#toConfigurationStr, wf:id=org.apache.oozie.DagELFunctions#wf_id, wf:name=org.apache.oozie.DagELFunctions#wf_name, wf:appPath=org.apache.oozie.DagELFunctions#wf_appPath, wf:conf=org.apache.oozie.DagELFunctions#wf_conf, wf:user=org.apache.oozie.DagELFunctions#wf_user, wf:group=org.apache.oozie.DagELFunctions#wf_group, wf:callback=org.apache.oozie.DagELFunctions#wf_callback, wf:transition=org.apache.oozie.DagELFunctions#wf_transition, wf:lastErrorNode=org.apache.oozie.DagELFunctions#wf_lastErrorNode, wf:errorCode=org.apache.oozie.DagELFunctions#wf_errorCode, wf:errorMessage=org.apache.oozie.DagELFunctions#wf_errorMessage, wf:run=org.apache.oozie.DagELFunctions#wf_run, wf:actionData=org.apache.oozie.DagELFunctions#wf_actionData, wf:actionExternalId=org.apache.oozie.DagELFunctions#wf_actionExternalId, wf:actionTrackerUri=org.apache.oozie.DagELFunctions#wf_actionTrackerUri, wf:actionExternalStatus=org.apache.oozie.DagELFunctions#wf_actionExternalStatus, hadoop:counters=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_counters, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf, fs:exists=org.apache.oozie.action.hadoop.FsELFunctions#fs_exists, fs:isDir=org.apache.oozie.action.hadoop.FsELFunctions#fs_isDir, fs:dirSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_dirSize, fs:fileSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_fileSize, fs:blockSize=org.apache.oozie.action.hadoop.FsELFunctions#fs_blockSize, hcat:exists=org.apache.oozie.coord.HCatELFunctions#hcat_exists EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength 100000 The maximum length of the workflow definition in bytes An error will be reported if the length exceeds the given maximum oozie.service.ELService.ext.functions.workflow EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.wf-sla-submit MINUTES=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_MINUTES, HOURS=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_HOURS, DAYS=org.apache.oozie.util.ELConstantsFunctions#SUBMIT_DAYS EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.wf-sla-submit EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.wf-sla-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.wf-sla-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. l oozie.service.ELService.constants.coord-job-submit-freq EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-job-submit-freq EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-job-submit-freq coord:days=org.apache.oozie.coord.CoordELFunctions#ph1_coord_days, coord:months=org.apache.oozie.coord.CoordELFunctions#ph1_coord_months, coord:hours=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hours, coord:minutes=org.apache.oozie.coord.CoordELFunctions#ph1_coord_minutes, coord:endOfDays=org.apache.oozie.coord.CoordELFunctions#ph1_coord_endOfDays, coord:endOfMonths=org.apache.oozie.coord.CoordELFunctions#ph1_coord_endOfMonths, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-submit-freq EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-job-wait-timeout EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.constants.coord-job-wait-timeout EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions without having to include all the built in ones. oozie.service.ELService.functions.coord-job-wait-timeout coord:days=org.apache.oozie.coord.CoordELFunctions#ph1_coord_days, coord:months=org.apache.oozie.coord.CoordELFunctions#ph1_coord_months, coord:hours=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hours, coord:minutes=org.apache.oozie.coord.CoordELFunctions#ph1_coord_minutes, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-wait-timeout EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions without having to include all the built in ones. oozie.service.ELService.constants.coord-job-submit-nofuncs MINUTE=org.apache.oozie.coord.CoordELConstants#SUBMIT_MINUTE, HOUR=org.apache.oozie.coord.CoordELConstants#SUBMIT_HOUR, DAY=org.apache.oozie.coord.CoordELConstants#SUBMIT_DAY, MONTH=org.apache.oozie.coord.CoordELConstants#SUBMIT_MONTH, YEAR=org.apache.oozie.coord.CoordELConstants#SUBMIT_YEAR EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-job-submit-nofuncs EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-job-submit-nofuncs coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-submit-nofuncs EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-job-submit-instances EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-job-submit-instances EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-job-submit-instances coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph1_coord_hoursInDay_echo, coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph1_coord_daysInMonth_echo, coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_tzOffset_echo, coord:current=org.apache.oozie.coord.CoordELFunctions#ph1_coord_current_echo, coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_currentRange_echo, coord:offset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_offset_echo, coord:latest=org.apache.oozie.coord.CoordELFunctions#ph1_coord_latest_echo, coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_latestRange_echo, coord:future=org.apache.oozie.coord.CoordELFunctions#ph1_coord_future_echo, coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph1_coord_futureRange_echo, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph1_coord_absolute_echo, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-submit-instances EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-job-submit-data EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-job-submit-data EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-job-submit-data coord:dataIn=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataIn_echo, coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataOut_echo, coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_wrap, coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actualTime_echo_wrap, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateTzOffset_echo, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actionId_echo, coord:name=org.apache.oozie.coord.CoordELFunctions#ph1_coord_name_echo, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:databaseIn=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseIn_echo, coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseOut_echo, coord:tableIn=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableIn_echo, coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableOut_echo, coord:dataInPartitionFilter=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionFilter_echo, coord:dataInPartitionMin=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionMin_echo, coord:dataInPartitionMax=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitionMax_echo, coord:dataInPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataInPartitions_echo, coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitions_echo, coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitionValue_echo, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-job-submit-data EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-sla-submit MINUTES=org.apache.oozie.coord.CoordELConstants#SUBMIT_MINUTES, HOURS=org.apache.oozie.coord.CoordELConstants#SUBMIT_HOURS, DAYS=org.apache.oozie.coord.CoordELConstants#SUBMIT_DAYS EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-sla-submit EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-sla-submit coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dataOut_echo, coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed, coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actualTime_echo_wrap, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateTzOffset_echo, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actionId_echo, coord:name=org.apache.oozie.coord.CoordELFunctions#ph1_coord_name_echo, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_databaseOut_echo, coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph1_coord_tableOut_echo, coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitions_echo, coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph1_coord_dataOutPartitionValue_echo, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-sla-submit EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-action-create EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-action-create EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-action-create coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph2_coord_hoursInDay, coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph2_coord_daysInMonth, coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_tzOffset, coord:current=org.apache.oozie.coord.CoordELFunctions#ph2_coord_current, coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_currentRange, coord:offset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_offset, coord:latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latestRange_echo, coord:future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_futureRange_echo, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph2_coord_actionId, coord:name=org.apache.oozie.coord.CoordELFunctions#ph2_coord_name, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_echo, coord:absoluteRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_range, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-action-create EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-action-create-inst EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-action-create-inst EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-action-create-inst coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph2_coord_hoursInDay, coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph2_coord_daysInMonth, coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_tzOffset, coord:current=org.apache.oozie.coord.CoordELFunctions#ph2_coord_current_echo, coord:currentRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_currentRange_echo, coord:offset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_offset_echo, coord:latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo, coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latestRange_echo, coord:future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo, coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_futureRange_echo, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:absolute=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_echo, coord:absoluteRange=org.apache.oozie.coord.CoordELFunctions#ph2_coord_absolute_range, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-action-create-inst EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-sla-create EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-sla-create MINUTES=org.apache.oozie.coord.CoordELConstants#SUBMIT_MINUTES, HOURS=org.apache.oozie.coord.CoordELConstants#SUBMIT_HOURS, DAYS=org.apache.oozie.coord.CoordELConstants#SUBMIT_DAYS EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-sla-create coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dataOut, coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_nominalTime, coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_actualTime, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_dateOffset, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph2_coord_dateTzOffset, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph2_coord_actionId, coord:name=org.apache.oozie.coord.CoordELFunctions#ph2_coord_name, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseOut, coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_tableOut, coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitions, coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitionValue, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-sla-create EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.constants.coord-action-start EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. oozie.service.ELService.ext.constants.coord-action-start EL constant declarations, separated by commas, format is [PREFIX:]NAME=CLASS#CONSTANT. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.functions.coord-action-start coord:hoursInDay=org.apache.oozie.coord.CoordELFunctions#ph3_coord_hoursInDay, coord:daysInMonth=org.apache.oozie.coord.CoordELFunctions#ph3_coord_daysInMonth, coord:tzOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_tzOffset, coord:latest=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latest, coord:latestRange=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latestRange, coord:future=org.apache.oozie.coord.CoordELFunctions#ph3_coord_future, coord:futureRange=org.apache.oozie.coord.CoordELFunctions#ph3_coord_futureRange, coord:dataIn=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dataIn, coord:dataOut=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dataOut, coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime, coord:actualTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_actualTime, coord:dateOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateOffset, coord:dateTzOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateTzOffset, coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime, coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph3_coord_actionId, coord:name=org.apache.oozie.coord.CoordELFunctions#ph3_coord_name, coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf, coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user, coord:databaseIn=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseIn, coord:databaseOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_databaseOut, coord:tableIn=org.apache.oozie.coord.HCatELFunctions#ph3_coord_tableIn, coord:tableOut=org.apache.oozie.coord.HCatELFunctions#ph3_coord_tableOut, coord:dataInPartitionFilter=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionFilter, coord:dataInPartitionMin=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionMin, coord:dataInPartitionMax=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitionMax, coord:dataInPartitions=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataInPartitions, coord:dataOutPartitions=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitions, coord:dataOutPartitionValue=org.apache.oozie.coord.HCatELFunctions#ph3_coord_dataOutPartitionValue, hadoop:conf=org.apache.oozie.action.hadoop.HadoopELFunctions#hadoop_conf EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. oozie.service.ELService.ext.functions.coord-action-start EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ELService.latest-el.use-current-time false Determine whether to use the current time to determine the latest dependency or the action creation time. This is for backward compatibility with older oozie behaviour. oozie.service.UUIDService.generator counter random : generated UUIDs will be random strings. counter: generated UUIDs generated will be a counter postfixed with the system startup time. oozie.service.DBLiteWorkflowStoreService.status.metrics.collection.interval 5 Workflow Status metrics collection interval in minutes. oozie.service.DBLiteWorkflowStoreService.status.metrics.window 3600 Workflow Status metrics collection window in seconds. Workflow status will be instrumented for the window. oozie.db.schema.name oozie Oozie DataBase Name oozie.service.JPAService.create.db.schema false Creates Oozie DB. If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP. If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. oozie.service.JPAService.validate.db.connection true Validates DB connections from the DB connection pool. If the 'oozie.service.JPAService.create.db.schema' property is set to true, this property is ignored. oozie.service.JPAService.validate.db.connection.eviction.interval 300000 Validates DB connections from the DB connection pool. When validate db connection 'TestWhileIdle' is true, the number of milliseconds to sleep between runs of the idle object evictor thread. oozie.service.JPAService.validate.db.connection.eviction.num 10 Validates DB connections from the DB connection pool. When validate db connection 'TestWhileIdle' is true, the number of objects to examine during each run of the idle object evictor thread. oozie.service.JPAService.connection.data.source org.apache.commons.dbcp.BasicDataSource DataSource to be used for connection pooling. oozie.service.JPAService.connection.properties DataSource connection properties. oozie.service.JPAService.jdbc.driver org.apache.derby.jdbc.EmbeddedDriver JDBC driver class. oozie.service.JPAService.jdbc.url jdbc:derby:${oozie.data.dir}/${oozie.db.schema.name}-db;create=true JDBC URL. oozie.service.JPAService.jdbc.username sa DB user name. oozie.service.JPAService.jdbc.password DB user password. IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value, if empty Configuration assumes it is NULL. IMPORTANT: if the StoreServicePasswordService is active, it will reset this value with the value given in the console. oozie.service.JPAService.pool.max.active.conn 10 Max number of connections. oozie.service.SchemaService.wf.schemas oozie-workflow-0.1.xsd,oozie-workflow-0.2.xsd,oozie-workflow-0.2.5.xsd,oozie-workflow-0.3.xsd,oozie-workflow-0.4.xsd, oozie-workflow-0.4.5.xsd,oozie-workflow-0.5.xsd, shell-action-0.1.xsd,shell-action-0.2.xsd,shell-action-0.3.xsd, email-action-0.1.xsd,email-action-0.2.xsd, hive-action-0.2.xsd,hive-action-0.3.xsd,hive-action-0.4.xsd,hive-action-0.5.xsd, sqoop-action-0.2.xsd,sqoop-action-0.3.xsd,sqoop-action-0.4.xsd, ssh-action-0.1.xsd,ssh-action-0.2.xsd, distcp-action-0.1.xsd,distcp-action-0.2.xsd, oozie-sla-0.1.xsd,oozie-sla-0.2.xsd, hive2-action-0.1.xsd, spark-action-0.1.xsd List of schemas for workflows (separated by commas). oozie.service.SchemaService.wf.ext.schemas List of additional schemas for workflows (separated by commas). oozie.service.SchemaService.coord.schemas oozie-coordinator-0.1.xsd,oozie-coordinator-0.2.xsd,oozie-coordinator-0.3.xsd,oozie-coordinator-0.4.xsd, oozie-sla-0.1.xsd,oozie-sla-0.2.xsd List of schemas for coordinators (separated by commas). oozie.service.SchemaService.coord.ext.schemas List of additional schemas for coordinators (separated by commas). oozie.service.SchemaService.bundle.schemas oozie-bundle-0.1.xsd,oozie-bundle-0.2.xsd List of schemas for bundles (separated by commas). oozie.service.SchemaService.bundle.ext.schemas List of additional schemas for bundles (separated by commas). oozie.service.SchemaService.sla.schemas gms-oozie-sla-0.1.xsd,oozie-sla-0.2.xsd List of schemas for semantic validation for GMS SLA (separated by commas). oozie.service.SchemaService.sla.ext.schemas List of additional schemas for semantic validation for GMS SLA (separated by commas). oozie.service.CallbackService.base.url ${oozie.base.url}/callback Base callback URL used by ActionExecutors. oozie.service.CallbackService.early.requeue.max.retries 5 If Oozie receives a callback too early (while the action is in PREP state), it will requeue the command this many times to give the action time to transition to RUNNING. oozie.servlet.CallbackServlet.max.data.len 2048 Max size in characters for the action completion data output. oozie.external.stats.max.size -1 Max size in bytes for action stats. -1 means infinite value. oozie.JobCommand.job.console.url ${oozie.base.url}?job= Base console URL for a workflow job. oozie.service.ActionService.executor.classes org.apache.oozie.action.decision.DecisionActionExecutor, org.apache.oozie.action.hadoop.JavaActionExecutor, org.apache.oozie.action.hadoop.FsActionExecutor, org.apache.oozie.action.hadoop.MapReduceActionExecutor, org.apache.oozie.action.hadoop.PigActionExecutor, org.apache.oozie.action.hadoop.HiveActionExecutor, org.apache.oozie.action.hadoop.ShellActionExecutor, org.apache.oozie.action.hadoop.SqoopActionExecutor, org.apache.oozie.action.hadoop.DistcpActionExecutor, org.apache.oozie.action.hadoop.Hive2ActionExecutor, org.apache.oozie.action.ssh.SshActionExecutor, org.apache.oozie.action.oozie.SubWorkflowActionExecutor, org.apache.oozie.action.email.EmailActionExecutor, org.apache.oozie.action.hadoop.SparkActionExecutor List of ActionExecutors classes (separated by commas). Only action types with associated executors can be used in workflows. oozie.service.ActionService.executor.ext.classes List of ActionExecutors extension classes (separated by commas). Only action types with associated executors can be used in workflows. This property is a convenience property to add extensions to the built in executors without having to include all the built in ones. oozie.service.ActionCheckerService.action.check.interval 60 The frequency at which the ActionCheckService will run. oozie.service.ActionCheckerService.action.check.delay 600 The time, in seconds, between an ActionCheck for the same action. oozie.service.ActionCheckerService.callable.batch.size 10 This value determines the number of actions which will be batched together to be executed by a single thread. oozie.service.StatusTransitService.statusTransit.interval 60 The frequency in seconds at which the StatusTransitService will run. oozie.service.StatusTransitService.backward.support.for.coord.status false true, if coordinator job submits using 'uri:oozie:coordinator:0.1' and wants to keep Oozie 2.x status transit. if set true, 1. SUCCEEDED state in coordinator job means materialization done. 2. No DONEWITHERROR state in coordinator job 3. No PAUSED or PREPPAUSED state in coordinator job 4. PREPSUSPENDED becomes SUSPENDED in coordinator job oozie.service.StatusTransitService.backward.support.for.states.without.error true true, if you want to keep Oozie 3.2 status transit. Change it to false for Oozie 4.x releases. if set true, No states like RUNNINGWITHERROR, SUSPENDEDWITHERROR and PAUSEDWITHERROR for coordinator and bundle oozie.service.PauseTransitService.PauseTransit.interval 60 The frequency in seconds at which the PauseTransitService will run. oozie.action.max.output.data 2048 Max size in characters for output data. oozie.action.fs.glob.max 1000 Maximum number of globbed files. oozie.action.launcher.mapreduce.job.ubertask.enable true Enables Uber Mode for the launcher job in YARN/Hadoop 2 (no effect in Hadoop 1) for all action types by default. This can be overridden on a per-action-type basis by setting oozie.action.#action-type#.launcher.mapreduce.job.ubertask.enable in oozie-site.xml (where #action-type# is the action type; for example, "pig"). And that can be overridden on a per-action basis by setting oozie.launcher.mapreduce.job.ubertask.enable in an action's configuration section in a workflow. In summary, the priority is this: 1. action's configuration section in a workflow 2. oozie.action.#action-type#.launcher.mapreduce.job.ubertask.enable in oozie-site 3. oozie.action.launcher.mapreduce.job.ubertask.enable in oozie-site oozie.action.shell.launcher.mapreduce.job.ubertask.enable false The Shell action may have issues with the $PATH environment when using Uber Mode, and so Uber Mode is disabled by default for it. See oozie.action.launcher.mapreduce.job.ubertask.enable oozie.action.launcher.yarn.timeline-service.enabled false Enables/disables getting delegation tokens for ATS for the launcher job in YARN/Hadoop 2.6 (no effect in Hadoop 1) for all action types by default if tez-site.xml is present in distributed cache. This can be overridden on a per-action basis by setting oozie.launcher.yarn.timeline-service.enabled in an action's configuration section in a workflow. oozie.action.retries.max 3 The number of retries for executing an action in case of failure oozie.action.retry.interval 10 The interval between retries of an action in case of failure oozie.action.retry.policy periodic Retry policy of an action in case of failure. Possible values are periodic/exponential oozie.action.ssh.delete.remote.tmp.dir true If set to true, it will delete temporary directory at the end of execution of ssh action. oozie.action.ssh.http.command curl Command to use for callback to oozie, normally is 'curl' or 'wget'. The command must available in PATH environment variable of the USER@HOST box shell. oozie.action.ssh.http.command.post.options --data-binary @#stdout --request POST --header "content-type:text/plain" The callback command POST options. Used when the ouptut of the ssh action is captured. oozie.action.ssh.allow.user.at.host true Specifies whether the user specified by the ssh action is allowed or is to be replaced by the Job user oozie.action.subworkflow.max.depth 50 The maximum depth for subworkflows. For example, if set to 3, then a workflow can start subwf1, which can start subwf2, which can start subwf3; but if subwf3 tries to start subwf4, then the action will fail. This is helpful in preventing errant workflows from starting infintely recursive subworkflows. oozie.service.HadoopAccessorService.kerberos.enabled false Indicates if Oozie is configured to use Kerberos. local.realm LOCALHOST Kerberos Realm used by Oozie and Hadoop. Using 'local.realm' to be aligned with Hadoop configuration oozie.service.HadoopAccessorService.keytab.file ${user.home}/oozie.keytab Location of the Oozie user keytab file. oozie.service.HadoopAccessorService.kerberos.principal ${user.name}/localhost@${local.realm} Kerberos principal for Oozie service. oozie.service.HadoopAccessorService.jobTracker.whitelist Whitelisted job tracker for Oozie service. oozie.service.HadoopAccessorService.nameNode.whitelist Whitelisted job tracker for Oozie service. oozie.service.HadoopAccessorService.hadoop.configurations *=hadoop-conf Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of the Hadoop service (JobTracker, YARN, HDFS). The wildcard '*' configuration is used when there is no exact match for an authority. The HADOOP_CONF_DIR contains the relevant Hadoop *-site.xml files. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute (i.e. to point to Hadoop client conf/ directories in the local filesystem. oozie.service.HadoopAccessorService.action.configurations *=action-conf Comma separated AUTHORITY=ACTION_CONF_DIR, where AUTHORITY is the HOST:PORT of the Hadoop MapReduce service (JobTracker, YARN). The wildcard '*' configuration is used when there is no exact match for an authority. The ACTION_CONF_DIR may contain ACTION.xml files where ACTION is the action type ('java', 'map-reduce', 'pig', 'hive', 'sqoop', etc.). If the ACTION.xml file exists, its properties will be used as defaults properties for the action. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute (i.e. to point to Hadoop client conf/ directories in the local filesystem. oozie.service.HadoopAccessorService.action.configurations.load.default.resources true true means that default and site xml files of hadoop (core-default, core-site, hdfs-default, hdfs-site, mapred-default, mapred-site, yarn-default, yarn-site) are parsed into actionConf on Oozie server. false means that site xml files are not loaded on server, instead loaded on launcher node. This is only done for pig and hive actions which handle loading those files automatically from the classpath on launcher task. It defaults to true. oozie.credentials.credentialclasses A list of credential class mapping for CredentialsProvider oozie.actions.main.classnames distcp=org.apache.hadoop.tools.DistCp A list of class name mapping for Action classes oozie.service.WorkflowAppService.system.libpath /user/${user.name}/share/lib System library path to use for workflow applications. This path is added to workflow application if their job properties sets the property 'oozie.use.system.libpath' to true. oozie.command.default.lock.timeout 5000 Default timeout (in milliseconds) for commands for acquiring an exclusive lock on an entity. oozie.command.default.requeue.delay 10000 Default time (in milliseconds) for commands that are requeued for delayed execution. oozie.service.LiteWorkflowStoreService.user.retry.max 3 Automatic retry max count for workflow action is 3 in default. oozie.service.LiteWorkflowStoreService.user.retry.inteval 10 Automatic retry interval for workflow action is in minutes and the default value is 10 minutes. oozie.service.LiteWorkflowStoreService.user.retry.error.code JA008,JA009,JA017,JA018,JA019,FS009,FS008,FS014 Automatic retry interval for workflow action is handled for these specified error code: FS009, FS008 is file exists error when using chmod in fs action. FS014 is permission error in fs action JA018 is output directory exists error in workflow map-reduce action. JA019 is error while executing distcp action. JA017 is job not exists error in action executor. JA008 is FileNotFoundException in action executor. JA009 is IOException in action executor. ALL is the any kind of error in action executor. oozie.service.LiteWorkflowStoreService.user.retry.error.code.ext Automatic retry interval for workflow action is handled for these specified extra error code: ALL is the any kind of error in action executor. oozie.service.LiteWorkflowStoreService.node.def.version _oozie_inst_v_1 NodeDef default version, _oozie_inst_v_0 or _oozie_inst_v_1 oozie.authentication.type simple Defines authentication used for Oozie HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# oozie.server.authentication.type ${oozie.authentication.type} Defines authentication used for Oozie server communicating to other Oozie server over HTTP(s). Supported values are: simple | kerberos | #AUTHENTICATOR_CLASSNAME# oozie.authentication.token.validity 36000 Indicates how long (in seconds) an authentication token is valid before it has to be renewed. oozie.authentication.cookie.domain The domain to use for the HTTP cookie that stores the authentication token. In order to authentiation to work correctly across multiple hosts the domain must be correctly set. oozie.authentication.simple.anonymous.allowed true Indicates if anonymous requests are allowed when using 'simple' authentication. oozie.authentication.kerberos.principal HTTP/localhost@${local.realm} Indicates the Kerberos principal to be used for HTTP endpoint. The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification. oozie.authentication.kerberos.keytab ${oozie.service.HadoopAccessorService.keytab.file} Location of the keytab file with the credentials for the principal. Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop. oozie.authentication.kerberos.name.rules DEFAULT The kerberos names rules is to resolve kerberos principal names, refer to Hadoop's KerberosName for more details. oozie.coord.execution.none.tolerance 1 Default time tolerance in minutes after action nominal time for an action to be skipped when execution order is "NONE" oozie.coord.actions.default.length 1000 Default number of coordinator actions to be retrieved by the info command oozie.validate.ForkJoin true If true, fork and join should be validated at wf submission time. oozie.coord.action.get.all.attributes false Setting to true is not recommended as coord job/action info will bring all columns of the action in memory. Set it true only if backward compatibility for action/job info is required. oozie.service.HadoopAccessorService.supported.filesystems hdfs,hftp,webhdfs Enlist the different filesystems supported for federation. If wildcard "*" is specified, then ALL file schemes will be allowed. oozie.service.URIHandlerService.uri.handlers org.apache.oozie.dependency.FSURIHandler Enlist the different uri handlers supported for data availability checks. oozie.notification.url.connection.timeout 10000 Defines the timeout, in milliseconds, for Oozie HTTP notification callbacks. Oozie does HTTP notifications for workflow jobs which set the 'oozie.wf.action.notification.url', 'oozie.wf.worklfow.notification.url' and/or 'oozie.coord.action.notification.url' properties in their job.properties. Refer to section '5 Oozie Notifications' in the Workflow specification for details. oozie.hadoop-2.0.2-alpha.workaround.for.distributed.cache false Due to a bug in Hadoop 2.0.2-alpha, MAPREDUCE-4820, launcher jobs fail to set the distributed cache for the action job because the local JARs are implicitly included triggering a duplicate check. This flag removes the distributed cache files for the action as they'll be included from the local JARs of the JobClient (MRApps) submitting the action job from the launcher. oozie.service.EventHandlerService.filter.app.types workflow_job, coordinator_action The app-types among workflow/coordinator/bundle job/action for which for which events system is enabled. oozie.service.EventHandlerService.event.queue org.apache.oozie.event.MemoryEventQueue The implementation for EventQueue in use by the EventHandlerService. oozie.service.EventHandlerService.event.listeners org.apache.oozie.jms.JMSJobEventListener oozie.service.EventHandlerService.queue.size 10000 Maximum number of events to be contained in the event queue. oozie.service.EventHandlerService.worker.interval 30 The default interval (seconds) at which the worker threads will be scheduled to run and process events. oozie.service.EventHandlerService.batch.size 10 The batch size for batched draining per thread from the event queue. oozie.service.EventHandlerService.worker.threads 3 Number of worker threads to be scheduled to run and process events. oozie.sla.service.SLAService.capacity 5000 Maximum number of sla records to be contained in the memory structure. oozie.sla.service.SLAService.alert.events END_MISS Default types of SLA events for being alerted of. oozie.sla.service.SLAService.calculator.impl org.apache.oozie.sla.SLACalculatorMemory The implementation for SLACalculator in use by the SLAService. oozie.sla.service.SLAService.job.event.latency 90000 Time in milliseconds to account of latency of getting the job status event to compare against and decide sla miss/met oozie.sla.service.SLAService.check.interval 30 Time interval, in seconds, at which SLA Worker will be scheduled to run oozie.sla.disable.alerts.older.than 48 Time threshold, in HOURS, for disabling SLA alerting for jobs whose nominal time is older than this. oozie.zookeeper.connection.string localhost:2181 Comma-separated values of host:port pairs of the ZooKeeper servers. oozie.zookeeper.namespace oozie The namespace to use. All of the Oozie Servers that are planning on talking to each other should have the same namespace. oozie.zookeeper.connection.timeout 180 Default ZK connection timeout (in sec). If connection is lost for more than timeout, then Oozie server will shutdown itself if oozie.zookeeper.server.shutdown.ontimeout is true. oozie.zookeeper.server.shutdown.ontimeout true If true, Oozie server will shutdown itself on ZK connection timeout. oozie.http.hostname localhost Oozie server host name. oozie.http.port 11000 Oozie server port. oozie.instance.id ${oozie.http.hostname} Each Oozie server should have its own unique instance id. The default is system property =${OOZIE_HTTP_HOSTNAME}= (i.e. the hostname). oozie.service.ShareLibService.mapping.file Sharelib mapping files contains list of key=value, where key will be the sharelib name for the action and value is a comma separated list of DFS directories or jar files. Example. oozie.pig_10=hdfs:///share/lib/pig/pig-0.10.1/lib/ oozie.pig=hdfs:///share/lib/pig/pig-0.11.1/lib/ oozie.distcp=hdfs:///share/lib/hadoop-2.2.0/share/hadoop/tools/lib/hadoop-distcp-2.2.0.jar oozie.service.ShareLibService.fail.fast.on.startup false Fails server starup if sharelib initilzation fails. oozie.service.ShareLibService.purge.interval 1 How often, in days, Oozie should check for old ShareLibs and LauncherLibs to purge from HDFS. oozie.service.ShareLibService.temp.sharelib.retention.days 7 ShareLib retention time in days. oozie.action.ship.launcher.jar false Specifies whether launcher jar is shipped or not. oozie.action.jobinfo.enable false JobInfo will contain information of bundle, coordinator, workflow and actions. If enabled, hadoop job will have property(oozie.job.info) which value is multiple key/value pair separated by ",". This information can be used for analytics like how many oozie jobs are submitted for a particular period, what is the total number of failed pig jobs, etc from mapreduce job history logs and configuration. User can also add custom workflow property to jobinfo by adding property which prefix with "oozie.job.info." Eg. oozie.job.info="bundle.id=,bundle.name=,coord.name=,coord.nominal.time=,coord.name=,wf.id=, wf.name=,action.name=,action.type=,launcher=true" oozie.service.XLogStreamingService.max.log.scan.duration -1 Max log scan duration in hours. If log scan request end_date - start_date > value, then exception is thrown to reduce the scan duration. -1 indicate no limit. oozie.service.XLogStreamingService.actionlist.max.log.scan.duration -1 Max log scan duration in hours for coordinator job when list of actions are specified. If log streaming request end_date - start_date > value, then exception is thrown to reduce the scan duration. -1 indicate no limit. This setting is separate from max.log.scan.duration as we want to allow higher durations when actions are specified. oozie.service.JvmPauseMonitorService.warn-threshold.ms 10000 The JvmPauseMonitorService runs a thread that repeatedly tries to detect when the JVM pauses, which could indicate that the JVM or host machine is overloaded or other problems. This thread sleeps for 500ms; if it sleeps for significantly longer, then there is likely a problem. This property specifies the threadshold for when Oozie should log a WARN level message; there is also a counter named "jvm.pause.warn-threshold". oozie.service.JvmPauseMonitorService.info-threshold.ms 1000 The JvmPauseMonitorService runs a thread that repeatedly tries to detect when the JVM pauses, which could indicate that the JVM or host machine is overloaded or other problems. This thread sleeps for 500ms; if it sleeps for significantly longer, then there is likely a problem. This property specifies the threadshold for when Oozie should log an INFO level message; there is also a counter named "jvm.pause.info-threshold". oozie.service.ZKLocksService.locks.reaper.threshold 300 The frequency at which the ChildReaper will run. Duration should be in sec. Default is 5 min. oozie.service.ZKLocksService.locks.reaper.threads 2 Number of fixed threads used by ChildReaper to delete empty locks. oozie.service.AbandonedCoordCheckerService.check.interval 1440 Interval, in minutes, at which AbandonedCoordCheckerService should run. oozie.service.AbandonedCoordCheckerService.check.delay 60 Delay, in minutes, at which AbandonedCoordCheckerService should run. oozie.service.AbandonedCoordCheckerService.failure.limit 25 Failure limit. A job is considered to be abandoned/faulty if total number of actions in failed/timedout/suspended >= "Failure limit" and there are no succeeded action. oozie.service.AbandonedCoordCheckerService.kill.jobs false If true, AbandonedCoordCheckerService will kill abandoned coords. oozie.service.AbandonedCoordCheckerService.job.older.than 2880 In minutes, job will be considered as abandoned/faulty if job is older than this value. oozie.notification.proxy System level proxy setting for job notifications. oozie.wf.rerun.disablechild false By setting this option, workflow rerun will be disabled if parent workflow or coordinator exist and it will only rerun through parent. oozie.use.system.libpath false Default value of oozie.use.system.libpath. If user haven't specified =oozie.use.system.libpath= in the job.properties and this value is true and Oozie will include sharelib jars for workflow. oozie.service.PauseTransitService.callable.batch.size 10 This value determines the number of callable which will be batched together to be executed by a single thread. oozie.configuration.substitute.depth 20 This value determines the depth of substitution in configurations. If set -1, No limitation on substitution. oozie.service.SparkConfigurationService.spark.configurations *=spark-conf Comma separated AUTHORITY=SPARK_CONF_DIR, where AUTHORITY is the HOST:PORT of the ResourceManager of a YARN cluster. The wildcard '*' configuration is used when there is no exact match for an authority. The SPARK_CONF_DIR contains the relevant spark-defaults.conf properties file. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute. This is only used when the Spark master is set to either "yarn-client" or "yarn-cluster". oozie.email.attachment.enabled true This value determines whether to support email attachment of a file on HDFS. Set it false if there is any security concern. sahara-8.0.0/sahara/plugins/vanilla/edp_engine.py0000666000175100017510000000266013245514472022034 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import exceptions as ex from sahara.plugins import utils as u from sahara.plugins.vanilla import utils as vu from sahara.service.edp.oozie import engine as edp_engine class EdpOozieEngine(edp_engine.OozieJobEngine): def get_hdfs_user(self): return 'hadoop' def get_name_node_uri(self, cluster): return cluster['info']['HDFS']['NameNode'] def get_oozie_server_uri(self, cluster): return cluster['info']['JobFlow']['Oozie'] + "/oozie/" def get_oozie_server(self, cluster): return vu.get_oozie(cluster) def validate_job_execution(self, cluster, job, data): oo_count = u.get_instances_count(cluster, 'oozie') if oo_count != 1: raise ex.InvalidComponentCountException('oozie', '1', oo_count) super(EdpOozieEngine, self).validate_job_execution(cluster, job, data) sahara-8.0.0/sahara/plugins/utils.py0000666000175100017510000000732713245514472017456 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import itertools from oslo_utils import netutils from six.moves.urllib import parse as urlparse from sahara.i18n import _ from sahara.plugins import base as plugins_base from sahara.plugins import exceptions as ex def get_node_groups(cluster, node_process=None): return [ng for ng in cluster.node_groups if (node_process is None or node_process in ng.node_processes)] def get_instances_count(cluster, node_process=None): return sum([ng.count for ng in get_node_groups(cluster, node_process)]) def get_instances(cluster, node_process=None): nodes = get_node_groups(cluster, node_process) return list(itertools.chain(*[node.instances for node in nodes])) def get_instance(cluster, node_process): instances = get_instances(cluster, node_process) if len(instances) > 1: raise ex.InvalidComponentCountException( node_process, _('0 or 1'), len(instances)) return instances[0] if instances else None def generate_host_names(nodes): return "\n".join([n.hostname() for n in nodes]) def generate_fqdn_host_names(nodes): return "\n".join([n.fqdn() for n in nodes]) def get_port_from_address(address): parse_result = urlparse.urlparse(address) # urlparse do not parse values like 0.0.0.0:8000, # netutils do not parse values like http://localhost:8000, # so combine approach is using if parse_result.port: return parse_result.port else: return netutils.parse_host_port(address)[1] def instances_with_services(instances, node_processes): node_processes = set(node_processes) return list(filter( lambda x: node_processes.intersection( x.node_group.node_processes), instances)) def start_process_event_message(process): return _("Start the following process(es): {process}").format( process=process) def get_config_value_or_default( service=None, name=None, cluster=None, config=None): if not config: if not service or not name: raise RuntimeError(_("Unable to retrieve config details")) default_value = None else: service = config.applicable_target name = config.name default_value = config.default_value cluster_configs = cluster.cluster_configs if cluster_configs.get(service, {}).get(name, None) is not None: return cluster_configs.get(service, {}).get(name, None) # Try getting config from the cluster. for ng in cluster.node_groups: if (ng.configuration().get(service) and ng.configuration()[service].get(name)): return ng.configuration()[service][name] # Find and return the default if default_value is not None: return default_value plugin = plugins_base.PLUGINS.get_plugin(cluster.plugin_name) configs = plugin.get_all_configs(cluster.hadoop_version) for config in configs: if config.applicable_target == service and config.name == name: return config.default_value raise RuntimeError(_("Unable to get parameter '%(param_name)s' from " "service %(service)s"), {'param_name': name, 'service': service}) sahara-8.0.0/sahara/plugins/labels.py0000666000175100017510000002011213245514472017543 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from oslo_log import log as logging from sahara import conductor as cond from sahara import context from sahara import exceptions as ex from sahara.i18n import _ conductor = cond.API LOG = logging.getLogger(__name__) STABLE = { 'name': 'stable', 'mutable': False, 'description': "Indicates that plugin or its version are stable to be used" } DEPRECATED = { 'name': 'deprecated', 'mutable': False, 'description': "Plugin or its version is deprecated and will be removed " "in future releases. Please, consider to use another " "plugin or its version." } ENABLED = { 'name': 'enabled', 'mutable': True, 'description': "Plugin or its version is enabled and can be used by user." } HIDDEN = { 'name': 'hidden', 'mutable': True, 'description': "Existence of plugin or its version is hidden, but " "still can be used for cluster creation by CLI and " "directly by client." } PLUGIN_LABELS_SCOPE = 'plugin_labels' VERSION_LABELS_SCOPE = 'version_labels' MUTABLE = 'mutable' LABEL_OBJECT = { 'type': 'object', 'properties': { 'status': { 'type': 'boolean', } }, "additionalProperties": False, } class LabelHandler(object): def __init__(self, loaded_plugins): self.plugins = loaded_plugins def get_plugin_update_validation_jsonschema(self): schema = { 'type': 'object', "additionalProperties": False, 'properties': { VERSION_LABELS_SCOPE: { 'type': 'object', 'additionalProperties': False, }, }, } ln = [label['name'] for label in self.get_labels()] labels_descr_object = { 'type': 'object', "properties": {name: copy.deepcopy(LABEL_OBJECT) for name in ln}, "additionalProperties": False } schema['properties'][PLUGIN_LABELS_SCOPE] = copy.deepcopy( labels_descr_object) all_versions = [] for plugin_name in self.plugins.keys(): plugin = self.plugins[plugin_name] all_versions.extend(plugin.get_versions()) all_versions = set(all_versions) schema['properties'][VERSION_LABELS_SCOPE]['properties'] = { ver: copy.deepcopy(labels_descr_object) for ver in all_versions } return schema def get_default_label_details(self, plugin_name): plugin = self.plugins.get(plugin_name) return plugin.get_labels() def get_label_details(self, plugin_name): try: plugin = conductor.plugin_get(context.ctx(), plugin_name) except Exception: LOG.error("Unable to retrieve plugin data from database") plugin = None if not plugin: plugin = self.get_default_label_details(plugin_name) fields = ['name', 'id', 'updated_at', 'created_at'] for field in fields: if field in plugin: del plugin[field] return plugin def get_label_full_details(self, plugin_name): return self.expand_data(self.get_label_details(plugin_name)) def get_labels(self): return [HIDDEN, STABLE, ENABLED, DEPRECATED] def get_labels_map(self): return { label['name']: label for label in self.get_labels() } def expand_data(self, plugin): plugin_labels = plugin.get(PLUGIN_LABELS_SCOPE) labels_map = self.get_labels_map() for key in plugin_labels.keys(): key_desc = labels_map.get(key) plugin_labels[key].update(key_desc) del plugin_labels[key]['name'] for version in plugin.get(VERSION_LABELS_SCOPE): vers_labels = plugin.get(VERSION_LABELS_SCOPE).get(version) for key in vers_labels.keys(): key_desc = labels_map.get(key) vers_labels[key].update(key_desc) del vers_labels[key]['name'] return plugin def _validate_labels_update(self, default_data, update_values): for label in update_values.keys(): if label not in default_data.keys(): raise ex.InvalidDataException( _("Label '%s' can't be updated because it's not " "available for plugin or its version") % label) if not default_data[label][MUTABLE]: raise ex.InvalidDataException( _("Label '%s' can't be updated because it's not " "mutable") % label) def validate_plugin_update(self, plugin_name, values): plugin = self.plugins[plugin_name] # it's important to get full details since we have mutability default = self.get_label_full_details(plugin_name) if values.get(PLUGIN_LABELS_SCOPE): pl = values.get(PLUGIN_LABELS_SCOPE) self._validate_labels_update(default[PLUGIN_LABELS_SCOPE], pl) if values.get(VERSION_LABELS_SCOPE): vl = values.get(VERSION_LABELS_SCOPE) for version in vl.keys(): if version not in plugin.get_versions(): raise ex.InvalidDataException( _("Unknown plugin version '%(version)s' of " "%(plugin)s") % { 'version': version, 'plugin': plugin_name}) self._validate_labels_update( default[VERSION_LABELS_SCOPE][version], vl[version]) def update_plugin(self, plugin_name, values): ctx = context.ctx() current = self.get_label_details(plugin_name) if not conductor.plugin_get(ctx, plugin_name): current['name'] = plugin_name conductor.plugin_create(ctx, current) del current['name'] if values.get(PLUGIN_LABELS_SCOPE): for label in values.get(PLUGIN_LABELS_SCOPE).keys(): current[PLUGIN_LABELS_SCOPE][label].update( values.get(PLUGIN_LABELS_SCOPE).get(label)) else: del current[PLUGIN_LABELS_SCOPE] if values.get(VERSION_LABELS_SCOPE): vl = values.get(VERSION_LABELS_SCOPE) for version in vl.keys(): for label in vl.get(version).keys(): current[VERSION_LABELS_SCOPE][version][label].update( vl[version][label]) else: del current[VERSION_LABELS_SCOPE] conductor.plugin_update(context.ctx(), plugin_name, current) def validate_plugin_labels(self, plugin_name, version): details = self.get_label_details(plugin_name) plb = details.get(PLUGIN_LABELS_SCOPE, {}) if not plb.get('enabled', {}).get('status'): raise ex.InvalidReferenceException( _("Plugin %s is not enabled") % plugin_name) if plb.get('deprecated', {}).get('status', False): LOG.warning("Plugin %s is deprecated and can be removed in " "the next release", plugin_name) vlb = details.get(VERSION_LABELS_SCOPE, {}).get(version, {}) if not vlb.get('enabled', {}).get('status'): raise ex.InvalidReferenceException( _("Version %(version)s of plugin %(plugin)s is not enabled") % {'version': version, 'plugin': plugin_name}) if vlb.get('deprecated', {}).get('status', False): LOG.warning("Using version %(version)s of plugin %(plugin)s is " "deprecated and can removed in next release", {'version': version, 'plugin': plugin_name}) sahara-8.0.0/sahara/plugins/fake/0000775000175100017510000000000013245515027016634 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/fake/plugin.py0000666000175100017510000001113113245514472020506 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import context from sahara.i18n import _ from sahara.plugins import exceptions as pex from sahara.plugins.fake import edp_engine from sahara.plugins import kerberos as krb from sahara.plugins import provisioning as p from sahara.plugins import utils as plugin_utils class FakePluginProvider(p.ProvisioningPluginBase): def get_title(self): return "Fake Plugin" def get_description(self): return _("It's a fake plugin that aimed to work on the CirrOS images. " "It doesn't install Hadoop. It's needed to be able to test " "provisioning part of Sahara codebase itself.") def get_versions(self): return ["0.1"] def get_labels(self): return { 'plugin_labels': { 'enabled': {'status': True}, 'hidden': {'status': True}, }, 'version_labels': { '0.1': {'enabled': {'status': True}} } } def get_node_processes(self, hadoop_version): return { "HDFS": ["namenode", "datanode"], "MapReduce": ["tasktracker", "jobtracker"], "Kerberos": [], } def get_configs(self, hadoop_version): # returning kerberos configs return krb.get_config_list() def configure_cluster(self, cluster): with context.ThreadGroup() as tg: for instance in plugin_utils.get_instances(cluster): tg.spawn('fake-write-%s' % instance.id, self._write_ops, instance) def start_cluster(self, cluster): self.deploy_kerberos(cluster) with context.ThreadGroup() as tg: for instance in plugin_utils.get_instances(cluster): tg.spawn('fake-check-%s' % instance.id, self._check_ops, instance) def deploy_kerberos(self, cluster): all_instances = plugin_utils.get_instances(cluster) namenodes = plugin_utils.get_instances(cluster, 'namenode') server = None if len(namenodes) > 0: server = namenodes[0] elif len(all_instances) > 0: server = all_instances[0] if server: krb.deploy_infrastructure(cluster, server) def scale_cluster(self, cluster, instances): with context.ThreadGroup() as tg: for instance in instances: tg.spawn('fake-scaling-%s' % instance.id, self._all_check_ops, instance) def decommission_nodes(self, cluster, instances): pass def _write_ops(self, instance): with instance.remote() as r: # check typical SSH command r.execute_command('echo "Hello, world!"') # check write file data_1 = "sp@m" r.write_file_to('test_data', data_1, run_as_root=True) # check append file data_2 = " and eggs" r.append_to_file('test_data', data_2, run_as_root=True) # check replace string r.replace_remote_string('test_data', "eggs", "pony") def _check_ops(self, instance): expected_data = "sp@m and pony" with instance.remote() as r: actual_data = r.read_file_from('test_data', run_as_root=True) if actual_data.strip() != expected_data.strip(): raise pex.HadoopProvisionError("ACTUAL:\n%s\nEXPECTED:\n%s" % ( actual_data, expected_data)) def _all_check_ops(self, instance): self._write_ops(instance) self._check_ops(instance) def get_edp_engine(self, cluster, job_type): return edp_engine.FakeJobEngine() def get_edp_job_types(self, versions=None): res = {} for vers in self.get_versions(): if not versions or vers in versions: res[vers] = edp_engine.FakeJobEngine.get_supported_job_types() return res def get_edp_config_hints(self, job_type, version): if version in self.get_versions(): return edp_engine.FakeJobEngine.get_possible_job_config(job_type) sahara-8.0.0/sahara/plugins/fake/__init__.py0000666000175100017510000000000013245514472020740 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/fake/edp_engine.py0000666000175100017510000000230313245514472021306 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp import base_engine from sahara.utils import edp class FakeJobEngine(base_engine.JobEngine): def cancel_job(self, job_execution): pass def get_job_status(self, job_execution): pass def run_job(self, job_execution): return 'engine_job_id', edp.JOB_STATUS_SUCCEEDED, None def run_scheduled_job(self, job_execution): pass def validate_job_execution(self, cluster, job, data): pass @staticmethod def get_possible_job_config(job_type): return None @staticmethod def get_supported_job_types(): return edp.JOB_TYPES_ALL sahara-8.0.0/sahara/plugins/provisioning.py0000666000175100017510000003066413245514472021044 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import base as plugins_base from sahara.utils import resources class ProvisioningPluginBase(plugins_base.PluginInterface): @plugins_base.required def get_versions(self): """Get available plugin versions :returns: A sequence of strings representing the versions For example: ["1.0.0", "1.0.1"] """ pass @plugins_base.required def get_configs(self, hadoop_version): """Get default configuration for a given plugin version :param hadoop_version: String representing a plugin version :returns: A dict containing the configuration """ pass @plugins_base.required_with_default def get_labels(self): versions = self.get_versions() default = {'enabled': {'status': True}} return { 'plugin_labels': copy.deepcopy(default), 'version_labels': { version: copy.deepcopy(default) for version in versions } } @plugins_base.required def get_node_processes(self, hadoop_version): """Get node processes of a given plugin version :param hadoop_version: String containing a plugin version :returns: A dict where the keys are the core components of the plugin and the value is a sequence of node processes for that component For example: { "HDFS": ["namenode", "datanode"], "Spark": ["master", "slave"] } """ pass @plugins_base.required_with_default def get_required_image_tags(self, hadoop_version): return [self.name, hadoop_version] @plugins_base.required_with_default def validate(self, cluster): pass @plugins_base.required_with_default def validate_scaling(self, cluster, existing, additional): pass @plugins_base.required_with_default def update_infra(self, cluster): pass @plugins_base.required def configure_cluster(self, cluster): pass @plugins_base.required def start_cluster(self, cluster): pass @plugins_base.optional def scale_cluster(self, cluster, instances): pass @plugins_base.optional def get_edp_engine(self, cluster, job_type): pass @plugins_base.optional def get_edp_job_types(self, versions=None): return {} @plugins_base.optional def get_edp_config_hints(self, job_type, version): return {} @plugins_base.required_with_default def get_open_ports(self, node_group): return [] @plugins_base.required_with_default def decommission_nodes(self, cluster, instances): pass @plugins_base.optional def get_image_arguments(self, hadoop_version): """Gets the argument set taken by the plugin's image generator Note: If the plugin can generate or validate an image but takes no arguments, please return an empty sequence rather than NotImplemented for all versions that support image generation or validation. This is used as a flag to determine whether the plugin has implemented this optional feature. :returns: A sequence with items of type sahara.plugins.images.ImageArgument """ return NotImplemented @plugins_base.optional def pack_image(self, hadoop_version, remote, test_only=False, image_arguments=None): """Packs an image for registration in Glance and use by Sahara :param remote: A remote (usually of type sahara.cli.image_pack.api.ImageRemote) that serves as a handle to the image to modify. Note that this image will be modified in-place, not copied. :param test_only: If set to True, this method will only test to ensure that the image already meets the plugin's requirements. This can be used to test images without modification. If set to False per the default, this method will modify the image if any requirements are not met. :param image_arguments: A dict of image argument name to argument value. :raises: sahara.plugins.exceptions.ImageValidationError: If the method fails to modify the image to specification (if test_only is False), or if the method finds that the image does not meet the specification (if test_only is True). :raises: sahara.plugins.exceptions.ImageValidationSpecificationError: If the specification for image generation or validation is itself in error and cannot be executed without repair. """ pass @plugins_base.optional def validate_images(self, cluster, test_only=False, image_arguments=None): """Validates the image to be used by a cluster. :param cluster: The object handle to a cluster which has active instances ready to generate remote handles. :param test_only: If set to True, this method will only test to ensure that the image already meets the plugin's requirements. This can be used to test images without modification. If set to False per the default, this method will modify the image if any requirements are not met. :param image_arguments: A dict of image argument name to argument value. :raises: sahara.plugins.exceptions.ImageValidationError: If the method fails to modify the image to specification (if test_only is False), or if the method finds that the image does not meet the specification (if test_only is True). :raises: sahara.plugins.exceptions.ImageValidationSpecificationError: If the specification for image generation or validation is itself in error and cannot be executed without repair. """ pass @plugins_base.required_with_default def on_terminate_cluster(self, cluster): pass @plugins_base.optional def recommend_configs(self, cluster, scaling=False): pass @plugins_base.required_with_default def get_health_checks(self, cluster): return [] def get_all_configs(self, hadoop_version): common = list_of_common_configs() plugin_specific_configs = self.get_configs(hadoop_version) if plugin_specific_configs: common.extend(plugin_specific_configs) return common def get_version_details(self, version): details = {} configs = self.get_all_configs(version) details['configs'] = [c.dict for c in configs] details['node_processes'] = self.get_node_processes(version) details['required_image_tags'] = self.get_required_image_tags(version) return details def to_dict(self): res = super(ProvisioningPluginBase, self).to_dict() res['versions'] = self.get_versions() return res # Some helpers for plugins def _map_to_user_inputs(self, hadoop_version, configs): config_objs = self.get_all_configs(hadoop_version) # convert config objects to applicable_target -> config_name -> obj config_objs_map = {} for config_obj in config_objs: applicable_target = config_obj.applicable_target confs = config_objs_map.get(applicable_target, {}) confs[config_obj.name] = config_obj config_objs_map[applicable_target] = confs # iterate over all configs and append UserInputs to result list result = [] for applicable_target in configs: for config_name in configs[applicable_target]: confs = config_objs_map.get(applicable_target) if not confs: raise ex.ConfigurationError( _("Can't find applicable target " "'%(applicable_target)s' for '%(config_name)s'") % {"applicable_target": applicable_target, "config_name": config_name}) conf = confs.get(config_name) if not conf: raise ex.ConfigurationError( _("Can't find config '%(config_name)s' " "in '%(applicable_target)s'") % {"config_name": config_name, "applicable_target": applicable_target}) result.append(UserInput( conf, configs[applicable_target][config_name])) return sorted(result) class Config(resources.BaseResource): """Describes a single config parameter. Config type - could be 'str', 'integer', 'boolean', 'enum'. If config type is 'enum' then list of valid values should be specified in config_values property. Priority - integer parameter which helps to differentiate all configurations in the UI. Priority decreases from the lower values to higher values. For example: "some_conf", "map_reduce", "node", is_optional=True """ def __init__(self, name, applicable_target, scope, config_type="string", config_values=None, default_value=None, is_optional=False, description=None, priority=2): self.name = name self.description = description self.config_type = config_type self.config_values = config_values self.default_value = default_value self.applicable_target = applicable_target self.scope = scope self.is_optional = is_optional self.priority = priority def to_dict(self): res = super(Config, self).to_dict() # TODO(slukjanov): all custom fields from res return res def __lt__(self, other): return self.name < other.name def __repr__(self): return '' % (self.name, self.applicable_target) class UserInput(object): """Value provided by the user for a specific config entry.""" def __init__(self, config, value): self.config = config self.value = value def __eq__(self, other): return self.config == other.config and self.value == other.value def __lt__(self, other): return (self.config, self.value) < (other.config, other.value) def __repr__(self): return '' % (self.config.name, self.value) class ValidationError(object): """Describes what is wrong with one of the values provided by user.""" def __init__(self, config, message): self.config = config self.message = message def __repr__(self): return "" % self.config.name # COMMON FOR ALL PLUGINS CONFIGS XFS_ENABLED = Config( "Enable XFS", 'general', 'cluster', priority=1, default_value=True, config_type="bool", is_optional=True, description='Enables XFS for formatting' ) DISKS_PREPARING_TIMEOUT = Config( "Timeout for disk preparing", 'general', 'cluster', priority=1, default_value=300, config_type="int", is_optional=True, description='Timeout for preparing disks, formatting and mounting' ) NTP_URL = Config( "URL of NTP server", 'general', 'cluster', priority=1, default_value='', is_optional=True, description='URL of the NTP server for synchronization time on cluster' ' instances' ) NTP_ENABLED = Config( "Enable NTP service", 'general', 'cluster', priority=1, default_value=True, config_type="bool", description='Enables NTP service for synchronization time on cluster ' 'instances' ) HEAT_WAIT_CONDITION_TIMEOUT = Config( "Heat Wait Condition timeout", "general", "cluster", priority=1, config_type="int", default_value=3600, is_optional=True, description="The number of seconds to wait for the instance to boot") def list_of_common_configs(): return [DISKS_PREPARING_TIMEOUT, NTP_ENABLED, NTP_URL, HEAT_WAIT_CONDITION_TIMEOUT, XFS_ENABLED] sahara-8.0.0/sahara/plugins/resources/0000775000175100017510000000000013245515027017740 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/plugins/resources/create-principal-keytab0000666000175100017510000000040013245514472024361 0ustar zuulzuul00000000000000#!/bin/bash mkdir -p /tmp/sahara-kerberos/ kadmin -p %(admin_principal)s < 0: decided_status = common.HEALTH_STATUS_RED else: decided_status = common.HEALTH_STATUS_YELLOW return cond.cluster_verification_update( context.ctx(), ver.id, {'status': decided_status}) def verification_run(cluster): ctx = context.ctx() LOG.debug("Running verification for the cluster") ver, cluster = _prepare_verification_running(ctx, cluster) _execute_health_checks(ctx, cluster) return _decide_status_for_verification(ctx, ver) def handle_verification(cluster, values): cluster = cond.cluster_get(context.ctx(), cluster) context.set_current_cluster_id(cluster.id) values = {} if not values else values status = values.get('verification', {}).get('status', None) if status == common.VERIFICATIONS_START_OPS: verification_run(cluster) def update_verification_required(values): if values.get('verification', {}).get('status', None): return True return False def get_verification_periodic_interval(): return (CONF.cluster_verifications.verification_periodic_interval if CONF.cluster_verifications.verification_enable else -1) sahara-8.0.0/sahara/service/api/0000775000175100017510000000000013245515027016456 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/api/v2/0000775000175100017510000000000013245515027017005 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/api/v2/cluster_templates.py0000666000175100017510000000254013245514472023124 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor as c from sahara import context conductor = c.API # ClusterTemplate ops def get_cluster_templates(**kwargs): return conductor.cluster_template_get_all(context.ctx(), regex_search=True, **kwargs) def get_cluster_template(id): return conductor.cluster_template_get(context.ctx(), id) def create_cluster_template(values): return conductor.cluster_template_create(context.ctx(), values) def terminate_cluster_template(id): return conductor.cluster_template_destroy(context.ctx(), id) def update_cluster_template(id, values): return conductor.cluster_template_update(context.ctx(), id, values) def export_cluster_template(id): return conductor.cluster_template_get(context.ctx(), id) sahara-8.0.0/sahara/service/api/v2/data_sources.py0000666000175100017510000000225213245514472022041 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor as c from sahara import context conductor = c.API def get_data_sources(**kwargs): return conductor.data_source_get_all(context.ctx(), regex_search=True, **kwargs) def get_data_source(id): return conductor.data_source_get(context.ctx(), id) def delete_data_source(id): conductor.data_source_destroy(context.ctx(), id) def register_data_source(values): return conductor.data_source_create(context.ctx(), values) def data_source_update(id, values): return conductor.data_source_update(context.ctx(), id, values) sahara-8.0.0/sahara/service/api/v2/job_binaries.py0000666000175100017510000000266513245514472022023 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor as c from sahara import context from sahara.service.edp.job_binaries import manager as jb_manager conductor = c.API def create_job_binary(values): return conductor.job_binary_create(context.ctx(), values) def get_job_binaries(**kwargs): return conductor.job_binary_get_all(context.ctx(), regex_search=True, **kwargs) def get_job_binary(id): return conductor.job_binary_get(context.ctx(), id) def update_job_binary(id, values): return conductor.job_binary_update(context.ctx(), id, values) def delete_job_binary(id): conductor.job_binary_destroy(context.ctx(), id) def get_job_binary_data(id): job_binary = conductor.job_binary_get(context.ctx(), id) return jb_manager.JOB_BINARIES.get_job_binary(job_binary.type). \ get_raw_data(job_binary, with_context=True) sahara-8.0.0/sahara/service/api/v2/plugins.py0000666000175100017510000000166613245514472021056 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins import base as plugin_base # Plugins ops def get_plugins(): return plugin_base.PLUGINS.get_plugins(serialized=True) def get_plugin(plugin_name, version=None): return plugin_base.PLUGINS.serialize_plugin(plugin_name, version) def update_plugin(plugin_name, values): return plugin_base.PLUGINS.update_plugin(plugin_name, values) sahara-8.0.0/sahara/service/api/v2/images.py0000666000175100017510000000517013245514472020634 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor as c from sahara.utils.openstack import base as b from sahara.utils.openstack import images as sahara_images conductor = c.API # Image Registry def get_images(name, tags): return b.execute_with_retries( sahara_images.image_manager().list_registered, name, tags) def get_image(**kwargs): if len(kwargs) == 1 and 'id' in kwargs: return b.execute_with_retries( sahara_images.image_manager().get, kwargs['id']) else: return b.execute_with_retries( sahara_images.image_manager().find, **kwargs) def get_registered_image(id): return b.execute_with_retries( sahara_images.image_manager().get_registered_image, id) def register_image(image_id, username, description=None): manager = sahara_images.image_manager() b.execute_with_retries( manager.set_image_info, image_id, username, description) return b.execute_with_retries(manager.get, image_id) def unregister_image(image_id): manager = sahara_images.image_manager() b.execute_with_retries(manager.unset_image_info, image_id) return b.execute_with_retries(manager.get, image_id) def get_image_tags(image_id): return b.execute_with_retries( sahara_images.image_manager().get, image_id).tags def set_image_tags(image_id, tags): manager = sahara_images.image_manager() image_obj = b.execute_with_retries(manager.get, image_id) org_tags = frozenset(image_obj.tags) new_tags = frozenset(tags) to_add = list(new_tags - org_tags) to_remove = list(org_tags - new_tags) if to_add: b.execute_with_retries(manager.tag, image_id, to_add) if to_remove: b.execute_with_retries(manager.untag, image_id, to_remove) return b.execute_with_retries(manager.get, image_id) def remove_image_tags(image_id): manager = sahara_images.image_manager() image_obj = b.execute_with_retries(manager.get, image_id) tags = image_obj.tags b.execute_with_retries(manager.untag, image_id, tags) return b.execute_with_retries(manager.get, image_id) sahara-8.0.0/sahara/service/api/v2/clusters.py0000666000175100017510000001367013245514472021237 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import excutils import six from sahara import conductor as c from sahara import context from sahara.plugins import base as plugin_base from sahara.service import api from sahara.service.health import verification_base from sahara.service import quotas from sahara.utils import cluster as c_u from sahara.utils import general as g from sahara.utils.notification import sender conductor = c.API # Cluster ops def get_clusters(**kwargs): return conductor.cluster_get_all(context.ctx(), regex_search=True, **kwargs) def get_cluster(id, show_progress=False): return conductor.cluster_get(context.ctx(), id, show_progress) def scale_cluster(id, data): context.set_current_cluster_id(id) ctx = context.ctx() cluster = conductor.cluster_get(ctx, id) plugin = plugin_base.PLUGINS.get_plugin(cluster.plugin_name) existing_node_groups = data.get('resize_node_groups', []) additional_node_groups = data.get('add_node_groups', []) # the next map is the main object we will work with # to_be_enlarged : {node_group_id: desired_amount_of_instances} to_be_enlarged = {} node_group_instance_map = {} for ng in existing_node_groups: ng_id = g.find(cluster.node_groups, name=ng['name'])['id'] to_be_enlarged.update({ng_id: ng['count']}) if 'instances' in ng: node_group_instance_map.update({ng_id: ng['instances']}) additional = construct_ngs_for_scaling(cluster, additional_node_groups) cluster = conductor.cluster_get(ctx, cluster) _add_ports_for_auto_sg(ctx, cluster, plugin) try: cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_VALIDATING) quotas.check_scaling(cluster, to_be_enlarged, additional) plugin.recommend_configs(cluster, scaling=True) plugin.validate_scaling(cluster, to_be_enlarged, additional) except Exception as e: with excutils.save_and_reraise_exception(): c_u.clean_cluster_from_empty_ng(cluster) c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_ACTIVE, six.text_type(e)) # If we are here validation is successful. # So let's update to_be_enlarged map: to_be_enlarged.update(additional) for node_group in cluster.node_groups: if node_group.id not in to_be_enlarged: to_be_enlarged[node_group.id] = node_group.count api.OPS.provision_scaled_cluster(id, to_be_enlarged, node_group_instance_map) return cluster def create_cluster(values): plugin = plugin_base.PLUGINS.get_plugin(values['plugin_name']) return _cluster_create(values, plugin) def create_multiple_clusters(values): num_of_clusters = values['count'] clusters = [] plugin = plugin_base.PLUGINS.get_plugin(values['plugin_name']) for counter in range(num_of_clusters): cluster_dict = values.copy() cluster_name = cluster_dict['name'] cluster_dict['name'] = get_multiple_cluster_name(num_of_clusters, cluster_name, counter + 1) cluster = _cluster_create(cluster_dict, plugin) clusters.append(cluster.id) clusters_dict = {'clusters': clusters} return clusters_dict def _cluster_create(values, plugin): ctx = context.ctx() cluster = conductor.cluster_create(ctx, values) context.set_current_cluster_id(cluster.id) sender.status_notify(cluster.id, cluster.name, "New", "create") _add_ports_for_auto_sg(ctx, cluster, plugin) # validating cluster try: plugin.recommend_configs(cluster) cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_VALIDATING) quotas.check_cluster(cluster) plugin.validate(cluster) except Exception as e: with excutils.save_and_reraise_exception(): c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_ERROR, six.text_type(e)) api.OPS.provision_cluster(cluster.id) return cluster def get_multiple_cluster_name(num_of_clusters, name, counter): return "%%s-%%0%dd" % len(str(num_of_clusters)) % (name, counter) def _add_ports_for_auto_sg(ctx, cluster, plugin): for ng in cluster.node_groups: if ng.auto_security_group: ports = {'open_ports': plugin.get_open_ports(ng)} conductor.node_group_update(ctx, ng, ports) def terminate_cluster(id, force=False): context.set_current_cluster_id(id) cluster = c_u.change_cluster_status(id, c_u.CLUSTER_STATUS_DELETING) if cluster is None: return api.OPS.terminate_cluster(id, force) sender.status_notify(cluster.id, cluster.name, cluster.status, "delete") def update_cluster(id, values): if verification_base.update_verification_required(values): api.OPS.handle_verification(id, values) return conductor.cluster_get(context.ctx(), id) return conductor.cluster_update(context.ctx(), id, values) def construct_ngs_for_scaling(cluster, additional_node_groups): ctx = context.ctx() additional = {} for ng in additional_node_groups: count = ng['count'] ng['count'] = 0 ng_id = conductor.node_group_add(ctx, cluster, ng) additional.update({ng_id: count}) return additional sahara-8.0.0/sahara/service/api/v2/__init__.py0000666000175100017510000000000013245514472021111 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/api/v2/jobs.py0000666000175100017510000000661113245514472020325 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_log import log as logging from sahara import conductor as c from sahara import context from sahara import exceptions as ex from sahara.service import api from sahara.service.edp import job_manager as manager from sahara.utils import edp from sahara.utils import proxy as p conductor = c.API LOG = logging.getLogger(__name__) def execute_job(data): # Elements common to all job types job_template_id = data['job_template_id'] cluster_id = data['cluster_id'] configs = data.get('job_configs', {}) interface = data.get('interface', {}) # Not in Java job types but present for all others input_id = data.get('input_id', None) output_id = data.get('output_id', None) # Since we will use a unified class in the database, we pass # a superset for all job types # example configs['start'] = '2015-05-12T08:55Z' frequency = 5 mins # the job will starts from 2015-05-12T08:55Z, runs every 5 mins job_execution_info = data.get('job_execution_info', {}) configs['job_execution_info'] = job_execution_info job_ex_dict = {'input_id': input_id, 'output_id': output_id, 'job_id': job_template_id, 'cluster_id': cluster_id, 'info': {'status': edp.JOB_STATUS_PENDING}, 'job_configs': configs, 'extra': {}, 'interface': interface} job_execution = conductor.job_execution_create(context.ctx(), job_ex_dict) context.set_current_job_execution_id(job_execution.id) # check to use proxy user if p.job_execution_requires_proxy_user(job_execution): try: p.create_proxy_user_for_job_execution(job_execution) except ex.SaharaException as e: LOG.error("Can't run job execution. " "(Reasons: {reason})".format(reason=e)) conductor.job_execution_destroy(context.ctx(), job_execution) raise e api.OPS.run_edp_job(job_execution.id) return job_execution def job_execution_list(**kwargs): return conductor.job_execution_get_all(context.ctx(), regex_search=True, **kwargs) def get_job_execution(id, refresh_status=False): if refresh_status: return manager.get_job_status(id) else: return conductor.job_execution_get(context.ctx(), id) def update_job_execution(id, values): _update_status(values.pop("info", None), id) return conductor.job_execution_update(context.ctx(), id, values) def _update_status(info, id): if info: status = info.get("status", None) if status == edp.JOB_ACTION_SUSPEND: api.OPS.job_execution_suspend(id) if status == edp.JOB_ACTION_CANCEL: api.OPS.cancel_job_execution(id) def delete_job_execution(id): context.set_current_job_execution_id(id) api.OPS.delete_job_execution(id) sahara-8.0.0/sahara/service/api/v2/node_group_templates.py0000666000175100017510000000261113245514472023603 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor as c from sahara import context conductor = c.API # NodeGroupTemplate ops def get_node_group_templates(**kwargs): return conductor.node_group_template_get_all(context.ctx(), regex_search=True, **kwargs) def get_node_group_template(id): return conductor.node_group_template_get(context.ctx(), id) def create_node_group_template(values): return conductor.node_group_template_create(context.ctx(), values) def terminate_node_group_template(id): return conductor.node_group_template_destroy(context.ctx(), id) def update_node_group_template(id, values): return conductor.node_group_template_update(context.ctx(), id, values) def export_node_group_template(id): return conductor.node_group_template_get(context.ctx(), id) sahara-8.0.0/sahara/service/api/v2/job_templates.py0000666000175100017510000000237113245514472022217 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor as c from sahara import context from sahara.service.edp import job_manager as manager conductor = c.API def get_job_templates(**kwargs): return conductor.job_get_all(context.ctx(), regex_search=True, **kwargs) def get_job_template(id): return conductor.job_get(context.ctx(), id) def create_job_template(values): return conductor.job_create(context.ctx(), values) def update_job_template(id, values): return conductor.job_update(context.ctx(), id, values) def delete_job_template(job_id): return conductor.job_destroy(context.ctx(), job_id) def get_job_config_hints(job_type): return manager.get_job_config_hints(job_type) sahara-8.0.0/sahara/service/api/v2/job_types.py0000666000175100017510000000552313245514472021367 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six from sahara.plugins import base as plugin_base from sahara.utils import edp def get_job_types(**kwargs): # Return a dictionary of all the job types that can be run # by this instance of Sahara. For each job type, the value # will be a list of plugins that support the job type. For # each plugin, include a dictionary of the versions that # support the job type. # All entries in kwargs are expected to have list values hints = kwargs.get("hints", ["false"])[0].lower() == "true" plugin_names = kwargs.get("plugin", []) all_plugins = plugin_base.PLUGINS.get_plugins() if plugin_names: plugins = filter(lambda x: x.name in plugin_names, all_plugins) else: plugins = all_plugins job_types = kwargs.get("type", edp.JOB_TYPES_ALL) versions = kwargs.get("version", []) res = [] for job_type in job_types: # All job types supported by all versions of the plugin. # This is a dictionary where keys are plugin version # strings and values are lists of job types job_entry = {"name": job_type, "plugins": []} for plugin in plugins: types_for_plugin = plugin.get_edp_job_types(versions) # dict returns a new object so we are not modifying the plugin p = plugin.dict # Find only the versions of this plugin that support the job. # Additionally, instead of a list we want a dictionary of # plugin versions with corresponding config hints p["versions"] = {} for version, supported_types in six.iteritems(types_for_plugin): if job_type in supported_types: if hints: config_hints = plugin.get_edp_config_hints(job_type, version) else: config_hints = {} p["versions"][version] = config_hints # If we found at least one version of the plugin that # supports the job type, add the plugin to the result if p["versions"]: job_entry["plugins"].append(p) if job_entry["plugins"]: res.append(job_entry) return res sahara-8.0.0/sahara/service/api/__init__.py0000666000175100017510000000120713245514472020574 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. OPS = None def setup_api(ops): global OPS OPS = ops sahara-8.0.0/sahara/service/api/v11.py0000666000175100017510000002022413245514472017444 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging import six from sahara import conductor as c from sahara import context from sahara import exceptions as ex from sahara.plugins import base as plugin_base from sahara.service import api from sahara.service.edp.binary_retrievers import dispatch from sahara.service.edp import job_manager as manager from sahara.utils import edp from sahara.utils import proxy as p conductor = c.API LOG = logging.getLogger(__name__) CONF = cfg.CONF def get_job_types(**kwargs): # Return a dictionary of all the job types that can be run # by this instance of Sahara. For each job type, the value # will be a list of plugins that support the job type. For # each plugin, include a dictionary of the versions that # support the job type. # All entries in kwargs are expected to have list values hints = kwargs.get("hints", ["false"])[0].lower() == "true" plugin_names = kwargs.get("plugin", []) all_plugins = plugin_base.PLUGINS.get_plugins() if plugin_names: plugins = filter(lambda x: x.name in plugin_names, all_plugins) else: plugins = all_plugins job_types = kwargs.get("type", edp.JOB_TYPES_ALL) versions = kwargs.get("version", []) res = [] for job_type in job_types: # All job types supported by all versions of the plugin. # This is a dictionary where keys are plugin version # strings and values are lists of job types job_entry = {"name": job_type, "plugins": []} for plugin in plugins: types_for_plugin = plugin.get_edp_job_types(versions) # dict returns a new object so we are not modifying the plugin p = plugin.dict # Find only the versions of this plugin that support the job. # Additionally, instead of a list we want a dictionary of # plugin versions with corresponding config hints p["versions"] = {} for version, supported_types in six.iteritems(types_for_plugin): if job_type in supported_types: if hints: config_hints = plugin.get_edp_config_hints(job_type, version) else: config_hints = {} p["versions"][version] = config_hints # If we found at least one version of the plugin that # supports the job type, add the plugin to the result if p["versions"]: job_entry["plugins"].append(p) if job_entry["plugins"]: res.append(job_entry) return res def get_job_config_hints(job_type): return manager.get_job_config_hints(job_type) def execute_job(job_id, data): # Elements common to all job types cluster_id = data['cluster_id'] configs = data.get('job_configs', {}) interface = data.get('interface', {}) # Not in Java job types but present for all others input_id = data.get('input_id', None) output_id = data.get('output_id', None) # Since we will use a unified class in the database, we pass # a superset for all job types # example configs['start'] = '2015-05-12T08:55Z' frequency = 5 mins # the job will starts from 2015-05-12T08:55Z, runs every 5 mins job_execution_info = data.get('job_execution_info', {}) configs['job_execution_info'] = job_execution_info job_ex_dict = {'input_id': input_id, 'output_id': output_id, 'job_id': job_id, 'cluster_id': cluster_id, 'info': {'status': edp.JOB_STATUS_PENDING}, 'job_configs': configs, 'extra': {}, 'interface': interface} job_execution = conductor.job_execution_create(context.ctx(), job_ex_dict) context.set_current_job_execution_id(job_execution.id) # check to use proxy user if p.job_execution_requires_proxy_user(job_execution): try: p.create_proxy_user_for_job_execution(job_execution) except ex.SaharaException as e: LOG.error("Can't run job execution. " "(Reasons: {reason})".format(reason=e)) conductor.job_execution_destroy(context.ctx(), job_execution) raise api.OPS.run_edp_job(job_execution.id) return job_execution def get_job_execution_status(id): return manager.get_job_status(id) def job_execution_list(**kwargs): return conductor.job_execution_get_all(context.ctx(), regex_search=True, **kwargs) def get_job_execution(id): return conductor.job_execution_get(context.ctx(), id) def cancel_job_execution(id): context.set_current_job_execution_id(id) job_execution = conductor.job_execution_get(context.ctx(), id) api.OPS.cancel_job_execution(id) return job_execution def update_job_execution(id, values): _update_status(values.pop("info", None), id) return conductor.job_execution_update(context.ctx(), id, values) def _update_status(info, id): if info: status = info.get("status", None) if status == edp.JOB_ACTION_SUSPEND: api.OPS.job_execution_suspend(id) def delete_job_execution(id): context.set_current_job_execution_id(id) api.OPS.delete_job_execution(id) def get_data_sources(**kwargs): return conductor.data_source_get_all(context.ctx(), regex_search=True, **kwargs) def get_data_source(id): return conductor.data_source_get(context.ctx(), id) def delete_data_source(id): conductor.data_source_destroy(context.ctx(), id) def register_data_source(values): return conductor.data_source_create(context.ctx(), values) def data_source_update(id, values): return conductor.data_source_update(context.ctx(), id, values) def get_jobs(**kwargs): return conductor.job_get_all(context.ctx(), regex_search=True, **kwargs) def get_job(id): return conductor.job_get(context.ctx(), id) def create_job(values): return conductor.job_create(context.ctx(), values) def update_job(id, values): return conductor.job_update(context.ctx(), id, values) def delete_job(job_id): return conductor.job_destroy(context.ctx(), job_id) def create_job_binary(values): return conductor.job_binary_create(context.ctx(), values) def get_job_binaries(**kwargs): return conductor.job_binary_get_all(context.ctx(), regex_search=True, **kwargs) def get_job_binary(id): return conductor.job_binary_get(context.ctx(), id) def update_job_binary(id, values): return conductor.job_binary_update(context.ctx(), id, values) def delete_job_binary(id): conductor.job_binary_destroy(context.ctx(), id) def create_job_binary_internal(values): return conductor.job_binary_internal_create(context.ctx(), values) def get_job_binary_internals(**kwargs): return conductor.job_binary_internal_get_all(context.ctx(), regex_search=True, **kwargs) def get_job_binary_internal(id): return conductor.job_binary_internal_get(context.ctx(), id) def delete_job_binary_internal(id): conductor.job_binary_internal_destroy(context.ctx(), id) def get_job_binary_internal_data(id): return conductor.job_binary_internal_get_raw_data(context.ctx(), id) def update_job_binary_internal(id, values): return conductor.job_binary_internal_update(context.ctx(), id, values) def get_job_binary_data(id): job_binary = conductor.job_binary_get(context.ctx(), id) return dispatch.get_raw_binary(job_binary, with_context=True) sahara-8.0.0/sahara/service/api/v10.py0000666000175100017510000002162713245514472017453 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_utils import excutils import six from sahara import conductor as c from sahara import context from sahara.plugins import base as plugin_base from sahara.service import api from sahara.service.health import verification_base from sahara.service import quotas from sahara.utils import cluster as c_u from sahara.utils import general as g from sahara.utils.notification import sender from sahara.utils.openstack import base as b from sahara.utils.openstack import images as sahara_images conductor = c.API CONF = cfg.CONF # Cluster ops def get_clusters(**kwargs): return conductor.cluster_get_all(context.ctx(), regex_search=True, **kwargs) def get_cluster(id, show_progress=False): return conductor.cluster_get(context.ctx(), id, show_progress) def scale_cluster(id, data): context.set_current_cluster_id(id) ctx = context.ctx() cluster = conductor.cluster_get(ctx, id) plugin = plugin_base.PLUGINS.get_plugin(cluster.plugin_name) existing_node_groups = data.get('resize_node_groups', []) additional_node_groups = data.get('add_node_groups', []) # the next map is the main object we will work with # to_be_enlarged : {node_group_id: desired_amount_of_instances} to_be_enlarged = {} for ng in existing_node_groups: ng_id = g.find(cluster.node_groups, name=ng['name'])['id'] to_be_enlarged.update({ng_id: ng['count']}) additional = construct_ngs_for_scaling(cluster, additional_node_groups) cluster = conductor.cluster_get(ctx, cluster) _add_ports_for_auto_sg(ctx, cluster, plugin) try: cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_VALIDATING) quotas.check_scaling(cluster, to_be_enlarged, additional) plugin.recommend_configs(cluster, scaling=True) plugin.validate_scaling(cluster, to_be_enlarged, additional) except Exception as e: with excutils.save_and_reraise_exception(): c_u.clean_cluster_from_empty_ng(cluster) c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_ACTIVE, six.text_type(e)) # If we are here validation is successful. # So let's update to_be_enlarged map: to_be_enlarged.update(additional) for node_group in cluster.node_groups: if node_group.id not in to_be_enlarged: to_be_enlarged[node_group.id] = node_group.count api.OPS.provision_scaled_cluster(id, to_be_enlarged) return cluster def create_cluster(values): plugin = plugin_base.PLUGINS.get_plugin(values['plugin_name']) return _cluster_create(values, plugin) def create_multiple_clusters(values): num_of_clusters = values['count'] clusters = [] plugin = plugin_base.PLUGINS.get_plugin(values['plugin_name']) for counter in range(num_of_clusters): cluster_dict = values.copy() cluster_name = cluster_dict['name'] cluster_dict['name'] = get_multiple_cluster_name(num_of_clusters, cluster_name, counter + 1) cluster = _cluster_create(cluster_dict, plugin) clusters.append(cluster.id) clusters_dict = {'clusters': clusters} return clusters_dict def _cluster_create(values, plugin): ctx = context.ctx() cluster = conductor.cluster_create(ctx, values) context.set_current_cluster_id(cluster.id) sender.status_notify(cluster.id, cluster.name, "New", "create") _add_ports_for_auto_sg(ctx, cluster, plugin) # validating cluster try: plugin.recommend_configs(cluster) cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_VALIDATING) quotas.check_cluster(cluster) plugin.validate(cluster) except Exception as e: with excutils.save_and_reraise_exception(): c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_ERROR, six.text_type(e)) api.OPS.provision_cluster(cluster.id) return cluster def get_multiple_cluster_name(num_of_clusters, name, counter): return "%%s-%%0%dd" % len(str(num_of_clusters)) % (name, counter) def _add_ports_for_auto_sg(ctx, cluster, plugin): for ng in cluster.node_groups: if ng.auto_security_group: ports = {'open_ports': plugin.get_open_ports(ng)} conductor.node_group_update(ctx, ng, ports) def terminate_cluster(id): context.set_current_cluster_id(id) cluster = c_u.change_cluster_status(id, c_u.CLUSTER_STATUS_DELETING) if cluster is None: return api.OPS.terminate_cluster(id) sender.status_notify(cluster.id, cluster.name, cluster.status, "delete") def update_cluster(id, values): if verification_base.update_verification_required(values): api.OPS.handle_verification(id, values) return conductor.cluster_get(context.ctx(), id) return conductor.cluster_update(context.ctx(), id, values) # ClusterTemplate ops def get_cluster_templates(**kwargs): return conductor.cluster_template_get_all(context.ctx(), regex_search=True, **kwargs) def get_cluster_template(id): return conductor.cluster_template_get(context.ctx(), id) def create_cluster_template(values): return conductor.cluster_template_create(context.ctx(), values) def terminate_cluster_template(id): return conductor.cluster_template_destroy(context.ctx(), id) def update_cluster_template(id, values): return conductor.cluster_template_update(context.ctx(), id, values) # NodeGroupTemplate ops def get_node_group_templates(**kwargs): return conductor.node_group_template_get_all(context.ctx(), regex_search=True, **kwargs) def get_node_group_template(id): return conductor.node_group_template_get(context.ctx(), id) def create_node_group_template(values): return conductor.node_group_template_create(context.ctx(), values) def terminate_node_group_template(id): return conductor.node_group_template_destroy(context.ctx(), id) def update_node_group_template(id, values): return conductor.node_group_template_update(context.ctx(), id, values) def export_node_group_template(id): return conductor.node_group_template_get(context.ctx(), id) # Plugins ops def get_plugins(): return plugin_base.PLUGINS.get_plugins(serialized=True) def get_plugin(plugin_name, version=None): return plugin_base.PLUGINS.serialize_plugin(plugin_name, version) def update_plugin(plugin_name, values): return plugin_base.PLUGINS.update_plugin(plugin_name, values) def construct_ngs_for_scaling(cluster, additional_node_groups): ctx = context.ctx() additional = {} for ng in additional_node_groups: count = ng['count'] ng['count'] = 0 ng_id = conductor.node_group_add(ctx, cluster, ng) additional.update({ng_id: count}) return additional # Image Registry def get_images(name, tags): return b.execute_with_retries( sahara_images.image_manager().list_registered, name, tags) def get_image(**kwargs): if len(kwargs) == 1 and 'id' in kwargs: return b.execute_with_retries( sahara_images.image_manager().get, kwargs['id']) else: return b.execute_with_retries( sahara_images.image_manager().find, **kwargs) def get_registered_image(image_id): return b.execute_with_retries( sahara_images.image_manager().get_registered_image, image_id) def register_image(image_id, username, description=None): manager = sahara_images.image_manager() b.execute_with_retries( manager.set_image_info, image_id, username, description) return b.execute_with_retries(manager.get, image_id) def unregister_image(image_id): manager = sahara_images.image_manager() b.execute_with_retries(manager.unset_image_info, image_id) return b.execute_with_retries(manager.get, image_id) def add_image_tags(image_id, tags): manager = sahara_images.image_manager() b.execute_with_retries(manager.tag, image_id, tags) return b.execute_with_retries(manager.get, image_id) def remove_image_tags(image_id, tags): manager = sahara_images.image_manager() b.execute_with_retries(manager.untag, image_id, tags) return b.execute_with_retries(manager.get, image_id) sahara-8.0.0/sahara/service/validation.py0000666000175100017510000001423213245514472020420 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools from oslo_utils import reflection from sahara import exceptions as ex from sahara.i18n import _ from sahara.utils import api as u from sahara.utils import api_validator from sahara.utils import types def _get_path(path): if path: path_string = path[0] for x in path[1:]: path_string += '[%s]' % str(x) return path_string + ': ' return '' def _generate_error(errors): message = [_get_path(list(e.path)) + e.message for e in errors] if message: return ex.SaharaException('\n'.join(message), "VALIDATION_ERROR") def validate_pagination_limit(): request_args = u.get_request_args() if 'limit' in request_args: if types.is_int(request_args['limit']): if not int(request_args['limit']) > 0: raise ex.SaharaException( _("'limit' must be positive integer"), 400) else: raise ex.SaharaException( _("'limit' must be positive integer"), 400) def get_sorting_field(): request_args = u.get_request_args() if 'sort_by' in request_args: sort_by = request_args['sort_by'] if sort_by: sort_by = sort_by[1:] if sort_by[0] == '-' else sort_by return sort_by return None def validate_sorting_clusters(): field = get_sorting_field() if field is None: return if field not in ['id', 'name', 'plugin_name', 'hadoop_version', 'status']: raise ex.SaharaException( _("Unknown field for sorting %s") % field, 400) def validate_sorting_cluster_templates(): field = get_sorting_field() if field is None: return if field not in ['id', 'name', 'plugin_name', 'hadoop_version', 'created_at', 'updated_at']: raise ex.SaharaException( _("Unknown field for sorting %s") % field, 400) def validate_sorting_node_group_templates(): field = get_sorting_field() if field is None: return if field not in ['id', 'name', 'plugin_name', 'hadoop_version', 'created_at', 'updated_at']: raise ex.SaharaException( _("Unknown field for sorting %s") % field, 400) def validate_sorting_job_binaries(): field = get_sorting_field() if field is None: return if field not in ['id', 'name', 'created_at', 'updated_at']: raise ex.SaharaException( _("Unknown field for sorting %s") % field, 400) def validate_sorting_job_binary_internals(): field = get_sorting_field() if field is None: return if field not in ['id', 'name', 'created_at', 'updated_at']: raise ex.SaharaException( _("Unknown field for sorting %s") % field, 400) def validate_sorting_data_sources(): field = get_sorting_field() if field is None: return if field not in ['id', 'name', 'type', 'created_at', 'updated_at']: raise ex.SaharaException( _("Unknown field for sorting %s") % field, 400) def validate_sorting_jobs(): field = get_sorting_field() if field is None: return if field not in ['id', 'name', 'type', 'created_at', 'updated_at']: raise ex.SaharaException( _("Unknown field for sorting %s") % field, 400) def validate_sorting_job_executions(): field = get_sorting_field() if field is None: return if field not in ['id', 'job_template', 'cluster', 'status']: raise ex.SaharaException( _("Unknown field for sorting %s") % field, 400) def validate(schema, *validators): def decorator(func): @functools.wraps(func) def handler(*args, **kwargs): request_data = u.request_data() try: if schema: validator = api_validator.ApiValidator(schema) errors = validator.iter_errors(request_data) error = _generate_error(errors) if error: return u.bad_request(error) if validators: for validator in validators: validator(**kwargs) except ex.SaharaException as e: return u.bad_request(e) except Exception as e: return u.internal_error( 500, "Error occurred during validation", e) return func(*args, **kwargs) return handler return decorator def check_exists(get_func, *id_prop, **get_args): def decorator(func): @functools.wraps(func) def handler(*args, **kwargs): if id_prop and not get_args: get_args['id'] = id_prop[0] if 'marker' in id_prop: if 'marker' not in u.get_request_args(): return func(*args, **kwargs) kwargs['marker'] = u.get_request_args()['marker'] get_kwargs = {} for get_arg in get_args: get_kwargs[get_arg] = kwargs[get_args[get_arg]] obj = None try: obj = get_func(**get_kwargs) except Exception as e: cls_name = reflection.get_class_name(e, fully_qualified=False) if 'notfound' not in cls_name.lower(): raise e if obj is None: e = ex.NotFoundException(get_kwargs, _('Object with %s not found')) return u.not_found(e) if 'marker' in kwargs: del(kwargs['marker']) return func(*args, **kwargs) return handler return decorator sahara-8.0.0/sahara/service/validations/0000775000175100017510000000000013245515027020222 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/validations/cluster_templates.py0000666000175100017510000000721613245514472024346 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.api import v10 as api import sahara.service.validations.base as b from sahara.service.validations import shares def check_cluster_template_create(data, **kwargs): plugin_version = 'hadoop_version' if data.get('plugin_version'): plugin_version = 'plugin_version' b.check_cluster_template_unique_name(data['name']) b.check_plugin_name_exists(data['plugin_name']) b.check_plugin_supports_version(data['plugin_name'], data[plugin_version]) if data.get('default_image_id'): b.check_image_registered(data['default_image_id']) b.check_required_image_tags(data['plugin_name'], data[plugin_version], data['default_image_id']) b.check_all_configurations(data) if data.get('anti_affinity'): b.check_node_processes(data['plugin_name'], data[plugin_version], data['anti_affinity']) if data.get('neutron_management_network'): b.check_network_exists(data['neutron_management_network']) if data.get('shares'): shares.check_shares(data['shares']) def check_cluster_template_usage(cluster_template_id, **kwargs): users = [] for cluster in api.get_clusters(): if cluster_template_id == cluster.cluster_template_id: users.append(cluster.name) if users: raise ex.InvalidReferenceException( _("Cluster template %(id)s in use by %(clusters)s") % {'id': cluster_template_id, 'clusters': ', '.join(users)}) def check_cluster_template_update(cluster_template_id, data, **kwargs): plugin_version = 'hadoop_version' if data.get('plugin_version'): plugin_version = 'plugin_version' if data.get('plugin_name') and not data.get(plugin_version): raise ex.InvalidReferenceException( _("You must specify a %s value " "for your plugin_name") % plugin_version) if data.get('plugin_name'): plugin = data['plugin_name'] version = data[plugin_version] b.check_plugin_name_exists(plugin) b.check_plugin_supports_version(plugin, version) b.check_all_configurations(data) else: cluster_template = api.get_cluster_template(cluster_template_id) plugin = cluster_template.plugin_name if data.get(plugin_version): version = data.get(plugin_version) b.check_plugin_supports_version(plugin, version) else: version = cluster_template.hadoop_version if data.get('default_image_id'): b.check_image_registered(data['default_image_id']) b.check_required_image_tags(plugin, version, data['default_image_id']) if data.get('anti_affinity'): b.check_node_processes(plugin, version, data['anti_affinity']) if data.get('neutron_management_network'): b.check_network_exists(data['neutron_management_network']) if data.get('shares'): shares.check_shares(data['shares']) sahara-8.0.0/sahara/service/validations/clusters_schema.py0000666000175100017510000001047513245514472023774 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import sahara.exceptions as ex from sahara.service.health import verification_base import sahara.service.validations.cluster_template_schema as ct_schema from sahara.service.validations import shares def _build_node_groups_schema(): schema = copy.deepcopy(ct_schema.CLUSTER_TEMPLATE_SCHEMA) return schema['properties']['node_groups'] def _build_cluster_schema(api_version='v1'): if api_version == 'v1': cluster_schema = copy.deepcopy(ct_schema.CLUSTER_TEMPLATE_SCHEMA) elif api_version == 'v2': cluster_schema = copy.deepcopy(ct_schema.CLUSTER_TEMPLATE_SCHEMA_V2) else: raise ex.InvalidDataException('Invalid API version %s' % api_version) cluster_schema['properties'].update({ "is_transient": { "type": "boolean" }, "user_keypair_id": { "type": "string", "format": "valid_keypair_name", }, "cluster_template_id": { "type": "string", "format": "uuid", }}) return cluster_schema CLUSTER_SCHEMA = _build_cluster_schema() CLUSTER_SCHEMA_V2 = _build_cluster_schema('v2') MULTIPLE_CLUSTER_SCHEMA = copy.deepcopy(CLUSTER_SCHEMA) MULTIPLE_CLUSTER_SCHEMA['properties'].update({ "count": { "type": "integer" }}) MULTIPLE_CLUSTER_SCHEMA['required'].append('count') MULTIPLE_CLUSTER_SCHEMA_V2 = copy.deepcopy(CLUSTER_SCHEMA_V2) MULTIPLE_CLUSTER_SCHEMA_V2['properties'].update({ "count": { "type": "integer" }}) MULTIPLE_CLUSTER_SCHEMA_V2['required'].append('count') CLUSTER_UPDATE_SCHEMA = { "type": "object", "properties": { "description": { "type": ["string", "null"] }, "name": { "type": "string", "minLength": 1, "maxLength": 50, "format": "valid_name_hostname", }, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], }, "verification": { "type": "object", "properties": { "status": { "enum": verification_base.get_possible_ops(), } }, }, "shares": copy.deepcopy(shares.SHARE_SCHEMA), }, "additionalProperties": False, "required": [] } CLUSTER_SCALING_SCHEMA = { "type": "object", "properties": { "resize_node_groups": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string", }, "count": { "type": "integer", "minimum": 0, }, }, "additionalProperties": False, "required": [ "name", "count", ] }, "minItems": 1 }, "add_node_groups": _build_node_groups_schema(), }, "additionalProperties": False, "anyOf": [ { "required": ["resize_node_groups"] }, { "required": ["add_node_groups"] } ] } CLUSTER_SCALING_SCHEMA_V2 = copy.deepcopy(CLUSTER_SCALING_SCHEMA) CLUSTER_SCALING_SCHEMA_V2['properties']['resize_node_groups'][ 'items']['properties'].update( { "instances": { "type": "array", "items": { "type": "string", }, } }) CLUSTER_DELETE_SCHEMA_V2 = { "type": "object", "properties": { "force": { "type": "boolean" } }, "additionalProperties": False, "required": [] } sahara-8.0.0/sahara/service/validations/shares.py0000666000175100017510000000461113245514472022070 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.edp.utils import shares from sahara.utils.openstack import manila SHARE_SCHEMA = { "type": "array", "items": { "type": "object", "properties": { "id": { "type": "string", "format": "uuid" }, "path": { "type": ["string", "null"] }, "access_level": { "type": ["string", "null"], "enum": ["rw", "ro"], "default": "rw" } }, "additionalProperties": False, "required": [ "id" ] } } def check_shares(data): if not data: return paths = (share.get('path') for share in data) paths = [path for path in paths if path is not None] if len(paths) != len(set(paths)): raise ex.InvalidDataException( _('Multiple shares cannot be mounted to the same path.')) for path in paths: if not path.startswith('/') or '\x00' in path: raise ex.InvalidDataException( _('Paths must be absolute Linux paths starting with "/"' 'and may not contain nulls.')) client = manila.client() for share in data: manila_share = manila.get_share(client, share['id']) if not manila_share: raise ex.InvalidReferenceException( _("Requested share id %s does not exist.") % share['id']) share_type = manila_share.share_proto if share_type not in shares.SUPPORTED_SHARE_TYPES: raise ex.InvalidReferenceException( _("Requested share id %(id)s is of type %(type)s, which is " "not supported by Sahara.") % {"id": share['id'], "type": share_type}) sahara-8.0.0/sahara/service/validations/cluster_template_schema.py0000666000175100017510000000775713245514472025515 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara.service.validations import node_group_template_schema as ngt_schema from sahara.service.validations import shares def _build_ng_schema_for_cluster_tmpl(): cl_tmpl_ng_schema = copy.deepcopy(ngt_schema.NODE_GROUP_TEMPLATE_SCHEMA) cl_tmpl_ng_schema["properties"].update({"count": {"type": "integer"}}) cl_tmpl_ng_schema["required"] = ["name", "flavor_id", "node_processes", "count"] del cl_tmpl_ng_schema["properties"]["plugin_name"] del cl_tmpl_ng_schema["properties"]["hadoop_version"] return cl_tmpl_ng_schema _cluster_tmpl_ng_schema = _build_ng_schema_for_cluster_tmpl() def _build_ng_tmpl_schema_for_cluster_template(): cl_tmpl_ng_tmpl_schema = copy.deepcopy(_cluster_tmpl_ng_schema) cl_tmpl_ng_tmpl_schema["properties"].update( { "node_group_template_id": { "type": "string", "format": "uuid", } }) cl_tmpl_ng_tmpl_schema["required"] = ["node_group_template_id", "name", "count"] return cl_tmpl_ng_tmpl_schema _cluster_tmpl_ng_tmpl_schema = _build_ng_tmpl_schema_for_cluster_template() CLUSTER_TEMPLATE_SCHEMA = { "type": "object", "properties": { "name": { "type": "string", "minLength": 1, "maxLength": 50, "format": "valid_name_hostname", }, "plugin_name": { "type": "string", }, "hadoop_version": { "type": "string", }, "default_image_id": { "type": ["string", "null"], "format": "uuid", }, "cluster_configs": { "type": ["configs", "null"], }, "node_groups": { "type": ["array", "null"], "items": { "oneOf": [_cluster_tmpl_ng_tmpl_schema, _cluster_tmpl_ng_schema] } }, "anti_affinity": { "type": ["array", "null"], "items": { "type": "string", }, }, "description": { "type": ["string", "null"], }, "neutron_management_network": { "type": ["string", "null"], "format": "uuid" }, "shares": copy.deepcopy(shares.SHARE_SCHEMA), "use_autoconfig": { "type": ["boolean", "null"], }, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], }, "domain_name": { "type": ["string", "null"], } }, "additionalProperties": False, "required": [ "name", "plugin_name", "hadoop_version", ] } # APIv2: renaming hadoop_version -> plugin_version CLUSTER_TEMPLATE_SCHEMA_V2 = copy.deepcopy(CLUSTER_TEMPLATE_SCHEMA) del CLUSTER_TEMPLATE_SCHEMA_V2["properties"]["hadoop_version"] CLUSTER_TEMPLATE_SCHEMA_V2["required"].remove("hadoop_version") CLUSTER_TEMPLATE_SCHEMA_V2["properties"].update({ "plugin_version": { "type": "string", }}) CLUSTER_TEMPLATE_SCHEMA_V2["required"].append("plugin_version") CLUSTER_TEMPLATE_UPDATE_SCHEMA = copy.copy(CLUSTER_TEMPLATE_SCHEMA) CLUSTER_TEMPLATE_UPDATE_SCHEMA["required"] = [] CLUSTER_TEMPLATE_UPDATE_SCHEMA_V2 = copy.copy(CLUSTER_TEMPLATE_SCHEMA_V2) CLUSTER_TEMPLATE_UPDATE_SCHEMA_V2["required"] = [] sahara-8.0.0/sahara/service/validations/base.py0000666000175100017510000004036013245514472021516 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import collections import novaclient.exceptions as nova_ex from oslo_config import cfg from oslo_utils import uuidutils import six from sahara import conductor as cond from sahara import context import sahara.exceptions as ex from sahara.i18n import _ import sahara.plugins.base as plugin_base from sahara.service.api import v10 as api from sahara.utils import general as g import sahara.utils.openstack.cinder as cinder from sahara.utils.openstack import images as sahara_images import sahara.utils.openstack.neutron as neutron import sahara.utils.openstack.nova as nova CONF = cfg.CONF conductor = cond.API MAX_HOSTNAME_LENGTH = 64 def _get_plugin_configs(plugin_name, hadoop_version, scope=None): pl_confs = {} for config in plugin_base.PLUGINS.get_plugin( plugin_name).get_all_configs(hadoop_version): if pl_confs.get(config.applicable_target): pl_confs[config.applicable_target].append(config.name) else: pl_confs[config.applicable_target] = [config.name] return pl_confs def _check_duplicates(lst, message): invalid = [] lst = collections.Counter(lst) for key, value in six.iteritems(lst): if value > 1: invalid.append(key) if len(invalid) > 0: raise ex.InvalidDataException(message % invalid) # Common validation checks def check_plugin_name_exists(name): if name not in [p.name for p in api.get_plugins()]: raise ex.InvalidReferenceException( _("Sahara doesn't contain plugin with name '%s'") % name) def check_plugin_supports_version(p_name, version): if version not in plugin_base.PLUGINS.get_plugin(p_name).get_versions(): raise ex.InvalidReferenceException( _("Requested plugin '%(name)s' doesn't support version " "'%(version)s'") % {'name': p_name, 'version': version}) def check_plugin_labels(p_name, version): plugin_base.PLUGINS.validate_plugin_labels(p_name, version) def check_image_registered(image_id): if image_id not in ( [i.id for i in sahara_images.image_manager().list_registered()]): raise ex.InvalidReferenceException( _("Requested image '%s' is not registered") % image_id) def check_node_group_configs(plugin_name, hadoop_version, ng_configs, plugin_configs=None): # TODO(aignatov): Should have scope and config type validations pl_confs = plugin_configs or _get_plugin_configs(plugin_name, hadoop_version) for app_target, configs in ng_configs.items(): if app_target not in pl_confs: raise ex.InvalidReferenceException( _("Plugin doesn't contain applicable target '%s'") % app_target) for name, values in configs.items(): if name not in pl_confs[app_target]: raise ex.InvalidReferenceException( _("Plugin's applicable target '%(target)s' doesn't " "contain config with name '%(name)s'") % {'target': app_target, 'name': name}) def check_all_configurations(data): plugin_version = 'hadoop_version' if data.get('plugin_version'): plugin_version = 'plugin_version' pl_confs = _get_plugin_configs(data['plugin_name'], data[plugin_version]) if data.get('cluster_configs'): check_node_group_configs(data['plugin_name'], data[plugin_version], data['cluster_configs'], plugin_configs=pl_confs) if data.get('node_groups'): check_duplicates_node_groups_names(data['node_groups']) for ng in data['node_groups']: check_node_group_basic_fields(data['plugin_name'], data[plugin_version], ng, pl_confs) # NodeGroup related checks def check_node_group_basic_fields(plugin_name, hadoop_version, ng, plugin_configs=None): if ng.get('node_group_template_id'): ng_tmpl_id = ng['node_group_template_id'] check_node_group_template_exists(ng_tmpl_id) ng_tmpl = api.get_node_group_template(ng_tmpl_id).to_wrapped_dict() check_node_group_basic_fields(plugin_name, hadoop_version, ng_tmpl['node_group_template'], plugin_configs) if ng.get('node_configs'): check_node_group_configs(plugin_name, hadoop_version, ng['node_configs'], plugin_configs) if ng.get('flavor_id'): check_flavor_exists(ng['flavor_id']) if ng.get('node_processes'): check_node_processes(plugin_name, hadoop_version, ng['node_processes']) if ng.get('image_id'): check_image_registered(ng['image_id']) if ng.get('volumes_per_node'): if not cinder.check_cinder_exists(): raise ex.InvalidReferenceException(_("Cinder is not supported")) if ng.get('volumes_availability_zone'): check_volume_availability_zone_exist( ng['volumes_availability_zone']) if ng.get('volume_type'): check_volume_type_exists(ng['volume_type']) if not ng.get('volumes_size'): raise ex.InvalidReferenceException( _("You must specify a volumes_size parameter")) if ng.get('floating_ip_pool'): check_floatingip_pool_exists(ng['floating_ip_pool']) if ng.get('security_groups'): check_security_groups_exist(ng['security_groups']) if ng.get('availability_zone'): check_availability_zone_exist(ng['availability_zone']) def check_flavor_exists(flavor_id): flavor_list = nova.client().flavors.list() if flavor_id not in [flavor.id for flavor in flavor_list]: raise ex.NotFoundException( flavor_id, _("Requested flavor '%s' not found")) def check_security_groups_exist(security_groups): security_group_list = neutron.client().list_security_groups() allowed_groups = set() for sg in security_group_list['security_groups']: allowed_groups.add(sg['name']) allowed_groups.add(sg['id']) for sg in security_groups: if sg not in allowed_groups: raise ex.NotFoundException( sg, _("Security group '%s' not found")) def check_floatingip_pool_exists(pool_id): network = None network = neutron.get_network(pool_id) if not network: raise ex.NotFoundException(pool_id, _("Floating IP pool %s not found")) def check_node_processes(plugin_name, version, node_processes): _check_duplicates(node_processes, _("Duplicates in node processes have " "been detected: %s")) plugin_processes = [] for process in plugin_base.PLUGINS.get_plugin( plugin_name).get_node_processes(version).values(): plugin_processes += process plugin_processes = set(plugin_processes) invalid_processes = [] for node_process in node_processes: if node_process not in plugin_processes: invalid_processes.append(node_process) if len(invalid_processes) > 0: raise ex.InvalidReferenceException( _("Plugin doesn't support the following node processes: %s") % sorted(invalid_processes)) def check_duplicates_node_groups_names(node_groups): ng_names = [ng['name'] for ng in node_groups] _check_duplicates( ng_names, _("Duplicates in node group names are detected: %s")) def check_availability_zone_exist(az): az_list = nova.client().availability_zones.list(False) az_names = [a.zoneName for a in az_list] if az not in az_names: raise ex.NotFoundException( az, _("Nova availability zone '%s' not found")) def check_volume_availability_zone_exist(az): az_list = cinder.client().availability_zones.list() az_names = [a.zoneName for a in az_list] if az not in az_names: raise ex.NotFoundException( az, _("Cinder availability zone '%s' not found")) def check_volume_type_exists(volume_type): volume_types = cinder.client().volume_types.list() volume_types = list(filter(lambda x: x.name == volume_type, volume_types)) if len(volume_types) == 1 and volume_types[0].name == volume_type: return raise ex.NotFoundException(volume_type, _("Volume type '%s' not found")) # Cluster creation related checks def check_cluster_unique_name(cluster_name): if cluster_name in [cluster.name for cluster in api.get_clusters(name=cluster_name)]: raise ex.NameAlreadyExistsException( _("Cluster with name '%s' already exists") % cluster_name) def check_cluster_hostnames_lengths(cluster_name, node_groups): for ng in node_groups: longest_hostname = g.generate_instance_name(cluster_name, ng['name'], ng['count']) longest_hostname += '.' longest_hostname += CONF.node_domain if len(longest_hostname) > MAX_HOSTNAME_LENGTH: raise ex.InvalidDataException( _("Composite hostname %(host)s in provisioned cluster exceeds" " maximum limit %(limit)s characters") % {'host': longest_hostname, 'limit': MAX_HOSTNAME_LENGTH}) def check_keypair_exists(keypair): try: nova.client().keypairs.get(keypair) except nova_ex.NotFound: raise ex.NotFoundException( keypair, _("Requested keypair '%s' not found")) def check_network_exists(net_id): if not neutron.get_network(net_id): raise ex.NotFoundException(net_id, _("Network %s not found")) # Cluster templates related checks def check_cluster_template_unique_name(cluster_tmpl_name): if cluster_tmpl_name in [cluster_tmpl.name for cluster_tmpl in api.get_cluster_templates( name=cluster_tmpl_name)]: raise ex.NameAlreadyExistsException( _("Cluster template with name '%s' already exists") % cluster_tmpl_name) def check_cluster_template_exists(cluster_template_id): if not api.get_cluster_template(id=cluster_template_id): raise ex.NotFoundException( cluster_template_id, _("Cluster template with id '%s' not found")) def check_node_groups_in_cluster_templates(cluster_name, plugin_name, hadoop_version, cluster_template_id): c_t = api.get_cluster_template(id=cluster_template_id) n_groups = c_t.to_wrapped_dict()['cluster_template']['node_groups'] for node_group in n_groups: check_node_group_basic_fields(plugin_name, hadoop_version, node_group) check_cluster_hostnames_lengths(cluster_name, n_groups) # NodeGroup templates related checks def check_node_group_template_unique_name(ng_tmpl_name): if ng_tmpl_name in [ng_tmpl.name for ng_tmpl in api.get_node_group_templates(name=ng_tmpl_name)]: raise ex.NameAlreadyExistsException( _("NodeGroup template with name '%s' already exists") % ng_tmpl_name) def check_node_group_template_exists(ng_tmpl_id): if not api.get_node_group_template(id=ng_tmpl_id): raise ex.NotFoundException( ng_tmpl_id, _("NodeGroup template with id '%s' not found")) def _get_floating_ip_pool(node_group): # tmckay-fp I think this would be the method we needed to # get floating_ip_pool for instances if node_group.get('floating_ip_pool'): return node_group['floating_ip_pool'] if node_group.get('node_group_template_id'): ctx = context.ctx() ngt = conductor.node_group_template_get( ctx, node_group['node_group_template_id']) if ngt.get('floating_ip_pool'): return ngt['floating_ip_pool'] return None # Cluster scaling def check_resize(cluster, r_node_groups): ng_map = {} for ng in cluster.node_groups: ng_map[ng.name] = ng check_duplicates_node_groups_names(r_node_groups) for ng in r_node_groups: if ng['name'] not in ng_map.keys(): raise ex.InvalidReferenceException( _("Cluster doesn't contain node group with name '%s'") % ng['name']) node_group = ng_map[ng['name']] if node_group.get('node_group_template_id'): ng_tmpl_id = node_group['node_group_template_id'] check_node_group_template_exists(ng_tmpl_id) ng_tmp = api.get_node_group_template(ng_tmpl_id).to_wrapped_dict() check_node_group_basic_fields(cluster.plugin_name, cluster.hadoop_version, ng_tmp['node_group_template']) for scaling_ng in r_node_groups: current_count = ng_map[scaling_ng['name']].count new_count = scaling_ng['count'] count_diff = current_count - new_count if 'instances' in scaling_ng: if len(scaling_ng['instances']) > count_diff: raise ex.InvalidDataException( _("Number of specific instances (%(instance)s) to" " delete can not be greater than the count difference" " (%(count)s during scaling") % {'instance': str(len(scaling_ng['instances'])), 'count': str(count_diff)}) else: if len(scaling_ng['instances']) > 0: is_uuid = uuidutils.is_uuid_like( scaling_ng['instances'][0]) if is_uuid: for instance in scaling_ng['instances']: if not uuidutils.is_uuid_like(instance): raise ex.InvalidReferenceException( _("You can only reference instances by" " Name or UUID, not both on the same" " request")) else: for instance in scaling_ng['instances']: if uuidutils.is_uuid_like(instance): raise ex.InvalidReferenceException( _("You can only reference instances by" " Name or UUID, not both on the same" " request")) _check_duplicates(scaling_ng['instances'], _("Duplicate entry for instances to" " delete")) def check_add_node_groups(cluster, add_node_groups): cluster_ng_names = [ng.name for ng in cluster.node_groups] check_duplicates_node_groups_names(add_node_groups) pl_confs = _get_plugin_configs(cluster.plugin_name, cluster.hadoop_version) for ng in add_node_groups: if ng['name'] in cluster_ng_names: raise ex.InvalidReferenceException( _("Can't add new nodegroup. Cluster already has nodegroup with" " name '%s'") % ng['name']) check_node_group_basic_fields(cluster.plugin_name, cluster.hadoop_version, ng, pl_confs) # Tags def check_required_image_tags(plugin_name, hadoop_version, image_id): image = api.get_image(id=image_id) plugin = plugin_base.PLUGINS.get_plugin(plugin_name) req_tags = set(plugin.get_required_image_tags(hadoop_version)) req_tags = list(req_tags.difference(set(image.tags))) if req_tags: raise ex.InvalidReferenceException( _("Requested image '%(image)s' doesn't contain required" " tags: %(tags)s") % {'image': image_id, 'tags': req_tags}) sahara-8.0.0/sahara/service/validations/plugins.py0000666000175100017510000000214713245514472022266 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import sahara.exceptions as ex from sahara.i18n import _ from sahara.plugins import base def plugin_update_validation_jsonschema(): return base.PLUGINS.get_plugin_update_validation_jsonschema() def check_convert_to_template(plugin_name, version, **kwargs): raise ex.InvalidReferenceException( _("Requested plugin '%s' doesn't support converting config files " "to cluster templates") % plugin_name) def check_plugin_update(plugin_name, data, **kwargs): base.PLUGINS.validate_plugin_update(plugin_name, data) sahara-8.0.0/sahara/service/validations/images.py0000666000175100017510000000234313245514472022050 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. image_register_schema = { "type": "object", "properties": { "username": { "type": "string", }, "description": { "type": "string", }, }, "additionalProperties": False, "required": ["username"] } image_tags_schema = { "type": "object", "properties": { "tags": { "type": "array", "items": { "type": "string", "format": "valid_tag" }, }, }, "additionalProperties": False, "required": ["tags"] } def check_image_register(data, **kwargs): pass def check_tags(data, **kwargs): pass sahara-8.0.0/sahara/service/validations/clusters.py0000666000175100017510000001022013245514472022440 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from sahara import context import sahara.exceptions as ex from sahara.i18n import _ from sahara.service.api import v10 as api from sahara.service.health import verification_base from sahara.service.validations import acl import sahara.service.validations.base as b CONF = cfg.CONF def check_cluster_create(data, **kwargs): b.check_cluster_unique_name(data['name']) _check_cluster_create(data) def check_multiple_clusters_create(data, **kwargs): _check_cluster_create(data) for counter in range(data['count']): cluster_name = api.get_multiple_cluster_name(data['count'], data['name'], counter + 1) b.check_cluster_unique_name(cluster_name) def _check_cluster_create(data): plugin_version = 'hadoop_version' if data.get('plugin_version'): plugin_version = 'plugin_version' b.check_plugin_name_exists(data['plugin_name']) b.check_plugin_supports_version(data['plugin_name'], data[plugin_version]) b.check_plugin_labels( data['plugin_name'], data[plugin_version]) if data.get('cluster_template_id'): ct_id = data['cluster_template_id'] b.check_cluster_template_exists(ct_id) if not data.get('node_groups'): b.check_node_groups_in_cluster_templates(data['name'], data['plugin_name'], data[plugin_version], ct_id) if data.get('user_keypair_id'): b.check_keypair_exists(data['user_keypair_id']) default_image_id = _get_cluster_field(data, 'default_image_id') if default_image_id: b.check_image_registered(default_image_id) b.check_required_image_tags(data['plugin_name'], data[plugin_version], default_image_id) else: raise ex.NotFoundException('default_image_id', _("'%s' field is not found")) b.check_all_configurations(data) if data.get('anti_affinity'): b.check_node_processes(data['plugin_name'], data[plugin_version], data['anti_affinity']) if data.get('node_groups'): b.check_cluster_hostnames_lengths(data['name'], data['node_groups']) neutron_net_id = _get_cluster_field(data, 'neutron_management_network') if neutron_net_id: b.check_network_exists(neutron_net_id) else: raise ex.NotFoundException('neutron_management_network', _("'%s' field is not found")) def _get_cluster_field(cluster, field): if cluster.get(field): return cluster[field] if cluster.get('cluster_template_id'): cluster_template = api.get_cluster_template( id=cluster['cluster_template_id']) if cluster_template.get(field): return cluster_template[field] return None def check_cluster_delete(cluster_id, **kwargs): cluster = api.get_cluster(cluster_id) acl.check_tenant_for_delete(context.current(), cluster) acl.check_protected_from_delete(cluster) def check_cluster_update(cluster_id, data, **kwargs): cluster = api.get_cluster(cluster_id) verification = verification_base.validate_verification_ops( cluster, data) acl.check_tenant_for_update(context.current(), cluster) if not verification: acl.check_protected_from_update(cluster, data) sahara-8.0.0/sahara/service/validations/__init__.py0000666000175100017510000000000013245514472022326 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/validations/node_group_templates.py0000666000175100017510000000754013245514472025026 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.api import v10 as api import sahara.service.validations.base as b from sahara.service.validations import shares def check_node_group_template_create(data, **kwargs): plugin_version = 'hadoop_version' if data.get('plugin_version'): plugin_version = 'plugin_version' b.check_node_group_template_unique_name(data['name']) b.check_plugin_name_exists(data['plugin_name']) b.check_plugin_supports_version(data['plugin_name'], data[plugin_version]) b.check_node_group_basic_fields(data['plugin_name'], data[plugin_version], data) if data.get('image_id'): b.check_image_registered(data['image_id']) b.check_required_image_tags(data['plugin_name'], data[plugin_version], data['image_id']) if data.get('shares'): shares.check_shares(data['shares']) def check_node_group_template_usage(node_group_template_id, **kwargs): cluster_users = [] template_users = [] for cluster in api.get_clusters(): if (node_group_template_id in [node_group.node_group_template_id for node_group in cluster.node_groups]): cluster_users += [cluster.name] for cluster_template in api.get_cluster_templates(): if (node_group_template_id in [node_group.node_group_template_id for node_group in cluster_template.node_groups]): template_users += [cluster_template.name] if cluster_users or template_users: raise ex.InvalidReferenceException( _("Node group template %(template)s is in use by " "cluster templates: %(users)s; and clusters: %(clusters)s") % {'template': node_group_template_id, 'users': template_users and ', '.join(template_users) or 'N/A', 'clusters': cluster_users and ', '.join(cluster_users) or 'N/A'}) def check_node_group_template_update(node_group_template_id, data, **kwargs): plugin_version = 'hadoop_version' if data.get('plugin_version'): plugin_version = 'plugin_version' if data.get('plugin_name') and not data.get(plugin_version): raise ex.InvalidReferenceException( _("You must specify a %s value " "for your plugin_name") % plugin_version) if data.get('plugin_name'): plugin = data.get('plugin_name') version = data.get(plugin_version) b.check_plugin_name_exists(plugin) b.check_plugin_supports_version(plugin, version) else: ngt = api.get_node_group_template(node_group_template_id) plugin = ngt.plugin_name if data.get(plugin_version): version = data.get(plugin_version) b.check_plugin_supports_version(plugin, version) else: version = ngt.hadoop_version if data.get('image_id'): b.check_image_registered(data['image_id']) b.check_required_image_tags(plugin, version, data['image_id']) b.check_node_group_basic_fields(plugin, version, data) if data.get('shares'): shares.check_shares(data['shares']) sahara-8.0.0/sahara/service/validations/node_group_template_schema.py0000666000175100017510000000730413245514472026161 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara.service.validations import shares NODE_GROUP_TEMPLATE_SCHEMA = { "type": "object", "properties": { "name": { "type": "string", "minLength": 1, "maxLength": 50, "format": "valid_name_hostname", }, "flavor_id": { "type": "flavor", }, "plugin_name": { "type": "string", }, "hadoop_version": { "type": "string", }, "node_processes": { "type": "array", "items": { "type": "string", }, "minItems": 1 }, "image_id": { "type": ["string", "null"], "format": "uuid", }, "node_configs": { "type": ["configs", "null"], }, "volumes_per_node": { "type": "integer", "minimum": 0, }, "volumes_size": { "type": ["integer", "null"], "minimum": 1, }, "volume_type": { "type": ["string", "null"], }, "volumes_availability_zone": { "type": ["string", "null"], }, "volume_mount_prefix": { "type": ["string", "null"], "format": "posix_path", }, "description": { "type": ["string", "null"], }, "floating_ip_pool": { "type": ["string", "null"], }, "security_groups": { "type": ["array", "null"], "items": {"type": "string"} }, "auto_security_group": { "type": ["boolean", "null"], }, "availability_zone": { "type": ["string", "null"], }, "is_proxy_gateway": { "type": ["boolean", "null"], }, "volume_local_to_instance": { "type": ["boolean", "null"] }, "shares": copy.deepcopy(shares.SHARE_SCHEMA), "use_autoconfig": { "type": ["boolean", "null"] }, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], } }, "additionalProperties": False, "required": [ "name", "flavor_id", "plugin_name", "hadoop_version", "node_processes", ] } # APIv2: renaming hadoop_version -> plugin_version NODE_GROUP_TEMPLATE_SCHEMA_V2 = copy.deepcopy(NODE_GROUP_TEMPLATE_SCHEMA) del NODE_GROUP_TEMPLATE_SCHEMA_V2["properties"]["hadoop_version"] NODE_GROUP_TEMPLATE_SCHEMA_V2["required"].remove("hadoop_version") NODE_GROUP_TEMPLATE_SCHEMA_V2["properties"].update({ "plugin_version": { "type": "string", }}) NODE_GROUP_TEMPLATE_SCHEMA_V2["required"].append("plugin_version") # For an update we do not require any fields but we want the given # fields to be validated NODE_GROUP_TEMPLATE_UPDATE_SCHEMA = copy.copy(NODE_GROUP_TEMPLATE_SCHEMA) NODE_GROUP_TEMPLATE_UPDATE_SCHEMA["required"] = [] NODE_GROUP_TEMPLATE_UPDATE_SCHEMA_V2 = copy.copy(NODE_GROUP_TEMPLATE_SCHEMA_V2) NODE_GROUP_TEMPLATE_UPDATE_SCHEMA_V2["required"] = [] sahara-8.0.0/sahara/service/validations/edp/0000775000175100017510000000000013245515027020772 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/validations/edp/job_schema.py0000666000175100017510000000441213245514472023444 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.validations.edp import job_interface as j_i from sahara.utils import edp JOB_SCHEMA = { "type": "object", "properties": { "name": { "type": "string", "minLength": 1, "maxLength": 50, "format": "valid_name" }, "description": { "type": "string" }, "type": { "type": "string", "enum": edp.JOB_TYPES_ALL, }, "mains": { "type": "array", "uniqueItems": True, "items": { "type": "string", "minLength": 1, } }, "libs": { "type": "array", "uniqueItems": True, "items": { "type": "string", "minLength": 1, } }, "streaming": { "type": "boolean" }, "interface": j_i.INTERFACE_ARGUMENT_SCHEMA, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], } }, "additionalProperties": False, "required": [ "name", "type", ] } JOB_UPDATE_SCHEMA = { "type": "object", "properties": { "name": { "type": "string", "minLength": 1, "maxLength": 50, "format": "valid_name" }, "description": { "type": ["string", "null"] }, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], } }, "additionalProperties": False, "required": [] } sahara-8.0.0/sahara/service/validations/edp/data_source.py0000666000175100017510000000515413245514472023647 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re from oslo_config import cfg from sahara import conductor as c from sahara import context import sahara.exceptions as ex from sahara.i18n import _ import sahara.service.edp.data_sources.manager as ds_manager import sahara.service.validations.edp.base as b CONF = cfg.CONF def check_data_source_create(data, **kwargs): b.check_data_source_unique_name(data['name']) _check_data_source(data) def _check_datasource_placeholder(url): if url is None: return total_length = 0 substrings = re.findall(r"%RANDSTR\(([\-]?\d+)\)%", url) for length in map(int, substrings): if length <= 0: raise ex.InvalidDataException(_("Requested RANDSTR length" " must be positive.")) total_length += length if total_length > 1024: raise ex.InvalidDataException(_("Requested RANDSTR length is" " too long, please choose a " "value less than 1024.")) def _check_data_source(data): _check_datasource_placeholder(data["url"]) if data["type"] in CONF.data_source_types: ds_manager.DATA_SOURCES.get_data_source(data["type"]).validate(data) def check_data_source_update(data, data_source_id): ctx = context.ctx() jobs = c.API.job_execution_get_all(ctx) pending_jobs = [job for job in jobs if job.info["status"] == "PENDING"] for job in pending_jobs: if data_source_id in job.data_source_urls: raise ex.UpdateFailedException( _("DataSource is used in a " "PENDING Job and can not be updated.")) ds = c.API.data_source_get(ctx, data_source_id) if 'name' in data and data['name'] != ds.name: b.check_data_source_unique_name(data['name']) check_data = {'type': data.get('type', None) or ds.type, 'url': data.get('url', None) or ds.url, 'credentials': data.get( 'credentials', None) or ds.credentials} _check_data_source(check_data) sahara-8.0.0/sahara/service/validations/edp/data_source_schema.py0000666000175100017510000000301013245514472025154 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import sahara.service.validations.edp.base as b DATA_SOURCE_SCHEMA = { "type": "object", "properties": { "name": { "type": "string", "minLength": 1, "maxLength": 50, "format": "valid_name" }, "description": { "type": "string" }, "type": b.data_source_type, "url": { "type": "string", }, "credentials": { "type": "object" }, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], } }, "additionalProperties": False, "required": [ "name", "type", "url" ] } # For an update we do not require any fields but we want the given # fields to be validated DATA_SOURCE_UPDATE_SCHEMA = copy.copy(DATA_SOURCE_SCHEMA) DATA_SOURCE_UPDATE_SCHEMA["required"] = [] sahara-8.0.0/sahara/service/validations/edp/job_execution.py0000666000175100017510000001605613245514472024216 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six import time from oslo_utils import timeutils from sahara import conductor as c from sahara import context from sahara import exceptions as ex from sahara.i18n import _ from sahara.plugins import base as plugin_base from sahara.service.edp import job_utils as j_u from sahara.service.validations import acl from sahara.service.validations import base as val_base import sahara.service.validations.edp.base as b import sahara.service.validations.edp.job_interface as j_i from sahara.utils import cluster as c_u from sahara.utils import edp conductor = c.API def _is_main_class_present(data): if data: val = data.get( 'job_configs', {}).get( 'configs', {}).get('edp.java.main_class', None) return val and isinstance(val, six.string_types) return False def check_main_class_present(data, job): if not _is_main_class_present(data): raise ex.InvalidDataException( _('%s job must specify edp.java.main_class') % job.type) def _is_topology_name_present(data): if data: val = data.get( 'job_configs', {}).get( 'configs', {}).get('topology_name', None) return val and isinstance(val, six.string_types) return False def check_topology_name_present(data, job): if not _is_main_class_present(data): raise ex.InvalidDataException( _('%s job must specify topology_name') % job.type) def _streaming_present(data): try: streaming = set(('edp.streaming.mapper', 'edp.streaming.reducer')) configs = set(data['job_configs']['configs']) return streaming.intersection(configs) == streaming except Exception: return False def check_streaming_present(data, job): if not _streaming_present(data): raise ex.InvalidDataException( _("%s job must specify streaming mapper and reducer") % job.type) def check_scheduled_job_execution_info(job_execution_info): start = job_execution_info.get('start', None) if start is None: raise ex.InvalidDataException(_( "Scheduled job must specify start time")) try: start = time.strptime(start, "%Y-%m-%d %H:%M:%S") start = timeutils.datetime.datetime.fromtimestamp(time.mktime(start)) except Exception: raise ex.InvalidDataException(_("Invalid Time Format")) now_time = timeutils.utcnow() if timeutils.delta_seconds(now_time, start) < 0: raise ex.InvalidJobExecutionInfoException(_( "Job start time should be later than now")) def check_job_execution(data, job_templates_id=None): ctx = context.ctx() job_execution_info = data.get('job_execution_info', {}) cluster = conductor.cluster_get(ctx, data['cluster_id']) if not cluster: raise ex.InvalidReferenceException( _("Cluster with id '%s' doesn't exist") % data['cluster_id']) val_base.check_plugin_labels(cluster.plugin_name, cluster.hadoop_version) jt_err_msg = _("Job template with id '%s' doesn't exist") if job_templates_id is None: job = conductor.job_get(ctx, data['job_template_id']) if not job: raise ex.InvalidReferenceException( jt_err_msg % data['job_template_id']) else: job = conductor.job_get(ctx, job_templates_id) if not job: raise ex.InvalidReferenceException( jt_err_msg % job_templates_id) plugin = plugin_base.PLUGINS.get_plugin(cluster.plugin_name) edp_engine = plugin.get_edp_engine(cluster, job.type) if not edp_engine: raise ex.InvalidReferenceException( _("Cluster with id '%(cluster_id)s' doesn't support job type " "'%(job_type)s'") % {"cluster_id": cluster.id, "job_type": job.type}) j_i.check_execution_interface(data, job) edp_engine.validate_job_execution(cluster, job, data) if 'job_execution_type' in job_execution_info: j_type = job_execution_info.get('job_execution_type', 'workflow') if j_type == 'scheduled': check_scheduled_job_execution_info(job_execution_info) def check_data_sources(data, job): if not ('input_id' in data and 'output_id' in data): raise ex.InvalidDataException(_("%s job requires 'input_id' " "and 'output_id'") % job.type) b.check_data_source_exists(data['input_id']) b.check_data_source_exists(data['output_id']) b.check_data_sources_are_different(data['input_id'], data['output_id']) def check_job_execution_cancel(job_id, **kwargs): ctx = context.current() je = conductor.job_execution_get(ctx, job_id) if je.tenant_id != ctx.tenant_id: raise ex.CancelingFailed( _("Job execution with id '%s' cannot be canceled " "because it wasn't created in this tenant") % job_id) if je.is_protected: raise ex.CancelingFailed( _("Job Execution with id '%s' cannot be canceled " "because it's marked as protected") % job_id) def check_job_execution_delete(job_id, **kwargs): ctx = context.current() je = conductor.job_execution_get(ctx, job_id) acl.check_tenant_for_delete(ctx, je) acl.check_protected_from_delete(je) def check_job_execution_update(job_id, data, **kwargs): ctx = context.current() je = conductor.job_execution_get(ctx, job_id) acl.check_tenant_for_update(ctx, je) acl.check_protected_from_update(je, data) def check_job_status_update(job_id, data): ctx = context.ctx() job_execution = conductor.job_execution_get(ctx, job_id) # check we are updating status if 'info' in data: if len(data) != 1: raise ex.InvalidJobStatus(_("Invalid status parameter")) cluster = conductor.cluster_get(ctx, job_execution.cluster_id) if cluster is None or cluster.status != c_u.CLUSTER_STATUS_ACTIVE: raise ex.InvalidDataException( _("Suspending operation can not be performed on an inactive or " "non-existent cluster")) job_templates_id = conductor.job_execution_get(ctx, job_id).job_id job_type = conductor.job_get(ctx, job_templates_id).type engine = j_u.get_plugin(cluster).get_edp_engine(cluster, job_type) if 'info' in data: if data['info']['status'] == edp.JOB_ACTION_SUSPEND: if not engine.does_engine_implement('suspend_job'): raise ex.InvalidReferenceException( _("Engine doesn't support suspending job feature")) sahara-8.0.0/sahara/service/validations/edp/job_binary_schema.py0000666000175100017510000000275613245514472025021 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy JOB_BINARY_SCHEMA = { "type": "object", "properties": { "name": { "type": "string", "minLength": 1, "maxLength": 50, "format": "valid_name" }, "description": { "type": "string" }, "url": { "type": "string", "format": "valid_job_location" }, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], }, # extra is simple_config for now because we may need not only # user-password it the case of external storage "extra": { "type": "simple_config", } }, "additionalProperties": False, "required": [ "name", "url" ] } JOB_BINARY_UPDATE_SCHEMA = copy.copy(JOB_BINARY_SCHEMA) JOB_BINARY_UPDATE_SCHEMA["required"] = [] sahara-8.0.0/sahara/service/validations/edp/job_interface.py0000666000175100017510000001621513245514472024150 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils import six from six.moves.urllib import parse as urlparse import sahara.exceptions as e from sahara.i18n import _ from sahara.service.validations.edp import base as b from sahara.utils import edp DATA_TYPE_STRING = "string" DATA_TYPE_NUMBER = "number" DATA_TYPE_DATA_SOURCE = "data_source" DATA_TYPES = [DATA_TYPE_STRING, DATA_TYPE_NUMBER, DATA_TYPE_DATA_SOURCE] DEFAULT_DATA_TYPE = DATA_TYPE_STRING INTERFACE_ARGUMENT_SCHEMA = { "type": ["array", "null"], "uniqueItems": True, "items": { "type": "object", "properties": { "name": { "type": "string", "minLength": 1 }, "description": { "type": ["string", "null"] }, "mapping_type": { "type": "string", "enum": ["args", "configs", "params"] }, "location": { "type": "string", "minLength": 1 }, "value_type": { "type": "string", "enum": DATA_TYPES, "default": "string" }, "required": { "type": "boolean" }, "default": { "type": ["string", "null"] } }, "additionalProperties": False, "required": ["name", "mapping_type", "location", "required"] } } def _check_job_interface(data, interface): names = set(arg["name"] for arg in interface) if len(names) != len(interface): raise e.InvalidDataException( _("Name must be unique within the interface for any job.")) mapping_types = set(arg["mapping_type"] for arg in interface) acceptable_types = edp.JOB_TYPES_ACCEPTABLE_CONFIGS[data["type"]] if any(m_type not in acceptable_types for m_type in mapping_types): args = {"mapping_types": str(list(acceptable_types)), "job_type": data["type"]} raise e.InvalidDataException( _("Only mapping types %(mapping_types)s are allowed for job type " "%(job_type)s.") % args) positional_args = [arg for arg in interface if arg["mapping_type"] == "args"] if not all(six.text_type(arg["location"]).isnumeric() for arg in positional_args): raise e.InvalidDataException( _("Locations of positional arguments must be an unbroken integer " "sequence ascending from 0.")) locations = set(int(arg["location"]) for arg in positional_args) if not all(i in locations for i in range(len(locations))): raise e.InvalidDataException( _("Locations of positional arguments must be an unbroken integer " "sequence ascending from 0.")) not_required = (arg for arg in positional_args if not arg["required"]) if not all(arg.get("default", None) for arg in not_required): raise e.InvalidDataException( _("Positional arguments must be given default values if they are " "not required.")) mappings = ((arg["mapping_type"], arg["location"]) for arg in interface) if len(set(mappings)) != len(interface): raise e.InvalidDataException( _("The combination of mapping type and location must be unique " "within the interface for any job.")) for arg in interface: if "value_type" not in arg: arg["value_type"] = DEFAULT_DATA_TYPE default = arg.get("default", None) if default is not None: _validate_value(arg["value_type"], default) def check_job_interface(data, **kwargs): interface = data.get("interface", []) if interface: _check_job_interface(data, interface) def _validate_data_source(value): if uuidutils.is_uuid_like(value): b.check_data_source_exists(value) else: if not urlparse.urlparse(value).scheme: raise e.InvalidDataException( _("Data source value '%s' is neither a valid data source ID " "nor a valid URL.") % value) def _validate_number(value): if not six.text_type(value).isnumeric(): raise e.InvalidDataException( _("Value '%s' is not a valid number.") % value) def _validate_string(value): if not isinstance(value, six.string_types): raise e.InvalidDataException( _("Value '%s' is not a valid string.") % value) _value_type_validators = { DATA_TYPE_STRING: _validate_string, DATA_TYPE_NUMBER: _validate_number, DATA_TYPE_DATA_SOURCE: _validate_data_source } def _validate_value(type, value): _value_type_validators[type](value) def check_execution_interface(data, job): job_int = {arg.name: arg for arg in job.interface} execution_int = data.get("interface", None) if not (job_int or execution_int): return if job_int and execution_int is None: raise e.InvalidDataException( _("An interface was specified with the template for this job. " "Please pass an interface map with this job (even if empty).")) execution_names = set(execution_int.keys()) definition_names = set(job_int.keys()) not_found_names = execution_names - definition_names if not_found_names: raise e.InvalidDataException( _("Argument names: %s were not found in the interface for this " "job.") % str(list(not_found_names))) required_names = {arg.name for arg in job.interface if arg.required} unset_names = required_names - execution_names if unset_names: raise e.InvalidDataException(_("Argument names: %s are required for " "this job.") % str(list(unset_names))) nonexistent = object() for name, value in six.iteritems(execution_int): arg = job_int[name] _validate_value(arg.value_type, value) if arg.mapping_type == "args": continue typed_configs = data.get("job_configs", {}).get(arg.mapping_type, {}) config_value = typed_configs.get(arg.location, nonexistent) if config_value is not nonexistent and config_value != value: args = {"name": name, "mapping_type": arg.mapping_type, "location": arg.location} raise e.InvalidDataException( _("Argument '%(name)s' was passed both through the interface " "and in location '%(mapping_type)s'.'%(location)s'. Please " "pass this through either the interface or the " "configuration maps, not both.") % args) sahara-8.0.0/sahara/service/validations/edp/base.py0000666000175100017510000000556613245514472022277 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """Cluster creation related checks""" from sahara import conductor as c from sahara import context from sahara import exceptions as ex from sahara.i18n import _ conductor = c.API data_source_type = { "type": "string", "enum": ["swift", "hdfs", "maprfs", "manila"] } job_configs = { "type": "object", "properties": { "configs": { "type": "simple_config", }, "params": { "type": "simple_config", }, "args": { "type": "array", "items": { "type": "string" } }, "job_execution_info": { "type": "simple_config", } }, "additionalProperties": False, } def check_data_source_unique_name(ds_name): if ds_name in [ds.name for ds in conductor.data_source_get_all(context.ctx(), name=ds_name)]: raise ex.NameAlreadyExistsException( _("Data source with name '%s' " "already exists") % ds_name) def check_data_source_exists(data_source_id): if not conductor.data_source_get(context.ctx(), data_source_id): raise ex.InvalidReferenceException( _("DataSource with id '%s' doesn't exist") % data_source_id) def check_job_unique_name(job_name): if job_name in [job.name for job in conductor.job_get_all(context.ctx(), name=job_name)]: raise ex.NameAlreadyExistsException(_("Job with name '%s' " "already exists") % job_name) def check_job_binary_internal_exists(jbi_id): if not conductor.job_binary_internal_get(context.ctx(), jbi_id): raise ex.InvalidReferenceException( _("JobBinaryInternal with id '%s' doesn't exist") % jbi_id) def check_data_sources_are_different(data_source_1_id, data_source_2_id): ds1 = conductor.data_source_get(context.ctx(), data_source_1_id) ds2 = conductor.data_source_get(context.ctx(), data_source_2_id) if ds1.type == ds2.type and ds1.url == ds2.url: raise ex.InvalidDataException(_('Provided input and output ' 'DataSources reference the same ' 'location: %s') % ds1.url) sahara-8.0.0/sahara/service/validations/edp/job_execution_schema.py0000666000175100017510000000411413245514472025526 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import sahara.service.validations.edp.base as b JOB_EXEC_SCHEMA = { "type": "object", "properties": { "input_id": { "type": "string", "format": "uuid", }, "output_id": { "type": "string", "format": "uuid", }, "cluster_id": { "type": "string", "format": "uuid", }, "interface": { "type": "simple_config", }, "job_configs": b.job_configs, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], } }, "additionalProperties": False, "required": [ "cluster_id" ] } JOB_EXEC_SCHEMA_V2 = copy.deepcopy(JOB_EXEC_SCHEMA) JOB_EXEC_SCHEMA_V2['properties'].update({ "job_template_id": { "type": "string", "format": "uuid", }}) JOB_EXEC_SCHEMA_V2['required'].append('job_template_id') JOB_EXEC_UPDATE_SCHEMA = { "type": "object", "properties": { "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], }, "info": { "type": "simple_config", "properties": { "status": { "enum": ["suspend", "cancel"] } }, "additionalProperties": False } }, "additionalProperties": False, "required": [] } sahara-8.0.0/sahara/service/validations/edp/job_binary_internal_schema.py0000666000175100017510000000177513245514472026715 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. JOB_BINARY_UPDATE_SCHEMA = { "type": "object", "properties": { "name": { "type": "string", "minLength": 1, "maxLength": 50, "format": "valid_name" }, "is_public": { "type": ["boolean", "null"], }, "is_protected": { "type": ["boolean", "null"], } }, "additionalProperties": False, "required": [] } sahara-8.0.0/sahara/service/validations/edp/__init__.py0000666000175100017510000000000013245514472023076 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/validations/edp/job_binary_internal.py0000666000175100017510000000264013245514472025365 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import sahara.exceptions as e from sahara.i18n import _ from sahara.service.edp.job_binaries import manager as jb_manager from sahara.utils import api_validator as a CONF = cfg.CONF def is_internal_db_enabled(): return 'internal-db' in jb_manager.JOB_BINARIES.get_job_binaries() def check_job_binary_internal(data, **kwargs): if not is_internal_db_enabled(): raise e.BadJobBinaryInternalException( _("Sahara internal db is disabled for storing job binaries.")) if not (type(data) is str and len(data) > 0): raise e.BadJobBinaryInternalException() if "name" in kwargs: name = kwargs["name"] if not a.validate_name_format(name): raise e.BadJobBinaryInternalException(_("%s is not a valid name") % name) sahara-8.0.0/sahara/service/validations/edp/job_binary.py0000666000175100017510000000172013245514472023467 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging import sahara.service.edp.job_binaries.manager as jb_manager CONF = cfg.CONF LOG = logging.getLogger(__name__) def check_job_binary(data, **kwargs): job_binary_url = data.get("url", None) if job_binary_url: jb_manager.JOB_BINARIES.get_job_binary_by_url(job_binary_url). \ validate(data, **kwargs) sahara-8.0.0/sahara/service/validations/edp/job.py0000666000175100017510000000514413245514472022127 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import sahara.exceptions as e from sahara.i18n import _ from sahara.service.api import v11 as api from sahara.service.validations.edp import job_interface as j_i from sahara.utils import edp def _check_binaries(values): for job_binary in values: if not api.get_job_binary(job_binary): raise e.NotFoundException(job_binary, _("Job binary '%s' does not exist")) def check_mains_libs(data, **kwargs): mains = data.get("mains", []) libs = data.get("libs", []) job_type, subtype = edp.split_job_type(data.get("type")) streaming = (job_type == edp.JOB_TYPE_MAPREDUCE and subtype == edp.JOB_SUBTYPE_STREAMING) # These types must have a value in mains and may also use libs if job_type in [edp.JOB_TYPE_PIG, edp.JOB_TYPE_HIVE, edp.JOB_TYPE_SHELL, edp.JOB_TYPE_SPARK, edp.JOB_TYPE_STORM, edp.JOB_TYPE_PYLEUS]: if not mains: if job_type in [edp.JOB_TYPE_SPARK, edp.JOB_TYPE_STORM, edp.JOB_TYPE_PYLEUS]: msg = _( "%s job requires main application jar") % data.get("type") else: msg = _("%s flow requires main script") % data.get("type") raise e.InvalidDataException(msg) # Check for overlap if set(mains).intersection(set(libs)): raise e.InvalidDataException(_("'mains' and 'libs' overlap")) else: # Java and MapReduce require libs, but MapReduce.Streaming does not if not streaming and not libs: raise e.InvalidDataException(_("%s flow requires libs") % data.get("type")) if mains: raise e.InvalidDataException(_("%s flow does not use mains") % data.get("type")) # Make sure that all referenced binaries exist _check_binaries(mains) _check_binaries(libs) def check_interface(data, **kwargs): j_i.check_job_interface(data, **kwargs) sahara-8.0.0/sahara/service/validations/clusters_scaling.py0000666000175100017510000000655013245514472024153 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import context import sahara.exceptions as ex from sahara.i18n import _ import sahara.plugins.base as plugin_base from sahara.service import api as service_api from sahara.service.api import v10 as api from sahara.service.validations import acl import sahara.service.validations.base as b from sahara.utils import cluster as c_u def check_cluster_scaling(data, cluster_id, **kwargs): ctx = context.current() cluster = api.get_cluster(id=cluster_id) if cluster is None: raise ex.NotFoundException( {'id': cluster_id}, _('Object with %s not found')) b.check_plugin_labels( cluster.plugin_name, cluster.hadoop_version) acl.check_tenant_for_update(ctx, cluster) acl.check_protected_from_update(cluster, data) cluster_engine = cluster.sahara_info.get( 'infrastructure_engine') if cluster.sahara_info else None engine_type_and_version = service_api.OPS.get_engine_type_and_version() if (not cluster_engine and not engine_type_and_version.startswith('direct')): raise ex.InvalidReferenceException( _("Cluster created before Juno release " "can't be scaled with %(engine)s engine") % {"engine": engine_type_and_version}) if (cluster.sahara_info and cluster_engine != engine_type_and_version): raise ex.InvalidReferenceException( _("Cluster created with %(old_engine)s infrastructure engine " "can't be scaled with %(new_engine)s engine") % {"old_engine": cluster.sahara_info.get('infrastructure_engine'), "new_engine": engine_type_and_version}) if not (plugin_base.PLUGINS.is_plugin_implements(cluster.plugin_name, 'scale_cluster') and ( plugin_base.PLUGINS.is_plugin_implements(cluster.plugin_name, 'decommission_nodes'))): raise ex.InvalidReferenceException( _("Requested plugin '%s' doesn't support cluster scaling feature") % cluster.plugin_name) if cluster.status != c_u.CLUSTER_STATUS_ACTIVE: raise ex.InvalidReferenceException( _("Cluster cannot be scaled not in 'Active' status. " "Cluster status: %s") % cluster.status) if cluster.user_keypair_id: b.check_keypair_exists(cluster.user_keypair_id) if cluster.default_image_id: b.check_image_registered(cluster.default_image_id) if data.get("resize_node_groups"): b.check_resize(cluster, data['resize_node_groups']) if data.get("add_node_groups"): b.check_add_node_groups(cluster, data['add_node_groups']) b.check_cluster_hostnames_lengths(cluster.name, data['add_node_groups']) sahara-8.0.0/sahara/service/validations/acl.py0000666000175100017510000000436213245514472021345 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six from sahara import exceptions as ex from sahara.i18n import _ def check_tenant_for_delete(context, object): if object.tenant_id != context.tenant_id: raise ex.DeletionFailed( _("{object} with id '{id}' could not be deleted because " "it wasn't created in this tenant").format( object=type(object).__name__, id=object.id)) def check_tenant_for_update(context, object): if object.tenant_id != context.tenant_id: raise ex.UpdateFailedException( object.id, _("{object} with id '%s' could not be updated because " "it wasn't created in this tenant").format( object=type(object).__name__)) def check_protected_from_delete(object): if object.is_protected: raise ex.DeletionFailed( _("{object} with id '{id}' could not be deleted because " "it's marked as protected").format( object=type(object).__name__, id=object.id)) def check_protected_from_update(object, data): if object.is_protected and data.get('is_protected', True): # Okay, the only thing we can allow here is a change # to 'is_public', so we have to make sure no other values # are changing if 'is_public' in data: obj = object.to_dict() if all(k == 'is_public' or ( k in obj and obj[k] == v) for k, v in six.iteritems(data)): return raise ex.UpdateFailedException( object.id, _("{object} with id '%s' could not be updated " "because it's marked as protected").format( object=type(object).__name__)) sahara-8.0.0/sahara/service/quotas.py0000666000175100017510000001406213245514472017603 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six from sahara import context from sahara import exceptions as ex from sahara.i18n import _ from sahara.utils.openstack import base as b from sahara.utils.openstack import cinder as cinder_client from sahara.utils.openstack import neutron as neutron_client from sahara.utils.openstack import nova as nova_client CONF = cfg.CONF UNLIMITED = 'unlimited' def _is_unlimited(limit): return limit == -1 def _get_zero_limits(): return { 'ram': 0, 'cpu': 0, 'instances': 0, 'floatingips': 0, 'security_groups': 0, 'security_group_rules': 0, 'ports': 0, 'volumes': 0, 'volume_gbs': 0 } def check_cluster(cluster): req_limits = _get_req_cluster_limits(cluster) _check_limits(req_limits) def check_scaling(cluster, to_be_enlarged, additional): req_limits = _get_req_scaling_limits(cluster, to_be_enlarged, additional) _check_limits(req_limits) def _check_limits(req_limits): limits_name_map = { 'ram': _("RAM"), 'cpu': _("VCPU"), 'instances': _("instance"), 'floatingips': _("floating ip"), 'security_groups': _("security group"), 'security_group_rules': _("security group rule"), 'ports': _("port"), 'volumes': _("volume"), 'volume_gbs': _("volume storage") } avail_limits = _get_avail_limits() for quota, quota_name in six.iteritems(limits_name_map): if avail_limits[quota] != UNLIMITED: if avail_limits[quota] < req_limits[quota]: raise ex.QuotaException(quota_name, req_limits[quota], avail_limits[quota]) def _get_req_cluster_limits(cluster): req_limits = _get_zero_limits() for ng in cluster.node_groups: _update_limits_for_ng(req_limits, ng, ng.count) return req_limits def _get_req_scaling_limits(cluster, to_be_enlarged, additional): ng_id_map = to_be_enlarged.copy() ng_id_map.update(additional) req_limits = _get_zero_limits() for ng in cluster.node_groups: if ng_id_map.get(ng.id): _update_limits_for_ng(req_limits, ng, ng_id_map[ng.id] - ng.count) return req_limits def _update_limits_for_ng(limits, ng, count): sign = lambda x: (1, -1)[x < 0] nova = nova_client.client() limits['instances'] += count flavor = b.execute_with_retries(nova.flavors.get, ng.flavor_id) limits['ram'] += flavor.ram * count limits['cpu'] += flavor.vcpus * count # tmckay-fp this is fine, it will be zero without it if ng.floating_ip_pool: limits['floatingips'] += count if ng.volumes_per_node: limits['volumes'] += ng.volumes_per_node * count limits['volume_gbs'] += ng.volumes_per_node * ng.volumes_size * count if ng.auto_security_group: limits['security_groups'] += sign(count) # NOTE: +3 - all traffic for private network limits['security_group_rules'] += ( (len(ng.open_ports) + 3) * sign(count)) limits['ports'] += count def _get_avail_limits(): limits = _get_zero_limits() limits.update(_get_nova_limits()) limits.update(_get_neutron_limits()) if cinder_client.check_cinder_exists(): limits.update(_get_cinder_limits()) return limits def _sub_limit(total, used): if _is_unlimited(total): return UNLIMITED else: return total - used def _get_nova_limits(): limits = {} nova = nova_client.client() lim = b.execute_with_retries(nova.limits.get).to_dict()['absolute'] limits['ram'] = _sub_limit(lim['maxTotalRAMSize'], lim['totalRAMUsed']) limits['cpu'] = _sub_limit(lim['maxTotalCores'], lim['totalCoresUsed']) limits['instances'] = _sub_limit(lim['maxTotalInstances'], lim['totalInstancesUsed']) return limits def _get_neutron_limits(): limits = {} neutron = neutron_client.client() tenant_id = context.ctx().tenant_id total_lim = b.execute_with_retries(neutron.show_quota, tenant_id)['quota'] # tmckay-fp here we would just get the limits all the time usage_fip = b.execute_with_retries( neutron.list_floatingips, tenant_id=tenant_id)['floatingips'] limits['floatingips'] = _sub_limit(total_lim['floatingip'], len(usage_fip)) usage_sg = b.execute_with_retries( neutron.list_security_groups, tenant_id=tenant_id).get( 'security_groups', []) limits['security_groups'] = _sub_limit(total_lim['security_group'], len(usage_sg)) usage_sg_rules = b.execute_with_retries( neutron.list_security_group_rules, tenant_id=tenant_id).get( 'security_group_rules', []) limits['security_group_rules'] = _sub_limit( total_lim['security_group_rule'], len(usage_sg_rules)) usage_ports = b.execute_with_retries( neutron.list_ports, tenant_id=tenant_id)['ports'] limits['ports'] = _sub_limit(total_lim['port'], len(usage_ports)) return limits def _get_cinder_limits(): avail_limits = {} cinder = cinder_client.client() lim = {} for l in b.execute_with_retries(cinder.limits.get).absolute: lim[l.name] = l.value avail_limits['volumes'] = _sub_limit(lim['maxTotalVolumes'], lim['totalVolumesUsed']) avail_limits['volume_gbs'] = _sub_limit(lim['maxTotalVolumeGigabytes'], lim['totalGigabytesUsed']) return avail_limits sahara-8.0.0/sahara/service/ops.py0000666000175100017510000004076613245514472017102 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools from oslo_config import cfg from oslo_log import log as logging import oslo_messaging as messaging from oslo_utils import uuidutils import six from sahara import conductor as c from sahara import context from sahara import exceptions from sahara.i18n import _ from sahara.plugins import base as plugin_base from sahara.service.edp import job_manager from sahara.service.edp.utils import shares from sahara.service.health import verification_base as ver_base from sahara.service import ntp_service from sahara.service import trusts from sahara.utils import cluster as c_u from sahara.utils import remote from sahara.utils import rpc as rpc_utils conductor = c.API CONF = cfg.CONF LOG = logging.getLogger(__name__) INFRA = None def setup_ops(engine): global INFRA INFRA = engine class LocalOps(object): def provision_cluster(self, cluster_id): context.spawn("cluster-creating-%s" % cluster_id, _provision_cluster, cluster_id) def provision_scaled_cluster(self, cluster_id, node_group_id_map, node_group_instance_map=None): context.spawn("cluster-scaling-%s" % cluster_id, _provision_scaled_cluster, cluster_id, node_group_id_map, node_group_instance_map) def terminate_cluster(self, cluster_id, force=False): context.spawn("cluster-terminating-%s" % cluster_id, terminate_cluster, cluster_id, force) def run_edp_job(self, job_execution_id): context.spawn("Starting Job Execution %s" % job_execution_id, _run_edp_job, job_execution_id) def cancel_job_execution(self, job_execution_id): context.spawn("Canceling Job Execution %s" % job_execution_id, _cancel_job_execution, job_execution_id) def delete_job_execution(self, job_execution_id): context.spawn("Deleting Job Execution %s" % job_execution_id, _delete_job_execution, job_execution_id) def handle_verification(self, cluster_id, values): context.spawn('Handling Verification for cluster %s' % cluster_id, _handle_verification, cluster_id, values) def get_engine_type_and_version(self): return INFRA.get_type_and_version() def job_execution_suspend(self, job_execution_id): context.spawn("Suspend Job Execution %s" % job_execution_id, _suspend_job_execution, job_execution_id) class RemoteOps(rpc_utils.RPCClient): def __init__(self): target = messaging.Target(topic='sahara-ops', version='1.0') super(RemoteOps, self).__init__(target) def provision_cluster(self, cluster_id): self.cast('provision_cluster', cluster_id=cluster_id) def provision_scaled_cluster(self, cluster_id, node_group_id_map, node_group_instance_map=None): self.cast('provision_scaled_cluster', cluster_id=cluster_id, node_group_id_map=node_group_id_map, node_group_instance_map=node_group_instance_map) def terminate_cluster(self, cluster_id, force=False): self.cast('terminate_cluster', cluster_id=cluster_id, force=force) def run_edp_job(self, job_execution_id): self.cast('run_edp_job', job_execution_id=job_execution_id) def cancel_job_execution(self, job_execution_id): self.cast('cancel_job_execution', job_execution_id=job_execution_id) def delete_job_execution(self, job_execution_id): self.cast('delete_job_execution', job_execution_id=job_execution_id) def handle_verification(self, cluster_id, values): self.cast('handle_verification', cluster_id=cluster_id, values=values) def get_engine_type_and_version(self): return self.call('get_engine_type_and_version') def job_execution_suspend(self, job_execution_id): self.cast('job_execution_suspend', job_execution_id=job_execution_id) def request_context(func): @functools.wraps(func) def wrapped(self, ctx, *args, **kwargs): context.set_ctx(context.Context(**ctx)) return func(self, *args, **kwargs) return wrapped class OpsServer(rpc_utils.RPCServer): def __init__(self): target = messaging.Target(topic='sahara-ops', server=uuidutils.generate_uuid(), version='1.0') super(OpsServer, self).__init__(target) @request_context def provision_cluster(self, cluster_id): _provision_cluster(cluster_id) @request_context def provision_scaled_cluster(self, cluster_id, node_group_id_map, node_group_instance_map=None): _provision_scaled_cluster(cluster_id, node_group_id_map, node_group_instance_map) @request_context def terminate_cluster(self, cluster_id, force=False): terminate_cluster(cluster_id, force) @request_context def run_edp_job(self, job_execution_id): _run_edp_job(job_execution_id) @request_context def cancel_job_execution(self, job_execution_id): _cancel_job_execution(job_execution_id) @request_context def delete_job_execution(self, job_execution_id): _delete_job_execution(job_execution_id) @request_context def handle_verification(self, cluster_id, values): _handle_verification(cluster_id, values) @request_context def get_engine_type_and_version(self): return INFRA.get_type_and_version() @request_context def job_execution_suspend(self, job_execution_id): _suspend_job_execution(job_execution_id) def _setup_trust_for_cluster(cluster): cluster = conductor.cluster_get(context.ctx(), cluster) trusts.create_trust_for_cluster(cluster) trusts.use_os_admin_auth_token(cluster) def ops_error_handler(description): def decorator(f): @functools.wraps(f) def wrapper(cluster_id, *args, **kwds): ctx = context.ctx() try: # Clearing status description before executing c_u.change_cluster_status_description(cluster_id, "") f(cluster_id, *args, **kwds) except Exception as ex: # something happened during cluster operation cluster = conductor.cluster_get(ctx, cluster_id) # check if cluster still exists (it might have been removed) if (cluster is None or cluster.status == c_u.CLUSTER_STATUS_DELETING): LOG.debug("Cluster was deleted or marked for deletion. " "Canceling current operation.") return msg = six.text_type(ex) LOG.exception("Error during operating on cluster (reason: " "{reason})".format(reason=msg)) try: # trying to rollback desc = description.format(reason=msg) if _rollback_cluster(cluster, ex): c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_ACTIVE, desc) else: c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_ERROR, desc) except Exception as rex: cluster = conductor.cluster_get(ctx, cluster_id) # check if cluster still exists (it might have been # removed during rollback) if (cluster is None or cluster.status == c_u.CLUSTER_STATUS_DELETING): LOG.debug("Cluster was deleted or marked for deletion." " Canceling current operation.") return LOG.exception( "Error during rollback of cluster (reason:" " {reason})".format(reason=six.text_type(rex))) desc = "{0}, {1}".format(msg, six.text_type(rex)) c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_ERROR, description.format(reason=desc)) return wrapper return decorator def _rollback_cluster(cluster, reason): _setup_trust_for_cluster(cluster) context.set_step_type(_("Engine: rollback cluster")) return INFRA.rollback_cluster(cluster, reason) def _prepare_provisioning(cluster_id): ctx = context.ctx() cluster = conductor.cluster_get(ctx, cluster_id) plugin = plugin_base.PLUGINS.get_plugin(cluster.plugin_name) for nodegroup in cluster.node_groups: update_dict = {} update_dict["image_username"] = INFRA.get_node_group_image_username( nodegroup) conductor.node_group_update(ctx, nodegroup, update_dict) _setup_trust_for_cluster(cluster) cluster = conductor.cluster_get(ctx, cluster_id) return ctx, cluster, plugin def _update_sahara_info(ctx, cluster): sahara_info = { 'infrastructure_engine': INFRA.get_type_and_version(), 'remote': remote.get_remote_type_and_version()} return conductor.cluster_update( ctx, cluster, {'sahara_info': sahara_info}) @ops_error_handler( _("Creating cluster failed for the following reason(s): {reason}")) def _provision_cluster(cluster_id): ctx, cluster, plugin = _prepare_provisioning(cluster_id) cluster = _update_sahara_info(ctx, cluster) # updating cluster infra cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_INFRAUPDATING) plugin.update_infra(cluster) # creating instances and configuring them cluster = conductor.cluster_get(ctx, cluster_id) context.set_step_type(_("Engine: create cluster")) INFRA.create_cluster(cluster) # configure cluster cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_CONFIGURING) context.set_step_type(_("Plugin: configure cluster")) if hasattr(plugin, 'validate_images'): plugin.validate_images(cluster, test_only=False) shares.mount_shares(cluster) plugin.configure_cluster(cluster) # starting prepared and configured cluster ntp_service.configure_ntp(cluster_id) cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_STARTING) context.set_step_type(_("Plugin: start cluster")) plugin.start_cluster(cluster) # cluster is now up and ready cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_ACTIVE) # schedule execution pending job for cluster for je in conductor.job_execution_get_all(ctx, cluster_id=cluster.id): job_manager.run_job(je.id) _refresh_health_for_cluster(cluster_id) @ops_error_handler( _("Scaling cluster failed for the following reason(s): {reason}")) def _provision_scaled_cluster(cluster_id, node_group_id_map, node_group_instance_map=None): ctx, cluster, plugin = _prepare_provisioning(cluster_id) # Decommissioning surplus nodes with the plugin cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_DECOMMISSIONING) try: instances_to_delete = [] for node_group in cluster.node_groups: new_count = node_group_id_map[node_group.id] if new_count < node_group.count: if (node_group_instance_map and node_group.id in node_group_instance_map): for instance_ref in node_group_instance_map[ node_group.id]: instance = _get_instance_obj(node_group.instances, instance_ref) instances_to_delete.append(instance) while node_group.count - new_count > len(instances_to_delete): instances_to_delete.append(_get_random_instance_from_ng( node_group.instances, instances_to_delete)) if instances_to_delete: context.set_step_type(_("Plugin: decommission cluster")) plugin.decommission_nodes(cluster, instances_to_delete) # Scaling infrastructure cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_SCALING) context.set_step_type(_("Engine: scale cluster")) instance_ids = INFRA.scale_cluster(cluster, node_group_id_map, instances_to_delete) # Setting up new nodes with the plugin if instance_ids: ntp_service.configure_ntp(cluster_id, instance_ids) cluster = c_u.change_cluster_status( cluster, c_u.CLUSTER_STATUS_CONFIGURING) instances = c_u.get_instances(cluster, instance_ids) context.set_step_type(_("Plugin: scale cluster")) plugin.scale_cluster(cluster, instances) c_u.change_cluster_status(cluster, c_u.CLUSTER_STATUS_ACTIVE) _refresh_health_for_cluster(cluster_id) except Exception as e: c_u.change_cluster_status(cluster, c_u.CLUSTER_STATUS_ACTIVE, six.text_type(e)) def _get_instance_obj(instances, instance_ref): for instance in instances: if (instance.instance_id == instance_ref or instance.instance_name == instance_ref): return instance raise exceptions.NotFoundException(str(instance_ref), _("Instance %s not found")) def _get_random_instance_from_ng(instances, instances_to_delete): # instances list doesn't order by creating date, so we should # sort it to make sure deleted instances same as heat deleted. insts = sorted(instances, key=lambda x: int(x['instance_name'].split('-')[-1])) for instance in reversed(insts): if instance not in instances_to_delete: return instance @ops_error_handler( _("Terminating cluster failed for the following reason(s): {reason}")) def terminate_cluster(cluster_id, force=False): ctx = context.ctx() _setup_trust_for_cluster(cluster_id) job_manager.update_job_statuses(cluster_id=cluster_id) cluster = conductor.cluster_get(ctx, cluster_id) plugin = plugin_base.PLUGINS.get_plugin(cluster.plugin_name) context.set_step_type(_("Plugin: shutdown cluster")) plugin.on_terminate_cluster(cluster) context.set_step_type(_("Engine: shutdown cluster")) INFRA.shutdown_cluster(cluster, force) trusts.delete_trust_from_cluster(cluster) conductor.cluster_destroy(ctx, cluster) def _run_edp_job(job_execution_id): job_manager.run_job(job_execution_id) def _suspend_job_execution(job_execution_id): job_manager.suspend_job(job_execution_id) def _cancel_job_execution(job_execution_id): job_manager.cancel_job(job_execution_id) def _delete_job_execution(job_execution_id): try: job_execution = job_manager.cancel_job(job_execution_id) if not job_execution: # job_execution was deleted already, nothing to do return except exceptions.CancelingFailed: LOG.error("Job execution can't be cancelled in time. " "Deleting it anyway.") conductor.job_execution_destroy(context.ctx(), job_execution_id) def _refresh_health_for_cluster(cluster_id): st_dict = {'verification': {'status': 'START'}} try: ver_base.validate_verification_start(cluster_id) ver_base.handle_verification(cluster_id, st_dict) except ver_base.CannotVerifyError: LOG.debug("Cannot verify cluster because verifications are disabled " "or cluster already is verifying") except Exception: # if occasional error occurred, there is no reason to move # cluster into error state LOG.debug("Skipping refreshing cluster health") ver_base.clean_verification_data(cluster_id) def _handle_verification(cluster_id, values): ver_base.handle_verification(cluster_id, values) sahara-8.0.0/sahara/service/sessions.py0000666000175100017510000001373313245514472020141 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from keystoneauth1 import session as keystone from oslo_config import cfg from oslo_log import log as logging from sahara import exceptions as ex from sahara.i18n import _ CONF = cfg.CONF LOG = logging.getLogger(__name__) _SESSION_CACHE = None SESSION_TYPE_CINDER = 'cinder' SESSION_TYPE_KEYSTONE = 'keystone' SESSION_TYPE_NEUTRON = 'neutron' SESSION_TYPE_NOVA = 'nova' SESSION_TYPE_GLANCE = 'glance' SESSION_TYPE_INSECURE = 'insecure' SESSION_TYPE_HEAT = 'heat' def cache(): global _SESSION_CACHE if not _SESSION_CACHE: _SESSION_CACHE = SessionCache() return _SESSION_CACHE class SessionCache(object): '''A cache of keystone Session objects When a requested Session is not currently cached, it will be acquired from specific information in this module. Sessions should be referenced by their OpenStack project name and not the service name, this is to allow for multiple service implementations while retaining the ability to generate Session objects. In all cases, the constant values in this module should be used to communicate the session type. ''' def __init__(self): '''create a new SessionCache''' self._sessions = {} self._session_funcs = { SESSION_TYPE_CINDER: self.get_cinder_session, SESSION_TYPE_KEYSTONE: self.get_keystone_session, SESSION_TYPE_NEUTRON: self.get_neutron_session, SESSION_TYPE_NOVA: self.get_nova_session, SESSION_TYPE_GLANCE: self.get_glance_session, SESSION_TYPE_INSECURE: self.get_insecure_session, SESSION_TYPE_HEAT: self.get_heat_session, } def _set_session(self, session_type, session): '''Set the session for a given type. :param session_type: the type of session to set. :param session: the session to associate with the type ''' self._sessions[session_type] = session def get_session(self, session_type=SESSION_TYPE_INSECURE): '''Return a Session for the requested type :param session_type: the type of Session to get, if None an insecure session will be returned. :raises SaharaException: if the requested session type is not found. ''' session_function = self._session_funcs.get(session_type) if session_function: return session_function() else: LOG.error('Requesting an unknown session type (type: {type})'. format(type=session_type)) raise ex.SaharaException( _('Session type {type} not recognized'). format(type=session_type)) def get_insecure_session(self): session = self._sessions.get(SESSION_TYPE_INSECURE) if not session: session = keystone.Session(verify=False) self._set_session(SESSION_TYPE_INSECURE, session) return session def get_cinder_session(self): session = self._sessions.get(SESSION_TYPE_CINDER) if not session: if not CONF.cinder.api_insecure: session = keystone.Session( verify=CONF.cinder.ca_file or True) else: session = self.get_insecure_session() self._set_session(SESSION_TYPE_CINDER, session) return session def get_keystone_session(self): session = self._sessions.get(SESSION_TYPE_KEYSTONE) if not session: if not CONF.keystone.api_insecure: session = keystone.Session( verify=CONF.keystone.ca_file or True) else: session = self.get_insecure_session() self._set_session(SESSION_TYPE_KEYSTONE, session) return session def get_neutron_session(self): session = self._sessions.get(SESSION_TYPE_NEUTRON) if not session: if not CONF.neutron.api_insecure: session = keystone.Session( verify=CONF.neutron.ca_file or True) else: session = self.get_insecure_session() self._set_session(SESSION_TYPE_NEUTRON, session) return session def get_nova_session(self): session = self._sessions.get(SESSION_TYPE_NOVA) if not session: if not CONF.nova.api_insecure: session = keystone.Session( verify=CONF.nova.ca_file or True) else: session = self.get_insecure_session() self._set_session(SESSION_TYPE_NOVA, session) return session def get_glance_session(self): session = self._sessions.get(SESSION_TYPE_GLANCE) if not session: if not CONF.glance.api_insecure: session = keystone.Session(verify=CONF.glance.ca_file or True) else: session = self.get_insecure_session() self._set_session(SESSION_TYPE_GLANCE, session) return session def get_heat_session(self): session = self._sessions.get(SESSION_TYPE_HEAT) if not session: if not CONF.heat.api_insecure: session = keystone.Session(verify=CONF.heat.ca_file or True) else: session = self.get_insecure_session() self._set_session(SESSION_TYPE_HEAT, session) return session def token_for_auth(self, auth): return self.get_keystone_session().get_auth_headers(auth).get( 'X-Auth-Token') sahara-8.0.0/sahara/service/ntp_service.py0000666000175100017510000000717613245514472020620 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging from sahara import conductor as cond from sahara import context from sahara.plugins import provisioning as common_configs from sahara.utils import cluster as c_u CONF = cfg.CONF LOG = logging.getLogger(__name__) conductor = cond.API ntp_opts = [ cfg.StrOpt('default_ntp_server', default="pool.ntp.org", help="Default ntp server for time sync") ] CONF.register_opts(ntp_opts) def _sudo(remote, cmd): remote.execute_command(cmd, run_as_root=True) def _restart_ntp(remote): distrib = remote.get_os_distrib() cmd = "service %s restart" if distrib == 'ubuntu': cmd = cmd % "ntp" else: cmd = cmd % "ntpd" _sudo(remote, cmd) def _verify_installation(remote): distrib = remote.get_os_distrib() if distrib == 'ubuntu': return remote.execute_command("dpkg -s ntp") else: return remote.execute_command("rpm -q ntp") def _check_ntp_installed(remote): try: exit_code, stdout = _verify_installation(remote) if exit_code != 0: return False return True except Exception: return False def _configure_ntp_on_instance(instance, url): with context.set_current_instance_id(instance.instance_id): LOG.debug("Configuring ntp server") with instance.remote() as r: if not _check_ntp_installed(r): # missing ntp service LOG.warning("Unable to configure NTP service") return r.append_to_file( "/etc/ntp.conf", "server {url}".format(url=url), run_as_root=True) _restart_ntp(r) try: _sudo(r, "ntpdate -u {url}".format(url=url)) except Exception as e: LOG.debug("Update time on VM failed with error: %s", e) LOG.info("NTP successfully configured") def is_ntp_enabled(cluster): target = common_configs.NTP_ENABLED.applicable_target name = common_configs.NTP_ENABLED.name cl_configs = cluster.cluster_configs if target not in cl_configs or name not in cl_configs[target]: return common_configs.NTP_ENABLED.default_value return cl_configs[target][name] def retrieve_ntp_server_url(cluster): target = common_configs.NTP_URL.applicable_target name = common_configs.NTP_URL.name cl_configs = cluster.cluster_configs if target not in cl_configs or name not in cl_configs[target]: return CONF.default_ntp_server return cl_configs[target][name] def configure_ntp(cluster_id, instance_ids=None): cluster = conductor.cluster_get(context.ctx(), cluster_id) if not is_ntp_enabled(cluster): LOG.debug("Don't configure NTP on cluster") return instances = c_u.get_instances(cluster, instance_ids) url = retrieve_ntp_server_url(cluster) with context.ThreadGroup() as tg: for instance in instances: tg.spawn("configure-ntp-%s" % instance.instance_name, _configure_ntp_on_instance, instance, url) sahara-8.0.0/sahara/service/trusts.py0000666000175100017510000001471113245514472017634 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging from oslo_serialization import jsonutils as json import six from sahara import conductor as c from sahara import context from sahara import exceptions as ex from sahara.i18n import _ from sahara.utils.openstack import keystone conductor = c.API CONF = cfg.CONF LOG = logging.getLogger(__name__) def create_trust(trustor, trustee, role_names, impersonation=True, project_id=None, allow_redelegation=False): '''Create a trust and return it's identifier :param trustor: The user delegating the trust, this is an auth plugin. :param trustee: The user consuming the trust, this is an auth plugin. :param role_names: A list of role names to be assigned. :param impersonation: Should the trustee impersonate trustor, default is True. :param project_id: The project that the trust will be scoped into, default is the trustor's project id. :param allow_redelegation: Allow redelegation parameter for cluster trusts. :returns: A valid trust id. :raises CreationFailed: If the trust cannot be created. ''' if project_id is None: project_id = keystone.project_id_from_auth(trustor) try: trustor_user_id = keystone.user_id_from_auth(trustor) trustee_user_id = keystone.user_id_from_auth(trustee) client = keystone.client_from_auth(trustor) trust = client.trusts.create(trustor_user=trustor_user_id, trustee_user=trustee_user_id, impersonation=impersonation, role_names=role_names, project=project_id, allow_redelegation=allow_redelegation) LOG.debug('Created trust {trust_id}'.format( trust_id=six.text_type(trust.id))) return trust.id except Exception as e: LOG.error('Unable to create trust (reason: {reason})'.format(reason=e)) raise ex.CreationFailed(_('Failed to create trust')) def create_trust_for_cluster(cluster, expires=True): '''Create a trust for a cluster This delegates a trust from the current user to the Sahara admin user based on the current context roles, and then adds the trust identifier to the cluster object. :param expires: The trust will expire if this is set to True. ''' ctx = context.current() cluster = conductor.cluster_get(ctx, cluster) if CONF.use_identity_api_v3 and not cluster.trust_id: trustor = keystone.auth() trustee = keystone.auth_for_admin( project_name=CONF.trustee.project_name) trust_id = create_trust(trustor=trustor, trustee=trustee, role_names=ctx.roles, allow_redelegation=True) conductor.cluster_update(ctx, cluster, {'trust_id': trust_id}) def delete_trust(trustee, trust_id): '''Delete a trust from a trustee :param trustee: The user to delete the trust from, this is an auth plugin. :param trust_id: The identifier of the trust to delete. :raises DeletionFailed: If the trust cannot be deleted. ''' try: client = keystone.client_from_auth(trustee) client.trusts.delete(trust_id) LOG.debug('Deleted trust {trust_id}'.format( trust_id=six.text_type(trust_id))) except Exception as e: LOG.error('Unable to delete trust (reason: {reason})'.format(reason=e)) raise ex.DeletionFailed( _('Failed to delete trust {0}').format(trust_id)) def delete_trust_from_cluster(cluster): '''Delete a trust from a cluster If the cluster has a trust delegated to it, then delete it and set the trust id to None. :param cluster: The cluster to delete the trust from. ''' ctx = context.current() cluster = conductor.cluster_get(ctx, cluster) if CONF.use_identity_api_v3 and cluster.trust_id: keystone_auth = keystone.auth_for_admin(trust_id=cluster.trust_id) delete_trust(keystone_auth, cluster.trust_id) conductor.cluster_update(ctx, cluster, {'trust_id': None}) def use_os_admin_auth_token(cluster): '''Set the current context to the admin user's trust scoped token This will configure the current context to the admin user's identity with the cluster's tenant. It will also generate an authentication token based on the admin user and a delegated trust associated with the cluster. :param cluster: The cluster to use for tenant and trust identification. ''' ctx = context.current() cluster = conductor.cluster_get(ctx, cluster) if CONF.use_identity_api_v3 and cluster.trust_id: ctx.username = CONF.trustee.username ctx.tenant_id = cluster.tenant_id ctx.auth_plugin = keystone.auth_for_admin( trust_id=cluster.trust_id) ctx.auth_token = context.get_auth_token() ctx.service_catalog = json.dumps( keystone.service_catalog_from_auth(ctx.auth_plugin)) def get_os_admin_auth_plugin(cluster): '''Return an admin auth plugin based on the cluster trust id or project If a trust id is available for the cluster, then it is used to create an auth plugin scoped to the trust. If not, the project name from the current context is used to scope the auth plugin. :param cluster: The id of the cluster to use for trust identification. ''' ctx = context.current() cluster = conductor.cluster_get(ctx, cluster) if CONF.use_identity_api_v3 and cluster.trust_id: return keystone.auth_for_admin(trust_id=cluster.trust_id) return keystone.auth_for_admin(project_name=ctx.tenant_name) sahara-8.0.0/sahara/service/__init__.py0000666000175100017510000000000013245514472020011 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/periodic.py0000666000175100017510000002447113245514472020072 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools import random from oslo_config import cfg from oslo_log import log from oslo_service import periodic_task from oslo_service import threadgroup from oslo_utils import timeutils import six from sahara import conductor as c from sahara import context from sahara.service.api import v10 as api from sahara.service import coordinator from sahara.service.edp import job_manager from sahara.service.health import verification_base as vb from sahara.service import trusts from sahara.utils import cluster as c_u from sahara.utils import edp from sahara.utils import proxy as p LOG = log.getLogger(__name__) periodic_opts = [ cfg.BoolOpt('periodic_enable', default=True, help='Enable periodic tasks.'), cfg.IntOpt('periodic_fuzzy_delay', default=60, help='Range in seconds to randomly delay when starting the' ' periodic task scheduler to reduce stampeding.' ' (Disable by setting to 0).'), cfg.IntOpt('periodic_interval_max', default=60, help='Max interval size between periodic tasks execution in ' 'seconds.'), cfg.IntOpt('min_transient_cluster_active_time', default=30, help='Minimal "lifetime" in seconds for a transient cluster. ' 'Cluster is guaranteed to be "alive" within this time ' 'period.'), cfg.IntOpt('cleanup_time_for_incomplete_clusters', default=0, help='Maximal time (in hours) for clusters allowed to be in ' 'states other than "Active", "Deleting" or "Error". If a ' 'cluster is not in "Active", "Deleting" or "Error" state ' 'and last update of it was longer than ' '"cleanup_time_for_incomplete_clusters" hours ago then it ' 'will be deleted automatically. (0 value means that ' 'automatic clean up is disabled).'), cfg.StrOpt('periodic_coordinator_backend_url', help='The backend URL to use for distributed periodic tasks ' 'coordination.'), cfg.IntOpt('periodic_workers_number', default=1, help='Number of threads to run periodic tasks.'), ] CONF = cfg.CONF CONF.register_opts(periodic_opts) conductor = c.API def get_time_since_last_update(cluster): cluster_updated_at = timeutils.normalize_time( timeutils.parse_isotime(cluster.updated_at)) current_time = timeutils.utcnow() spacing = timeutils.delta_seconds(cluster_updated_at, current_time) return spacing def terminate_cluster(ctx, cluster, description): if CONF.use_identity_api_v3 and cluster.trust_id: trusts.use_os_admin_auth_token(cluster) context.set_current_cluster_id(cluster.id) LOG.debug('Terminating {description} cluster ' 'in "{status}" state'.format(status=cluster.status, description=description)) try: api.terminate_cluster(cluster.id) except Exception as e: LOG.warning( 'Failed to terminate {description} cluster in "{status}" ' 'state: {error}.'.format(error=six.text_type(e), status=cluster.status, description=description)) else: if (cluster.status != c_u.CLUSTER_STATUS_AWAITINGTERMINATION): conductor.cluster_update( ctx, cluster, {'status': c_u.CLUSTER_STATUS_AWAITINGTERMINATION}) def set_context(func): @functools.wraps(func) def handler(self, ctx): ctx = context.get_admin_context() context.set_ctx(ctx) func(self, ctx) context.set_ctx(None) return handler def _make_periodic_tasks(): '''Return the periodic tasks object This function creates the periodic tasks class object, it is wrapped in this manner to allow easier control of enabling and disabling tasks. ''' zombie_task_spacing = 300 if CONF.use_domain_for_proxy_users else -1 heartbeat_interval = (CONF.coordinator_heartbeat_interval if CONF.periodic_coordinator_backend_url else -1) class SaharaPeriodicTasks(periodic_task.PeriodicTasks): hr = coordinator.HashRing( CONF.periodic_coordinator_backend_url, 'sahara-periodic-tasks') def __init__(self): super(SaharaPeriodicTasks, self).__init__(CONF) @periodic_task.periodic_task( spacing=heartbeat_interval, run_immediately=True) @set_context def heartbeat(self, ctx): self.hr.heartbeat() @periodic_task.periodic_task(spacing=45) @set_context def update_job_statuses(self, ctx): LOG.debug('Updating job statuses') all_je = conductor.job_execution_get_all(ctx, end_time=None) je_to_manage = self.hr.get_subset(all_je) for job in je_to_manage: job_manager.update_job_status(job.id) @periodic_task.periodic_task(spacing=90) @set_context def terminate_unneeded_transient_clusters(self, ctx): LOG.debug('Terminating unneeded transient clusters') all_clusters = conductor.cluster_get_all( ctx, status=c_u.CLUSTER_STATUS_ACTIVE, is_transient=True) clusters_to_manage = self.hr.get_subset(all_clusters) for cluster in clusters_to_manage: jc = conductor.job_execution_count(ctx, end_time=None, cluster_id=cluster.id) if jc > 0: continue spacing = get_time_since_last_update(cluster) if spacing < CONF.min_transient_cluster_active_time: continue terminate_cluster(ctx, cluster, description='transient') # Add event log info cleanup context.ctx().current_instance_info = context.InstanceInfo() @periodic_task.periodic_task(spacing=zombie_task_spacing) @set_context def check_for_zombie_proxy_users(self, ctx): all_users = p.proxy_domain_users_list() users_to_manage = self.hr.get_subset(all_users) for user in users_to_manage: if user.name.startswith('job_'): je_id = user.name[4:] je = conductor.job_execution_get(ctx, je_id) if je is None or (je.info['status'] in edp.JOB_STATUSES_TERMINATED): LOG.debug('Found zombie proxy user {username}'.format( username=user.name)) p.proxy_user_delete(user_id=user.id) @periodic_task.periodic_task(spacing=3600) @set_context def terminate_incomplete_clusters(self, ctx): if CONF.cleanup_time_for_incomplete_clusters <= 0: return LOG.debug('Terminating old clusters in non-final state') # NOTE(alazarev) Retrieving all clusters once in hour for now. # Criteria support need to be implemented in sahara db API to # have SQL filtering. all_clusters = [ cluster for cluster in conductor.cluster_get_all(ctx) if (cluster.status not in [ c_u.CLUSTER_STATUS_ACTIVE, c_u.CLUSTER_STATUS_ERROR, c_u.CLUSTER_STATUS_DELETING]) ] clusters_to_manage = self.hr.get_subset(all_clusters) for cluster in clusters_to_manage: spacing = get_time_since_last_update(cluster) if spacing < CONF.cleanup_time_for_incomplete_clusters * 3600: continue terminate_cluster(ctx, cluster, description='incomplete') # Add event log info cleanup context.ctx().current_instance_info = context.InstanceInfo() @periodic_task.periodic_task( spacing=vb.get_verification_periodic_interval()) @set_context def run_verifications(self, ctx): LOG.debug("Executing health checks for the clusters") start_dict = {'verification': {'status': 'START'}} all_clusters = conductor.cluster_get_all( ctx, status=c_u.CLUSTER_STATUS_ACTIVE) clusters_to_manage = self.hr.get_subset(all_clusters) for cluster in clusters_to_manage: try: vb.validate_verification_start(cluster) api.update_cluster(cluster.id, start_dict) except vb.CannotVerifyError: LOG.debug("Skipping running verification " "on the cluster %s", cluster.name) return SaharaPeriodicTasks() def setup(): if CONF.periodic_enable: if CONF.periodic_fuzzy_delay: initial_delay = random.randint(0, CONF.periodic_fuzzy_delay) LOG.debug("Starting periodic tasks with initial delay {seconds} " "seconds".format(seconds=initial_delay)) else: initial_delay = None tg = threadgroup.ThreadGroup() workers_number = (CONF.periodic_workers_number if CONF.periodic_coordinator_backend_url else 1) for t in range(workers_number): pt = _make_periodic_tasks() tg.add_dynamic_timer( pt.run_periodic_tasks, initial_delay=initial_delay, periodic_interval_max=CONF.periodic_interval_max, context=None) sahara-8.0.0/sahara/service/heat/0000775000175100017510000000000013245515027016626 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/heat/templates.py0000666000175100017510000005523713245514472021217 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from oslo_config import cfg from oslo_log import log as logging from oslo_serialization import jsonutils as json import six import yaml from sahara.plugins import provisioning as plugin_provisioning from sahara.service.heat import commons as heat_common from sahara.utils import cluster as cl from sahara.utils import general as g from sahara.utils.openstack import base as b from sahara.utils.openstack import heat as h from sahara.utils.openstack import neutron CONF = cfg.CONF LOG = logging.getLogger(__name__) SSH_PORT = 22 INSTANCE_RESOURCE_NAME = "inst" SERVER_GROUP_NAMES = "servgroups" AUTO_SECURITY_GROUP_PARAM_NAME = "autosecgroup" INTERNAL_DESIGNATE_REC = "internal_designate_record" INTERNAL_DESIGNATE_REV_REC = "internal_designate_reverse_record" EXTERNAL_DESIGNATE_REC = "external_designate_record" EXTERNAL_DESIGNATE_REV_REC = "external_designate_reverse_record" # TODO(vgridnev): Using insecure flag until correct way to pass certificate # will be invented WAIT_CONDITION_SCRIPT_TEMPLATE = ''' while true; do wc_notify --insecure --data-binary '{"status": "SUCCESS"}' if [ $? -eq 0 ]; then break fi sleep 10 done ''' heat_engine_opts = [ cfg.BoolOpt( 'heat_enable_wait_condition', default=True, help="Enable wait condition feature to reduce polling during cluster " "creation") ] CONF.register_opts(heat_engine_opts) def _get_inst_name(ng): return { "list_join": [ '-', [ng.cluster.name.lower(), ng.name.lower(), {"get_param": "instance_index"}] ] } def _get_inst_domain_name(domain): return { "list_join": [ '.', [{"get_attr": [INSTANCE_RESOURCE_NAME, "name"]}, domain]] } def _get_aa_group_name(cluster, server_group_index): return g.generate_aa_group_name(cluster.name, server_group_index) def _get_port_name(ng): return { "list_join": [ '-', [ng.cluster.name.lower(), ng.name.lower(), {"get_param": "instance_index"}, "port"] ] } def _get_floating_name(ng): return { "list_join": [ '-', [ng.cluster.name.lower(), ng.name.lower(), {"get_param": "instance_index"}, "floating"] ] } def _get_floating_assoc_name(ng): return { "list_join": [ '-', [ng.cluster.name.lower(), ng.name.lower(), {"get_param": "instance_index"}, "floating", "assoc"] ] } def _get_volume_name(ng): return { "list_join": [ '-', [ng.cluster.name.lower(), ng.name.lower(), {"get_param": "instance_index"}, "volume", {"get_param": "volume_index"}] ] } def _get_wc_handle_name(inst_name): return '%s-wc-handle' % inst_name def _get_wc_waiter_name(inst_name): return '%s-wc-waiter' % inst_name def _get_index_from_inst_name(inst_name): return inst_name.split('-')[-1] class ClusterStack(object): def __init__(self, cluster): self.cluster = cluster self.node_groups_extra = {} self.files = {} self.last_updated_time = None self.base_info = ( "Data Processing Cluster by Sahara\n" "Sahara cluster name: {cluster}\n" "Sahara engine: {version}".format( cluster=cluster.name, version=heat_common.HEAT_ENGINE_VERSION) ) self._current_sg_index = 1 def _node_group_description(self, ng): return "{info}\nNode group {node_group}".format( info=self.base_info, node_group=ng.name) def _asg_for_node_group_description(self, ng): return ("{info}\n" "Auto security group for Sahara Node Group: " "{node_group}".format(info=self.base_info, node_group=ng.name)) def _volume_for_node_group_description(self, ng): return ("{info}\n" "Volume for Sahara Node Group {node_group}".format( node_group=ng.name, info=self.base_info)) def add_node_group_extra(self, node_group_id, node_count, gen_userdata_func, instances_to_delete=None): self.node_groups_extra[node_group_id] = { 'node_count': node_count, 'gen_userdata_func': gen_userdata_func, 'instances_to_delete': instances_to_delete } def _get_main_template(self, instances_to_delete=None): outputs = {} resources = self._serialize_resources(outputs, instances_to_delete) return yaml.safe_dump({ "heat_template_version": heat_common.HEAT_TEMPLATE_VERSION, "description": self.base_info, "resources": resources, "outputs": outputs }) def instantiate(self, update_existing, disable_rollback=True, instances_to_delete=None): main_tmpl = self._get_main_template(instances_to_delete) kwargs = { 'stack_name': self.cluster.stack_name, 'timeout_mins': 180, 'disable_rollback': disable_rollback, 'parameters': {}, 'template': main_tmpl, 'files': self.files } if CONF.heat_stack_tags: kwargs['tags'] = ",".join(CONF.heat_stack_tags) log_kwargs = copy.deepcopy(kwargs) log_kwargs['template'] = yaml.safe_load(log_kwargs['template']) for filename in log_kwargs['files'].keys(): log_kwargs['files'][filename] = yaml.safe_load( log_kwargs['files'][filename]) log_kwargs = json.dumps(log_kwargs) if not update_existing: LOG.debug("Creating Heat stack with args: \n{args}" .format(args=log_kwargs)) b.execute_with_retries(h.client().stacks.create, **kwargs) else: stack = h.get_stack(self.cluster.stack_name) self.last_updated_time = stack.updated_time LOG.debug("Updating Heat stack {stack} with args: \n" "{args}".format(stack=stack, args=log_kwargs)) b.execute_with_retries(stack.update, **kwargs) def _get_server_group_name(self): index = self._current_sg_index # computing server group index in round robin fashion if index < self.cluster.anti_affinity_ratio: self._current_sg_index = (index + 1) else: self._current_sg_index = 1 return _get_aa_group_name(self.cluster, self._current_sg_index) def _need_aa_server_group(self, node_group): for node_process in node_group.node_processes: if node_process in self.cluster.anti_affinity: return True return False def _get_anti_affinity_scheduler_hints(self, node_group): if not self._need_aa_server_group(node_group): return {} return { "scheduler_hints": { "group": { "get_param": [SERVER_GROUP_NAMES, {"get_param": "instance_index"}] } } } def _serialize_resources(self, outputs, instances_to_delete=None): resources = {} if self.cluster.anti_affinity: # Creating server groups equal to the anti_affinity_ratio for i in range(1, self.cluster.anti_affinity_ratio): resources.update(self._serialize_aa_server_group(i)) for ng in self.cluster.node_groups: resources.update(self._serialize_ng_group(ng, outputs, instances_to_delete)) for ng in self.cluster.node_groups: resources.update(self._serialize_auto_security_group(ng)) return resources def _serialize_ng_group(self, ng, outputs, instances_to_delete=None): ng_file_name = "file://" + ng.name + ".yaml" self.files[ng_file_name] = self._serialize_ng_file(ng) outputs[ng.name + "-instances"] = { "value": {"get_attr": [ng.name, "instance"]}} properties = {"instance_index": "%index%"} if ng.cluster.anti_affinity: ng_count = ng.count # assuming instance_index also start from index 0 for i in range(0, ng_count - 1): server_group_name = self._get_server_group_name() server_group_resource = { "get_resource": server_group_name } properties[SERVER_GROUP_NAMES].insert(i, server_group_resource) if ng.auto_security_group: properties[AUTO_SECURITY_GROUP_PARAM_NAME] = { 'get_resource': g.generate_auto_security_group_name(ng)} removal_policies = [] if self.node_groups_extra[ng.id]['instances_to_delete']: resource_list = [] for name in self.node_groups_extra[ng.id]['instances_to_delete']: resource_list.append(_get_index_from_inst_name(name)) removal_policies.append({'resource_list': resource_list}) return { ng.name: { "type": "OS::Heat::ResourceGroup", "properties": { "count": self.node_groups_extra[ng.id]['node_count'], "removal_policies": removal_policies, "resource_def": { "type": ng_file_name, "properties": properties } } } } def _serialize_ng_file(self, ng): parameters = {"instance_index": {"type": "string"}} if ng.cluster.anti_affinity: parameters[SERVER_GROUP_NAMES] = {"type": "comma_delimited_list", "default": []} if ng.auto_security_group: parameters[AUTO_SECURITY_GROUP_PARAM_NAME] = {'type': "string"} return yaml.safe_dump({ "heat_template_version": heat_common.HEAT_TEMPLATE_VERSION, "description": self._node_group_description(ng), "parameters": parameters, "resources": self._serialize_instance(ng), "outputs": { "instance": {"value": { "physical_id": {"get_resource": INSTANCE_RESOURCE_NAME}, "name": {"get_attr": [INSTANCE_RESOURCE_NAME, "name"]} }}} }) def _serialize_auto_security_group(self, ng): if not ng.auto_security_group: return {} security_group_name = g.generate_auto_security_group_name(ng) security_group_description = self._asg_for_node_group_description(ng) res_type = "OS::Neutron::SecurityGroup" desc_key = "description" rules_key = "rules" create_rule = lambda ip_version, cidr, proto, from_port, to_port: { "ethertype": "IPv{}".format(ip_version), "remote_ip_prefix": cidr, "protocol": proto, "port_range_min": six.text_type(from_port), "port_range_max": six.text_type(to_port)} rules = self._serialize_auto_security_group_rules(ng, create_rule) return { security_group_name: { "type": res_type, "properties": { desc_key: security_group_description, rules_key: rules } } } def _serialize_auto_security_group_rules(self, ng, create_rule): rules = [] for port in ng.open_ports: rules.append(create_rule(4, '0.0.0.0/0', 'tcp', port, port)) rules.append(create_rule(6, '::/0', 'tcp', port, port)) rules.append(create_rule(4, '0.0.0.0/0', 'tcp', SSH_PORT, SSH_PORT)) rules.append(create_rule(6, '::/0', 'tcp', SSH_PORT, SSH_PORT)) # open all traffic for private networks for cidr in neutron.get_private_network_cidrs(ng.cluster): ip_ver = 6 if ':' in cidr else 4 for protocol in ['tcp', 'udp']: rules.append(create_rule(ip_ver, cidr, protocol, 1, 65535)) rules.append(create_rule(ip_ver, cidr, 'icmp', 0, 255)) return rules @staticmethod def _get_wait_condition_timeout(ng): configs = ng.cluster.cluster_configs timeout_cfg = plugin_provisioning.HEAT_WAIT_CONDITION_TIMEOUT cfg_target = timeout_cfg.applicable_target cfg_name = timeout_cfg.name return int(configs.get(cfg_target, {}).get(cfg_name, timeout_cfg.default_value)) def _serialize_designate_records(self): if not self.cluster.use_designate_feature(): return {} hostname = _get_inst_domain_name(self.cluster.domain_name) return { INTERNAL_DESIGNATE_REC: { 'type': 'OS::Designate::Record', 'properties': { 'name': hostname, 'type': 'A', 'data': {'get_attr': [ INSTANCE_RESOURCE_NAME, 'networks', 'private', 0]}, 'domain': self.cluster.domain_name } }, EXTERNAL_DESIGNATE_REC: { 'type': 'OS::Designate::Record', 'properties': { 'name': hostname, 'type': 'A', 'data': {'get_attr': ['floating_ip', 'ip']}, 'domain': self.cluster.domain_name } } } def _serialize_designate_reverse_records(self): if not self.cluster.use_designate_feature(): return {} def _generate_reversed_ip(ip): return { 'list_join': [ '.', [ {'str_split': ['.', ip, 3]}, {'str_split': ['.', ip, 2]}, {'str_split': ['.', ip, 1]}, {'str_split': ['.', ip, 0]}, 'in-addr.arpa.' ] ] } hostname = _get_inst_domain_name(self.cluster.domain_name) return { INTERNAL_DESIGNATE_REV_REC: { 'type': 'OS::Designate::Record', 'properties': { 'name': _generate_reversed_ip({'get_attr': [ INSTANCE_RESOURCE_NAME, 'networks', 'private', 0]}), 'type': 'PTR', 'data': hostname, 'domain': 'in-addr.arpa.' } }, EXTERNAL_DESIGNATE_REV_REC: { 'type': 'OS::Designate::Record', 'properties': { 'name': _generate_reversed_ip( {'get_attr': ['floating_ip', 'ip']}), 'type': 'PTR', 'data': hostname, 'domain': 'in-addr.arpa.' } } } def _serialize_instance(self, ng): resources = {} properties = {} inst_name = _get_inst_name(ng) private_net = self.cluster.neutron_management_network sec_groups = self._get_security_groups(ng) # Check if cluster contains user key-pair and include it to template. if self.cluster.user_keypair_id: properties["key_name"] = self.cluster.user_keypair_id port_name = _get_port_name(ng) resources.update(self._serialize_port( port_name, private_net, sec_groups)) properties["networks"] = [{"port": {"get_resource": "port"}}] if ng.floating_ip_pool: resources.update(self._serialize_neutron_floating(ng)) gen_userdata_func = self.node_groups_extra[ng.id]['gen_userdata_func'] key_script = gen_userdata_func(ng, inst_name) if CONF.heat_enable_wait_condition: etc_hosts = cl.etc_hosts_entry_for_service('orchestration') if etc_hosts: etc_hosts = "echo '%s' | sudo tee -a /etc/hosts" % etc_hosts tml = [key_script, WAIT_CONDITION_SCRIPT_TEMPLATE] if etc_hosts: tml = [key_script, etc_hosts, WAIT_CONDITION_SCRIPT_TEMPLATE] userdata = { "str_replace": { "template": "\n".join(tml), "params": { "wc_notify": { "get_attr": [ _get_wc_handle_name(ng.name), "curl_cli" ] } } } } else: userdata = key_script if ng.availability_zone: properties["availability_zone"] = ng.availability_zone properties.update(self._get_anti_affinity_scheduler_hints(ng)) properties.update({ "name": inst_name, "flavor": six.text_type(ng.flavor_id), "image": ng.get_image_id(), "admin_user": ng.image_username, "user_data": userdata }) resources.update({ INSTANCE_RESOURCE_NAME: { "type": "OS::Nova::Server", "properties": properties } }) resources.update(self._serialize_designate_records()) resources.update(self._serialize_designate_reverse_records()) resources.update(self._serialize_volume(ng)) resources.update(self._serialize_wait_condition(ng)) return resources def _serialize_wait_condition(self, ng): if not CONF.heat_enable_wait_condition: return {} return { _get_wc_handle_name(ng.name): { "type": "OS::Heat::WaitConditionHandle" }, _get_wc_waiter_name(ng.name): { "type": "OS::Heat::WaitCondition", "depends_on": INSTANCE_RESOURCE_NAME, "properties": { "timeout": self._get_wait_condition_timeout(ng), "handle": {"get_resource": _get_wc_handle_name(ng.name)} } } } def _serialize_neutron_floating(self, ng): return { "floating_ip": { "type": "OS::Neutron::FloatingIP", "properties": { "floating_network_id": ng.floating_ip_pool, "port_id": {"get_resource": "port"} } } } def _serialize_port(self, port_name, fixed_net_id, security_groups): properties = { "network_id": fixed_net_id, "replacement_policy": "AUTO", "name": port_name } if security_groups: properties["security_groups"] = security_groups return { "port": { "type": "OS::Neutron::Port", "properties": properties, } } def _serialize_volume(self, ng): if not ng.volumes_size or not ng.volumes_per_node: return {} volume_file_name = "file://" + ng.name + "-volume.yaml" self.files[volume_file_name] = self._serialize_volume_file(ng) return { ng.name: { "type": "OS::Heat::ResourceGroup", "properties": { "count": ng.volumes_per_node, "resource_def": { "type": volume_file_name, "properties": { "volume_index": "%index%", "instance_index": {"get_param": "instance_index"}, "instance": {"get_resource": INSTANCE_RESOURCE_NAME}} } } } } def _serialize_volume_file(self, ng): volume_name = _get_volume_name(ng) properties = { "name": volume_name, "size": six.text_type(ng.volumes_size) } if ng.volume_type: properties["volume_type"] = ng.volume_type if ng.volumes_availability_zone: properties["availability_zone"] = ng.volumes_availability_zone if ng.volume_local_to_instance: properties["scheduler_hints"] = { "local_to_instance": {"get_param": "instance"}} return yaml.safe_dump({ "heat_template_version": heat_common.HEAT_TEMPLATE_VERSION, "description": self._volume_for_node_group_description(ng), "parameters": { "volume_index": { "type": "string" }, "instance_index": { "type": "string" }, "instance": { "type": "string" }}, "resources": { "volume": { "type": "OS::Cinder::Volume", "properties": properties }, "volume-attachment": { "type": "OS::Cinder::VolumeAttachment", "properties": { "instance_uuid": {"get_param": "instance"}, "volume_id": {"get_resource": "volume"}, } }}, "outputs": {} }) def _get_security_groups(self, node_group): node_group_sg = list(node_group.security_groups or []) if node_group.auto_security_group: node_group_sg += [ {"get_param": AUTO_SECURITY_GROUP_PARAM_NAME} ] return node_group_sg def _serialize_aa_server_group(self, server_group_index): server_group_name = _get_aa_group_name(self.cluster, server_group_index) return { server_group_name: { "type": "OS::Nova::ServerGroup", "properties": { "name": server_group_name, "policies": ["anti-affinity"] } } } def get_node_group_instances(self, node_group): cluster = node_group.cluster outputs = h.get_stack_outputs(cluster) for output in outputs: if output['output_key'] == node_group.name + "-instances": return output["output_value"] return [] sahara-8.0.0/sahara/service/heat/heat_engine.py0000666000175100017510000002442013245514472021455 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from heatclient import exc as heat_exc from oslo_config import cfg from oslo_log import log as logging from sahara import conductor as c from sahara import context from sahara.i18n import _ from sahara.service import engine as e from sahara.service.heat import commons as heat_common from sahara.service.heat import templates as ht from sahara.service import volumes from sahara.utils import cluster as c_u from sahara.utils import cluster_progress_ops as cpo from sahara.utils.openstack import heat conductor = c.API CONF = cfg.CONF LOG = logging.getLogger(__name__) CREATE_STAGES = [c_u.CLUSTER_STATUS_SPAWNING, c_u.CLUSTER_STATUS_WAITING, c_u.CLUSTER_STATUS_PREPARING] SCALE_STAGES = [c_u.CLUSTER_STATUS_SCALING_SPAWNING, c_u.CLUSTER_STATUS_SCALING_WAITING, c_u.CLUSTER_STATUS_SCALING_PREPARING] ROLLBACK_STAGES = [c_u.CLUSTER_STATUS_ROLLBACK_SPAWNING, c_u.CLUSTER_STATUS_ROLLBACK_WAITING, c_u.CLUSTER_STATUS_ROLLBACK__PREPARING] heat_engine_opts = [ cfg.ListOpt('heat_stack_tags', default=['data-processing-cluster'], help="List of tags to be used during operating with stack.") ] CONF.register_opts(heat_engine_opts) class HeatEngine(e.Engine): def get_type_and_version(self): return heat_common.HEAT_ENGINE_VERSION def create_cluster(self, cluster): self._update_rollback_strategy(cluster, shutdown=True) target_count = self._get_ng_counts(cluster) self._nullify_ng_counts(cluster) cluster = self._generate_heat_stack_name(cluster) self._launch_instances(cluster, target_count, CREATE_STAGES) self._update_rollback_strategy(cluster) @staticmethod def _generate_heat_stack_name(cluster): cluster = conductor.cluster_get(context.ctx(), cluster) hsn = cluster.name + cluster.id[:8] extra = cluster.extra.to_dict() if cluster.extra else {} extra['heat_stack_name'] = hsn conductor.cluster_update(context.ctx(), cluster, {'extra': extra}) return conductor.cluster_get(context.ctx(), cluster) def _get_ng_counts(self, cluster): count = {} for node_group in cluster.node_groups: count[node_group.id] = node_group.count return count def _nullify_ng_counts(self, cluster): ctx = context.ctx() for node_group in cluster.node_groups: conductor.node_group_update(ctx, node_group, {"count": 0}) def scale_cluster(self, cluster, target_count, instances_to_delete=None): ctx = context.ctx() rollback_count = self._get_ng_counts(cluster) self._update_rollback_strategy(cluster, rollback_count=rollback_count, target_count=target_count) inst_ids = self._launch_instances( cluster, target_count, SCALE_STAGES, update_stack=True, disable_rollback=False, instances_to_delete=instances_to_delete) cluster = conductor.cluster_get(ctx, cluster) c_u.clean_cluster_from_empty_ng(cluster) self._update_rollback_strategy(cluster) return inst_ids def rollback_cluster(self, cluster, reason): rollback_info = cluster.rollback_info or {} self._update_rollback_strategy(cluster) if rollback_info.get('shutdown', False): self._rollback_cluster_creation(cluster, reason) LOG.warning("Cluster creation rollback " "(reason: {reason})".format(reason=reason)) return False rollback_count = rollback_info.get('rollback_count', {}).copy() target_count = rollback_info.get('target_count', {}).copy() if rollback_count or target_count: self._rollback_cluster_scaling( cluster, rollback_count, target_count, reason) LOG.warning("Cluster scaling rollback " "(reason: {reason})".format(reason=reason)) return True return False def _update_rollback_strategy(self, cluster, shutdown=False, rollback_count=None, target_count=None): rollback_info = {} if shutdown: rollback_info['shutdown'] = shutdown if rollback_count: rollback_info['rollback_count'] = rollback_count if target_count: rollback_info['target_count'] = target_count cluster = conductor.cluster_update( context.ctx(), cluster, {'rollback_info': rollback_info}) return cluster def _populate_cluster(self, cluster, stack): ctx = context.ctx() old_ids = [i.instance_id for i in c_u.get_instances(cluster)] new_ids = [] for node_group in cluster.node_groups: instances = stack.get_node_group_instances(node_group) for instance in instances: nova_id = instance['physical_id'] if nova_id not in old_ids: name = instance['name'] inst = { "instance_id": nova_id, "instance_name": name } if cluster.use_designate_feature(): inst.update( {"dns_hostname": name + '.' + cluster.domain_name[:-1]}) instance_id = conductor.instance_add(ctx, node_group, inst) new_ids.append(instance_id) return new_ids def _rollback_cluster_creation(self, cluster, ex): """Shutdown all instances and update cluster status.""" self.shutdown_cluster(cluster) def _rollback_cluster_scaling(self, cluster, rollback_count, target_count, ex): """Attempt to rollback cluster scaling. Our rollback policy for scaling is as follows: We shut down nodes created during scaling, but we don't try to to get back decommissioned nodes. I.e. during the rollback we only shut down nodes and not launch them. That approach should maximize the chance of rollback success. """ for ng in rollback_count: if rollback_count[ng] > target_count[ng]: rollback_count[ng] = target_count[ng] self._launch_instances(cluster, rollback_count, ROLLBACK_STAGES, update_stack=True) def shutdown_cluster(self, cluster, force=False): """Shutdown specified cluster and all related resources.""" if force: heat_shutdown = heat.abandon_stack else: heat_shutdown = heat.delete_stack try: heat_shutdown(cluster) except heat_exc.HTTPNotFound: LOG.warning('Did not find stack for cluster.') except heat_exc.BadRequest: LOG.error("Can't force delete cluster.", exc_info=True) finally: self._clean_job_executions(cluster) self._remove_db_objects(cluster) @cpo.event_wrapper( True, step=_('Create Heat stack'), param=('cluster', 1)) def _create_instances(self, cluster, target_count, update_stack=False, disable_rollback=True, instances_to_delete=None): stack = ht.ClusterStack(cluster) self._update_instance_count(stack, cluster, target_count, instances_to_delete) stack.instantiate(update_existing=update_stack, disable_rollback=disable_rollback) heat.wait_stack_completion( cluster, is_update=update_stack, last_updated_time=stack.last_updated_time) return self._populate_cluster(cluster, stack) def _launch_instances(self, cluster, target_count, stages, update_stack=False, disable_rollback=True, instances_to_delete=None): # create all instances cluster = c_u.change_cluster_status(cluster, stages[0]) inst_ids = self._create_instances( cluster, target_count, update_stack, disable_rollback, instances_to_delete) # wait for all instances are up and networks ready cluster = c_u.change_cluster_status(cluster, stages[1]) instances = c_u.get_instances(cluster, inst_ids) self._await_networks(cluster, instances) # prepare all instances cluster = c_u.change_cluster_status(cluster, stages[2]) instances = c_u.get_instances(cluster, inst_ids) volumes.mount_to_instances(instances) self._configure_instances(cluster) return inst_ids def _update_instance_count(self, stack, cluster, target_count, instances_to_delete=None): ctx = context.ctx() instances_name_to_delete = {} if instances_to_delete: for instance in instances_to_delete: node_group_id = instance['node_group']['id'] if node_group_id not in instances_name_to_delete: instances_name_to_delete[node_group_id] = [] instances_name_to_delete[node_group_id].append( instance['instance_name']) for node_group in cluster.node_groups: count = target_count[node_group.id] stack.add_node_group_extra( node_group.id, count, self._generate_user_data_script, instances_name_to_delete.get(node_group.id, None)) for inst in node_group.instances: if (instances_to_delete and node_group.id in instances_name_to_delete): if (inst.instance_name in instances_name_to_delete[node_group.id]): conductor.instance_remove(ctx, inst) sahara-8.0.0/sahara/service/heat/__init__.py0000666000175100017510000000000013245514472020732 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/heat/commons.py0000666000175100017510000000121513245514472020657 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. HEAT_ENGINE_VERSION = 'heat.3.0' HEAT_TEMPLATE_VERSION = '2016-04-08' sahara-8.0.0/sahara/service/edp/0000775000175100017510000000000013245515027016455 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/s3_common.py0000666000175100017510000000572213245514472020737 0ustar zuulzuul00000000000000# Copyright 2017 Massachusetts Open Cloud # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import botocore.exceptions import botocore.session from oslo_config import cfg import six import sahara.exceptions as ex from sahara.i18n import _ from sahara.service.castellan import utils as key_manager S3_JB_PREFIX = "s3://" CONF = cfg.CONF def _get_s3_client(extra): sess = botocore.session.get_session() secretkey = key_manager.get_secret(extra['secretkey']) return sess.create_client( 's3', # TODO(jfreud): investigate region name support region_name=None, # TODO(jfreud): investigate configurable verify verify=False, endpoint_url=extra['endpoint'], aws_access_key_id=extra['accesskey'], aws_secret_access_key=secretkey ) def _get_names_from_job_binary_url(url): parse = six.moves.urllib.parse.urlparse(url) return (parse.netloc + parse.path).split('/', 1) def _get_raw_job_binary_data(job_binary, conn): names = _get_names_from_job_binary_url(job_binary.url) bucket, obj = names try: size = conn.head_object(Bucket=bucket, Key=obj)['ContentLength'] # We have bytes, but want kibibytes: total_KB = size / 1024.0 if total_KB > CONF.job_binary_max_KB: raise ex.DataTooBigException( round(total_KB, 1), CONF.job_binary_max_KB, _("Size of S3 object (%(size)sKB) is greater " "than maximum (%(maximum)sKB)")) body = conn.get_object(Bucket=bucket, Key=obj)['Body'].read() except ex.DataTooBigException: raise except Exception: raise ex.S3ClientException("Couldn't get object from s3") return body def _validate_job_binary_url(job_binary_url): if not job_binary_url.startswith(S3_JB_PREFIX): # Sanity check raise ex.BadJobBinaryException( _("URL for binary in S3 must start with %s") % S3_JB_PREFIX) names = _get_names_from_job_binary_url(job_binary_url) if len(names) == 1: # we have a bucket instead of an individual object raise ex.BadJobBinaryException( _("URL for binary in S3 must specify an object not a bucket")) def get_raw_job_binary_data(job_binary): _validate_job_binary_url(job_binary.url) try: conn = _get_s3_client(job_binary.extra) except Exception: raise ex.S3ClientException("Couldn't create boto client") return _get_raw_job_binary_data(job_binary, conn) sahara-8.0.0/sahara/service/edp/utils/0000775000175100017510000000000013245515027017615 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/utils/shares.py0000666000175100017510000002645213245514472021472 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import collections import itertools from oslo_log import log import six from sahara import context from sahara.utils.openstack import manila LOG = log.getLogger(__name__) def mount_shares(cluster): """Mounts all shares specified for the cluster and any of its node groups. - In the event that a specific share is configured for both the cluster and a specific node group, configuration at the node group level will be ignored. - In the event that utilities required to mount the share are not already installed on the node, this method may fail if the node cannot access the internet. - This method will not remove already-mounted shares. - This method will not remove or remount (or currently, reconfigure) shares already mounted to the desired local mount point. :param cluster: The cluster model. """ node_groups = (ng for ng in cluster.node_groups if ng.shares) ng_mounts = [_mount(ng, share_config) for ng in node_groups for share_config in ng.shares] c_mounts = [_mount(ng, share_config) for ng in cluster.node_groups for share_config in cluster.shares or []] if not (ng_mounts or c_mounts): return ng_mounts_by_share_id = _group_mounts_by_share_id(ng_mounts) c_mounts_by_share_id = _group_mounts_by_share_id(c_mounts) all_share_ids = (set(ng_mounts_by_share_id.keys()) | set(c_mounts_by_share_id.keys())) mounts_by_share_id = { share_id: c_mounts_by_share_id.get(share_id) or ng_mounts_by_share_id[share_id] for share_id in all_share_ids} all_mounts = itertools.chain(*mounts_by_share_id.values()) mounts_by_ng_id = _group_mounts_by_ng_id(all_mounts) client = manila.client() handlers_by_share_id = {id: _ShareHandler.create_from_id(id, client) for id in all_share_ids} for mounts in mounts_by_ng_id.values(): node_group_shares = _NodeGroupShares(mounts[0].node_group) for mount in mounts: share_id = mount.share_config['id'] node_group_shares.add_share(mount.share_config, handlers_by_share_id[share_id]) node_group_shares.mount_shares_to_node_group() def unmount_shares(cluster, unmount_share_list): """Unmounts all shares in unmount_share_list on the given cluster :param cluster: The cluster model. :param unmount_share_list: list of shares to unmount """ client = manila.client() unmount_share_ids = (set(s['id'] for s in unmount_share_list)) handlers_by_share_id = {id: _ShareHandler.create_from_id(id, client) for id in unmount_share_ids} for share in unmount_share_list: for ng in cluster.node_groups: for instance in ng.instances: handlers_by_share_id[share['id']].unmount_from_instance( instance.remote(), share) _mount = collections.namedtuple('Mount', ['node_group', 'share_config']) def _group_mounts(mounts, grouper): result = collections.defaultdict(list) for mount in mounts: result[grouper(mount)].append(mount) return result def _group_mounts_by_share_id(mounts): return _group_mounts(mounts, lambda mount: mount.share_config['id']) def _group_mounts_by_ng_id(mounts): return _group_mounts(mounts, lambda mount: mount.node_group['id']) class _NodeGroupShares(object): """Organizes share mounting for a single node group.""" _share = collections.namedtuple('Share', ['share_config', 'handler']) def __init__(self, node_group): self.node_group = node_group self.shares = [] def add_share(self, share_config, handler): """Adds a share to mount; add all shares before mounting.""" self.shares.append(self._share(share_config, handler)) def mount_shares_to_node_group(self): """Mounts all configured shares to the node group.""" for instance in self.node_group.instances: with context.set_current_instance_id(instance.instance_id): self._mount_shares_to_instance(instance) def _mount_shares_to_instance(self, instance): # Note: Additional iteration here is critical: based on # experimentation, failure to execute allow_access before spawning # the remote results in permission failure. for share in self.shares: share.handler.allow_access_to_instance(instance, share.share_config) with instance.remote() as remote: share_types = set(type(share.handler) for share in self.shares) for share_type in share_types: share_type.setup_instance(remote) for share in self.shares: share.handler.mount_to_instance(remote, share.share_config) @six.add_metaclass(abc.ABCMeta) class _ShareHandler(object): """Handles mounting of a single share to any number of instances.""" @classmethod def setup_instance(cls, remote): """Prepares an instance to mount this type of share.""" pass @classmethod def create_from_id(cls, share_id, client): """Factory method for creation from a share_id of unknown type.""" share = manila.get_share(client, share_id, raise_on_error=True) mounter_class = _share_types[share.share_proto] return mounter_class(share, client) def __init__(self, share, client): self.share = share self.client = client def allow_access_to_instance(self, instance, share_config): """Mounts a specific share to a specific instance.""" access_level = self._get_access_level(share_config) accesses = list(filter(lambda x: (x.access_type == 'ip' and x.access_to == instance.internal_ip), self.share.access_list())) if accesses: access = accesses[0] if access.access_level not in ('ro', 'rw'): LOG.warning("Unknown permission level {access_level} on share " "id {share_id} for ip {ip}. Leaving pre-existing " "permissions.".format( access_level=access.access_level, share_id=self.share.id, ip=instance.internal_ip)) elif access.access_level == 'ro' and access_level == 'rw': self.share.deny(access.id) self.share.allow('ip', instance.internal_ip, access_level) else: self.share.allow('ip', instance.internal_ip, access_level) @abc.abstractmethod def mount_to_instance(self, remote, share_info): """Mounts the share to the instance as configured.""" pass @abc.abstractmethod def unmount_from_instance(self, remote, share_info): """Unmounts the share from the instance.""" pass def _get_access_level(self, share_config): return share_config.get('access_level', 'rw') def _default_mount(self): return '/mnt/{0}'.format(self.share.id) def _get_path(self, share_info): return share_info.get('path', self._default_mount()) class _NFSMounter(_ShareHandler): """Handles mounting of a single NFS share to any number of instances.""" _DEBIAN_INSTALL = "dpkg -s nfs-common || apt-get -y install nfs-common" _REDHAT_INSTALL = "rpm -q nfs-utils || yum install -y nfs-utils" _NFS_CHECKS = { "centos": _REDHAT_INSTALL, "fedora": _REDHAT_INSTALL, "redhatenterpriseserver": _REDHAT_INSTALL, "redhat": _REDHAT_INSTALL, "ubuntu": _DEBIAN_INSTALL } _MKDIR_COMMAND = 'mkdir -p %s' _MOUNT_COMMAND = ("mount | grep '%(remote)s' | grep '%(local)s' | " "grep nfs || mount -t nfs %(access_arg)s %(remote)s " "%(local)s") _UNMOUNT_COMMAND = ("umount -f %s ") _RMDIR_COMMAND = 'rmdir %s' @classmethod def setup_instance(cls, remote): """Prepares an instance to mount this type of share.""" distro = remote.get_os_distrib() if distro in cls._NFS_CHECKS: command = cls._NFS_CHECKS[distro] remote.execute_command(command, run_as_root=True) else: LOG.warning("Cannot verify installation of NFS mount tools for " "unknown distro {distro}.".format(distro=distro)) def mount_to_instance(self, remote, share_info): """Mounts the share to the instance as configured.""" local_path = self._get_path(share_info) access_level = self._get_access_level(share_info) access_arg = '-w' if access_level == 'rw' else '-r' remote.execute_command(self._MKDIR_COMMAND % local_path, run_as_root=True) mount_command = self._MOUNT_COMMAND % { "remote": self.share.export_location, "local": local_path, "access_arg": access_arg} remote.execute_command(mount_command, run_as_root=True) def unmount_from_instance(self, remote, share_info): """Unmounts the share from the instance.""" local_path = self._get_path(share_info) unmount_command = self._UNMOUNT_COMMAND % local_path rmdir_command = self._RMDIR_COMMAND % local_path remote.execute_command(unmount_command, run_as_root=True) remote.execute_command(rmdir_command, run_as_root=True) _share_types = {"NFS": _NFSMounter} SUPPORTED_SHARE_TYPES = _share_types.keys() def make_share_path(mount_point, path): return "{0}{1}".format(mount_point, path) def default_mount(share_id): client = manila.client() return _ShareHandler.create_from_id(share_id, client)._default_mount() def get_share_path(url, shares): # url example: 'manila://ManilaShare-uuid/path_to_file' url = six.moves.urllib.parse.urlparse(url) # using list() as a python2/3 workaround share_list = list(filter(lambda s: s['id'] == url.netloc, shares)) if not share_list: # Share id is not in the share list, let the caller # determine a default path if possible path = None else: # We will always select the first one. Let the # caller determine whether duplicates are okay mount_point = share_list[0].get('path', None) # Do this in two steps instead of passing the default # expression to get(), because it's a big side effect if mount_point is None: # The situation here is that the user specified a # share without a path, so the default mnt was used # during cluster provisioning. mount_point = default_mount(share_list[0]['id']) path = make_share_path(mount_point, url.path) return path sahara-8.0.0/sahara/service/edp/utils/__init__.py0000666000175100017510000000000013245514472021721 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/hdfs_helper.py0000666000175100017510000001074713245514472021330 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils import six from six.moves.urllib import parse as urlparse from sahara import conductor as c from sahara import context from sahara.plugins import exceptions as ex from sahara.plugins import utils as u from sahara.utils import cluster as cluster_utils conductor = c.API HBASE_COMMON_LIB_PATH = "/user/sahara-hbase-lib" def create_hbase_common_lib(r): r.execute_command( 'sudo su - -c "hdfs dfs -mkdir -p %s" hdfs' % ( HBASE_COMMON_LIB_PATH)) ret_code, stdout = r.execute_command( 'hbase classpath') if ret_code == 0: paths = stdout.split(':') for p in paths: if p.endswith(".jar"): r.execute_command('sudo su - -c "hdfs fs -put -p %s %s" hdfs' % (p, HBASE_COMMON_LIB_PATH)) else: raise ex.RequiredServiceMissingException('hbase') def put_file_to_hdfs(r, file, file_name, path, hdfs_user): tmp_file_name = '%s.%s' % (file_name, six.text_type( uuidutils.generate_uuid())) r.write_file_to('/tmp/%s' % tmp_file_name, file) move_from_local(r, '/tmp/%s' % tmp_file_name, path + '/' + file_name, hdfs_user) def copy_from_local(r, source, target, hdfs_user): r.execute_command('sudo su - -c "hdfs dfs -copyFromLocal ' '%s %s" %s' % (source, target, hdfs_user)) def move_from_local(r, source, target, hdfs_user): # using copyFromLocal followed by rm to address permission issues that # arise when image user is not the same as hdfs user (permissions-wise). r.execute_command('sudo su - -c "hdfs dfs -copyFromLocal %(source)s ' '%(target)s" %(user)s && sudo rm -f %(source)s' % {"source": source, "target": target, "user": hdfs_user}) def create_dir_hadoop1(r, dir_name, hdfs_user): r.execute_command( 'sudo su - -c "hdfs dfs -mkdir %s" %s' % (dir_name, hdfs_user)) def create_dir_hadoop2(r, dir_name, hdfs_user): r.execute_command( 'sudo su - -c "hdfs dfs -mkdir -p %s" %s' % (dir_name, hdfs_user)) def _get_cluster_hosts_information(host, cluster): for clust in conductor.cluster_get_all(context.ctx()): if clust.id == cluster.id: continue for i in u.get_instances(clust): if i.instance_name == host: return cluster_utils.generate_etc_hosts(clust) return None def _is_cluster_configured(cluster, host_info): inst = u.get_instances(cluster)[0] cat_etc_hosts = 'cat /etc/hosts' with inst.remote() as r: exit_code, etc_hosts = r.execute_command(cat_etc_hosts) return all(host in etc_hosts for host in host_info) def configure_cluster_for_hdfs(cluster, data_source_url): host = urlparse.urlparse(data_source_url).hostname etc_hosts_information = _get_cluster_hosts_information(host, cluster) if etc_hosts_information is None: # Ip address hasn't been resolved, the last chance is for VM itself return # If the cluster was already configured for this data source # there's no need to configure it again if _is_cluster_configured(cluster, etc_hosts_information.splitlines()): return etc_hosts_update = ('/tmp/etc-hosts-update' '.%s' % six.text_type(uuidutils.generate_uuid())) tmp_etc_hosts = ('/tmp/etc-hosts' '.%s' % six.text_type(uuidutils.generate_uuid())) update_etc_hosts_cmd = ( 'cat %(etc_hosts_update)s /etc/hosts | ' 'sort | uniq > %(tmp_etc_hosts)s && ' 'cat %(tmp_etc_hosts)s > /etc/hosts && ' 'rm -f %(tmp_etc_hosts)s %(etc_hosts_update)s' % {'etc_hosts_update': etc_hosts_update, 'tmp_etc_hosts': tmp_etc_hosts}) for inst in u.get_instances(cluster): with inst.remote() as r: r.write_file_to(etc_hosts_update, etc_hosts_information) r.execute_command(update_etc_hosts_cmd, run_as_root=True) sahara-8.0.0/sahara/service/edp/storm/0000775000175100017510000000000013245515027017621 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/storm/__init__.py0000666000175100017510000000000013245514472021725 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/storm/engine.py0000666000175100017510000003074213245514472021453 0ustar zuulzuul00000000000000# Copyright (c) 2014 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from oslo_config import cfg from oslo_utils import uuidutils from sahara import conductor as c from sahara import context from sahara import exceptions as e from sahara.i18n import _ from sahara.plugins import utils as plugin_utils from sahara.service.edp import base_engine from sahara.service.edp.job_binaries import manager as jb_manager from sahara.service.edp import job_utils from sahara.service.validations.edp import job_execution as j from sahara.utils import cluster as cluster_utils from sahara.utils import edp from sahara.utils import files from sahara.utils import remote conductor = c.API CONF = cfg.CONF class StormJobEngine(base_engine.JobEngine): def __init__(self, cluster): self.cluster = cluster def _get_topology_and_inst_id(self, job_id): try: topology_name, inst_id = job_id.split("@", 1) if topology_name and inst_id: return (topology_name, inst_id) except Exception: pass return "", "" def _get_instance_if_running(self, job_execution): topology_name, inst_id = self._get_topology_and_inst_id( job_execution.engine_job_id) if not topology_name or not inst_id or ( job_execution.info['status'] in edp.JOB_STATUSES_TERMINATED): return None, None # TODO(tmckay): well, if there is a list index out of range # error here it probably means that the instance is gone. If we # have a job execution that is not terminated, and the instance # is gone, we should probably change the status somehow. # For now, do nothing. try: instance = cluster_utils.get_instances(self.cluster, [inst_id])[0] except Exception: instance = None return topology_name, instance def _get_topology_name(self, job_execution): topology_name, inst_id = self._get_topology_and_inst_id( job_execution.engine_job_id) return topology_name def _set_topology_name(self, job_execution, name): return self._generate_topology_name(name) def _generate_topology_name(self, name): return name + "_" + uuidutils.generate_uuid() def _get_job_status_from_remote(self, job_execution, retries=3): topology_name, inst_id = self._get_instance_if_running( job_execution) if topology_name is None or inst_id is None: return edp.JOB_STATUSES_TERMINATED topology_name = self._get_topology_name(job_execution) master = plugin_utils.get_instance(self.cluster, "nimbus") cmd = ( "%(storm)s -c nimbus.host=%(host)s " "list | grep %(topology_name)s | awk '{print $2}'") % ( { "storm": "/usr/local/storm/bin/storm", "host": master.hostname(), "topology_name": topology_name }) for i in range(retries): with remote.get_remote(master) as r: ret, stdout = r.execute_command("%s " % (cmd)) # If the status is ACTIVE is there, it's still running if stdout.strip() == "ACTIVE": return {"status": edp.JOB_STATUS_RUNNING} else: if i == retries - 1: return {"status": edp.JOB_STATUS_KILLED} context.sleep(10) def _job_script(self, python_version): path = "service/edp/resources/launch_command.py" return files.get_file_text(path).replace( '{{PYTHON_VERSION}}', python_version) def _prepare_job_binaries(self, job_binaries, r): for jb in job_binaries: jb_manager.JOB_BINARIES.get_job_binary_by_url(jb.url). \ prepare_cluster(jb, remote=r) def _upload_job_files(self, where, job_dir, job, job_configs): def upload(r, dir, job_file, proxy_configs): path = jb_manager.JOB_BINARIES. \ get_job_binary_by_url(job_file.url). \ copy_binary_to_cluster(job_file, proxy_configs=proxy_configs, remote=r, context=context.ctx()) return path uploaded_paths = [] with remote.get_remote(where) as r: mains = list(job.mains) if job.mains else [] libs = list(job.libs) if job.libs else [] job_binaries = mains + libs self._prepare_job_binaries(job_binaries, r) for job_file in job_binaries: uploaded_paths.append( upload(r, job_dir, job_file, job_configs.get('proxy_configs'))) return uploaded_paths def _exec_cmd_on_remote_instance(self, master, cmd): if master is not None: with remote.get_remote(master) as r: ret, stdout = r.execute_command("%s > /dev/null 2>&1 & echo $!" % cmd) return ret, stdout def cancel_job(self, job_execution): topology_name, instance = self._get_instance_if_running(job_execution) if topology_name is None or instance is None: return None topology_name = self._get_topology_name(job_execution) master = plugin_utils.get_instance(self.cluster, "nimbus") cmd = ( '%(storm_kill)s -c nimbus.host=%(host)s %(topology_name)s') % ( { "storm_kill": "/usr/local/storm/bin/storm kill", "host": master.hostname(), "topology_name": topology_name }) ret, stdout = self._exec_cmd_on_remote_instance(instance, cmd) if ret == 0: # We had some effect, check the status return self._get_job_status_from_remote(job_execution) def get_job_status(self, job_execution): topology_name, instance = self._get_instance_if_running(job_execution) if instance is not None: return self._get_job_status_from_remote(job_execution, retries=3) def _execute_remote_job(self, master, wf_dir, cmd): # If an exception is raised here, the job_manager will mark # the job failed and log the exception # The redirects of stdout and stderr will preserve output in the wf_dir with remote.get_remote(master) as r: # Upload the command launch script launch = os.path.join(wf_dir, "launch_command") python_version = r.get_python_version() r.write_file_to(launch, self._job_script(python_version)) r.execute_command("chmod +x %s" % launch) ret, stdout = r.execute_command( "cd %s; ./launch_command %s > /dev/null 2>&1 & echo $!" % (wf_dir, cmd)) return ret, stdout def _build_command(self, paths, updated_job_configs, host, topology_name): app_jar = paths.pop(0) job_class = updated_job_configs["configs"]["edp.java.main_class"] args = updated_job_configs.get('args', []) args = " ".join([arg for arg in args]) if args: args = " " + args cmd = ( '%(storm_jar)s -c nimbus.host=%(host)s %(job_jar)s ' '%(main_class)s %(topology_name)s%(args)s' % ( { "storm_jar": "/usr/local/storm/bin/storm jar", "main_class": job_class, "job_jar": app_jar, "host": host, "topology_name": topology_name, "args": args })) return cmd def run_job(self, job_execution): ctx = context.ctx() job = conductor.job_get(ctx, job_execution.job_id) # This will be a dictionary of tuples, (native_url, runtime_url) # keyed by data_source id data_source_urls = {} additional_sources, updated_job_configs = ( job_utils.resolve_data_source_references(job_execution.job_configs, job_execution.id, data_source_urls, self.cluster) ) job_execution = conductor.job_execution_update( ctx, job_execution, {"data_source_urls": job_utils.to_url_dict(data_source_urls)}) # Now that we've recorded the native urls, we can switch to the # runtime urls data_source_urls = job_utils.to_url_dict(data_source_urls, runtime=True) job_utils.prepare_cluster_for_ds(additional_sources, self.cluster, updated_job_configs, data_source_urls) # We'll always run the driver program on the master master = plugin_utils.get_instance(self.cluster, "nimbus") # TODO(tmckay): wf_dir should probably be configurable. # The only requirement is that the dir is writable by the image user wf_dir = job_utils.create_workflow_dir(master, '/tmp/storm-edp', job, job_execution.id, "700") paths = self._upload_job_files(master, wf_dir, job, updated_job_configs) topology_name = self._set_topology_name(job_execution, job.name) # Launch the storm job using storm jar host = master.hostname() cmd = self._build_command(paths, updated_job_configs, host, topology_name) job_execution = conductor.job_execution_get(ctx, job_execution.id) if job_execution.info['status'] == edp.JOB_STATUS_TOBEKILLED: return (None, edp.JOB_STATUS_KILLED, None) ret, stdout = self._execute_remote_job(master, wf_dir, cmd) if ret == 0: # Success, we'll add the wf_dir in job_execution.extra and store # topology_name@instance_id as the job id # We know the job is running so return "RUNNING" return (topology_name + "@" + master.id, edp.JOB_STATUS_RUNNING, {'storm-path': wf_dir}) # Hmm, no execption but something failed. # Since we're using backgrounding with redirect, this is unlikely. raise e.EDPError(_("Storm job execution failed. Exit status = " "%(status)s, stdout = %(stdout)s") % {'status': ret, 'stdout': stdout}) def run_scheduled_job(self, job_execution): raise e.NotImplementedException(_("Currently Storm engine does not" " support scheduled EDP jobs")) def validate_job_execution(self, cluster, job, data): j.check_main_class_present(data, job) @staticmethod def get_possible_job_config(job_type): return {'job_config': {'configs': [], 'args': []}} @staticmethod def get_supported_job_types(): return [edp.JOB_TYPE_STORM] class StormPyleusJobEngine(StormJobEngine): def _build_command(self, paths, updated_job_configs, host, topology_name): jar_file = paths.pop(0) cmd = ("{pyleus} -n {nimbus_host} {jar_file}").format( pyleus='pyleus submit', nimbus_host=host, jar_file=jar_file) return cmd def validate_job_execution(self, cluster, job, data): j.check_topology_name_present(data, job) def _set_topology_name(self, job_execution, name): topology_name = job_execution["configs"]["topology_name"] return topology_name def _execute_remote_job(self, master, wf_dir, cmd): with remote.get_remote(master) as r: ret, stdout = r.execute_command( "cd %s; %s > /dev/null 2>&1 & echo $!" % (wf_dir, cmd)) return ret, stdout @staticmethod def get_possible_job_config(job_type): return {'job_config': {'configs': [], 'args': []}} @staticmethod def get_supported_job_types(): return [edp.JOB_TYPE_PYLEUS] sahara-8.0.0/sahara/service/edp/shares.py0000666000175100017510000002645213245514472020332 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import collections import itertools from oslo_log import log import six from sahara import context from sahara.utils.openstack import manila LOG = log.getLogger(__name__) def mount_shares(cluster): """Mounts all shares specified for the cluster and any of its node groups. - In the event that a specific share is configured for both the cluster and a specific node group, configuration at the node group level will be ignored. - In the event that utilities required to mount the share are not already installed on the node, this method may fail if the node cannot access the internet. - This method will not remove already-mounted shares. - This method will not remove or remount (or currently, reconfigure) shares already mounted to the desired local mount point. :param cluster: The cluster model. """ node_groups = (ng for ng in cluster.node_groups if ng.shares) ng_mounts = [_mount(ng, share_config) for ng in node_groups for share_config in ng.shares] c_mounts = [_mount(ng, share_config) for ng in cluster.node_groups for share_config in cluster.shares or []] if not (ng_mounts or c_mounts): return ng_mounts_by_share_id = _group_mounts_by_share_id(ng_mounts) c_mounts_by_share_id = _group_mounts_by_share_id(c_mounts) all_share_ids = (set(ng_mounts_by_share_id.keys()) | set(c_mounts_by_share_id.keys())) mounts_by_share_id = { share_id: c_mounts_by_share_id.get(share_id) or ng_mounts_by_share_id[share_id] for share_id in all_share_ids} all_mounts = itertools.chain(*mounts_by_share_id.values()) mounts_by_ng_id = _group_mounts_by_ng_id(all_mounts) client = manila.client() handlers_by_share_id = {id: _ShareHandler.create_from_id(id, client) for id in all_share_ids} for mounts in mounts_by_ng_id.values(): node_group_shares = _NodeGroupShares(mounts[0].node_group) for mount in mounts: share_id = mount.share_config['id'] node_group_shares.add_share(mount.share_config, handlers_by_share_id[share_id]) node_group_shares.mount_shares_to_node_group() def unmount_shares(cluster, unmount_share_list): """Unmounts all shares in unmount_share_list on the given cluster :param cluster: The cluster model. :param unmount_share_list: list of shares to unmount """ client = manila.client() unmount_share_ids = (set(s['id'] for s in unmount_share_list)) handlers_by_share_id = {id: _ShareHandler.create_from_id(id, client) for id in unmount_share_ids} for share in unmount_share_list: for ng in cluster.node_groups: for instance in ng.instances: handlers_by_share_id[share['id']].unmount_from_instance( instance.remote(), share) _mount = collections.namedtuple('Mount', ['node_group', 'share_config']) def _group_mounts(mounts, grouper): result = collections.defaultdict(list) for mount in mounts: result[grouper(mount)].append(mount) return result def _group_mounts_by_share_id(mounts): return _group_mounts(mounts, lambda mount: mount.share_config['id']) def _group_mounts_by_ng_id(mounts): return _group_mounts(mounts, lambda mount: mount.node_group['id']) class _NodeGroupShares(object): """Organizes share mounting for a single node group.""" _share = collections.namedtuple('Share', ['share_config', 'handler']) def __init__(self, node_group): self.node_group = node_group self.shares = [] def add_share(self, share_config, handler): """Adds a share to mount; add all shares before mounting.""" self.shares.append(self._share(share_config, handler)) def mount_shares_to_node_group(self): """Mounts all configured shares to the node group.""" for instance in self.node_group.instances: with context.set_current_instance_id(instance.instance_id): self._mount_shares_to_instance(instance) def _mount_shares_to_instance(self, instance): # Note: Additional iteration here is critical: based on # experimentation, failure to execute allow_access before spawning # the remote results in permission failure. for share in self.shares: share.handler.allow_access_to_instance(instance, share.share_config) with instance.remote() as remote: share_types = set(type(share.handler) for share in self.shares) for share_type in share_types: share_type.setup_instance(remote) for share in self.shares: share.handler.mount_to_instance(remote, share.share_config) @six.add_metaclass(abc.ABCMeta) class _ShareHandler(object): """Handles mounting of a single share to any number of instances.""" @classmethod def setup_instance(cls, remote): """Prepares an instance to mount this type of share.""" pass @classmethod def create_from_id(cls, share_id, client): """Factory method for creation from a share_id of unknown type.""" share = manila.get_share(client, share_id, raise_on_error=True) mounter_class = _share_types[share.share_proto] return mounter_class(share, client) def __init__(self, share, client): self.share = share self.client = client def allow_access_to_instance(self, instance, share_config): """Mounts a specific share to a specific instance.""" access_level = self._get_access_level(share_config) accesses = list(filter(lambda x: (x.access_type == 'ip' and x.access_to == instance.internal_ip), self.share.access_list())) if accesses: access = accesses[0] if access.access_level not in ('ro', 'rw'): LOG.warning("Unknown permission level {access_level} on share " "id {share_id} for ip {ip}. Leaving pre-existing " "permissions.".format( access_level=access.access_level, share_id=self.share.id, ip=instance.internal_ip)) elif access.access_level == 'ro' and access_level == 'rw': self.share.deny(access.id) self.share.allow('ip', instance.internal_ip, access_level) else: self.share.allow('ip', instance.internal_ip, access_level) @abc.abstractmethod def mount_to_instance(self, remote, share_info): """Mounts the share to the instance as configured.""" pass @abc.abstractmethod def unmount_from_instance(self, remote, share_info): """Unmounts the share from the instance.""" pass def _get_access_level(self, share_config): return share_config.get('access_level', 'rw') def _default_mount(self): return '/mnt/{0}'.format(self.share.id) def _get_path(self, share_info): return share_info.get('path', self._default_mount()) class _NFSMounter(_ShareHandler): """Handles mounting of a single NFS share to any number of instances.""" _DEBIAN_INSTALL = "dpkg -s nfs-common || apt-get -y install nfs-common" _REDHAT_INSTALL = "rpm -q nfs-utils || yum install -y nfs-utils" _NFS_CHECKS = { "centos": _REDHAT_INSTALL, "fedora": _REDHAT_INSTALL, "redhatenterpriseserver": _REDHAT_INSTALL, "redhat": _REDHAT_INSTALL, "ubuntu": _DEBIAN_INSTALL } _MKDIR_COMMAND = 'mkdir -p %s' _MOUNT_COMMAND = ("mount | grep '%(remote)s' | grep '%(local)s' | " "grep nfs || mount -t nfs %(access_arg)s %(remote)s " "%(local)s") _UNMOUNT_COMMAND = ("umount -f %s ") _RMDIR_COMMAND = 'rmdir %s' @classmethod def setup_instance(cls, remote): """Prepares an instance to mount this type of share.""" distro = remote.get_os_distrib() if distro in cls._NFS_CHECKS: command = cls._NFS_CHECKS[distro] remote.execute_command(command, run_as_root=True) else: LOG.warning("Cannot verify installation of NFS mount tools for " "unknown distro {distro}.".format(distro=distro)) def mount_to_instance(self, remote, share_info): """Mounts the share to the instance as configured.""" local_path = self._get_path(share_info) access_level = self._get_access_level(share_info) access_arg = '-w' if access_level == 'rw' else '-r' remote.execute_command(self._MKDIR_COMMAND % local_path, run_as_root=True) mount_command = self._MOUNT_COMMAND % { "remote": self.share.export_location, "local": local_path, "access_arg": access_arg} remote.execute_command(mount_command, run_as_root=True) def unmount_from_instance(self, remote, share_info): """Unmounts the share from the instance.""" local_path = self._get_path(share_info) unmount_command = self._UNMOUNT_COMMAND % local_path rmdir_command = self._RMDIR_COMMAND % local_path remote.execute_command(unmount_command, run_as_root=True) remote.execute_command(rmdir_command, run_as_root=True) _share_types = {"NFS": _NFSMounter} SUPPORTED_SHARE_TYPES = _share_types.keys() def make_share_path(mount_point, path): return "{0}{1}".format(mount_point, path) def default_mount(share_id): client = manila.client() return _ShareHandler.create_from_id(share_id, client)._default_mount() def get_share_path(url, shares): # url example: 'manila://ManilaShare-uuid/path_to_file' url = six.moves.urllib.parse.urlparse(url) # using list() as a python2/3 workaround share_list = list(filter(lambda s: s['id'] == url.netloc, shares)) if not share_list: # Share id is not in the share list, let the caller # determine a default path if possible path = None else: # We will always select the first one. Let the # caller determine whether duplicates are okay mount_point = share_list[0].get('path', None) # Do this in two steps instead of passing the default # expression to get(), because it's a big side effect if mount_point is None: # The situation here is that the user specified a # share without a path, so the default mnt was used # during cluster provisioning. mount_point = default_mount(share_list[0]['id']) path = make_share_path(mount_point, url.path) return path sahara-8.0.0/sahara/service/edp/job_binaries/0000775000175100017510000000000013245515027021103 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/opts.py0000666000175100017510000000173113245514472022451 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # File contains job binaries opts to avoid cyclic imports issue from oslo_config import cfg opts = [ cfg.ListOpt('job_binary_types', default=['swift', 'manila', 'internal-db', 's3'], help='List of job binary types to be loaded. Sahara ' 'preserves the order of the list when returning it.'), ] CONF = cfg.CONF CONF.register_opts(opts) sahara-8.0.0/sahara/service/edp/job_binaries/s3/0000775000175100017510000000000013245515027021430 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/s3/implementation.py0000666000175100017510000000345613245514472025044 0ustar zuulzuul00000000000000# Copyright (c) 2017 Massachusetts Open Cloud # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six import six.moves.urllib.parse as urlparse import sahara.exceptions as ex from sahara.i18n import _ from sahara.service.edp.job_binaries.base import JobBinaryType from sahara.service.edp import s3_common class S3Type(JobBinaryType): def copy_binary_to_cluster(self, job_binary, **kwargs): r = kwargs.pop('remote') dst = self._generate_valid_path(job_binary) raw = self.get_raw_data(job_binary) r.write_file_to(dst, raw) return dst def validate_job_location_format(self, url): url = urlparse.urlparse(url) return url.scheme == "s3" and url.hostname def validate(self, data, **kwargs): # We only check on create, not update if not kwargs.get('job_binary_id', None): s3_common._validate_job_binary_url(data['url']) extra = data.get("extra", {}) if (six.viewkeys(extra) != {"accesskey", "secretkey", "endpoint"}): raise ex.InvalidDataException( _("Configs 'accesskey', 'secretkey', and 'endpoint'" " must be provided.")) def get_raw_data(self, job_binary, **kwargs): return s3_common.get_raw_job_binary_data(job_binary) sahara-8.0.0/sahara/service/edp/job_binaries/s3/__init__.py0000666000175100017510000000000013245514472023534 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/manager.py0000666000175100017510000000537513245514472023106 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging import six import six.moves.urllib.parse as urlparse from stevedore import enabled from sahara import conductor as cond from sahara import exceptions as ex from sahara.i18n import _ conductor = cond.API LOG = logging.getLogger(__name__) CONF = cfg.CONF class JobBinaryManager(object): def __init__(self): self.job_binaries = {} self._load_job_binaries() def _load_job_binaries(self): config_jb = CONF.job_binary_types extension_manager = enabled.EnabledExtensionManager( check_func=lambda ext: ext.name in config_jb, namespace='sahara.job_binary.types', invoke_on_load=True ) for ext in extension_manager.extensions: if ext.name in self.job_binaries: raise ex.ConfigurationError( _("Job binary with name '%s' already exists.") % ext.name) ext.obj.name = ext.name self.job_binaries[ext.name] = ext.obj LOG.info("Job binary name {jb_name} loaded {entry_point}".format( jb_name=ext.name, entry_point=ext.entry_point_target)) if len(self.job_binaries) < len(config_jb): loaded_jb = set(six.iterkeys(self.job_binaries)) requested_jb = set(config_jb) raise ex.ConfigurationError( _("Job binaries couldn't be loaded: %s") % ", ".join(requested_jb - loaded_jb)) def get_job_binaries(self): config_jb = CONF.job_binary_types return [self.get_job_binary(name).name for name in config_jb] def get_job_binary(self, name): res = self.job_binaries.get(name) if res is None: raise ex.InvalidDataException (_("Invalid job binary")) return res def get_job_binary_by_url(self, url): url = urlparse.urlparse(url) if not url.scheme: raise ex.InvalidDataException (_("Job binary url must have a scheme")) return self.get_job_binary(url.scheme) JOB_BINARIES = None def setup_job_binaries(): global JOB_BINARIES JOB_BINARIES = JobBinaryManager() sahara-8.0.0/sahara/service/edp/job_binaries/swift/0000775000175100017510000000000013245515027022237 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/swift/implementation.py0000666000175100017510000001111313245514472025640 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six import six.moves.urllib.parse as urlparse import swiftclient import sahara.exceptions as ex from sahara.i18n import _ from sahara.service.castellan import utils as key_manager from sahara.service.edp.job_binaries.base import JobBinaryType from sahara.swift import utils as su from sahara.utils.openstack import swift as sw CONF = cfg.CONF class SwiftType(JobBinaryType): def copy_binary_to_cluster(self, job_binary, **kwargs): r = kwargs.pop('remote') dst = self._generate_valid_path(job_binary) raw = self.get_raw_data(job_binary, **kwargs) r.write_file_to(dst, raw) return dst def validate_job_location_format(self, url): url = urlparse.urlparse(url) return url.scheme == "swift" and url.hostname def validate(self, data, **kwargs): if not kwargs.get('job_binary_id', None): extra = data.get("extra", {}) # Should not be checked during job binary update if (not extra.get("user") or not extra.get("password")) and ( not CONF.use_domain_for_proxy_users): raise ex.BadJobBinaryException() def get_raw_data(self, job_binary, **kwargs): self._validate_job_binary_url(job_binary) proxy_configs = kwargs.pop('proxy_configs', None) with_context = kwargs.pop('with_context', False) if not with_context: conn_kwargs = {} if proxy_configs: conn_kwargs.update(username=proxy_configs.get( 'proxy_username'), password=key_manager.get_secret( proxy_configs.get('proxy_password')), trust_id=proxy_configs.get( 'proxy_trust_id')) else: conn_kwargs.update(username=job_binary.extra.get('user'), password=key_manager.get_secret( job_binary.extra.get('password'))) conn = sw.client(**conn_kwargs) else: conn = sw.client_from_token() raw = self._get_raw_data(job_binary, conn) return raw def _get_names_from_url(self, url): parse = six.moves.urllib.parse.urlparse(url) return (parse.netloc + parse.path).split('/', 1) def _get_raw_data(self, job_binary, conn): names = self._get_names_from_url(job_binary.url) container, obj = names # if container name has '.sahara' suffix we need to strip it if container.endswith(su.SWIFT_URL_SUFFIX): container = container[:-len(su.SWIFT_URL_SUFFIX)] try: # First check the size headers = conn.head_object(container, obj) total_KB = int(headers.get('content-length', 0)) / 1024.0 if total_KB > CONF.job_binary_max_KB: raise ex.DataTooBigException( round(total_KB, 1), CONF.job_binary_max_KB, _("Size of swift object (%(size)sKB) is greater " "than maximum (%(maximum)sKB)")) headers, body = conn.get_object(container, obj) except swiftclient.ClientException as e: raise ex.SwiftClientException(six.text_type(e)) return body def _validate_job_binary_url(self, job_binary): if not (job_binary.url.startswith(su.SWIFT_INTERNAL_PREFIX)): # This should have been guaranteed already, # but we'll check just in case. raise ex.BadJobBinaryException( _("Url for binary in internal swift must start with %s") % su.SWIFT_INTERNAL_PREFIX) names = self._get_names_from_url(job_binary.url) if len(names) == 1: # a container has been requested, this is currently unsupported raise ex.BadJobBinaryException( _('Url for binary in internal swift must specify an ' 'object not a container')) sahara-8.0.0/sahara/service/edp/job_binaries/swift/__init__.py0000666000175100017510000000000013245514472024343 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/base.py0000666000175100017510000000650213245514472022377 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import six from sahara import exceptions as ex from sahara.plugins import base as plugins_base @six.add_metaclass(abc.ABCMeta) class JobBinaryType(object): @plugins_base.required_with_default def prepare_cluster(self, job_binary, **kwargs): """Makes a cluster ready to use this job binary Different implementations for each job binary type for Manila it will be mount the share, for Swift verify credentials, and so on :param job_binary: The object handle to a job binary :returns: None """ pass @plugins_base.required def copy_binary_to_cluster(self, job_binary, **kwargs): """Get the path for the binary in a cluster If necessary, pull binary data from the binary store, and copy that data to a useful path on the cluster. Then returns a valid FS path for the job binary in the cluster :param job_binary: The object handle to a job binary :returns: String representing the local path """ # TODO(mariannelm): currently for the job binaries it's true # that the raw data must be available at a FS path in the cluster, but # for most of the job binary types there's no need to keep this data # in the cluster after the job is done, so it would be a good thing to # have a method responsible for removing the job binary raw data # after the end of the job return None @plugins_base.required_with_default def validate(self, data, **kwargs): """Method that validate the data passed through the API This method will be executed during the job binary creation and update :raise: If data is invalid, InvalidDataException """ pass @plugins_base.optional def _validate_url(self, url): """Auxiliary method used by the validate method""" pass @plugins_base.required_with_default def validate_job_location_format(self, entry): """Checks whether or not the API entry is valid :param entry: String that represents a job binary url :returns: True if this entry is valid, False otherwhise """ return True @plugins_base.required_with_default def get_raw_data(self, job_binary, **kwargs): """Get the raw binary Used only by the API, if the type doesn't support this operation it should raise NotImplementedException :param job_binary: The object handle to a job binary :returns: Raw binary """ raise ex.NotImplementedException() @plugins_base.optional def _generate_valid_path(self, job_binary): """Generates a valid FS path for the binary be placed""" return '/tmp/' + job_binary.name sahara-8.0.0/sahara/service/edp/job_binaries/__init__.py0000666000175100017510000000000013245514472023207 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/manila/0000775000175100017510000000000013245515027022344 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/manila/implementation.py0000666000175100017510000000663313245514472025760 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils import six.moves.urllib.parse as urlparse from sahara import conductor as c from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.edp.job_binaries.base import JobBinaryType from sahara.service.edp import job_utils from sahara.service.edp.utils import shares as shares_service from sahara.utils.openstack import manila as m conductor = c.API class ManilaType(JobBinaryType): def copy_binary_to_cluster(self, job_binary, **kwargs): remote = kwargs.pop('remote') path = self._get_share_path(job_binary, remote) # if path is None, then it was mounted in the default path # by the prepare_cluster method so just construct the # default path and return it if path is None: url = urlparse.urlparse(job_binary.url) share_id = url.netloc mount_point = shares_service.default_mount(share_id) path = shares_service.make_share_path(mount_point, url.path) return path def prepare_cluster(self, job_binary, **kwargs): remote = kwargs.pop('remote') path = self._get_share_path(job_binary, remote) if path is None: path = job_utils.mount_share_at_default_path( job_binary.url, remote.instance.node_group.cluster) def _get_share_path(self, job_binary, remote): shares = [] if remote.instance.node_group.cluster.shares: shares.extend(remote.instance.node_group.cluster.shares) if remote.instance.node_group.shares: shares.extend(remote.instance.node_group.shares) path = shares_service.get_share_path(job_binary.url, shares) return path def validate_job_location_format(self, url): if url.startswith(m.MANILA_PREFIX): url = urlparse.urlparse(url) return (uuidutils.is_uuid_like(url.netloc) and url.path) else: return False def validate(self, data, **kwargs): self._validate_url(data['url']) def _validate_url(self, url): if len(url) == 0: raise ex.InvalidDataException(_("Manila url must not be empty")) url = urlparse.urlparse(url) if url.scheme != "manila": raise ex.InvalidDataException(_("Manila url scheme must be" " 'manila'")) if not uuidutils.is_uuid_like(url.netloc): raise ex.InvalidDataException(_("Manila url netloc must be a" " uuid")) if not url.path: raise ex.InvalidDataException(_("Manila url path must not be" " empty")) def get_raw_data(self, job_binary, **kwargs): raise ex.NotImplementedException('Manila does not implement this ' 'method') sahara-8.0.0/sahara/service/edp/job_binaries/manila/__init__.py0000666000175100017510000000000013245514472024450 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/internal_db/0000775000175100017510000000000013245515027023364 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/job_binaries/internal_db/implementation.py0000666000175100017510000000476313245514472027002 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_utils import uuidutils import six.moves.urllib.parse as urlparse from sahara import conductor as c from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.edp.job_binaries.base import JobBinaryType import sahara.service.validations.edp.base as b CONF = cfg.CONF conductor = c.API class InternalDBType(JobBinaryType): def copy_binary_to_cluster(self, job_binary, **kwargs): # url example: 'internal-db://JobBinaryInternal-UUID' r = kwargs.pop('remote') dst = self._generate_valid_path(job_binary) raw = self.get_raw_data(job_binary, **kwargs) r.write_file_to(dst, raw) return dst def get_raw_data(self, job_binary, **kwargs): context = kwargs.pop('context') # url example: 'internal-db://JobBinaryInternal-UUID' binary_internal_id = job_binary.url[len("internal-db://"):] return conductor.job_binary_internal_get_raw_data(context, binary_internal_id) def validate_job_location_format(self, url): try: self._validate_url(url) except ex.InvalidDataException: return False return True def validate(self, data, **kwargs): self._validate_url(data['url']) internal_uid = data['url'].replace("internal-db://", '') b.check_job_binary_internal_exists(internal_uid) def _validate_url(self, url): if len(url) == 0: raise ex.InvalidDataException( _("Internal data base url must not be empty")) url = urlparse.urlparse(url) if url.scheme != "internal-db": raise ex.InvalidDataException( _("URL scheme must be 'internal-db'")) if not uuidutils.is_uuid_like(url.netloc): raise ex.InvalidDataException( _("Internal data base url netloc must be a uuid")) sahara-8.0.0/sahara/service/edp/job_binaries/internal_db/__init__.py0000666000175100017510000000000013245514472025470 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/base_engine.py0000666000175100017510000000324013245514472021272 0ustar zuulzuul00000000000000# Copyright (c) 2014 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import six from sahara import conductor as c conductor = c.API def optional(fun): fun.__not_implemented__ = True return fun @six.add_metaclass(abc.ABCMeta) class JobEngine(object): @abc.abstractmethod def cancel_job(self, job_execution): pass @abc.abstractmethod def get_job_status(self, job_execution): pass @abc.abstractmethod def run_job(self, job_execution): pass @abc.abstractmethod def run_scheduled_job(self, job_execution): pass @abc.abstractmethod def validate_job_execution(self, cluster, job, data): pass @staticmethod @abc.abstractmethod def get_possible_job_config(job_type): return None @staticmethod @abc.abstractmethod def get_supported_job_types(): return None @optional def suspend_job(self, job_execution): pass def does_engine_implement(self, fun_name): fun = getattr(self, fun_name) if not (fun and callable(fun)): return False return not hasattr(fun, '__not_implemented__') sahara-8.0.0/sahara/service/edp/spark/0000775000175100017510000000000013245515027017575 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/spark/__init__.py0000666000175100017510000000000013245514472021701 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/spark/engine.py0000666000175100017510000004237413245514472021433 0ustar zuulzuul00000000000000# Copyright (c) 2014 OpenStack Foundation # Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os from oslo_config import cfg from oslo_utils import uuidutils from sahara import conductor as c from sahara import context from sahara import exceptions as e from sahara.i18n import _ from sahara.service.castellan import utils as key_manager from sahara.service.edp import base_engine from sahara.service.edp.job_binaries import manager as jb_manager from sahara.service.edp import job_utils from sahara.service.validations.edp import job_execution as j from sahara.swift import swift_helper as sw from sahara.swift import utils as su from sahara.utils import cluster as c_u from sahara.utils import edp from sahara.utils import files from sahara.utils import remote from sahara.utils import xmlutils conductor = c.API CONF = cfg.CONF class SparkJobEngine(base_engine.JobEngine): def __init__(self, cluster): self.cluster = cluster # We'll always run the driver program on the master self.master = None # These parameters depend on engine that is used self.plugin_params = {"master": "", "spark-user": "", "deploy-mode": "", "spark-submit": "", "driver-class-path": "", } def _get_pid_and_inst_id(self, job_id): try: pid, inst_id = job_id.split("@", 1) if pid and inst_id: return (pid, inst_id) except Exception: pass return "", "" def _get_instance_if_running(self, job_execution): pid, inst_id = self._get_pid_and_inst_id(job_execution.engine_job_id) if not pid or not inst_id or ( job_execution.info['status'] in edp.JOB_STATUSES_TERMINATED): return None, None # TODO(tmckay): well, if there is a list index out of range # error here it probably means that the instance is gone. If we # have a job execution that is not terminated, and the instance # is gone, we should probably change the status somehow. # For now, do nothing. try: instance = c_u.get_instances(self.cluster, [inst_id])[0] except Exception: instance = None return pid, instance def _get_result_file(self, r, job_execution): result = os.path.join(job_execution.extra['spark-path'], "result") return r.execute_command("cat %s" % result, raise_when_error=False) def _check_pid(self, r, pid): ret, stdout = r.execute_command("ps hp %s" % pid, raise_when_error=False) return ret def _get_job_status_from_remote(self, r, pid, job_execution): # If the pid is there, it's still running if self._check_pid(r, pid) == 0: return {"status": edp.JOB_STATUS_RUNNING} # The process ended. Look in the result file to get the exit status ret, stdout = self._get_result_file(r, job_execution) if ret == 0: exit_status = stdout.strip() if exit_status == "0": return {"status": edp.JOB_STATUS_SUCCEEDED} # SIGINT will yield either -2 or 130 elif exit_status in ["-2", "130"]: return {"status": edp.JOB_STATUS_KILLED} # Well, process is done and result is missing or unexpected return {"status": edp.JOB_STATUS_DONEWITHERROR} def _job_script(self, python_version): path = "service/edp/resources/launch_command.py" return files.get_file_text(path).replace( '{{PYTHON_VERSION}}', python_version) def _upload_wrapper_xml(self, where, job_dir, job_configs): xml_name = 'spark.xml' proxy_configs = job_configs.get('proxy_configs') configs = {} if proxy_configs: configs[sw.HADOOP_SWIFT_USERNAME] = proxy_configs.get( 'proxy_username') configs[sw.HADOOP_SWIFT_PASSWORD] = key_manager.get_secret( proxy_configs.get('proxy_password')) configs[sw.HADOOP_SWIFT_TRUST_ID] = proxy_configs.get( 'proxy_trust_id') configs[sw.HADOOP_SWIFT_DOMAIN_NAME] = CONF.proxy_user_domain_name else: cfgs = job_configs.get('configs', {}) targets = [sw.HADOOP_SWIFT_USERNAME, sw.HADOOP_SWIFT_PASSWORD] configs = {k: cfgs[k] for k in targets if k in cfgs} content = xmlutils.create_hadoop_xml(configs) with remote.get_remote(where) as r: dst = os.path.join(job_dir, xml_name) r.write_file_to(dst, content) return xml_name def _prepare_job_binaries(self, job_binaries, r): for jb in job_binaries: jb_manager.JOB_BINARIES.get_job_binary_by_url(jb.url). \ prepare_cluster(jb, remote=r) def _upload_job_files(self, where, job_dir, job, job_configs): def upload(r, dir, job_file, proxy_configs): path = jb_manager.JOB_BINARIES. \ get_job_binary_by_url(job_file.url). \ copy_binary_to_cluster(job_file, proxy_configs=proxy_configs, remote=r, context=context.ctx()) return path def upload_builtin(r, dir, builtin): dst = os.path.join(dir, builtin['name']) r.write_file_to(dst, builtin['raw']) return dst builtin_libs = [] if edp.is_adapt_spark_for_swift_enabled( job_configs.get('configs', {})): path = 'service/edp/resources/edp-spark-wrapper.jar' name = 'builtin-%s.jar' % uuidutils.generate_uuid() builtin_libs = [{'raw': files.get_file_text(path), 'name': name}] uploaded_paths = [] builtin_paths = [] with remote.get_remote(where) as r: mains = list(job.mains) if job.mains else [] libs = list(job.libs) if job.libs else [] job_binaries = mains + libs self._prepare_job_binaries(job_binaries, r) for job_file in job_binaries: uploaded_paths.append( upload(r, job_dir, job_file, job_configs.get('proxy_configs'))) for builtin in builtin_libs: builtin_paths.append( upload_builtin(r, job_dir, builtin)) return uploaded_paths, builtin_paths def _check_driver_class_path(self, job_configs, param_dict, wf_dir): overridden = edp.spark_driver_classpath( job_configs.get('configs', {})) if overridden: param_dict['driver-class-path'] = ( " --driver-class-path " + overridden) return if not param_dict.get('wrapper_jar'): # no need in driver classpath if swift as datasource is not used param_dict['driver-class-path'] = "" return cp = param_dict['driver-class-path'] or "" if param_dict['deploy-mode'] == 'client' and not ( cp.startswith(":") or cp.endswith(":")): cp += ":" + wf_dir param_dict['driver-class-path'] = " --driver-class-path " + cp def cancel_job(self, job_execution): pid, instance = self._get_instance_if_running(job_execution) if instance is not None: with remote.get_remote(instance) as r: ret, stdout = r.execute_command("kill -SIGINT %s" % pid, raise_when_error=False) if ret == 0: # We had some effect, check the status return self._get_job_status_from_remote(r, pid, job_execution) def get_job_status(self, job_execution): pid, instance = self._get_instance_if_running(job_execution) if instance is not None: with remote.get_remote(instance) as r: return self._get_job_status_from_remote(r, pid, job_execution) def _build_command(self, wf_dir, paths, builtin_paths, updated_job_configs): indep_params = {} # TODO(tmckay): for now, paths[0] is always assumed to be the app # jar and we generate paths in order (mains, then libs). # When we have a Spark job type, we can require a "main" and set # the app jar explicitly to be "main" indep_params["app_jar"] = paths.pop(0) indep_params["job_class"] = ( updated_job_configs["configs"]["edp.java.main_class"]) if self.plugin_params.get('drivers-to-jars', None): paths.extend(self.plugin_params['drivers-to-jars']) # If we uploaded builtins then we are using a wrapper jar. It will # be the first one on the builtin list and the original app_jar needs # to be added to the 'additional' jars if builtin_paths: indep_params["wrapper_jar"] = builtin_paths.pop(0) indep_params["wrapper_class"] = ( 'org.openstack.sahara.edp.SparkWrapper') wrapper_xml = self._upload_wrapper_xml(self.master, wf_dir, updated_job_configs) indep_params["wrapper_args"] = "%s %s" % ( wrapper_xml, indep_params["job_class"]) indep_params["addnl_files"] = wrapper_xml indep_params["addnl_jars"] = ",".join( [indep_params["wrapper_jar"]] + paths + builtin_paths) else: indep_params["addnl_jars"] = ",".join(paths) # All additional jars are passed with the --jars option if indep_params["addnl_jars"]: indep_params["addnl_jars"] = ( " --jars " + indep_params["addnl_jars"]) # Launch the spark job using spark-submit and deploy_mode = client # TODO(tmckay): we need to clean up wf_dirs on long running clusters # TODO(tmckay): probably allow for general options to spark-submit indep_params["args"] = updated_job_configs.get('args', []) indep_params["args"] = " ".join([su.inject_swift_url_suffix(arg) for arg in indep_params["args"]]) if indep_params.get("args"): indep_params["args"] = (" " + indep_params["args"]) mutual_dict = self.plugin_params.copy() mutual_dict.update(indep_params) # Handle driver classpath. Because of the way the hadoop # configuration is handled in the wrapper class, using # wrapper_xml, the working directory must be on the classpath self._check_driver_class_path(updated_job_configs, mutual_dict, wf_dir) if mutual_dict.get("wrapper_jar"): # Substrings which may be empty have spaces # embedded if they are non-empty cmd = ( '%(spark-user)s%(spark-submit)s%(driver-class-path)s' ' --files %(addnl_files)s' ' --class %(wrapper_class)s%(addnl_jars)s' ' --master %(master)s' ' --deploy-mode %(deploy-mode)s' ' %(app_jar)s %(wrapper_args)s%(args)s') % dict( mutual_dict) else: cmd = ( '%(spark-user)s%(spark-submit)s%(driver-class-path)s' ' --class %(job_class)s%(addnl_jars)s' ' --master %(master)s' ' --deploy-mode %(deploy-mode)s' ' %(app_jar)s%(args)s') % dict( mutual_dict) return cmd def run_job(self, job_execution): ctx = context.ctx() job = conductor.job_get(ctx, job_execution.job_id) # This will be a dictionary of tuples, (native_url, runtime_url) # keyed by data_source id data_source_urls = {} additional_sources, updated_job_configs = ( job_utils.resolve_data_source_references(job_execution.job_configs, job_execution.id, data_source_urls, self.cluster) ) job_execution = conductor.job_execution_update( ctx, job_execution, {"data_source_urls": job_utils.to_url_dict(data_source_urls)}) # Now that we've recorded the native urls, we can switch to the # runtime urls data_source_urls = job_utils.to_url_dict(data_source_urls, runtime=True) job_utils.prepare_cluster_for_ds(additional_sources, self.cluster, updated_job_configs, data_source_urls) # It is needed in case we are working with Spark plugin self.plugin_params['master'] = ( self.plugin_params['master'] % {'host': self.master.hostname()}) # TODO(tmckay): wf_dir should probably be configurable. # The only requirement is that the dir is writable by the image user wf_dir = job_utils.create_workflow_dir(self.master, '/tmp/spark-edp', job, job_execution.id, "700") paths, builtin_paths = self._upload_job_files( self.master, wf_dir, job, updated_job_configs) # We can shorten the paths in this case since we'll run out of wf_dir paths = [os.path.basename(p) if p.startswith(wf_dir) else p for p in paths] builtin_paths = [os.path.basename(p) for p in builtin_paths] cmd = self._build_command(wf_dir, paths, builtin_paths, updated_job_configs) job_execution = conductor.job_execution_get(ctx, job_execution.id) if job_execution.info['status'] == edp.JOB_STATUS_TOBEKILLED: return (None, edp.JOB_STATUS_KILLED, None) # If an exception is raised here, the job_manager will mark # the job failed and log the exception # The redirects of stdout and stderr will preserve output in the wf_dir with remote.get_remote(self.master) as r: # Upload the command launch script launch = os.path.join(wf_dir, "launch_command") python_version = r.get_python_version() r.write_file_to(launch, self._job_script(python_version)) r.execute_command("chmod u+rwx,g+rx,o+rx %s" % wf_dir) r.execute_command("chmod +x %s" % launch) ret, stdout = r.execute_command( "cd %s; ./launch_command %s > /dev/null 2>&1 & echo $!" % (wf_dir, cmd)) if ret == 0: # Success, we'll add the wf_dir in job_execution.extra and store # pid@instance_id as the job id # We know the job is running so return "RUNNING" return (stdout.strip() + "@" + self.master.id, edp.JOB_STATUS_RUNNING, {'spark-path': wf_dir}) # Hmm, no execption but something failed. # Since we're using backgrounding with redirect, this is unlikely. raise e.EDPError(_("Spark job execution failed. Exit status = " "%(status)s, stdout = %(stdout)s") % {'status': ret, 'stdout': stdout}) def run_scheduled_job(self, job_execution): raise e.NotImplementedException(_("Currently Spark engine does not" " support scheduled EDP jobs")) def validate_job_execution(self, cluster, job, data): j.check_main_class_present(data, job) @staticmethod def get_possible_job_config(job_type): return {'job_config': {'configs': [], 'args': []}} @staticmethod def get_supported_job_types(): return [edp.JOB_TYPE_SPARK] class SparkShellJobEngine(SparkJobEngine): def _build_command(self, wf_dir, paths, builtin_paths, updated_job_configs): main_script = paths.pop(0) args = " ".join(updated_job_configs.get('args', [])) env_params = "" params = updated_job_configs.get('params', {}) for key, value in params.items(): env_params += "{key}={value} ".format(key=key, value=value) cmd = ("{env_params}{cmd} {main_script} {args}".format( cmd='/bin/sh', main_script=main_script, env_params=env_params, args=args)) return cmd def validate_job_execution(self, cluster, job, data): # Shell job doesn't require any special validation pass @staticmethod def get_possible_job_config(job_type): return {'job_config': {'configs': {}, 'args': [], 'params': {}}} @staticmethod def get_supported_job_types(): return [edp.JOB_TYPE_SHELL] sahara-8.0.0/sahara/service/edp/__init__.py0000666000175100017510000000000013245514472020561 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/oozie/0000775000175100017510000000000013245515027017602 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/0000775000175100017510000000000013245515027023173 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/workflow_factory.py0000666000175100017510000003222313245514472027155 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six from sahara import conductor as c from sahara import context from sahara.service.castellan import utils as key_manager from sahara.service.edp.oozie.workflow_creator import hive_workflow from sahara.service.edp.oozie.workflow_creator import java_workflow from sahara.service.edp.oozie.workflow_creator import mapreduce_workflow from sahara.service.edp.oozie.workflow_creator import pig_workflow from sahara.service.edp.oozie.workflow_creator import shell_workflow from sahara.swift import swift_helper as sw from sahara.swift import utils as su from sahara.utils import edp from sahara.utils import xmlutils conductor = c.API CONF = cfg.CONF class BaseFactory(object): def _separate_edp_configs(self, job_dict): configs = {} edp_configs = {} if 'configs' in job_dict: for k, v in six.iteritems(job_dict['configs']): if k.startswith('edp.'): edp_configs[k] = v elif not k.startswith('oozie.'): # 'oozie.' configs have been written to the properties file configs[k] = v return configs, edp_configs def _prune_edp_configs(self, job_dict): if job_dict is None: return {}, {} # Rather than copy.copy, we make this by hand # to avoid FrozenClassError when we update 'configs' pruned_job_dict = {} for k, v in six.iteritems(job_dict): pruned_job_dict[k] = v # Separate out "edp." configs into its own dictionary configs, edp_configs = self._separate_edp_configs(job_dict) # Prune the new job_dict so it does not hold "edp." configs pruned_job_dict['configs'] = configs return pruned_job_dict, edp_configs def _update_dict(self, dest, src): if src is not None: for key, value in six.iteritems(dest): if hasattr(value, "update"): new_vals = src.get(key, {}) value.update(new_vals) def update_job_dict(self, job_dict, exec_dict): pruned_exec_dict, edp_configs = self._prune_edp_configs(exec_dict) self._update_dict(job_dict, pruned_exec_dict) # Add the separated "edp." configs to the job_dict job_dict['edp_configs'] = edp_configs # Args are listed, not named. Simply replace them. job_dict['args'] = pruned_exec_dict.get('args', []) # Find all swift:// paths in args, configs, and params and # add the .sahara suffix to the container if it is not there # already job_dict['args'] = [ # TODO(tmckay) args for Pig can actually be -param name=value # and value could conceivably contain swift paths su.inject_swift_url_suffix(arg) for arg in job_dict['args']] for k, v in six.iteritems(job_dict.get('configs', {})): job_dict['configs'][k] = su.inject_swift_url_suffix(v) for k, v in six.iteritems(job_dict.get('params', {})): job_dict['params'][k] = su.inject_swift_url_suffix(v) def get_configs(self, input_data, output_data, proxy_configs=None): configs = {} if proxy_configs: configs[sw.HADOOP_SWIFT_USERNAME] = proxy_configs.get( 'proxy_username') configs[sw.HADOOP_SWIFT_PASSWORD] = key_manager.get_secret( proxy_configs.get('proxy_password')) configs[sw.HADOOP_SWIFT_TRUST_ID] = proxy_configs.get( 'proxy_trust_id') configs[sw.HADOOP_SWIFT_DOMAIN_NAME] = CONF.proxy_user_domain_name return configs for src in (input_data, output_data): if src.type == "swift" and hasattr(src, "credentials"): if "user" in src.credentials: configs[sw.HADOOP_SWIFT_USERNAME] = src.credentials['user'] if "password" in src.credentials: configs[sw.HADOOP_SWIFT_PASSWORD] = ( key_manager.get_secret(src.credentials['password'])) break return configs def get_params(self, input_data, output_data, data_source_urls): return {'INPUT': data_source_urls[input_data.id], 'OUTPUT': data_source_urls[output_data.id]} class PigFactory(BaseFactory): def __init__(self, job): super(PigFactory, self).__init__() self.name = self.get_script_name(job) def get_script_name(self, job): return conductor.job_main_name(context.ctx(), job) def get_workflow_xml(self, cluster, job_configs, input_data, output_data, hdfs_user, data_source_urls): proxy_configs = job_configs.get('proxy_configs') job_dict = {'configs': self.get_configs(input_data, output_data, proxy_configs), 'params': self.get_params(input_data, output_data, data_source_urls), 'args': []} self.update_job_dict(job_dict, job_configs) creator = pig_workflow.PigWorkflowCreator() creator.build_workflow_xml(self.name, configuration=job_dict['configs'], params=job_dict['params'], arguments=job_dict['args']) return creator.get_built_workflow_xml() class HiveFactory(BaseFactory): def __init__(self, job): super(HiveFactory, self).__init__() self.name = self.get_script_name(job) def get_script_name(self, job): return conductor.job_main_name(context.ctx(), job) def get_workflow_xml(self, cluster, job_configs, input_data, output_data, hdfs_user, data_source_urls): proxy_configs = job_configs.get('proxy_configs') job_dict = {'configs': self.get_configs(input_data, output_data, proxy_configs), 'params': self.get_params(input_data, output_data, data_source_urls)} self.update_job_dict(job_dict, job_configs) creator = hive_workflow.HiveWorkflowCreator() creator.build_workflow_xml(self.name, edp.get_hive_shared_conf_path(hdfs_user), configuration=job_dict['configs'], params=job_dict['params']) return creator.get_built_workflow_xml() class MapReduceFactory(BaseFactory): def get_configs(self, input_data, output_data, proxy_configs, data_source_urls): configs = super(MapReduceFactory, self).get_configs(input_data, output_data, proxy_configs) configs['mapred.input.dir'] = data_source_urls[input_data.id] configs['mapred.output.dir'] = data_source_urls[output_data.id] return configs def _get_streaming(self, job_dict): prefix = 'edp.streaming.' return {k[len(prefix):]: v for (k, v) in six.iteritems( job_dict['edp_configs']) if k.startswith(prefix)} def get_workflow_xml(self, cluster, job_configs, input_data, output_data, hdfs_user, data_source_urls): proxy_configs = job_configs.get('proxy_configs') job_dict = {'configs': self.get_configs(input_data, output_data, proxy_configs, data_source_urls)} self.update_job_dict(job_dict, job_configs) creator = mapreduce_workflow.MapReduceWorkFlowCreator() creator.build_workflow_xml(configuration=job_dict['configs'], streaming=self._get_streaming(job_dict)) return creator.get_built_workflow_xml() class JavaFactory(BaseFactory): def _get_java_configs(self, job_dict): main_class = job_dict['edp_configs']['edp.java.main_class'] java_opts = job_dict['edp_configs'].get('edp.java.java_opts', None) args = job_dict['args'] if edp.is_adapt_for_oozie_enabled(job_dict['edp_configs']): if args: args = [main_class] + args else: args = [main_class] main_class = 'org.openstack.sahara.edp.MainWrapper' return main_class, java_opts, args def get_configs(self, proxy_configs=None): configs = {} if proxy_configs: configs[sw.HADOOP_SWIFT_USERNAME] = proxy_configs.get( 'proxy_username') configs[sw.HADOOP_SWIFT_PASSWORD] = key_manager.get_secret( proxy_configs.get('proxy_password')) configs[sw.HADOOP_SWIFT_TRUST_ID] = proxy_configs.get( 'proxy_trust_id') configs[sw.HADOOP_SWIFT_DOMAIN_NAME] = CONF.proxy_user_domain_name return configs return configs def get_workflow_xml(self, cluster, job_configs, *args, **kwargs): proxy_configs = job_configs.get('proxy_configs') job_dict = {'configs': self.get_configs(proxy_configs=proxy_configs), 'args': []} self.update_job_dict(job_dict, job_configs) main_class, java_opts, args = self._get_java_configs(job_dict) creator = java_workflow.JavaWorkflowCreator() creator.build_workflow_xml(main_class, configuration=job_dict['configs'], java_opts=java_opts, arguments=args) return creator.get_built_workflow_xml() class ShellFactory(BaseFactory): def __init__(self, job): self.name, self.file_names = self.get_file_names(job) def get_file_names(self, job): ctx = context.ctx() return (conductor.job_main_name(ctx, job), conductor.job_lib_names(ctx, job)) def get_configs(self): return {'configs': {}, 'params': {}, 'args': []} def get_workflow_xml(self, cluster, job_configs, *args, **kwargs): job_dict = self.get_configs() self.update_job_dict(job_dict, job_configs) creator = shell_workflow.ShellWorkflowCreator() creator.build_workflow_xml(self.name, configuration=job_dict['configs'], env_vars=job_dict['params'], arguments=job_dict['args'], files=self.file_names) return creator.get_built_workflow_xml() def _get_creator(job): def make_PigFactory(): return PigFactory(job) def make_HiveFactory(): return HiveFactory(job) def make_ShellFactory(): return ShellFactory(job) type_map = { edp.JOB_TYPE_HIVE: make_HiveFactory, edp.JOB_TYPE_JAVA: JavaFactory, edp.JOB_TYPE_MAPREDUCE: MapReduceFactory, edp.JOB_TYPE_MAPREDUCE_STREAMING: MapReduceFactory, edp.JOB_TYPE_PIG: make_PigFactory, edp.JOB_TYPE_SHELL: make_ShellFactory } return type_map[job.type]() def get_workflow_xml(job, cluster, job_configs, *args, **kwargs): return _get_creator(job).get_workflow_xml( cluster, job_configs, *args, **kwargs) def get_possible_job_config(job_type): if not edp.compare_job_type(job_type, *edp.JOB_TYPES_ALL): return None if edp.compare_job_type(job_type, edp.JOB_TYPE_JAVA): return {'job_config': {'configs': [], 'args': []}} if edp.compare_job_type(job_type, edp.JOB_TYPE_SHELL): return {'job_config': {'configs': [], 'params': {}, 'args': []}} if edp.compare_job_type(job_type, edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_PIG): cfg = xmlutils.load_hadoop_xml_defaults( 'service/edp/resources/mapred-default.xml') if edp.compare_job_type(job_type, edp.JOB_TYPE_MAPREDUCE): cfg += get_possible_mapreduce_configs() elif edp.compare_job_type(job_type, edp.JOB_TYPE_HIVE): cfg = xmlutils.load_hadoop_xml_defaults( 'service/edp/resources/hive-default.xml') config = {'configs': cfg} if edp.compare_job_type(job_type, edp.JOB_TYPE_PIG, edp.JOB_TYPE_HIVE): config.update({'params': {}}) if edp.compare_job_type(job_type, edp.JOB_TYPE_PIG): config.update({'args': []}) return {'job_config': config} def get_possible_mapreduce_configs(): '''return a list of possible configuration values for map reduce jobs.''' cfg = xmlutils.load_hadoop_xml_defaults( 'service/edp/resources/mapred-job-config.xml') return cfg sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/base_workflow.py0000666000175100017510000000636613245514472026431 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import xml.dom.minidom as xml import sahara.exceptions as ex from sahara.i18n import _ from sahara.utils import xmlutils as x class OozieWorkflowCreator(object): doc = None tag_name = "no-op" def __init__(self, name): self.doc = x.load_xml_document("service/edp/resources/workflow.xml", strip=True) self.tag_name = name x.add_child(self.doc, 'action', self.tag_name) ok_elem = xml.parseString('<%s to="%s"/>' % ("ok", "end")) x.add_element(self.doc, 'action', ok_elem.firstChild) error_elem = xml.parseString('<%s to="%s"/>' % ("error", "fail")) x.add_element(self.doc, 'action', error_elem.firstChild) x.add_text_element_to_tag(self.doc, self.tag_name, 'job-tracker', "${jobTracker}") x.add_text_element_to_tag(self.doc, self.tag_name, 'name-node', "${nameNode}") def _add_to_prepare_element(self, element, paths): if element not in ['delete', 'mkdir']: raise ex.NotFoundException(element, _('"%s" child cannot be ' 'added to prepare element')) prop = x.get_and_create_if_not_exist(self.doc, self.tag_name, 'prepare') for path in paths: elem = xml.parseString('<%s path="%s"/>' % (element, path)) prop.appendChild(elem.firstChild) def _add_to_streaming_element(self, element, path): if element not in ['mapper', 'reducer']: raise ex.NotFoundException(element, _('"%s" child cannot be added ' 'to streaming element')) x.get_and_create_if_not_exist(self.doc, self.tag_name, 'streaming') x.add_text_element_to_tag(self.doc, 'streaming', element, path) def _add_configuration_elements(self, configuration): if configuration: x.add_properties_to_configuration(self.doc, self.tag_name, configuration) def _add_job_xml_element(self, job_xml): if job_xml: x.add_text_element_to_tag(self.doc, self.tag_name, 'job-xml', job_xml) def _add_files_and_archives(self, files, archives): if files: x.add_tagged_list(self.doc, self.tag_name, 'file', files) if archives: x.add_tagged_list(self.doc, self.tag_name, 'archive', archives) def get_built_workflow_xml(self): return self.doc.toprettyxml(indent=" ") sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/__init__.py0000666000175100017510000000000013245514472025277 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/pig_workflow.py0000666000175100017510000000333213245514472026264 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp.oozie.workflow_creator import base_workflow from sahara.utils import xmlutils as x class PigWorkflowCreator(base_workflow.OozieWorkflowCreator): def __init__(self): super(PigWorkflowCreator, self).__init__('pig') def build_workflow_xml(self, script_name, prepare=None, job_xml=None, configuration=None, params=None, arguments=None, files=None, archives=None): prepare = prepare or {} params = params or {} arguments = arguments or [] files = files or [] archives = archives or [] for k in sorted(prepare): self._add_to_prepare_element(k, prepare[k]) self._add_job_xml_element(job_xml) self._add_configuration_elements(configuration) x.add_text_element_to_tag(self.doc, self.tag_name, 'script', script_name) x.add_equal_separated_dict(self.doc, self.tag_name, 'param', params) for arg in arguments: x.add_text_element_to_tag(self.doc, self.tag_name, 'argument', arg) self._add_files_and_archives(files, archives) sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/mapreduce_workflow.py0000666000175100017510000000301713245514472027452 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp.oozie.workflow_creator import base_workflow class MapReduceWorkFlowCreator(base_workflow.OozieWorkflowCreator): def __init__(self): super(MapReduceWorkFlowCreator, self).__init__('map-reduce') def build_workflow_xml(self, prepare=None, job_xml=None, configuration=None, files=None, archives=None, streaming=None): prepare = prepare or {} files = files or [] archives = archives or [] streaming = streaming or {} for k in sorted(prepare): self._add_to_prepare_element(k, prepare[k]) # TODO(aignatov): Need to add PIPES workflow for k in sorted(streaming): self._add_to_streaming_element(k, streaming[k]) self._add_job_xml_element(job_xml) self._add_configuration_elements(configuration) self._add_files_and_archives(files, archives) sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/shell_workflow.py0000666000175100017510000000344013245514472026614 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp.oozie.workflow_creator import base_workflow from sahara.utils import xmlutils as x class ShellWorkflowCreator(base_workflow.OozieWorkflowCreator): SHELL_XMLNS = {"xmlns": "uri:oozie:shell-action:0.1"} def __init__(self): super(ShellWorkflowCreator, self).__init__('shell') def build_workflow_xml(self, script_name, prepare=None, job_xml=None, configuration=None, env_vars=None, arguments=None, files=None): x.add_attributes_to_element(self.doc, self.tag_name, self.SHELL_XMLNS) prepare = prepare or {} env_vars = env_vars or {} arguments = arguments or [] files = files or [] for k in sorted(prepare): self._add_to_prepare_element(k, prepare[k]) self._add_configuration_elements(configuration) x.add_text_element_to_tag(self.doc, self.tag_name, 'exec', script_name) for arg in arguments: x.add_text_element_to_tag(self.doc, self.tag_name, 'argument', arg) x.add_equal_separated_dict(self.doc, self.tag_name, 'env-var', env_vars) self._add_files_and_archives(files + [script_name], []) sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/hive_workflow.py0000666000175100017510000000325513245514472026444 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp.oozie.workflow_creator import base_workflow from sahara.utils import xmlutils as x class HiveWorkflowCreator(base_workflow.OozieWorkflowCreator): def __init__(self): super(HiveWorkflowCreator, self).__init__('hive') hive_elem = self.doc.getElementsByTagName('hive')[0] hive_elem.setAttribute('xmlns', 'uri:oozie:hive-action:0.2') def build_workflow_xml(self, script, job_xml, prepare=None, configuration=None, params=None, files=None, archives=None): prepare = prepare or {} params = params or {} files = files or [] archives = archives or [] for k in sorted(prepare): self._add_to_prepare_element(k, prepare[k]) self._add_job_xml_element(job_xml) self._add_configuration_elements(configuration) x.add_text_element_to_tag(self.doc, self.tag_name, 'script', script) x.add_equal_separated_dict(self.doc, self.tag_name, 'param', params) self._add_files_and_archives(files, archives) sahara-8.0.0/sahara/service/edp/oozie/workflow_creator/java_workflow.py0000666000175100017510000000366113245514472026433 0ustar zuulzuul00000000000000# Copyright (c) 2013 RedHat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp.oozie.workflow_creator import base_workflow from sahara.utils import xmlutils as x class JavaWorkflowCreator(base_workflow.OozieWorkflowCreator): def __init__(self): super(JavaWorkflowCreator, self).__init__('java') def build_workflow_xml(self, main_class, prepare=None, job_xml=None, configuration=None, java_opts=None, arguments=None, files=None, archives=None): prepare = prepare or {} arguments = arguments or [] files = files or [] archives = archives or [] for k in sorted(prepare): self._add_to_prepare_element(k, prepare[k]) self._add_job_xml_element(job_xml) self._add_configuration_elements(configuration) x.add_text_element_to_tag(self.doc, self.tag_name, 'main-class', main_class) if java_opts: x.add_text_element_to_tag(self.doc, self.tag_name, 'java-opts', java_opts) for arg in arguments: x.add_text_element_to_tag(self.doc, self.tag_name, 'arg', arg) self._add_files_and_archives(files, archives) sahara-8.0.0/sahara/service/edp/oozie/oozie.py0000666000175100017510000001225713245514472021315 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import re from oslo_serialization import jsonutils as json from oslo_utils import uuidutils import six from six.moves.urllib import parse as urlparse import sahara.exceptions as ex @six.add_metaclass(abc.ABCMeta) class BaseOozieClient(object): def __init__(self, url, oozie_server): self.job_url = url + "/v2/job/%s" self.jobs_url = url + "/v2/jobs" self.oozie_server = oozie_server self.port = urlparse.urlparse(url).port @abc.abstractmethod def add_job(self, job_config, job_execution): pass @abc.abstractmethod def manage_job(self, job_execution, action, job_id=None): pass @abc.abstractmethod def get_job_info(self, job_execution, job_id=None): pass def kill_job(self, job_execution): self.manage_job(job_execution, 'kill') def run_job(self, job_execution, job_id): self.manage_job(job_execution, 'start', job_id=job_id) class OozieClient(BaseOozieClient): def add_job(self, job_config, job_execution): return self.post( job_execution, self.jobs_url, data=job_config, headers={ "Content-Type": "application/xml;charset=UTF-8"}) def manage_job(self, job_execution, action, job_id=None): job_id = job_id if job_id else job_execution.engine_job_id url = self.job_url % job_id + "?action=" + action self.put(job_execution, url) def get_job_info(self, job_execution, job_id=None): job_id = job_id if job_id else job_execution.engine_job_id url = self.job_url % job_id + "?show=info" return self.get(job_execution, url) def _get_http_session(self, info=None): return self.oozie_server.remote().get_http_client(self.port, info=info) def post(self, job_execution, url, data, headers): session = self._get_http_session(job_execution.extra.get('neutron')) resp = session.post(url, data=data, headers=headers) _check_status_code(resp, 201) return get_json(resp)['id'] def put(self, job_execution, url): session = self._get_http_session(job_execution.extra.get('neutron')) resp = session.put(url) _check_status_code(resp, 200) def get(self, job_execution, url): session = self._get_http_session(job_execution.extra.get('neutron')) resp = session.get(url) _check_status_code(resp, 200) return get_json(resp) class RemoteOozieClient(OozieClient): def __init__(self, url, oozie_server, hdfs_user): self.hdfs_user = hdfs_user self.oozie_url = url.replace( urlparse.urlparse(url).hostname, oozie_server.fqdn()) super(RemoteOozieClient, self).__init__(url, oozie_server) def _oozie(self, cmd): return ( "sudo su - -c 'oozie -Doozie.auth.token.cache=false " "{cmd} -oozie {oozie}' {user}".format( cmd=cmd, oozie=self.oozie_url, user=self.hdfs_user)) def add_job(self, job_config, job_execution): with self.oozie_server.remote() as r: name = "/tmp/%s.xml" % uuidutils.generate_uuid()[:8] r.write_file_to(name, job_config) cmd = self._oozie("job -submit -config %s" % name) cmd += " | awk '{ print $2 }'" code, stdout = r.execute_command(cmd) stdout = stdout.strip() return stdout def manage_job(self, job_execution, action, job_id=None): job_id = job_id if job_id else job_execution.engine_job_id cmd = self._oozie("job -%s %s" % (action, job_id)) with self.oozie_server.remote() as r: r.execute_command(cmd) def get_job_info(self, job_execution, job_id=None): job_id = job_id if job_id else job_execution.engine_job_id cmd = self._oozie("job -info %s" % job_id) cmd += " | grep Status | head -n 1 | awk '{ print $3 }'" with self.oozie_server.remote() as r: code, stdout = r.execute_command(cmd) return {'status': stdout.strip()} def _check_status_code(resp, expected_code): if resp.status_code != expected_code: resp_text = resp.text # cleaning tomcat error message message = resp_text.split("
")[1] message = message.replace("

", "\n") message = re.sub('<[^<]+?>', ' ', message) raise ex.OozieException(message) def get_json(response): """Provides backward compatibility for old versions of requests library.""" json_field_or_function = getattr(response, 'json', None) if callable(json_field_or_function): return response.json() else: return json.loads(response.content) sahara-8.0.0/sahara/service/edp/oozie/__init__.py0000666000175100017510000000000013245514472021706 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/oozie/engine.py0000666000175100017510000004525613245514472021442 0ustar zuulzuul00000000000000# Copyright (c) 2014 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import os import xml.dom.minidom as xml from oslo_config import cfg from oslo_utils import uuidutils import six from sahara import conductor as c from sahara import context from sahara.service.edp import base_engine from sahara.service.edp import hdfs_helper as h from sahara.service.edp.job_binaries import manager as jb_manager from sahara.service.edp import job_utils from sahara.service.edp.oozie import oozie as o from sahara.service.edp.oozie.workflow_creator import workflow_factory from sahara.service.validations.edp import job_execution as j from sahara.utils import edp from sahara.utils import remote from sahara.utils import xmlutils as x CONF = cfg.CONF conductor = c.API @six.add_metaclass(abc.ABCMeta) class OozieJobEngine(base_engine.JobEngine): def __init__(self, cluster): self.cluster = cluster self.plugin = job_utils.get_plugin(self.cluster) def get_remote_client(self): return o.RemoteOozieClient(self.get_oozie_server_uri(self.cluster), self.get_oozie_server(self.cluster), self.get_hdfs_user()) def get_client(self): # by default engine will return standard oozie client implementation return o.OozieClient(self.get_oozie_server_uri(self.cluster), self.get_oozie_server(self.cluster)) def _get_oozie_job_params(self, hdfs_user, path_to_workflow, oozie_params, use_hbase_lib, scheduled_params=None, job_dir=None, job_execution_type=None): oozie_libpath_key = "oozie.libpath" oozie_libpath = "" rm_path = self.get_resource_manager_uri(self.cluster) nn_path = self.get_name_node_uri(self.cluster) hbase_common_lib_path = "%s%s" % (nn_path, h.HBASE_COMMON_LIB_PATH) if use_hbase_lib: if oozie_libpath_key in oozie_params: oozie_libpath = "%s,%s" % (oozie_params.get(oozie_libpath_key, ""), hbase_common_lib_path) else: oozie_libpath = hbase_common_lib_path if job_execution_type == "scheduled": app_path = "oozie.coord.application.path" job_parameters = { "start": scheduled_params.get('start'), "end": scheduled_params.get('end'), "frequency": scheduled_params.get('frequency'), "workflowAppUri": "%s%s" % (nn_path, job_dir), app_path: "%s%s" % (nn_path, job_dir)} else: app_path = "oozie.wf.application.path" job_parameters = { app_path: "%s%s" % (nn_path, path_to_workflow)} job_parameters["nameNode"] = nn_path job_parameters["user.name"] = hdfs_user job_parameters["jobTracker"] = rm_path job_parameters[oozie_libpath_key] = oozie_libpath job_parameters["oozie.use.system.libpath"] = "true" # Don't let the application path be overwritten, that can't # possibly make any sense if app_path in oozie_params: del oozie_params[app_path] if oozie_libpath_key in oozie_params: del oozie_params[oozie_libpath_key] job_parameters.update(oozie_params) return job_parameters def _upload_workflow_file(self, where, job_dir, wf_xml, hdfs_user): with remote.get_remote(where) as r: h.put_file_to_hdfs(r, wf_xml, "workflow.xml", job_dir, hdfs_user) return "%s/workflow.xml" % job_dir def _upload_coordinator_file(self, where, job_dir, wf_xml, hdfs_user): with remote.get_remote(where) as r: h.put_file_to_hdfs(r, wf_xml, "coordinator.xml", job_dir, hdfs_user) return "%s/coordinator.xml" % job_dir def cancel_job(self, job_execution): if job_execution.engine_job_id is not None: client = self.get_client() client.kill_job(job_execution) return client.get_job_info(job_execution) def get_job_status(self, job_execution): if job_execution.engine_job_id is not None: return self.get_client().get_job_info(job_execution) def _prepare_run_job(self, job_execution): ctx = context.ctx() # This will be a dictionary of tuples, (native_url, runtime_url) # keyed by data_source id data_source_urls = {} prepared_job_params = {} job = conductor.job_get(ctx, job_execution.job_id) input_source, output_source = job_utils.get_input_output_data_sources( job_execution, job, data_source_urls, self.cluster) # Updated_job_configs will be a copy of job_execution.job_configs with # any name or uuid references to data_sources resolved to paths # assuming substitution is enabled. # If substitution is not enabled then updated_job_configs will # just be a reference to job_execution.job_configs to avoid a copy. # Additional_sources will be a list of any data_sources found. additional_sources, updated_job_configs = ( job_utils.resolve_data_source_references(job_execution.job_configs, job_execution.id, data_source_urls, self.cluster) ) job_execution = conductor.job_execution_update( ctx, job_execution, {"data_source_urls": job_utils.to_url_dict(data_source_urls)}) # Now that we've recorded the native urls, we can switch to the # runtime urls data_source_urls = job_utils.to_url_dict(data_source_urls, runtime=True) data_sources = additional_sources + [input_source, output_source] job_utils.prepare_cluster_for_ds(data_sources, self.cluster, updated_job_configs, data_source_urls) proxy_configs = updated_job_configs.get('proxy_configs') configs = updated_job_configs.get('configs', {}) use_hbase_lib = configs.get('edp.hbase_common_lib', {}) # Extract all the 'oozie.' configs so that they can be set in the # job properties file. These are config values for Oozie itself, # not the job code oozie_params = {} for k in list(configs): if k.startswith('oozie.'): oozie_params[k] = configs[k] external_hdfs_urls = self._resolve_external_hdfs_urls( job_execution.job_configs) for url in external_hdfs_urls: h.configure_cluster_for_hdfs(self.cluster, url) hdfs_user = self.get_hdfs_user() # TODO(tmckay): this should probably be "get_namenode" # but that call does not exist in the oozie engine api now. oozie_server = self.get_oozie_server(self.cluster) wf_dir = self._create_hdfs_workflow_dir(oozie_server, job) self._upload_job_files_to_hdfs(oozie_server, wf_dir, job, configs, proxy_configs) wf_xml = workflow_factory.get_workflow_xml( job, self.cluster, updated_job_configs, input_source, output_source, hdfs_user, data_source_urls) path_to_workflow = self._upload_workflow_file(oozie_server, wf_dir, wf_xml, hdfs_user) prepared_job_params['context'] = ctx prepared_job_params['hdfs_user'] = hdfs_user prepared_job_params['path_to_workflow'] = path_to_workflow prepared_job_params['use_hbase_lib'] = use_hbase_lib prepared_job_params['job_execution'] = job_execution prepared_job_params['oozie_params'] = oozie_params prepared_job_params['wf_dir'] = wf_dir prepared_job_params['oozie_server'] = oozie_server return prepared_job_params def run_job(self, job_execution): prepared_job_params = self._prepare_run_job(job_execution) path_to_workflow = prepared_job_params['path_to_workflow'] hdfs_user = prepared_job_params['hdfs_user'] oozie_params = prepared_job_params['oozie_params'] use_hbase_lib = prepared_job_params['use_hbase_lib'] ctx = prepared_job_params['context'] job_execution = prepared_job_params['job_execution'] job_params = self._get_oozie_job_params(hdfs_user, path_to_workflow, oozie_params, use_hbase_lib) client = self.get_client() oozie_job_id = client.add_job(x.create_hadoop_xml(job_params), job_execution) job_execution = conductor.job_execution_get(ctx, job_execution.id) if job_execution.info['status'] == edp.JOB_STATUS_TOBEKILLED: return (None, edp.JOB_STATUS_KILLED, None) conductor.job_execution_update( context.ctx(), job_execution.id, {'info': {'status': edp.JOB_STATUS_READYTORUN}, 'engine_job_id': oozie_job_id}) client.run_job(job_execution, oozie_job_id) try: status = client.get_job_info(job_execution, oozie_job_id)['status'] except Exception: status = None return (oozie_job_id, status, None) def run_scheduled_job(self, job_execution): prepared_job_params = self._prepare_run_job(job_execution) oozie_server = prepared_job_params['oozie_server'] wf_dir = prepared_job_params['wf_dir'] hdfs_user = prepared_job_params['hdfs_user'] oozie_params = prepared_job_params['oozie_params'] use_hbase_lib = prepared_job_params['use_hbase_lib'] ctx = prepared_job_params['context'] job_execution = prepared_job_params['job_execution'] coord_configs = {"jobTracker": "${jobTracker}", "nameNode": "${nameNode}"} coord_xml = self._create_coordinator_xml(coord_configs) self._upload_coordinator_file(oozie_server, wf_dir, coord_xml, hdfs_user) job_params = self._get_oozie_job_params( hdfs_user, None, oozie_params, use_hbase_lib, job_execution.job_configs.job_execution_info, wf_dir, "scheduled") client = self.get_client() oozie_job_id = client.add_job(x.create_hadoop_xml(job_params), job_execution) job_execution = conductor.job_execution_get(ctx, job_execution.id) if job_execution.info['status'] == edp.JOB_STATUS_TOBEKILLED: return (None, edp.JOB_STATUS_KILLED, None) try: status = client.get_job_status(job_execution, oozie_job_id)['status'] except Exception: status = None return (oozie_job_id, status, None) @abc.abstractmethod def get_hdfs_user(self): pass @abc.abstractmethod def create_hdfs_dir(self, remote, dir_name): pass @abc.abstractmethod def get_oozie_server_uri(self, cluster): pass @abc.abstractmethod def get_oozie_server(self, cluster): pass @abc.abstractmethod def get_name_node_uri(self, cluster): pass @abc.abstractmethod def get_resource_manager_uri(self, cluster): pass def validate_job_execution(self, cluster, job, data): # Shell job type requires no specific fields if job.type == edp.JOB_TYPE_SHELL: return # All other types except Java require input and output # objects and Java require main class if job.type == edp.JOB_TYPE_JAVA: j.check_main_class_present(data, job) else: j.check_data_sources(data, job) job_type, subtype = edp.split_job_type(job.type) if job_type == edp.JOB_TYPE_MAPREDUCE and ( subtype == edp.JOB_SUBTYPE_STREAMING): j.check_streaming_present(data, job) @staticmethod def get_possible_job_config(job_type): return workflow_factory.get_possible_job_config(job_type) @staticmethod def get_supported_job_types(): return [edp.JOB_TYPE_HIVE, edp.JOB_TYPE_JAVA, edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_MAPREDUCE_STREAMING, edp.JOB_TYPE_PIG, edp.JOB_TYPE_SHELL] def _prepare_job_binaries(self, job_binaries, r): for jb in job_binaries: jb_manager.JOB_BINARIES.get_job_binary_by_url(jb.url). \ prepare_cluster(jb, remote=r) def _upload_job_files_to_hdfs(self, where, job_dir, job, configs, proxy_configs=None): mains = list(job.mains) if job.mains else [] libs = list(job.libs) if job.libs else [] builtin_libs = edp.get_builtin_binaries(job, configs) uploaded_paths = [] hdfs_user = self.get_hdfs_user() job_dir_suffix = 'lib' if job.type != edp.JOB_TYPE_SHELL else '' lib_dir = os.path.join(job_dir, job_dir_suffix) with remote.get_remote(where) as r: job_binaries = mains + libs self._prepare_job_binaries(job_binaries, r) # upload mains uploaded_paths.extend(self._upload_job_binaries(r, mains, proxy_configs, hdfs_user, job_dir)) # upload libs if len(libs) and job_dir_suffix: # HDFS 2.2.0 fails to put file if the lib dir does not exist self.create_hdfs_dir(r, lib_dir) uploaded_paths.extend(self._upload_job_binaries(r, libs, proxy_configs, hdfs_user, lib_dir)) # upload buitin_libs for lib in builtin_libs: h.put_file_to_hdfs(r, lib['raw'], lib['name'], lib_dir, hdfs_user) uploaded_paths.append(lib_dir + lib['name']) return uploaded_paths def _upload_job_binaries(self, r, job_binaries, proxy_configs, hdfs_user, job_dir): uploaded_paths = [] for jb in job_binaries: path = jb_manager.JOB_BINARIES. \ get_job_binary_by_url(jb.url). \ copy_binary_to_cluster(jb, proxy_configs=proxy_configs, remote=r, context=context.ctx()) h.copy_from_local(r, path, job_dir, hdfs_user) uploaded_paths.append(path) return uploaded_paths def _create_hdfs_workflow_dir(self, where, job): constructed_dir = '/user/%s/' % self.get_hdfs_user() constructed_dir = self._add_postfix(constructed_dir) constructed_dir += '%s/%s' % (job.name, uuidutils.generate_uuid()) with remote.get_remote(where) as r: self.create_hdfs_dir(r, constructed_dir) return constructed_dir def _create_coordinator_xml(self, coord_configs, config_filter=None, appname='coord'): doc = xml.Document() # Create the base element coord = doc.createElement('coordinator-app') coord.attributes['name'] = appname coord.attributes['start'] = "${start}" coord.attributes['end'] = "${end}" coord.attributes['frequency'] = "${frequency}" coord.attributes['timezone'] = 'UTC' coord.attributes['xmlns'] = 'uri:oozie:coordinator:0.2' doc.appendChild(coord) action = doc.createElement('action') workflow = doc.createElement('workflow') coord.appendChild(action) action.appendChild(workflow) x.add_text_element_to_tag(doc, "workflow", 'app-path', "${workflowAppUri}") configuration = doc.createElement('configuration') workflow.appendChild(configuration) default_configs = [] if config_filter is not None: default_configs = [cfg['name'] for cfg in config_filter] for name in sorted(coord_configs): if name in default_configs or config_filter is None: x.add_property_to_configuration(doc, name, coord_configs[name]) # Return newly created XML return doc.toprettyxml(indent=" ") def _add_postfix(self, constructed_dir): def _append_slash_if_needed(path): if path[-1] != '/': path += '/' return path constructed_dir = _append_slash_if_needed(constructed_dir) if CONF.job_workflow_postfix: constructed_dir = ''.join([str(constructed_dir), str(CONF.job_workflow_postfix)]) return _append_slash_if_needed(constructed_dir) def _resolve_external_hdfs_urls(self, job_configs): external_hdfs_urls = [] for k, v in six.iteritems(job_configs.get('configs', {})): if isinstance(v, six.string_types) and v.startswith("hdfs://"): external_hdfs_urls.append(v) for k, v in six.iteritems(job_configs.get('params', {})): if isinstance(v, six.string_types) and v.startswith("hdfs://"): external_hdfs_urls.append(v) for v in job_configs.get('args', []): if isinstance(v, six.string_types) and v.startswith("hdfs://"): external_hdfs_urls.append(v) return external_hdfs_urls def suspend_job(self, job_execution): return self._manage_job(job_execution, edp.JOB_ACTION_SUSPEND) def _manage_job(self, job_execution, action): if job_execution.oozie_job_id is not None: client = self.get_client() if action == edp.JOB_ACTION_SUSPEND: client.suspend_job(job_execution) return client.get_job_status(job_execution) sahara-8.0.0/sahara/service/edp/job_utils.py0000666000175100017510000002351713245514472021036 0ustar zuulzuul00000000000000# Copyright (c) 2014 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_utils import uuidutils import six from sahara import conductor as c from sahara import context from sahara.plugins import base as plugin_base from sahara.service.edp.data_sources import manager as ds_manager from sahara.service.edp.utils import shares as shares_service from sahara.utils import remote opts = [ cfg.StrOpt('job_workflow_postfix', default='', help="Postfix for storing jobs in hdfs. Will be " "added to '/user//' path.") ] CONF = cfg.CONF CONF.register_opts(opts) conductor = c.API # Prefix used to mark data_source name references in arg lists DATA_SOURCE_PREFIX = "datasource://" DATA_SOURCE_SUBST_NAME = "edp.substitute_data_source_for_name" DATA_SOURCE_SUBST_UUID = "edp.substitute_data_source_for_uuid" def get_plugin(cluster): return plugin_base.PLUGINS.get_plugin(cluster.plugin_name) def get_data_source(ds_name): return ds_manager.DATA_SOURCES.get_data_source(ds_name) def create_workflow_dir(where, path, job, use_uuid=None, chmod=""): if use_uuid is None: use_uuid = uuidutils.generate_uuid() constructed_dir = _append_slash_if_needed(path) constructed_dir += '%s/%s' % (job.name, use_uuid) with remote.get_remote(where) as r: if chmod: r.execute_command("mkdir -p -m %s %s" % (chmod, constructed_dir)) else: r.execute_command("mkdir -p %s" % constructed_dir) return constructed_dir def _get_data_source_urls(ds, cluster, job_exec_id): # returns a tuple (native_url, runtime_url) return get_data_source(ds.type).get_urls(ds.url, cluster, job_exec_id) def get_input_output_data_sources(job_execution, job, data_source_urls, cluster=None): def _construct(ctx, ds_id): job_exec_id = job_execution.id source = conductor.data_source_get(ctx, ds_id) if source and source.id not in data_source_urls: data_source_urls[source.id] = _get_data_source_urls(source, cluster, job_exec_id) return source ctx = context.ctx() input_source = _construct(ctx, job_execution.input_id) output_source = _construct(ctx, job_execution.output_id) return input_source, output_source def _append_slash_if_needed(path): if path[-1] != '/': path += '/' return path def may_contain_data_source_refs(job_configs): def _check_data_source_ref_option(option): truth = job_configs and ( job_configs.get('configs', {}).get(option)) # Config values specified in the UI may be # passed as strings return truth in (True, 'True') return ( _check_data_source_ref_option(DATA_SOURCE_SUBST_NAME), _check_data_source_ref_option(DATA_SOURCE_SUBST_UUID)) def _data_source_ref_search(job_configs, func, prune=lambda x: x): """Return a list of unique values in job_configs filtered by func(). Loop over the 'args', 'configs' and 'params' elements in job_configs and return a list of all values for which func(value) is True. Optionally provide a 'prune' function that is applied to values before they are added to the return value. """ args = set([prune(arg) for arg in job_configs.get( 'args', []) if func(arg)]) configs = set([prune(val) for val in six.itervalues( job_configs.get('configs', {})) if func(val)]) params = set([prune(val) for val in six.itervalues( job_configs.get('params', {})) if func(val)]) return list(args | configs | params) def find_possible_data_source_refs_by_name(job_configs): """Find string values in job_configs starting with 'datasource://'. Loop over the 'args', 'configs', and 'params' elements of job_configs to find all values beginning with the prefix 'datasource://'. Return a list of unique values with the prefix removed. Note that for 'configs' and 'params', which are dictionaries, only the values are considered and the keys are not relevant. """ def startswith(arg): return isinstance( arg, six.string_types) and arg.startswith(DATA_SOURCE_PREFIX) return _data_source_ref_search(job_configs, startswith, prune=lambda x: x[len(DATA_SOURCE_PREFIX):]) def find_possible_data_source_refs_by_uuid(job_configs): """Find string values in job_configs which are uuids. Return a list of unique values in the 'args', 'configs', and 'params' elements of job_configs which have the form of a uuid. Note that for 'configs' and 'params', which are dictionaries, only the values are considered and the keys are not relevant. """ return _data_source_ref_search(job_configs, uuidutils.is_uuid_like) def resolve_data_source_references(job_configs, job_exec_id, data_source_urls, cluster=None): """Resolve possible data_source references in job_configs. Look for any string values in the 'args', 'configs', and 'params' elements of job_configs which start with 'datasource://' or have the form of a uuid. For values beginning with 'datasource://', strip off the prefix and search for a DataSource object with a name that matches the value. For values having the form of a uuid, search for a DataSource object with an id that matches the value. If a DataSource object is found for the value, replace the value with the URL from the DataSource object. If any DataSource objects are found which reference swift paths and contain credentials, set credential configuration values in job_configs (use the first set of swift credentials found). If no values are resolved, return an empty list and a reference to job_configs. If any values are resolved, return a list of the referenced data_source objects and a copy of job_configs with all of the references replaced with URLs. """ by_name, by_uuid = may_contain_data_source_refs(job_configs) if not (by_name or by_uuid): return [], job_configs ctx = context.ctx() ds_seen = {} new_configs = {} def _resolve(value): kwargs = {} if by_name and isinstance( value, six.string_types) and value.startswith(DATA_SOURCE_PREFIX): value = value[len(DATA_SOURCE_PREFIX):] kwargs['name'] = value elif by_uuid and uuidutils.is_uuid_like(value): kwargs['id'] = value if kwargs: # Name and id are both unique constraints so if there # is more than 1 something is really wrong ds = conductor.data_source_get_all(ctx, **kwargs) if len(ds) == 1: ds = ds[0] ds_seen[ds.id] = ds if ds.id not in data_source_urls: data_source_urls[ds.id] = _get_data_source_urls( ds, cluster, job_exec_id) return data_source_urls[ds.id][1] return value # Loop over configs/params/args and look up each value as a data_source. # If we find it, replace the value. In all cases, we've produced a # copy which is not a FrozenClass type and can be updated. new_configs['configs'] = { k: _resolve(v) for k, v in six.iteritems( job_configs.get('configs', {}))} new_configs['params'] = { k: _resolve(v) for k, v in six.iteritems( job_configs.get('params', {}))} new_configs['args'] = [_resolve(a) for a in job_configs.get('args', [])] # If we didn't resolve anything we might as well return the original ds_seen = ds_seen.values() if not ds_seen: return [], job_configs # If there are proxy_configs we'll need to copy these, too, # so job_configs is complete if job_configs.get('proxy_configs'): new_configs['proxy_configs'] = { k: v for k, v in six.iteritems(job_configs.get('proxy_configs'))} return ds_seen, new_configs def prepare_cluster_for_ds(data_sources, cluster, job_configs, ds_urls): for ds in data_sources: if ds: get_data_source(ds.type).prepare_cluster( ds, cluster, job_configs=job_configs, runtime_url=ds_urls[ds.id]) def to_url_dict(data_source_urls, runtime=False): idx = 1 if runtime else 0 return {id: urls[idx] for id, urls in six.iteritems(data_source_urls)} def mount_share_at_default_path(url, cluster): # Automount this share to the cluster with default path # url example: 'manila://ManilaShare-uuid/path_to_file' share_id = six.moves.urllib.parse.urlparse(url).netloc if cluster.shares: cluster_shares = [dict(s) for s in cluster.shares] else: cluster_shares = [] needed_share = { 'id': share_id, 'path': shares_service.default_mount(share_id), 'access_level': 'rw' } cluster_shares.append(needed_share) cluster = conductor.cluster_update( context.ctx(), cluster, {'shares': cluster_shares}) shares_service.mount_shares(cluster) return shares_service.get_share_path(url, cluster.shares) sahara-8.0.0/sahara/service/edp/binary_retrievers/0000775000175100017510000000000013245515027022213 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/binary_retrievers/internal_swift.py0000666000175100017510000000670713245514472025634 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import functools from oslo_config import cfg import six import swiftclient import sahara.exceptions as ex from sahara.i18n import _ from sahara.service.castellan import utils as key_manager from sahara.swift import utils as su from sahara.utils.openstack import swift as sw CONF = cfg.CONF def _get_names_from_url(url): parse = six.moves.urllib.parse.urlparse(url) return (parse.netloc + parse.path).split('/', 1) def _get_raw_data(job_binary, conn): names = _get_names_from_url(job_binary.url) container, obj = names # if container name has '.sahara' suffix we need to strip it if container.endswith(su.SWIFT_URL_SUFFIX): container = container[:-len(su.SWIFT_URL_SUFFIX)] try: # First check the size headers = conn.head_object(container, obj) total_KB = int(headers.get('content-length', 0)) / 1024.0 if total_KB > CONF.job_binary_max_KB: raise ex.DataTooBigException( round(total_KB, 1), CONF.job_binary_max_KB, _("Size of swift object (%(size)sKB) is greater " "than maximum (%(maximum)sKB)")) headers, body = conn.get_object(container, obj) except swiftclient.ClientException as e: raise ex.SwiftClientException(six.text_type(e)) return body def _validate_job_binary_url(f): @functools.wraps(f) def wrapper(job_binary, *args, **kwargs): if not (job_binary.url.startswith(su.SWIFT_INTERNAL_PREFIX)): # This should have been guaranteed already, # but we'll check just in case. raise ex.BadJobBinaryException( _("Url for binary in internal swift must start with %s") % su.SWIFT_INTERNAL_PREFIX) names = _get_names_from_url(job_binary.url) if len(names) == 1: # a container has been requested, this is currently unsupported raise ex.BadJobBinaryException( _('Url for binary in internal swift must specify an object ' 'not a container')) return f(job_binary, *args, **kwargs) return wrapper @_validate_job_binary_url def get_raw_data(job_binary, proxy_configs=None): conn_kwargs = {} if proxy_configs: conn_kwargs.update(username=proxy_configs.get('proxy_username'), password=key_manager.get_secret( proxy_configs.get('proxy_password')), trust_id=proxy_configs.get('proxy_trust_id')) else: conn_kwargs.update(username=job_binary.extra.get('user'), password=key_manager.get_secret( job_binary.extra.get('password'))) conn = sw.client(**conn_kwargs) return _get_raw_data(job_binary, conn) @_validate_job_binary_url def get_raw_data_with_context(job_binary): conn = sw.client_from_token() return _get_raw_data(job_binary, conn) sahara-8.0.0/sahara/service/edp/binary_retrievers/s3_storage.py0000666000175100017510000000132313245514472024642 0ustar zuulzuul00000000000000# Copyright (c) 2017 Massachusetts Open Cloud # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp import s3_common def get_raw_data(job_binary): return s3_common.get_raw_job_binary_data(job_binary) sahara-8.0.0/sahara/service/edp/binary_retrievers/__init__.py0000666000175100017510000000000013245514472024317 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/binary_retrievers/sahara_db.py0000666000175100017510000000165113245514472024501 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import conductor as c conductor = c.API def get_raw_data(context, job_binary): # url example: 'internal-db://JobBinaryInternal-UUID' binary_internal_id = job_binary.url[len("internal-db://"):] return conductor.job_binary_internal_get_raw_data(context, binary_internal_id) sahara-8.0.0/sahara/service/edp/binary_retrievers/dispatch.py0000666000175100017510000000444113245514472024374 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara import context from sahara.service.edp.binary_retrievers import internal_swift as i_swift from sahara.service.edp.binary_retrievers import manila_share as manila from sahara.service.edp.binary_retrievers import s3_storage as s3 from sahara.service.edp.binary_retrievers import sahara_db as db from sahara.service.edp import s3_common from sahara.swift import utils as su from sahara.utils.openstack import manila as m def get_raw_binary(job_binary, proxy_configs=None, with_context=False, remote=None): '''Get the raw data for a job binary This will retrieve the raw data for a job binary from it's source. In the case of Swift based binaries there is a precedence of credentials for authenticating the client. Requesting a context based authentication takes precedence over proxy user which takes precedence over embedded credentials. :param job_binary: The job binary to retrieve :param proxy_configs: Proxy user configuration to use as credentials :param with_context: Use the current context as credentials :param remote: The remote contains node group and cluster information :returns: The raw data from a job binary ''' url = job_binary.url if url.startswith("internal-db://"): res = db.get_raw_data(context.ctx(), job_binary) if url.startswith(s3_common.S3_JB_PREFIX): res = s3.get_raw_data(job_binary) if url.startswith(su.SWIFT_INTERNAL_PREFIX): if with_context: res = i_swift.get_raw_data_with_context(job_binary) else: res = i_swift.get_raw_data(job_binary, proxy_configs) if url.startswith(m.MANILA_PREFIX): res = manila.get_file_info(job_binary, remote) return res sahara-8.0.0/sahara/service/edp/binary_retrievers/manila_share.py0000666000175100017510000000231213245514472025213 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.service.edp import job_utils from sahara.service.edp.utils import shares as shares_service def get_file_info(job_binary, remote): shares = [] if remote.instance.node_group.cluster.shares: shares.extend(remote.instance.node_group.cluster.shares) if remote.instance.node_group.shares: shares.extend(remote.instance.node_group.shares) path = shares_service.get_share_path(job_binary.url, shares) if path is None: path = job_utils.mount_share_at_default_path( job_binary.url, remote.instance.node_group.cluster) return {'type': 'path', 'path': path} sahara-8.0.0/sahara/service/edp/job_manager.py0000666000175100017510000002407313245514472021306 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import datetime from oslo_config import cfg from oslo_log import log from oslo_utils import timeutils from sahara import conductor as c from sahara import context from sahara import exceptions as e from sahara.i18n import _ from sahara.service.edp import job_utils from sahara.service.edp.oozie import engine as oozie_engine from sahara.service.edp.spark import engine as spark_engine from sahara.service.edp.storm import engine as storm_engine from sahara.utils import cluster as c_u from sahara.utils import edp from sahara.utils import proxy as p LOG = log.getLogger(__name__) CONF = cfg.CONF conductor = c.API ENGINES = [oozie_engine.OozieJobEngine, spark_engine.SparkJobEngine, storm_engine.StormJobEngine, storm_engine.StormPyleusJobEngine] def _get_job_type(job_execution): return conductor.job_get(context.ctx(), job_execution.job_id).type def get_job_engine(cluster, job_execution): return job_utils.get_plugin(cluster).get_edp_engine(cluster, _get_job_type( job_execution)) def _write_job_status(job_execution, job_info): update = {"info": job_info} if job_info['status'] in edp.JOB_STATUSES_TERMINATED: update['end_time'] = datetime.datetime.now() job_configs = p.delete_proxy_user_for_job_execution(job_execution) if job_configs: update['job_configs'] = job_configs return conductor.job_execution_update(context.ctx(), job_execution, update) def _update_job_status(engine, job_execution): job_info = engine.get_job_status(job_execution) if job_info is not None: job_execution = _write_job_status(job_execution, job_info) return job_execution def _update_job_execution_extra(cluster, job_execution): # tmckay-fp we can make this slightly more efficient in # the use_namespaces case by asking the engine if it knows # the submission machine, and checking if that machine has # a floating ip. if (CONF.use_namespaces or CONF.proxy_command): info = cluster.node_groups[0].instances[0].remote().get_neutron_info() extra = job_execution.extra.copy() extra['neutron'] = info job_execution = conductor.job_execution_update( context.ctx(), job_execution.id, {'extra': extra}) return job_execution def _run_job(job_execution_id): ctx = context.ctx() job_execution = conductor.job_execution_get(ctx, job_execution_id) cluster = conductor.cluster_get(ctx, job_execution.cluster_id) if cluster is None or cluster.status != c_u.CLUSTER_STATUS_ACTIVE: LOG.info("Can not run this job on a non-existant cluster or a " "inactive cluster.") return eng = get_job_engine(cluster, job_execution) if eng is None: raise e.EDPError(_("Cluster does not support job type %s") % _get_job_type(job_execution)) job_execution = _update_job_execution_extra(cluster, job_execution) # Job id is a string # Status is a string # Extra is a dictionary to add to extra in the job_execution if job_execution.job_configs.job_execution_info.get('job_execution_type' ) == 'scheduled': jid, status, extra = eng.run_scheduled_job(job_execution) else: jid, status, extra = eng.run_job(job_execution) # Set the job id and the start time # Optionally, update the status and the 'extra' field update_dict = {'engine_job_id': jid, 'start_time': datetime.datetime.now()} if status: update_dict['info'] = {'status': status} if extra: curr_extra = job_execution.extra.copy() if 'neutron' in curr_extra: curr_extra['neutron'] = curr_extra['neutron'].copy() curr_extra.update(extra) update_dict['extra'] = curr_extra job_execution = conductor.job_execution_update( ctx, job_execution, update_dict) def run_job(job_execution_id): try: _run_job(job_execution_id) except Exception as ex: LOG.exception("Can't run job execution (reason: {reason})".format( reason=ex)) job_execution = conductor.job_execution_get( context.ctx(), job_execution_id) if job_execution.engine_job_id is not None: cancel_job(job_execution_id) conductor.job_execution_update( context.ctx(), job_execution_id, {'info': {'status': edp.JOB_STATUS_FAILED}, 'start_time': datetime.datetime.now(), 'end_time': datetime.datetime.now()}) def cancel_job(job_execution_id): ctx = context.ctx() job_execution = conductor.job_execution_get(ctx, job_execution_id) if job_execution.info['status'] in edp.JOB_STATUSES_TERMINATED: LOG.info("Job execution is already finished and shouldn't be canceled") return job_execution cluster = conductor.cluster_get(ctx, job_execution.cluster_id) if cluster is None: LOG.info("Can not cancel this job on a non-existant cluster.") return job_execution engine = get_job_engine(cluster, job_execution) if engine is not None: job_execution = conductor.job_execution_update( ctx, job_execution_id, {'info': {'status': edp.JOB_STATUS_TOBEKILLED}}) timeout = CONF.job_canceling_timeout s_time = timeutils.utcnow() while timeutils.delta_seconds(s_time, timeutils.utcnow()) < timeout: if job_execution.info['status'] not in edp.JOB_STATUSES_TERMINATED: try: job_info = engine.cancel_job(job_execution) except Exception as ex: job_info = None LOG.warning("Error during cancel of job execution: " "{error}".format(error=ex)) if job_info is not None: job_execution = _write_job_status(job_execution, job_info) LOG.info("Job execution was canceled successfully") return job_execution context.sleep(3) job_execution = conductor.job_execution_get( ctx, job_execution_id) if not job_execution: LOG.info("Job execution was deleted. " "Canceling current operation.") return job_execution else: LOG.info("Job execution status: {status}").format( status=job_execution.info['status']) return job_execution else: raise e.CancelingFailed(_('Job execution %s was not canceled') % job_execution.id) def get_job_status(job_execution_id): ctx = context.ctx() job_execution = conductor.job_execution_get(ctx, job_execution_id) cluster = conductor.cluster_get(ctx, job_execution.cluster_id) if (cluster is not None and cluster.status == c_u.CLUSTER_STATUS_ACTIVE): engine = get_job_engine(cluster, job_execution) if engine is not None: job_execution = _update_job_status(engine, job_execution) return job_execution def update_job_status(job_execution_id): try: get_job_status(job_execution_id) except Exception as e: LOG.exception("Error during update job execution {job}: {error}" .format(job=job_execution_id, error=e)) def update_job_statuses(cluster_id=None): ctx = context.ctx() kwargs = {'end_time': None} if cluster_id: kwargs.update({'cluster_id': cluster_id}) for je in conductor.job_execution_get_all(ctx, **kwargs): update_job_status(je.id) def get_job_config_hints(job_type): for eng in ENGINES: if job_type in eng.get_supported_job_types(): return eng.get_possible_job_config(job_type) def suspend_job(job_execution_id): ctx = context.ctx() job_execution = conductor.job_execution_get(ctx, job_execution_id) if job_execution.info['status'] not in edp.JOB_STATUSES_SUSPENDIBLE: raise e.SuspendingFailed(_("Suspending operation can not be performed" " on status: {status}")).format( status=job_execution.info['status']) cluster = conductor.cluster_get(ctx, job_execution.cluster_id) engine = get_job_engine(cluster, job_execution) job_execution = conductor.job_execution_update( ctx, job_execution_id, { 'info': {'status': edp.JOB_STATUS_TOBESUSPENDED}}) try: job_info = engine.suspend_job(job_execution) except Exception as ex: job_info = None conductor.job_execution_update( ctx, job_execution_id, {'info': { 'status': edp.JOB_STATUS_SUSPEND_FAILED}}) raise e.SuspendingFailed(_("Error during suspending of job execution: " "{error}")).format(error=ex) if job_info is not None: job_execution = _write_job_status(job_execution, job_info) LOG.info("Job execution was suspended successfully") return job_execution conductor.job_execution_update( ctx, job_execution_id, {'info': { 'status': edp.JOB_STATUS_SUSPEND_FAILED}}) raise e.SuspendingFailed(_("Failed to suspend job execution" "{jid}")).format(jid=job_execution_id) sahara-8.0.0/sahara/service/edp/data_sources/0000775000175100017510000000000013245515027021131 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/opts.py0000666000175100017510000000173113245514472022477 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # File contains data sources opts to avoid cyclic imports issue from oslo_config import cfg opts = [ cfg.ListOpt('data_source_types', default=['swift', 'hdfs', 'maprfs', 'manila'], help='List of data sources types to be loaded. Sahara ' 'preserves the order of the list when returning it.'), ] CONF = cfg.CONF CONF.register_opts(opts) sahara-8.0.0/sahara/service/edp/data_sources/manager.py0000666000175100017510000000543713245514472023133 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging import six import six.moves.urllib.parse as urlparse from stevedore import enabled from sahara import conductor as cond from sahara import exceptions as ex from sahara.i18n import _ conductor = cond.API LOG = logging.getLogger(__name__) CONF = cfg.CONF class DataSourceManager(object): def __init__(self): self.data_sources = {} self._load_data_sources() def _load_data_sources(self): config_ds = CONF.data_source_types extension_manager = enabled.EnabledExtensionManager( check_func=lambda ext: ext.name in config_ds, namespace='sahara.data_source.types', invoke_on_load=True ) for ext in extension_manager.extensions: if ext.name in self.data_sources: raise ex.ConfigurationError( _("Data source with name '%s' already exists.") % ext.name) ext.obj.name = ext.name self.data_sources[ext.name] = ext.obj LOG.info("Data source name {ds_name} loaded {entry_point}".format( ds_name=ext.name, entry_point=ext.entry_point_target)) if len(self.data_sources) < len(config_ds): loaded_ds = set(six.iterkeys(self.data_sources)) requested_ds = set(config_ds) raise ex.ConfigurationError( _("Data sources couldn't be loaded: %s") % ", ".join(requested_ds - loaded_ds)) def get_data_sources(self): config_ds = CONF.data_source_types return [self.get_data_source(name).name for name in config_ds] def get_data_source(self, name): res = self.data_sources.get(name) if res is None: raise ex.InvalidDataException(_("Invalid data source")) return res def get_data_source_by_url(self, url): url = urlparse.urlparse(url) if not url.scheme: raise ex.InvalidDataException(_("Data source url must have a" " scheme")) return self.get_data_source(url.scheme) DATA_SOURCES = None def setup_data_sources(): global DATA_SOURCES DATA_SOURCES = DataSourceManager() sahara-8.0.0/sahara/service/edp/data_sources/swift/0000775000175100017510000000000013245515027022265 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/swift/implementation.py0000666000175100017510000000705213245514472025675 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six.moves.urllib.parse as urlparse from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.edp.data_sources.base import DataSourceType from sahara.swift import swift_helper as sw from sahara.swift import utils as su from sahara.utils.types import FrozenDict CONF = cfg.CONF class SwiftType(DataSourceType): def validate(self, data): self._validate_url(data['url']) if not CONF.use_domain_for_proxy_users and "credentials" not in data: raise ex.InvalidCredentials(_("No credentials provided for Swift")) if not CONF.use_domain_for_proxy_users and ( "user" not in data["credentials"]): raise ex.InvalidCredentials( _("User is not provided in credentials for Swift")) if not CONF.use_domain_for_proxy_users and ( "password" not in data["credentials"]): raise ex.InvalidCredentials( _("Password is not provided in credentials for Swift")) def _validate_url(self, url): if len(url) == 0: raise ex.InvalidDataException(_("Swift url must not be empty")) url = urlparse.urlparse(url) if url.scheme != "swift": raise ex.InvalidDataException(_("URL scheme must be 'swift'")) # The swift url suffix does not have to be included in the netloc. # However, if the swift suffix indicator is part of the netloc then # we require the right suffix. # Additionally, the path must be more than '/' if (su.SWIFT_URL_SUFFIX_START in url.netloc and not url.netloc.endswith(su.SWIFT_URL_SUFFIX)) or len(url.path) <= 1: raise ex.InvalidDataException( _("URL must be of the form swift://container%s/object") % su.SWIFT_URL_SUFFIX) def prepare_cluster(self, data_source, cluster, **kwargs): if hasattr(data_source, "credentials"): job_configs = kwargs.pop('job_configs') # if no data source was passed as a reference for the job, the # job_configs will not be changed (so it will be a FronzenDict) # and we don't need to change it as well if isinstance(job_configs, FrozenDict) or \ job_configs.get('configs', None) is None: return if not job_configs.get('proxy_configs'): username = data_source.credentials['user'] password = data_source.credentials['password'] # Don't overwrite if there is already a value here if job_configs['configs'].get(sw.HADOOP_SWIFT_USERNAME, None) \ is None and (username is not None): job_configs['configs'][sw.HADOOP_SWIFT_USERNAME] = username if job_configs['configs'].get(sw.HADOOP_SWIFT_PASSWORD, None) \ is None and (password is not None): job_configs['configs'][sw.HADOOP_SWIFT_PASSWORD] = password sahara-8.0.0/sahara/service/edp/data_sources/swift/__init__.py0000666000175100017510000000000013245514472024371 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/base.py0000666000175100017510000000716113245514472022427 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import random import re import string import six from sahara.plugins import base as plugins_base @six.add_metaclass(abc.ABCMeta) class DataSourceType(object): @plugins_base.required_with_default def construct_url(self, url, job_exec_id): """Resolve placeholders in the data source url Supported placeholders: * %RANDSTR(len)% - will be replaced with random string of lowercase letters of length `len` * %JOB_EXEC_ID% - will be replaced with the job execution ID :param url: String that represents an url with placeholders :param job_exec_id: Id of the job execution :returns: String that is an url without placeholders """ def _randstr(match): random_len = int(match.group(1)) return ''.join(random.choice(string.ascii_lowercase) for _ in six.moves.range(random_len)) url = url.replace("%JOB_EXEC_ID%", job_exec_id) url = re.sub(r"%RANDSTR\((\d+)\)%", _randstr, url) return url @plugins_base.required_with_default def prepare_cluster(self, data_source, cluster, **kwargs): """Makes a cluster ready to use this data source Different implementations for each data source, for HDFS will be configure the cluster, for Swift verify credentials, and so on :param data_source: The object handle to a data source :param cluster: The object handle to a cluster :returns: None """ pass @plugins_base.required_with_default def get_runtime_url(self, url, cluster): """Get the runtime url of the data source for a cluster It will construct a runtime url if needed, if it's not needed it will use the native url as runtime url :param url: String that represents an already constructed url :param cluster: The object handle to a cluster :returns: String representing the runtime url """ return url @plugins_base.required_with_default def get_urls(self, url, cluster, job_exec_id): """Get the native url and runtime url of a determined data source :param url: String that represents a url (constructed or not) :param cluster: The object handle to a cluster :param job_exec_id: Id of the job execution :returns: A tuple of the form (native_url, runtime_url), where the urls are Strings """ native_url = self.construct_url(url, job_exec_id) runtime_url = self.get_runtime_url(native_url, cluster) return (native_url, runtime_url) @plugins_base.required_with_default def validate(self, data): """Method that validates the data passed through the API This method will be executed during the data source creation and update :raise: If data is invalid, InvalidDataException """ pass @plugins_base.optional def _validate_url(self, url): """Auxiliary method used by the validate method""" pass sahara-8.0.0/sahara/service/edp/data_sources/hdfs/0000775000175100017510000000000013245515027022055 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/hdfs/implementation.py0000666000175100017510000000301613245514472025461 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six.moves.urllib.parse as urlparse from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.edp.data_sources.base import DataSourceType from sahara.service.edp import hdfs_helper as h class HDFSType(DataSourceType): def validate(self, data): self._validate_url(data['url']) def _validate_url(self, url): if len(url) == 0: raise ex.InvalidDataException(_("HDFS url must not be empty")) url = urlparse.urlparse(url) if url.scheme: if url.scheme != "hdfs": raise ex.InvalidDataException(_("URL scheme must be 'hdfs'")) if not url.hostname: raise ex.InvalidDataException( _("HDFS url is incorrect, cannot determine a hostname")) def prepare_cluster(self, data_source, cluster, **kwargs): runtime_url = kwargs.pop('runtime_url') h.configure_cluster_for_hdfs(cluster, runtime_url) sahara-8.0.0/sahara/service/edp/data_sources/hdfs/__init__.py0000666000175100017510000000000013245514472024161 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/__init__.py0000666000175100017510000000000013245514472023235 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/manila/0000775000175100017510000000000013245515027022372 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/manila/implementation.py0000666000175100017510000000513713245514472026004 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils import six.moves.urllib.parse as urlparse from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.edp.data_sources.base import DataSourceType from sahara.service.edp import job_utils from sahara.service.edp.utils import shares as shares_service class ManilaType(DataSourceType): def validate(self, data): self._validate_url(data['url']) def _validate_url(self, url): if len(url) == 0: raise ex.InvalidDataException(_("Manila url must not be empty")) url = urlparse.urlparse(url) if url.scheme != "manila": raise ex.InvalidDataException(_("Manila url scheme must be" " 'manila'")) if not uuidutils.is_uuid_like(url.netloc): raise ex.InvalidDataException(_("Manila url netloc must be a" " uuid")) if not url.path: raise ex.InvalidDataException(_("Manila url path must not be" " empty")) def _prepare_cluster(self, url, cluster): path = self._get_share_path(url, cluster.shares or []) if path is None: path = job_utils.mount_share_at_default_path(url, cluster) return path def get_runtime_url(self, url, cluster): # TODO(mariannelm): currently the get_runtime_url method is responsible # for preparing the cluster for the manila job type which is not the # best approach. In order to make a prepare_cluster method for manila # the edp/job_utils.py resolve_data_source_reference function must be # refactored path = self._prepare_cluster(url, cluster) # This gets us the mount point, but we need a file:// scheme to # indicate a local filesystem path return "file://{path}".format(path=path) def _get_share_path(self, url, shares): return shares_service.get_share_path(url, shares) sahara-8.0.0/sahara/service/edp/data_sources/manila/__init__.py0000666000175100017510000000000013245514472024476 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/maprfs/0000775000175100017510000000000013245515027022421 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/data_sources/maprfs/implementation.py0000666000175100017510000000223713245514472026031 0ustar zuulzuul00000000000000# Copyright (c) 2017 OpenStack Foundation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six.moves.urllib.parse as urlparse from sahara import exceptions as ex from sahara.i18n import _ from sahara.service.edp.data_sources.base import DataSourceType class MapRFSType(DataSourceType): def validate(self, data): self._validate_url(data['url']) def _validate_url(self, url): if len(url) == 0: raise ex.InvalidDataException(_("MapR FS url must not be empty")) url = urlparse.urlparse(url) if url.scheme: if url.scheme != "maprfs": raise ex.InvalidDataException(_("URL scheme must be 'maprfs'")) sahara-8.0.0/sahara/service/edp/data_sources/maprfs/__init__.py0000666000175100017510000000000013245514472024525 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/resources/0000775000175100017510000000000013245515027020467 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/edp/resources/edp-main-wrapper.jar0000666000175100017510000001172313245514472024346 0ustar zuulzuul00000000000000PK ²z…E META-INF/PK ±z…E„ÕÄÑg}META-INF/MANIFEST.MFóMÌËLK-.Ñ K-*ÎÌϳR0Ô3àår,JÎÈ,K-Bä¤V”+À$x¹œ‹RKRSt*­ “3R|ËRóx¹œJ3sJÀâÙ‰U¥Ù™‘]¯”lñæzñæF¼\¼\PK ¤t…Eorg/PK ¤t…Eorg/openstack/PK ¤t…Eorg/openstack/sahara/PK ¤t…Eorg/openstack/sahara/edp/PK ±z…Eñ43[W¼Aorg/openstack/sahara/edp/MainWrapper$WrapperSecurityManager.class¥U]SW~„lH¢â`©EM@³ÕŠÚ´•‚Fe&ÔÖ^¹lŽa%ìfv7ŠÅßÐjfÊL{׋þ(Ççìnc$Êt¦7çã=Ïû>Ïû1»ÿ¼ýãOWQKc‹£Èáº:ÝPËMuýZ-ßh(kXJ#…Å–Õ~K-·SøVíߥ¡áކï5¬ dä®Tœî¶lˆŸÆžÙž¬Ò¼â6$m¬/­Žg¯j¦c6¥'0U}n¾0–é4ú‡eä’íØÁ-³…#pÅljˆ$[µù°³³)½ s³EK¾êZfë±éÙêÁ–í Lþè™í¶ôê‡UéÇ‘ÞJËô}IÜrÕõš†Û–Ž˜Ö¶á›[¦g²Ñ6j¦íÄQæ>id­-im¯KoÇö}ÛuŒ8Ÿëa¼,÷eúhó¹´‚(AFÞ8ói?Ír@î*ë ìH]©¯™í¸ 3GˆP”£¡lÕAÒ*Ê”d‚K’^ݵd; ”—ñ¦ Ýëÿp¡È ÈÄæ¨5´qRKV+î) Œ˜®»Ï’k¶R”ë+gIiÓq WuL⤎ÏqN-yÓq'8>ƒSÑ&p²ï5ðl§y§c·ªÁ§*,”§€²1[år§¤&˜Á×pW-÷8±EÜWh¨ ”ÿÇœþä³®ÊÅaå.0è¡èrqÀçÁžN³/G<§72 / ‰8ñ1Böeî¿äŠY~(&ø1FZu…§´j ÷!Õ4îXmœæz†·5Ú‡j~b~áw í…Ø)®ãŒ,"ëÅ Ló6¡1C&„'•­QÓÇ|J?å91ÿ†þ†®¶…K] ï…ÂÞ‡.SÌ2ßCÏFN½Ðøs ©NçqÄC‘ Ù 1Ûq¹ˆ-²u‘8L¶B²Udq·/\,“ÅY‹˜'HQ¼d ¼ô+F^#“xƒ‘©7H@{r€Ô“}ŒæÓ]d¦ºÐócɽ&KÈ"Ýk¼¢=V³¶Åj©’»Uì@bÚÅê¡DrÏAq8±Ç±ž±Z‡¨MêëžUÉL¹®lÔü![m\\½Ì8Ì#•è€:—KvÐjèõûËòr> Eÿ£q¥™ÏŽæã@Ü–OûX‰´¢4?5 ‘ãô5©à’È%ñâüHª_45øB2ÜDw~, þ±Øðü}ôÌk”õå@,BÑXÚľüÏè]{€èÜø&úRSôËãÁXÿ–gLyf»SÁÔÃMìÿ ±lHËŠxw+T<ä‡úÙð8#~ =¾¡¬cĸx„P\üê}…ˆZ‡7qà³YÃÜåG>èç{Ã|îc.½Ì=Š>ô3׳ÄËÂÀFÃA\#}e¸FÏ;äá.ú”ÖŸ‘›Ïq_ /‘Ä7Å·ÃHûVaÜ#8N›.ÚGñ-ôÒ:E¶É"}Ƹ†ðÆé§ñö dˆ£‹¼È¿;¢(°p'ˆý.Ñ"ÁÊ”1ƒWè"®)¼Š×ø¯çuF¢góÖ îšžôÌòlÝâ¤ÀäßÈ#,ãVà´À³oøšsЦú"Q¼I— !M3ÐŒß*çI‡j¶1ûkáPK ²z…EMETA-INF/maven/PK ²z…E(META-INF/maven/org.openstack.sahara.edp/PK ²z…E9META-INF/maven/org.openstack.sahara.edp/edp-main-wrapper/PK ‘t…E¶‡Ø_»@META-INF/maven/org.openstack.sahara.edp/edp-main-wrapper/pom.xmlµU]oÛ6}÷¯¸ö°‘ètÅPdŠ ¯qQm‰DN³>´DÉl$R%©È^±ÿ¾KRJl7Š{²xï¹ç~ÒñëmSÃSšKqœFÓ˜ÈeÁEuܮކ¯‚×É$þ. '—¢®ÒHªŠÔ¥Éeúf¾Èæ!Vkñ·¢fZƒbŸ;®°Íõh‹µätÖ´©€VŠ¡ÏH[k¯¸ÁQ€–¥é©bÈRpm_wæ`PceØï>GE³ Ò,€_gYš Ç]ºz·¼]ÁÝìæf¶X¥ó –7ðf¹¸HWér§·0[|€ßÓÅÅ 0¦aÛVÙú±HnGÈ ;¯Œ±ƒJé Ò-ËyÉsìKT­T… °h™j¸¶‹ÔX^,5o¸¡ÆY¾j*rIhn7GÅÎ2 “u»&aˆj•üÄr¨8¡Ïƒa5 }`b9×Ë+ò2š¢1­Ãžm5ÄÛUö?9ä‹éô”üqu™ahCC.´±ªqqšŸig¾”¹+ûÂ?!œ!|xùqúqmu$˜"ndÁêAï‰#ˆÉÍK0®”ìÚ´H)’-³Uæ÷‘¦ªhÄŠ6&#Âã©2¼¤¹AzÃUö …ÈTLöœ=ÜÝäÔf‹Ùuön¹ŠÉh÷(A–Ì/®á7ú@a–Û‰À•½?wžÙÉBÊ?9‹‰û8œÁ=­p¡É'ŠÙŸŽ¶7»Q 5œéî¶=¾‰{>brhôÀBö¢–´Èd§r 7ªÃÄÇÖ¡¯›hÝñºˆ´óÍrü+Æ–Jökµ‡3Ô65ä{<Ç£ºð""Vx¼F¯{;p*ôA_ïÒ…åR”¼ê”í“=¾+\ôÏ1¾÷݆ªŠï¾ŸXɳ´8•ƒNÿ¯Æ7,¿×fW³ÿÚº÷ŒW9±ŠÂ{úý—5Å#®þ"QD6´² uÏKSjò”4ÂGdÀ#Å>{Iy½﹬zZkTã±ù Dª†šÄ1ßϸ7æ9ÿ7-c mapred.reduce.tasks -1 The default number of reduce tasks per job. Typically set to a prime close to the number of available hosts. Ignored when mapred.job.tracker is "local". Hadoop set this to 1 by default, whereas hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers. hive.exec.reducers.bytes.per.reducer 1000000000 size per reducer.The default is 1G, i.e if the input size is 10G, it will use 10 reducers. hive.exec.reducers.max 999 max number of reducers will be used. If the one specified in the configuration parameter mapred.reduce.tasks is negative, hive will use this one as the max number of reducers when automatically determine number of reducers. hive.cli.print.header false Whether to print the names of the columns in query output. hive.cli.print.current.db false Whether to include the current database in the hive prompt. hive.cli.prompt hive Command line prompt configuration value. Other hiveconf can be used in this configuration value. Variable substitution will only be invoked at the hive cli startup. hive.cli.pretty.output.num.cols -1 The number of columns to use when formatting output generated by the DESCRIBE PRETTY table_name command. If the value of this property is -1, then hive will use the auto-detected terminal width. hive.exec.scratchdir /tmp/hive-${user.name} Scratch space for Hive jobs hive.exec.local.scratchdir /tmp/${user.name} Local scratch space for Hive jobs hive.test.mode false whether hive is running in test mode. If yes, it turns on sampling and prefixes the output tablename hive.test.mode.prefix test_ if hive is running in test mode, prefixes the output table by this string hive.test.mode.samplefreq 32 if hive is running in test mode and table is not bucketed, sampling frequency hive.test.mode.nosamplelist if hive is running in test mode, dont sample the above comma seperated list of tables hive.metastore.uris Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore. javax.jdo.option.ConnectionURL jdbc:derby:;databaseName=metastore_db;create=true JDBC connect string for a JDBC metastore javax.jdo.option.ConnectionDriverName org.apache.derby.jdbc.EmbeddedDriver Driver class name for a JDBC metastore javax.jdo.PersistenceManagerFactoryClass org.datanucleus.jdo.JDOPersistenceManagerFactory class implementing the jdo persistence javax.jdo.option.DetachAllOnCommit true detaches all objects from session so that they can be used after transaction is committed javax.jdo.option.NonTransactionalRead true reads outside of transactions javax.jdo.option.ConnectionUserName APP username to use against metastore database javax.jdo.option.ConnectionPassword mine password to use against metastore database javax.jdo.option.Multithreaded true Set this to true if multiple threads access metastore through JDO concurrently. datanucleus.connectionPoolingType DBCP Uses a DBCP connection pool for JDBC metastore datanucleus.validateTables false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.validateColumns false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.validateConstraints false validates existing schema against code. turn this on if you want to verify existing schema datanucleus.storeManagerType rdbms metadata store type datanucleus.autoCreateSchema true creates necessary schema on a startup if one doesn't exist. set this to false, after creating it once datanucleus.autoStartMechanismMode checked throw exception if metadata tables are incorrect datanucleus.transactionIsolation read-committed Default transaction isolation level for identity generation. datanucleus.cache.level2 false Use a level 2 cache. Turn this off if metadata is changed independently of hive metastore server datanucleus.cache.level2.type SOFT SOFT=soft reference based cache, WEAK=weak reference based cache. datanucleus.identifierFactory datanucleus Name of the identifier factory to use when generating table/column names etc. 'datanucleus' is used for backward compatibility datanucleus.plugin.pluginRegistryBundleCheck LOG Defines what happens when plugin bundles are found and are duplicated [EXCEPTION|LOG|NONE] hive.metastore.warehouse.dir /user/hive/warehouse location of default database for the warehouse hive.metastore.execute.setugi false In unsecure mode, setting this property to true will cause the metastore to execute DFS operations using the client's reported user and group permissions. Note that this property must be set on both the client and server sides. Further note that its best effort. If client sets its to true and server sets it to false, client setting will be ignored. hive.metastore.event.listeners list of comma seperated listeners for metastore events. hive.metastore.partition.inherit.table.properties list of comma seperated keys occurring in table properties which will get inherited to newly created partitions. * implies all the keys will get inherited. hive.metadata.export.location When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, it is the location to which the metadata will be exported. The default is an empty string, which results in the metadata being exported to the current user's home directory on HDFS. hive.metadata.move.exported.metadata.to.trash When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, this setting determines if the metadata that is exported will subsequently be moved to the user's trash directory alongside the dropped table data. This ensures that the metadata will be cleaned up along with the dropped table data. hive.metastore.partition.name.whitelist.pattern Partition names will be checked against this regex pattern and rejected if not matched. hive.metastore.end.function.listeners list of comma separated listeners for the end of metastore functions. hive.metastore.event.expiry.duration 0 Duration after which events expire from events table (in seconds) hive.metastore.event.clean.freq 0 Frequency at which timer task runs to purge expired events in metastore(in seconds). hive.metastore.connect.retries 5 Number of retries while opening a connection to metastore hive.metastore.failure.retries 3 Number of retries upon failure of Thrift metastore calls hive.metastore.client.connect.retry.delay 1 Number of seconds for the client to wait between consecutive connection attempts hive.metastore.client.socket.timeout 20 MetaStore Client socket timeout in seconds hive.metastore.rawstore.impl org.apache.hadoop.hive.metastore.ObjectStore Name of the class that implements org.apache.hadoop.hive.metastore.rawstore interface. This class is used to store and retrieval of raw metadata objects such as table, database hive.metastore.batch.retrieve.max 300 Maximum number of objects (tables/partitions) can be retrieved from metastore in one batch. The higher the number, the less the number of round trips is needed to the Hive metastore server, but it may also cause higher memory requirement at the client side. hive.metastore.batch.retrieve.table.partition.max 1000 Maximum number of table partitions that metastore internally retrieves in one batch. hive.default.fileformat TextFile Default file format for CREATE TABLE statement. Options are TextFile and SequenceFile. Users can explicitly say CREATE TABLE ... STORED AS <TEXTFILE|SEQUENCEFILE> to override hive.fileformat.check true Whether to check file format or not when loading data files hive.map.aggr true Whether to use map-side aggregation in Hive Group By queries hive.groupby.skewindata false Whether there is skew in data to optimize group by queries hive.optimize.multigroupby.common.distincts true Whether to optimize a multi-groupby query with the same distinct. Consider a query like: from src insert overwrite table dest1 select col1, count(distinct colx) group by col1 insert overwrite table dest2 select col2, count(distinct colx) group by col2; With this parameter set to true, first we spray by the distinct value (colx), and then perform the 2 groups bys. This makes sense if map-side aggregation is turned off. However, with maps-side aggregation, it might be useful in some cases to treat the 2 inserts independently, thereby performing the query above in 2MR jobs instead of 3 (due to spraying by distinct key first). If this parameter is turned off, we dont consider the fact that the distinct key is the same across different MR jobs. hive.groupby.mapaggr.checkinterval 100000 Number of rows after which size of the grouping keys/aggregation classes is performed hive.mapred.local.mem 0 For local mode, memory of the mappers/reducers hive.mapjoin.followby.map.aggr.hash.percentmemory 0.3 Portion of total memory to be used by map-side grup aggregation hash table, when this group by is followed by map join hive.map.aggr.hash.force.flush.memory.threshold 0.9 The max memory to be used by map-side grup aggregation hash table, if the memory usage is higher than this number, force to flush data hive.map.aggr.hash.percentmemory 0.5 Portion of total memory to be used by map-side grup aggregation hash table hive.map.aggr.hash.min.reduction 0.5 Hash aggregation will be turned off if the ratio between hash table size and input rows is bigger than this number. Set to 1 to make sure hash aggregation is never turned off. hive.optimize.cp true Whether to enable column pruner hive.optimize.index.filter false Whether to enable automatic use of indexes hive.optimize.index.groupby false Whether to enable optimization of group-by queries using Aggregate indexes. hive.optimize.ppd true Whether to enable predicate pushdown hive.optimize.ppd.storage true Whether to push predicates down into storage handlers. Ignored when hive.optimize.ppd is false. hive.ppd.recognizetransivity true Whether to transitively replicate predicate filters over equijoin conditions. hive.optimize.groupby true Whether to enable the bucketed group by from bucketed partitions/tables. hive.optimize.skewjoin.compiletime false Whether to create a separate plan for skewed keys for the tables in the join. This is based on the skewed keys stored in the metadata. At compile time, the plan is broken into different joins: one for the skewed keys, and the other for the remaining keys. And then, a union is performed for the 2 joins generated above. So unless the same skewed key is present in both the joined tables, the join for the skewed key will be performed as a map-side join. The main difference between this paramater and hive.optimize.skewjoin is that this parameter uses the skew information stored in the metastore to optimize the plan at compile time itself. If there is no skew information in the metadata, this parameter will not have any affect. Both hive.optimize.skewjoin.compiletime and hive.optimize.skewjoin should be set to true. Ideally, hive.optimize.skewjoin should be renamed as hive.optimize.skewjoin.runtime, but not doing so for backward compatibility. If the skew information is correctly stored in the metadata, hive.optimize.skewjoin.compiletime would change the query plan to take care of it, and hive.optimize.skewjoin will be a no-op. hive.optimize.union.remove false Whether to remove the union and push the operators between union and the filesink above union. This avoids an extra scan of the output by union. This is independently useful for union queries, and specially useful when hive.optimize.skewjoin.compiletime is set to true, since an extra union is inserted. The merge is triggered if either of hive.merge.mapfiles or hive.merge.mapredfiles is set to true. If the user has set hive.merge.mapfiles to true and hive.merge.mapredfiles to false, the idea was the number of reducers are few, so the number of files anyway are small. However, with this optimization, we are increasing the number of files possibly by a big margin. So, we merge aggresively. hive.mapred.supports.subdirectories false Whether the version of hadoop which is running supports sub-directories for tables/partitions. Many hive optimizations can be applied if the hadoop version supports sub-directories for tables/partitions. It was added by MAPREDUCE-1501 hive.multigroupby.singlemr false Whether to optimize multi group by query to generate single M/R job plan. If the multi group by query has common group by keys, it will be optimized to generate single M/R job. hive.map.groupby.sorted false If the bucketing/sorting properties of the table exactly match the grouping key, whether to perform the group by in the mapper by using BucketizedHiveInputFormat. The only downside to this is that it limits the number of mappers to the number of files. hive.map.groupby.sorted.testmode false If the bucketing/sorting properties of the table exactly match the grouping key, whether to perform the group by in the mapper by using BucketizedHiveInputFormat. If the test mode is set, the plan is not converted, but a query property is set to denote the same. hive.new.job.grouping.set.cardinality 30 Whether a new map-reduce job should be launched for grouping sets/rollups/cubes. For a query like: select a, b, c, count(1) from T group by a, b, c with rollup; 4 rows are created per row: (a, b, c), (a, b, null), (a, null, null), (null, null, null). This can lead to explosion across map-reduce boundary if the cardinality of T is very high, and map-side aggregation does not do a very good job. This parameter decides if hive should add an additional map-reduce job. If the grouping set cardinality (4 in the example above), is more than this value, a new MR job is added under the assumption that the orginal group by will reduce the data size. hive.join.emit.interval 1000 How many rows in the right-most join operand Hive should buffer before emitting the join result. hive.join.cache.size 25000 How many rows in the joining tables (except the streaming table) should be cached in memory. hive.mapjoin.bucket.cache.size 100 How many values in each keys in the map-joined table should be cached in memory. hive.mapjoin.cache.numrows 25000 How many rows should be cached by jdbm for map join. hive.optimize.skewjoin false Whether to enable skew join optimization. The algorithm is as follows: At runtime, detect the keys with a large skew. Instead of processing those keys, store them temporarily in a hdfs directory. In a follow-up map-reduce job, process those skewed keys. The same key need not be skewed for all the tables, and so, the follow-up map-reduce job (for the skewed keys) would be much faster, since it would be a map-join. hive.skewjoin.key 100000 Determine if we get a skew key in join. If we see more than the specified number of rows with the same key in join operator, we think the key as a skew join key. hive.skewjoin.mapjoin.map.tasks 10000 Determine the number of map task used in the follow up map join job for a skew join. It should be used together with hive.skewjoin.mapjoin.min.split to perform a fine grained control. hive.skewjoin.mapjoin.min.split 33554432 Determine the number of map task at most used in the follow up map join job for a skew join by specifying the minimum split size. It should be used together with hive.skewjoin.mapjoin.map.tasks to perform a fine grained control. hive.mapred.mode nonstrict The mode in which the hive operations are being performed. In strict mode, some risky queries are not allowed to run. They include: Cartesian Product. No partition being picked up for a query. Comparing bigints and strings. Comparing bigints and doubles. Orderby without limit. hive.enforce.bucketmapjoin false If the user asked for bucketed map-side join, and it cannot be performed, should the query fail or not ? For eg, if the buckets in the tables being joined are not a multiple of each other, bucketed map-side join cannot be performed, and the query will fail if hive.enforce.bucketmapjoin is set to true. hive.exec.script.maxerrsize 100000 Maximum number of bytes a script is allowed to emit to standard error (per map-reduce task). This prevents runaway scripts from filling logs partitions to capacity hive.exec.script.allow.partial.consumption false When enabled, this option allows a user script to exit successfully without consuming all the data from the standard input. hive.script.operator.id.env.var HIVE_SCRIPT_OPERATOR_ID Name of the environment variable that holds the unique script operator ID in the user's transform function (the custom mapper/reducer that the user has specified in the query) hive.script.operator.truncate.env false Truncate each environment variable for external script in scripts operator to 20KB (to fit system limits) hive.exec.compress.output false This controls whether the final outputs of a query (to a local/hdfs file or a hive table) is compressed. The compression codec and other options are determined from hadoop config variables mapred.output.compress* hive.exec.compress.intermediate false This controls whether intermediate files produced by hive between multiple map-reduce jobs are compressed. The compression codec and other options are determined from hadoop config variables mapred.output.compress* hive.exec.parallel false Whether to execute jobs in parallel hive.exec.parallel.thread.number 8 How many jobs at most can be executed in parallel hive.exec.rowoffset false Whether to provide the row offset virtual column hive.task.progress false Whether Hive should periodically update task progress counters during execution. Enabling this allows task progress to be monitored more closely in the job tracker, but may impose a performance penalty. This flag is automatically set to true for jobs with hive.exec.dynamic.partition set to true. hive.hwi.war.file lib/hive-hwi-0.11.0.war This sets the path to the HWI war file, relative to ${HIVE_HOME}. hive.hwi.listen.host 0.0.0.0 This is the host address the Hive Web Interface will listen on hive.hwi.listen.port 9999 This is the port the Hive Web Interface will listen on hive.exec.pre.hooks Comma-separated list of pre-execution hooks to be invoked for each statement. A pre-execution hook is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.exec.post.hooks Comma-separated list of post-execution hooks to be invoked for each statement. A post-execution hook is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.exec.failure.hooks Comma-separated list of on-failure hooks to be invoked for each statement. An on-failure hook is specified as the name of Java class which implements the org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext interface. hive.metastore.init.hooks A comma separated list of hooks to be invoked at the beginning of HMSHandler initialization. Aninit hook is specified as the name of Java class which extends org.apache.hadoop.hive.metastore.MetaStoreInitListener. hive.client.stats.publishers Comma-separated list of statistics publishers to be invoked on counters on each job. A client stats publisher is specified as the name of a Java class which implements the org.apache.hadoop.hive.ql.stats.ClientStatsPublisher interface. hive.client.stats.counters Subset of counters that should be of interest for hive.client.stats.publishers (when one wants to limit their publishing). Non-display names should be used hive.merge.mapfiles true Merge small files at the end of a map-only job hive.merge.mapredfiles false Merge small files at the end of a map-reduce job hive.heartbeat.interval 1000 Send a heartbeat after this interval - used by mapjoin and filter operators hive.merge.size.per.task 256000000 Size of merged files at the end of the job hive.merge.smallfiles.avgsize 16000000 When the average output file size of a job is less than this number, Hive will start an additional map-reduce job to merge the output files into bigger files. This is only done for map-only jobs if hive.merge.mapfiles is true, and for map-reduce jobs if hive.merge.mapredfiles is true. hive.mapjoin.smalltable.filesize 25000000 The threshold for the input file size of the small tables; if the file size is smaller than this threshold, it will try to convert the common join into map join hive.ignore.mapjoin.hint true Ignore the mapjoin hint hive.mapjoin.localtask.max.memory.usage 0.90 This number means how much memory the local task can take to hold the key/value into in-memory hash table; If the local task's memory usage is more than this number, the local task will be abort by themself. It means the data of small table is too large to be hold in the memory. hive.mapjoin.followby.gby.localtask.max.memory.usage 0.55 This number means how much memory the local task can take to hold the key/value into in-memory hash table when this map join followed by a group by; If the local task's memory usage is more than this number, the local task will be abort by themself. It means the data of small table is too large to be hold in the memory. hive.mapjoin.check.memory.rows 100000 The number means after how many rows processed it needs to check the memory usage hive.auto.convert.join false Whether Hive enable the optimization about converting common join into mapjoin based on the input file size hive.auto.convert.join.noconditionaltask true Whether Hive enable the optimization about converting common join into mapjoin based on the input file size. If this paramater is on, and the sum of size for n-1 of the tables/partitions for a n-way join is smaller than the specified size, the join is directly converted to a mapjoin (there is no conditional task). hive.auto.convert.join.noconditionaltask.size 10000000 If hive.auto.convert.join.noconditionaltask is off, this parameter does not take affect. However, if it is on, and the sum of size for n-1 of the tables/partitions for a n-way join is smaller than this size, the join is directly converted to a mapjoin(there is no conditional task). The default is 10MB hive.optimize.mapjoin.mapreduce false If hive.auto.convert.join is off, this parameter does not take affect. If it is on, and if there are map-join jobs followed by a map-reduce job (for e.g a group by), each map-only job is merged with the following map-reduce job. hive.script.auto.progress false Whether Hive Tranform/Map/Reduce Clause should automatically send progress information to TaskTracker to avoid the task getting killed because of inactivity. Hive sends progress information when the script is outputting to stderr. This option removes the need of periodically producing stderr messages, but users should be cautious because this may prevent infinite loops in the scripts to be killed by TaskTracker. hive.script.serde org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe The default serde for trasmitting input data to and reading output data from the user scripts. hive.binary.record.max.length 1000 Read from a binary stream and treat each hive.binary.record.max.length bytes as a record. The last record before the end of stream can have less than hive.binary.record.max.length bytes hive.script.recordreader org.apache.hadoop.hive.ql.exec.TextRecordReader The default record reader for reading data from the user scripts. hive.script.recordwriter org.apache.hadoop.hive.ql.exec.TextRecordWriter The default record writer for writing data to the user scripts. hive.input.format org.apache.hadoop.hive.ql.io.CombineHiveInputFormat The default input format. Set this to HiveInputFormat if you encounter problems with CombineHiveInputFormat. hive.udtf.auto.progress false Whether Hive should automatically send progress information to TaskTracker when using UDTF's to prevent the task getting killed because of inactivity. Users should be cautious because this may prevent TaskTracker from killing tasks with infinte loops. hive.mapred.reduce.tasks.speculative.execution true Whether speculative execution for reducers should be turned on. hive.exec.counters.pull.interval 1000 The interval with which to poll the JobTracker for the counters the running job. The smaller it is the more load there will be on the jobtracker, the higher it is the less granular the caught will be. hive.querylog.location /tmp/${user.name} Location of Hive run time structured log file hive.querylog.enable.plan.progress true Whether to log the plan's progress every time a job's progress is checked. These logs are written to the location specified by hive.querylog.location hive.querylog.plan.progress.interval 60000 The interval to wait between logging the plan's progress in milliseconds. If there is a whole number percentage change in the progress of the mappers or the reducers, the progress is logged regardless of this value. The actual interval will be the ceiling of (this value divided by the value of hive.exec.counters.pull.interval) multiplied by the value of hive.exec.counters.pull.interval I.e. if it is not divide evenly by the value of hive.exec.counters.pull.interval it will be logged less frequently than specified. This only has an effect if hive.querylog.enable.plan.progress is set to true. hive.enforce.bucketing false Whether bucketing is enforced. If true, while inserting into the table, bucketing is enforced. hive.enforce.sorting false Whether sorting is enforced. If true, while inserting into the table, sorting is enforced. hive.optimize.bucketingsorting true If hive.enforce.bucketing or hive.enforce.sorting is true, dont create a reducer for enforcing bucketing/sorting for queries of the form: insert overwrite table T2 select * from T1; where T1 and T2 are bucketed/sorted by the same keys into the same number of buckets. hive.enforce.sortmergebucketmapjoin false If the user asked for sort-merge bucketed map-side join, and it cannot be performed, should the query fail or not ? hive.auto.convert.sortmerge.join false Will the join be automatically converted to a sort-merge join, if the joined tables pass the criteria for sort-merge join. hive.auto.convert.sortmerge.join.bigtable.selection.policy org.apache.hadoop.hive.ql.optimizer.AvgPartitionSizeBasedBigTableSelectorForAutoSMJ The policy to choose the big table for automatic conversion to sort-merge join. By default, the table with the largest partitions is assigned the big table. All policies are: . based on position of the table - the leftmost table is selected org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSMJ. . based on total size (all the partitions selected in the query) of the table org.apache.hadoop.hive.ql.optimizer.TableSizeBasedBigTableSelectorForAutoSMJ. . based on average size (all the partitions selected in the query) of the table org.apache.hadoop.hive.ql.optimizer.AvgPartitionSizeBasedBigTableSelectorForAutoSMJ. New policies can be added in future. hive.metastore.ds.connection.url.hook Name of the hook to use for retriving the JDO connection URL. If empty, the value in javax.jdo.option.ConnectionURL is used hive.metastore.ds.retry.attempts 1 The number of times to retry a metastore call if there were a connection error hive.metastore.ds.retry.interval 1000 The number of miliseconds between metastore retry attempts hive.metastore.server.min.threads 200 Minimum number of worker threads in the Thrift server's pool. hive.metastore.server.max.threads 100000 Maximum number of worker threads in the Thrift server's pool. hive.metastore.server.tcp.keepalive true Whether to enable TCP keepalive for the metastore server. Keepalive will prevent accumulation of half-open connections. hive.metastore.sasl.enabled false If true, the metastore thrift interface will be secured with SASL. Clients must authenticate with Kerberos. hive.metastore.thrift.framed.transport.enabled false If true, the metastore thrift interface will use TFramedTransport. When false (default) a standard TTransport is used. hive.metastore.kerberos.keytab.file The path to the Kerberos Keytab file containing the metastore thrift server's service principal. hive.metastore.kerberos.principal hive-metastore/_HOST@EXAMPLE.COM The service principal for the metastore thrift server. The special string _HOST will be replaced automatically with the correct host name. hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.MemoryTokenStore The delegation token store implementation. Set to org.apache.hadoop.hive.thrift.ZooKeeperTokenStore for load-balanced cluster. hive.cluster.delegation.token.store.zookeeper.connectString localhost:2181 The ZooKeeper token store connect string. hive.cluster.delegation.token.store.zookeeper.znode /hive/cluster/delegation The root path for token store data. hive.cluster.delegation.token.store.zookeeper.acl sasl:hive/host1@EXAMPLE.COM:cdrwa,sasl:hive/host2@EXAMPLE.COM:cdrwa ACL for token store entries. List comma separated all server principals for the cluster. hive.metastore.cache.pinobjtypes Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order List of comma separated metastore object types that should be pinned in the cache hive.optimize.reducededuplication true Remove extra map-reduce jobs if the data is already clustered by the same key which needs to be used again. This should always be set to true. Since it is a new feature, it has been made configurable. hive.optimize.reducededuplication.min.reducer 4 Reduce deduplication merges two RSs by moving key/parts/reducer-num of the child RS to parent RS. That means if reducer-num of the child RS is fixed (order by or forced bucketing) and small, it can make very slow, single MR. The optimization will be disabled if number of reducers is less than specified value. hive.exec.dynamic.partition true Whether or not to allow dynamic partitions in DML/DDL. hive.exec.dynamic.partition.mode strict In strict mode, the user must specify at least one static partition in case the user accidentally overwrites all partitions. hive.exec.max.dynamic.partitions 1000 Maximum number of dynamic partitions allowed to be created in total. hive.exec.max.dynamic.partitions.pernode 100 Maximum number of dynamic partitions allowed to be created in each mapper/reducer node. hive.exec.max.created.files 100000 Maximum number of HDFS files created by all mappers/reducers in a MapReduce job. hive.exec.default.partition.name __HIVE_DEFAULT_PARTITION__ The default partition name in case the dynamic partition column value is null/empty string or anyother values that cannot be escaped. This value must not contain any special character used in HDFS URI (e.g., ':', '%', '/' etc). The user has to be aware that the dynamic partition value should not contain this value to avoid confusions. hive.stats.dbclass jdbc:derby The default database that stores temporary hive statistics. hive.stats.autogather true A flag to gather statistics automatically during the INSERT OVERWRITE command. hive.stats.jdbcdriver org.apache.derby.jdbc.EmbeddedDriver The JDBC driver for the database that stores temporary hive statistics. hive.stats.dbconnectionstring jdbc:derby:;databaseName=TempStatsStore;create=true The default connection string for the database that stores temporary hive statistics. hive.stats.default.publisher The Java class (implementing the StatsPublisher interface) that is used by default if hive.stats.dbclass is not JDBC or HBase. hive.stats.default.aggregator The Java class (implementing the StatsAggregator interface) that is used by default if hive.stats.dbclass is not JDBC or HBase. hive.stats.jdbc.timeout 30 Timeout value (number of seconds) used by JDBC connection and statements. hive.stats.retries.max 0 Maximum number of retries when stats publisher/aggregator got an exception updating intermediate database. Default is no tries on failures. hive.stats.retries.wait 3000 The base waiting window (in milliseconds) before the next retry. The actual wait time is calculated by baseWindow * failues baseWindow * (failure 1) * (random number between [0.0,1.0]). hive.stats.reliable false Whether queries will fail because stats cannot be collected completely accurately. If this is set to true, reading/writing from/into a partition may fail becuase the stats could not be computed accurately. hive.stats.collect.tablekeys false Whether join and group by keys on tables are derived and maintained in the QueryPlan. This is useful to identify how tables are accessed and to determine if they should be bucketed. hive.stats.collect.scancols false Whether column accesses are tracked in the QueryPlan. This is useful to identify how tables are accessed and to determine if there are wasted columns that can be trimmed. hive.stats.ndv.error 20.0 Standard error expressed in percentage. Provides a tradeoff between accuracy and compute cost.A lower value for error indicates higher accuracy and a higher compute cost. hive.stats.key.prefix.max.length 200 Determines if when the prefix of the key used for intermediate stats collection exceeds a certain length, a hash of the key is used instead. If the value < 0 then hashing is never used, if the value >= 0 then hashing is used only when the key prefixes length exceeds that value. The key prefix is defined as everything preceding the task ID in the key. hive.support.concurrency false Whether hive supports concurrency or not. A zookeeper instance must be up and running for the default hive lock manager to support read-write locks. hive.lock.numretries 100 The number of times you want to try to get all the locks hive.unlock.numretries 10 The number of times you want to retry to do one unlock hive.lock.sleep.between.retries 60 The sleep time (in seconds) between various retries hive.zookeeper.quorum The list of zookeeper servers to talk to. This is only needed for read/write locks. hive.zookeeper.client.port 2181 The port of zookeeper servers to talk to. This is only needed for read/write locks. hive.zookeeper.session.timeout 600000 Zookeeper client's session timeout. The client is disconnected, and as a result, all locks released, if a heartbeat is not sent in the timeout. hive.zookeeper.namespace hive_zookeeper_namespace The parent node under which all zookeeper nodes are created. hive.zookeeper.clean.extra.nodes false Clean extra nodes at the end of the session. fs.har.impl org.apache.hadoop.hive.shims.HiveHarFileSystem The implementation for accessing Hadoop Archives. Note that this won't be applicable to Hadoop vers less than 0.20 hive.archive.enabled false Whether archiving operations are permitted hive.fetch.output.serde org.apache.hadoop.hive.serde2.DelimitedJSONSerDe The serde used by FetchTask to serialize the fetch output. hive.exec.mode.local.auto false Let hive determine whether to run in local mode automatically hive.exec.drop.ignorenonexistent true Do not report an error if DROP TABLE/VIEW specifies a non-existent table/view hive.exec.show.job.failure.debug.info true If a job fails, whether to provide a link in the CLI to the task with the most failures, along with debugging hints if applicable. hive.auto.progress.timeout 0 How long to run autoprogressor for the script/UDTF operators (in seconds). Set to 0 for forever. hive.hbase.wal.enabled true Whether writes to HBase should be forced to the write-ahead log. Disabling this improves HBase write performance at the risk of lost writes in case of a crash. hive.table.parameters.default Default property values for newly created tables hive.entity.separator @ Separator used to construct names of tables and partitions. For example, dbname@tablename@partitionname hive.ddl.createtablelike.properties.whitelist Table Properties to copy over when executing a Create Table Like. hive.variable.substitute true This enables substitution using syntax like ${var} ${system:var} and ${env:var}. hive.variable.substitute.depth 40 The maximum replacements the substitution engine will do. hive.conf.validation true Eables type checking for registered hive configurations hive.security.authorization.enabled false enable or disable the hive client authorization hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider the hive client authorization manager class name. The user defined authorization class should implement interface org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider. hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.DefaultHiveMetastoreAuthorizationProvider authorization manager class name to be used in the metastore for authorization. The user defined authorization class should implement interface org.apache.hadoop.hive.ql.security.authorization.HiveMetastoreAuthorizationProvider. hive.security.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator hive client authenticator manager class name. The user defined authenticator should implement interface org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider. hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator authenticator manager class name to be used in the metastore for authentication. The user defined authenticator should implement interface org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider. hive.security.authorization.createtable.user.grants the privileges automatically granted to some users whenever a table gets created. An example like "userX,userY:select;userZ:create" will grant select privilege to userX and userY, and grant create privilege to userZ whenever a new table created. hive.security.authorization.createtable.group.grants the privileges automatically granted to some groups whenever a table gets created. An example like "groupX,groupY:select;groupZ:create" will grant select privilege to groupX and groupY, and grant create privilege to groupZ whenever a new table created. hive.security.authorization.createtable.role.grants the privileges automatically granted to some roles whenever a table gets created. An example like "roleX,roleY:select;roleZ:create" will grant select privilege to roleX and roleY, and grant create privilege to roleZ whenever a new table created. hive.security.authorization.createtable.owner.grants the privileges automatically granted to the owner whenever a table gets created. An example like "select,drop" will grant select and drop privilege to the owner of the table hive.metastore.authorization.storage.checks false Should the metastore do authorization checks against the underlying storage for operations like drop-partition (disallow the drop-partition if the user in question doesn't have permissions to delete the corresponding directory on the storage). hive.error.on.empty.partition false Whether to throw an excpetion if dynamic partition insert generates empty results. hive.index.compact.file.ignore.hdfs false True the hdfs location stored in the index file will be igbored at runtime. If the data got moved or the name of the cluster got changed, the index data should still be usable. hive.optimize.index.filter.compact.minsize 5368709120 Minimum size (in bytes) of the inputs on which a compact index is automatically used. hive.optimize.index.filter.compact.maxsize -1 Maximum size (in bytes) of the inputs on which a compact index is automatically used. A negative number is equivalent to infinity. hive.index.compact.query.max.size 10737418240 The maximum number of bytes that a query using the compact index can read. Negative value is equivalent to infinity. hive.index.compact.query.max.entries 10000000 The maximum number of index entries to read during a query that uses the compact index. Negative value is equivalent to infinity. hive.index.compact.binary.search true Whether or not to use a binary search to find the entries in an index table that match the filter, where possible hive.exim.uri.scheme.whitelist hdfs,pfile A comma separated list of acceptable URI schemes for import and export. hive.lock.mapred.only.operation false This param is to control whether or not only do lock on queries that need to execute at least one mapred job. hive.limit.row.max.size 100000 When trying a smaller subset of data for simple LIMIT, how much size we need to guarantee each row to have at least. hive.limit.optimize.limit.file 10 When trying a smaller subset of data for simple LIMIT, maximum number of files we can sample. hive.limit.optimize.enable false Whether to enable to optimization to trying a smaller subset of data for simple LIMIT first. hive.limit.optimize.fetch.max 50000 Maximum number of rows allowed for a smaller subset of data for simple LIMIT, if it is a fetch query. Insert queries are not restricted by this limit. hive.rework.mapredwork false should rework the mapred work or not. This is first introduced by SymlinkTextInputFormat to replace symlink files with real paths at compile time. hive.exec.concatenate.check.index true If this sets to true, hive will throw error when doing 'alter table tbl_name [partSpec] concatenate' on a table/partition that has indexes on it. The reason the user want to set this to true is because it can help user to avoid handling all index drop, recreation, rebuild work. This is very helpful for tables with thousands of partitions. hive.sample.seednumber 0 A number used to percentage sampling. By changing this number, user will change the subsets of data sampled. hive.io.exception.handlers A list of io exception handler class names. This is used to construct a list exception handlers to handle exceptions thrown by record readers hive.autogen.columnalias.prefix.label _c String used as a prefix when auto generating column alias. By default the prefix label will be appended with a column position number to form the column alias. Auto generation would happen if an aggregate function is used in a select clause without an explicit alias. hive.autogen.columnalias.prefix.includefuncname false Whether to include function name in the column alias auto generated by hive. hive.exec.perf.logger org.apache.hadoop.hive.ql.log.PerfLogger The class responsible logging client side performance metrics. Must be a subclass of org.apache.hadoop.hive.ql.log.PerfLogger hive.start.cleanup.scratchdir false To cleanup the hive scratchdir while starting the hive server hive.output.file.extension String used as a file extension for output files. If not set, defaults to the codec extension for text files (e.g. ".gz"), or no extension otherwise. hive.insert.into.multilevel.dirs false Where to insert into multilevel directories like "insert directory '/HIVEFT25686/chinna/' from table" hive.warehouse.subdir.inherit.perms false Set this to true if the the table directories should inherit the permission of the warehouse or database directory instead of being created with the permissions derived from dfs umask hive.exec.job.debug.capture.stacktraces true Whether or not stack traces parsed from the task logs of a sampled failed task for each failed job should be stored in the SessionState hive.exec.driver.run.hooks A comma separated list of hooks which implement HiveDriverRunHook and will be run at the beginning and end of Driver.run, these will be run in the order specified hive.ddl.output.format text The data format to use for DDL output. One of "text" (for human readable text) or "json" (for a json object). hive.transform.escape.input false This adds an option to escape special chars (newlines, carriage returns and tabs) when they are passed to the user script. This is useful if the hive tables can contain data that contains special characters. hive.exec.rcfile.use.explicit.header true If this is set the header for RC Files will simply be RCF. If this is not set the header will be that borrowed from sequence files, e.g. SEQ- followed by the input and output RC File formats. hive.multi.insert.move.tasks.share.dependencies false If this is set all move tasks for tables/partitions (not directories) at the end of a multi-insert query will only begin once the dependencies for all these move tasks have been met. Advantages: If concurrency is enabled, the locks will only be released once the query has finished, so with this config enabled, the time when the table/partition is generated will be much closer to when the lock on it is released. Disadvantages: If concurrency is not enabled, with this disabled, the tables/partitions which are produced by this query and finish earlier will be available for querying much earlier. Since the locks are only released once the query finishes, this does not apply if concurrency is enabled. hive.fetch.task.conversion minimal Some select queries can be converted to single FETCH task minimizing latency. Currently the query should be single sourced not having any subquery and should not have any aggregations or distincts (which incurrs RS), lateral views and joins. 1. minimal : SELECT STAR, FILTER on partition columns, LIMIT only 2. more : SELECT, FILTER, LIMIT only (TABLESAMPLE, virtual columns) hive.hmshandler.retry.attempts 1 The number of times to retry a HMSHandler call if there were a connection error hive.hmshandler.retry.interval 1000 The number of miliseconds between HMSHandler retry attempts hive.server.read.socket.timeout 10 Timeout for the HiveServer to close the connection if no response from the client in N seconds, defaults to 10 seconds. hive.server.tcp.keepalive true Whether to enable TCP keepalive for the Hive server. Keepalive will prevent accumulation of half-open connections. hive.decode.partition.name false Whether to show the unquoted partition names in query results. hive.log4j.file Hive log4j configuration file. If the property is not set, then logging will be initialized using hive-log4j.properties found on the classpath. If the property is set, the value must be a valid URI (java.net.URI, e.g. "file:///tmp/my-logging.properties"), which you can then extract a URL from and pass to PropertyConfigurator.configure(URL). hive.exec.log4j.file Hive log4j configuration file for execution mode(sub command). If the property is not set, then logging will be initialized using hive-exec-log4j.properties found on the classpath. If the property is set, the value must be a valid URI (java.net.URI, e.g. "file:///tmp/my-logging.properties"), which you can then extract a URL from and pass to PropertyConfigurator.configure(URL). hive.exec.infer.bucket.sort false If this is set, when writing partitions, the metadata will include the bucketing/sorting properties with which the data was written if any (this will not overwrite the metadata inherited from the table if the table is bucketed/sorted) hive.exec.infer.bucket.sort.num.buckets.power.two false If this is set, when setting the number of reducers for the map reduce task which writes the final output files, it will choose a number which is a power of two, unless the user specifies the number of reducers to use using mapred.reduce.tasks. The number of reducers may be set to a power of two, only to be followed by a merge task meaning preventing anything from being inferred. With hive.exec.infer.bucket.sort set to true: Advantages: If this is not set, the number of buckets for partitions will seem arbitrary, which means that the number of mappers used for optimized joins, for example, will be very low. With this set, since the number of buckets used for any partition is a power of two, the number of mappers used for optimized joins will be the least number of buckets used by any partition being joined. Disadvantages: This may mean a much larger or much smaller number of reducers being used in the final map reduce job, e.g. if a job was originally going to take 257 reducers, it will now take 512 reducers, similarly if the max number of reducers is 511, and a job was going to use this many, it will now use 256 reducers. hive.groupby.orderby.position.alias false Whether to enable using Column Position Alias in Group By or Order By hive.server2.thrift.min.worker.threads 5 Minimum number of Thrift worker threads hive.server2.thrift.max.worker.threads 100 Maximum number of Thrift worker threads hive.server2.thrift.port 10000 Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT hive.server2.thrift.bind.host localhost Bind host on which to run the HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_BIND_HOST hive.server2.authentication NONE Client authentication types. NONE: no authentication check LDAP: LDAP/AD based authentication KERBEROS: Kerberos/GSSAPI authentication CUSTOM: Custom authentication provider (Use with property hive.server2.custom.authentication.class) hive.server2.custom.authentication.class Custom authentication class. Used when property 'hive.server2.authentication' is set to 'CUSTOM'. Provided class must be a proper implementation of the interface org.apache.hive.service.auth.PasswdAuthenticationProvider. HiveServer2 will call its Authenticate(user, passed) method to authenticate requests. The implementation may optionally extend the Hadoop's org.apache.hadoop.conf.Configured class to grab Hive's Configuration object. >hive.server2.authentication.kerberos.principal Kerberos server principal >hive.server2.authentication.kerberos.keytab Kerberos keytab file for server principal hive.server2.authentication.ldap.url LDAP connection URL hive.server2.authentication.ldap.baseDN LDAP base DN hive.server2.enable.doAs true Setting this property to true will have hive server2 execute hive operations as the user making the calls to it. sahara-8.0.0/sahara/service/edp/resources/edp-spark-wrapper.jar0000666000175100017510000000664513245514472024551 0ustar zuulzuul00000000000000PK "u5F META-INF/PK !u5FHÚÉÔmƒMETA-INF/MANIFEST.MFóMÌËLK-.Ñ K-*ÎÌϳR0Ô3àår,JÎÈ,K-Bä¤V”+À$x¹œJ3sJt*­Jr“³+y¹œ‹RKRSÀbމɩ ¾‰e©y ÆzFzF)º^)Ù K,ô âLy¹x¹PK "u5Forg/PK "u5Forg/openstack/PK "u5Forg/openstack/sahara/PK "u5Forg/openstack/sahara/edp/PK "u5F¡öàɾ+org/openstack/sahara/edp/SparkWrapper.classT]OA=C·Ý¶T¡* ¨(íB»ŠŸ|ØHL„"yÚi»Ðîn¦[Ÿå‹‰Ïþßw¶%¨ÆMvfï½çÜ9÷Þiüúz`¹(˜ c:Š Ra¤u:f”5«#EfC:†£x„Ç:æBK–my9†@*½Ã ­8%ÁЗ·l±Ñ¬¹ÍjäIä"¯ípi)»íÔ¼ªÕ`˜Î;²b:®°/™ ^å’›¢äš—Ë£]É]WÈE"Ô¹e3$SûùC~ÌÍ·+fÁ“–]Yôç²B º„z‹Ž]¶*+5Þ L¼â»ª ¯ê”F;‚R”k¢è™ë~ŒP%¢%ª¾×Û,Ý'˾€äåjOÝ‹Š¯žºdä(etõcQ¸žåØ OÈ.8MYo-ʼnw6!«ø1ÜDõ-Ë]^¬Šl•—Çͪ³+~MÉUžâõŸ—JoD™7kÞ–høÙiLWÄ0ô_í[ Ïñâ’óàÚ¡ãe óXбÃ^1Lþ×i8RmW¥sÒjŒ^vä¯Ó×dªËl» +RÞEç_wáì_ã¤ÿ5Õá¿ÅèZXö±sDÒæ;iõáÒ1mWúºë¢ƒMϪ™ËRòÓ†!ÝÓÍòá(¹‘ê’jm-ÝÅ‹qÜ ŸªzzÀÔe µŸ,“vF{ÐøöÉÇi ùÎ0´ÆZ `ö†$”"ÿ„NH`?Ñs†@~&¡ißÜ $B…ÏÐ×g™á½€˜#;²khsÛÈ$¢mX¯‚-hFÐ8?Cì;z‚#Z›4ô9JRÀ—4EG+1=TIˆ*HR SÉȉ{Gò>8%9×’…[ö‹yŒRI¬â6îPŽ ý5‘/@Ø1Ü%ŸFâ¸GŒ îKÙb޳…'üŠiäyHï„ß°ÉßPK "u5FMETA-INF/maven/PK "u5F(META-INF/maven/org.openstack.sahara.edp/PK "u5F:META-INF/maven/org.openstack.sahara.edp/edp-spark-wrapper/PK t5FÀÎgùRªAMETA-INF/maven/org.openstack.sahara.edp/edp-spark-wrapper/pom.xmlµUQoÛ6~÷¯¸ }Ø€Htº¢(2E…Û¸¨ÐÔ"§YŸ Z¢d6©’TdcØï‘”Û͆bÀž,Þ}üî»»OrüzÛÔpÏ”æRœ§Ñ4&rYpQ7«wá«àu2‰ à À%ϙЬ€NLÙ0˜µ4ÇŸ!sŸ<<¦ð«C*øídØÉº! tš!×PòšÛæ¬5Àä²ikNEΠçfãÊ $R|(äÚPDSÄ·x*÷q@Í¡cÚ3Bú¾¨SIU‘Ú£4¹LßÎÙGE³ Ò,€7³,ÍNã6]½_Þ¬àvv}=[¬ÒyËkx»\\¤«t¹ÀÓ;˜->Çtqq Ç„eضUV?Šäv„¬°óÊ;PJ/H·,ç%ϱ/Qu´bPI4‚Àv eªáÚ.R£¼YjÞpC‹üÐTäŠÐÜnŽŠe&ëvMÂ=Ô*ù•åÐqBŸÃjzÏÄþr®–É‹hŠNIJ{¶ÕüoWÙÿîϧÓSòçÇË ¯64äBëwOó3í—2w²¢ üÂÂû_¦_¦ÑVA‚%âF¬üž8‚˜ļãJÉ®M‹™"Ù2«2¿‹4ÝPE#V´1O•á%Í 0ê–ª»°WèD¦b²—õðáåMN­‚0[Ì®²÷ËULƸG Ú°d~q·žÈÙ ³Ô1q9Þïh… L¾R,öx´½Ø âUÙàn»ã÷"qŸ‹˜=°½¨%-2Ù©¯Õ±˜G Þ'Ѻãui—›ÕøWŒ•JöµÚ€CŽê›ê=œÇãÑÊxS 7Vv¼6ïû6à(Tè/ý¸:w-—¢äU§œIã˜ñ]á^_ÆdxÞOª*f|zx~d%OÒâT:ý¿ß°üN›]Íþkë>3¾º‰u¾—ÏþZSü÷áêoEdC )ñõèyiJM‹FøÑð@±Ï^R^/Å'.ëžÖÝx>¸"UCM☇ç'ÒóTþ§–1ž´wïàÖ£'“ïPK "u5F—‚æç~‡HMETA-INF/maven/org.openstack.sahara.edp/edp-spark-wrapper/pom.properties ʽ Â0à=OqÐ9!©u)tpÀHÁùlÏZ Éq‰ßÞŒ|Õ fáùƒ ®Tõ(:c€ÚkÚÆµv{ßCmÝV­$iŽ¡sÆ«ýuw÷Ç[¯&‰>]”ÉD¦2‹IøFAC#+”<¿pÈåêÄ(‹þ 2“¨?PK "u5F íAMETA-INF/PK !u5FHÚÉÔmƒ¤'META-INF/MANIFEST.MFPK "u5FýAÆorg/PK "u5FýAèorg/openstack/PK "u5FýAorg/openstack/sahara/PK "u5FýAGorg/openstack/sahara/edp/PK "u5F¡öàɾ+´~org/openstack/sahara/edp/SparkWrapper.classPK "u5Fÿÿ…META-INF/maven/PK "u5F(ÿÿ²META-INF/maven/org.openstack.sahara.edp/PK "u5F:ÿÿøMETA-INF/maven/org.openstack.sahara.edp/edp-spark-wrapper/PK t5FÀÎgùRªA´PMETA-INF/maven/org.openstack.sahara.edp/edp-spark-wrapper/pom.xmlPK "u5F—‚æç~‡H´ META-INF/maven/org.openstack.sahara.edp/edp-spark-wrapper/pom.propertiesPK ªå sahara-8.0.0/sahara/service/edp/resources/mapred-default.xml0000666000175100017510000021053413245514472024115 0ustar zuulzuul00000000000000 mapreduce.jobtracker.jobhistory.location If job tracker is static the history files are stored in this single well known place. If No value is set here, by default, it is in the local file system at ${hadoop.log.dir}/history. mapreduce.jobtracker.jobhistory.task.numberprogresssplits 12 Every task attempt progresses from 0.0 to 1.0 [unless it fails or is killed]. We record, for each task attempt, certain statistics over each twelfth of the progress range. You can change the number of intervals we divide the entire range of progress into by setting this property. Higher values give more precision to the recorded data, but costs more memory in the job tracker at runtime. Each increment in this attribute costs 16 bytes per running task. mapreduce.job.userhistorylocation User can specify a location to store the history files of a particular job. If nothing is specified, the logs are stored in output directory. The files are stored in "_logs/history/" in the directory. User can stop logging by giving the value "none". mapreduce.jobtracker.jobhistory.completed.location The completed job history files are stored at this single well known location. If nothing is specified, the files are stored at ${mapreduce.jobtracker.jobhistory.location}/done. mapreduce.job.committer.setup.cleanup.needed true true, if job needs job-setup and job-cleanup. false, otherwise mapreduce.task.io.sort.factor 10 The number of streams to merge at once while sorting files. This determines the number of open file handles. mapreduce.task.io.sort.mb 100 The total amount of buffer memory to use while sorting files, in megabytes. By default, gives each merge stream 1MB, which should minimize seeks. mapreduce.map.sort.spill.percent 0.80 The soft limit in the serialization buffer. Once reached, a thread will begin to spill the contents to disk in the background. Note that collection will not block if this threshold is exceeded while a spill is already in progress, so spills may be larger than this threshold when it is set to less than .5 mapreduce.jobtracker.address local The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. mapreduce.local.clientfactory.class.name org.apache.hadoop.mapred.LocalClientFactory This the client factory that is responsible for creating local job runner client mapreduce.jobtracker.http.address 0.0.0.0:50030 The job tracker http server address and port the server will listen on. If the port is 0 then the server will start on a free port. mapreduce.jobtracker.handler.count 10 The number of server threads for the JobTracker. This should be roughly 4% of the number of tasktracker nodes. mapreduce.tasktracker.report.address 127.0.0.1:0 The interface and port that task tracker server listens on. Since it is only connected to by the tasks, it uses the local interface. EXPERT ONLY. Should only be changed if your host does not have the loopback interface. mapreduce.cluster.local.dir ${hadoop.tmp.dir}/mapred/local The local directory where MapReduce stores intermediate data files. May be a comma-separated list of directories on different devices in order to spread disk i/o. Directories that do not exist are ignored. mapreduce.jobtracker.system.dir ${hadoop.tmp.dir}/mapred/system The directory where MapReduce stores control files. mapreduce.jobtracker.staging.root.dir ${hadoop.tmp.dir}/mapred/staging The root of the staging area for users' job files In practice, this should be the directory where users' home directories are located (usually /user) mapreduce.cluster.temp.dir ${hadoop.tmp.dir}/mapred/temp A shared directory for temporary files. mapreduce.tasktracker.local.dir.minspacestart 0 If the space in mapreduce.cluster.local.dir drops under this, do not ask for more tasks. Value in bytes. mapreduce.tasktracker.local.dir.minspacekill 0 If the space in mapreduce.cluster.local.dir drops under this, do not ask more tasks until all the current ones have finished and cleaned up. Also, to save the rest of the tasks we have running, kill one of them, to clean up some space. Start with the reduce tasks, then go with the ones that have finished the least. Value in bytes. mapreduce.jobtracker.expire.trackers.interval 600000 Expert: The time-interval, in miliseconds, after which a tasktracker is declared 'lost' if it doesn't send heartbeats. mapreduce.tasktracker.instrumentation org.apache.hadoop.mapred.TaskTrackerMetricsInst Expert: The instrumentation class to associate with each TaskTracker. mapreduce.tasktracker.resourcecalculatorplugin Name of the class whose instance will be used to query resource information on the tasktracker. The class must be an instance of org.apache.hadoop.util.ResourceCalculatorPlugin. If the value is null, the tasktracker attempts to use a class appropriate to the platform. Currently, the only platform supported is Linux. mapreduce.tasktracker.taskmemorymanager.monitoringinterval 5000 The interval, in milliseconds, for which the tasktracker waits between two cycles of monitoring its tasks' memory usage. Used only if tasks' memory management is enabled via mapred.tasktracker.tasks.maxmemory. mapreduce.tasktracker.tasks.sleeptimebeforesigkill 5000 The time, in milliseconds, the tasktracker waits for sending a SIGKILL to a task, after it has been sent a SIGTERM. This is currently not used on WINDOWS where tasks are just sent a SIGTERM. mapreduce.job.maps 2 The default number of map tasks per job. Ignored when mapreduce.jobtracker.address is "local". mapreduce.job.reduces 1 The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapreduce.jobtracker.address is "local". mapreduce.jobtracker.restart.recover false "true" to enable (job) recovery upon restart, "false" to start afresh mapreduce.jobtracker.jobhistory.block.size 3145728 The block size of the job history file. Since the job recovery uses job history, its important to dump job history to disk as soon as possible. Note that this is an expert level parameter. The default value is set to 3 MB. mapreduce.jobtracker.taskscheduler org.apache.hadoop.mapred.JobQueueTaskScheduler The class responsible for scheduling the tasks. mapreduce.job.reducer.preempt.delay.sec 0 The threshold in terms of seconds after which an unsatisfied mapper request triggers reducer preemption to free space. Default 0 implies that the reduces should be preempted immediately after allocation if there is currently no room for newly allocated mappers. mapreduce.job.max.split.locations 10 The max number of block locations to store for each split for locality calculation. mapreduce.job.split.metainfo.maxsize 10000000 The maximum permissible size of the split metainfo file. The JobTracker won't attempt to read split metainfo files bigger than the configured value. No limits if set to -1. mapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob The maximum number of running tasks for a job before it gets preempted. No limits if undefined. mapreduce.map.maxattempts 4 Expert: The maximum number of attempts per map task. In other words, framework will try to execute a map task these many number of times before giving up on it. mapreduce.reduce.maxattempts 4 Expert: The maximum number of attempts per reduce task. In other words, framework will try to execute a reduce task these many number of times before giving up on it. mapreduce.reduce.shuffle.fetch.retry.enabled ${yarn.nodemanager.recovery.enabled} Set to enable fetch retry during host restart. mapreduce.reduce.shuffle.fetch.retry.interval-ms 1000 Time of interval that fetcher retry to fetch again when some non-fatal failure happens because of some events like NM restart. mapreduce.reduce.shuffle.fetch.retry.timeout-ms 30000 Timeout value for fetcher to retry to fetch again when some non-fatal failure happens because of some events like NM restart. mapreduce.reduce.shuffle.retry-delay.max.ms 60000 The maximum number of ms the reducer will delay before retrying to download map data. mapreduce.reduce.shuffle.parallelcopies 5 The default number of parallel transfers run by reduce during the copy(shuffle) phase. mapreduce.reduce.shuffle.connect.timeout 180000 Expert: The maximum amount of time (in milli seconds) reduce task spends in trying to connect to a tasktracker for getting map output. mapreduce.reduce.shuffle.read.timeout 180000 Expert: The maximum amount of time (in milli seconds) reduce task waits for map output data to be available for reading after obtaining connection. mapreduce.shuffle.connection-keep-alive.enable false set to true to support keep-alive connections. mapreduce.shuffle.connection-keep-alive.timeout 5 The number of seconds a shuffle client attempts to retain http connection. Refer "Keep-Alive: timeout=" header in Http specification mapreduce.task.timeout 600000 The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string. A value of 0 disables the timeout. mapreduce.tasktracker.map.tasks.maximum 2 The maximum number of map tasks that will be run simultaneously by a task tracker. mapreduce.tasktracker.reduce.tasks.maximum 2 The maximum number of reduce tasks that will be run simultaneously by a task tracker. mapreduce.map.memory.mb 1024 The amount of memory to request from the scheduler for each map task. mapreduce.map.cpu.vcores 1 The number of virtual cores to request from the scheduler for each map task. mapreduce.reduce.memory.mb 1024 The amount of memory to request from the scheduler for each reduce task. mapreduce.reduce.cpu.vcores 1 The number of virtual cores to request from the scheduler for each reduce task. mapreduce.jobtracker.retiredjobs.cache.size 1000 The number of retired job status to keep in the cache. mapreduce.tasktracker.outofband.heartbeat false Expert: Set this to true to let the tasktracker send an out-of-band heartbeat on task-completion for better latency. mapreduce.jobtracker.jobhistory.lru.cache.size 5 The number of job history files loaded in memory. The jobs are loaded when they are first accessed. The cache is cleared based on LRU. mapreduce.jobtracker.instrumentation org.apache.hadoop.mapred.JobTrackerMetricsInst Expert: The instrumentation class to associate with each JobTracker. mapred.child.java.opts -Xmx200m Java opts for the task processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. mapred.child.env User added environment variables for the task processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit nodemanager's B env variable on Unix. 3) B=%B%;c This is inherit nodemanager's B env variable on Windows. mapreduce.admin.user.env Expert: Additional execution environment entries for map and reduce task processes. This is not an additive property. You must preserve the original value if you want your map and reduce tasks to have access to native libraries (compression, etc). When this value is empty, the command to set execution envrionment will be OS dependent: For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native. For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin. mapreduce.task.tmp.dir ./tmp To set the value of tmp directory for map and reduce tasks. If the value is an absolute path, it is directly assigned. Otherwise, it is prepended with task's working directory. The java tasks are executed with option -Djava.io.tmpdir='the absolute path of the tmp dir'. Pipes and streaming are set with environment variable, TMPDIR='the absolute path of the tmp dir' mapreduce.map.log.level INFO The logging level for the map task. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. mapreduce.reduce.log.level INFO The logging level for the reduce task. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. mapreduce.map.cpu.vcores 1 The number of virtual cores required for each map task. mapreduce.reduce.cpu.vcores 1 The number of virtual cores required for each reduce task. mapreduce.reduce.merge.inmem.threshold 1000 The threshold, in terms of the number of files for the in-memory merge process. When we accumulate threshold number of files we initiate the in-memory merge and spill to disk. A value of 0 or less than 0 indicates we want to DON'T have any threshold and instead depend only on the ramfs's memory consumption to trigger the merge. mapreduce.reduce.shuffle.merge.percent 0.66 The usage threshold at which an in-memory merge will be initiated, expressed as a percentage of the total memory allocated to storing in-memory map outputs, as defined by mapreduce.reduce.shuffle.input.buffer.percent. mapreduce.reduce.shuffle.input.buffer.percent 0.70 The percentage of memory to be allocated from the maximum heap size to storing map outputs during the shuffle. mapreduce.reduce.input.buffer.percent 0.0 The percentage of memory- relative to the maximum heap size- to retain map outputs during the reduce. When the shuffle is concluded, any remaining map outputs in memory must consume less than this threshold before the reduce can begin. mapreduce.reduce.shuffle.memory.limit.percent 0.25 Expert: Maximum percentage of the in-memory limit that a single shuffle can consume mapreduce.shuffle.ssl.enabled false Whether to use SSL for for the Shuffle HTTP endpoints. mapreduce.shuffle.ssl.file.buffer.size 65536 Buffer size for reading spills from file when using SSL. mapreduce.shuffle.max.connections 0 Max allowed connections for the shuffle. Set to 0 (zero) to indicate no limit on the number of connections. mapreduce.shuffle.max.threads 0 Max allowed threads for serving shuffle connections. Set to zero to indicate the default of 2 times the number of available processors (as reported by Runtime.availableProcessors()). Netty is used to serve requests, so a thread is not needed for each connection. mapreduce.shuffle.transferTo.allowed This option can enable/disable using nio transferTo method in the shuffle phase. NIO transferTo does not perform well on windows in the shuffle phase. Thus, with this configuration property it is possible to disable it, in which case custom transfer method will be used. Recommended value is false when running Hadoop on Windows. For Linux, it is recommended to set it to true. If nothing is set then the default value is false for Windows, and true for Linux. mapreduce.shuffle.transfer.buffer.size 131072 This property is used only if mapreduce.shuffle.transferTo.allowed is set to false. In that case, this property defines the size of the buffer used in the buffer copy code for the shuffle phase. The size of this buffer determines the size of the IO requests. mapreduce.reduce.markreset.buffer.percent 0.0 The percentage of memory -relative to the maximum heap size- to be used for caching values when using the mark-reset functionality. mapreduce.map.speculative true If true, then multiple instances of some map tasks may be executed in parallel. mapreduce.reduce.speculative true If true, then multiple instances of some reduce tasks may be executed in parallel. mapreduce.job.speculative.speculativecap 0.1 The max percent (0-1) of running tasks that can be speculatively re-executed at any time. mapreduce.job.map.output.collector.class org.apache.hadoop.mapred.MapTask$MapOutputBuffer The MapOutputCollector implementation(s) to use. This may be a comma-separated list of class names, in which case the map task will try to initialize each of the collectors in turn. The first to successfully initialize will be used. mapreduce.job.speculative.slowtaskthreshold 1.0 The number of standard deviations by which a task's ave progress-rates must be lower than the average of all running tasks' for the task to be considered too slow. mapreduce.job.speculative.slownodethreshold 1.0 The number of standard deviations by which a Task Tracker's ave map and reduce progress-rates (finishTime-dispatchTime) must be lower than the average of all successful map/reduce task's for the TT to be considered too slow to give a speculative task to. mapreduce.job.jvm.numtasks 1 How many tasks to run per jvm. If set to -1, there is no limit. mapreduce.job.ubertask.enable false Whether to enable the small-jobs "ubertask" optimization, which runs "sufficiently small" jobs sequentially within a single JVM. "Small" is defined by the following maxmaps, maxreduces, and maxbytes settings. Note that configurations for application masters also affect the "Small" definition - yarn.app.mapreduce.am.resource.mb must be larger than both mapreduce.map.memory.mb and mapreduce.reduce.memory.mb, and yarn.app.mapreduce.am.resource.cpu-vcores must be larger than both mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores to enable ubertask. Users may override this value. mapreduce.job.ubertask.maxmaps 9 Threshold for number of maps, beyond which job is considered too big for the ubertasking optimization. Users may override this value, but only downward. mapreduce.job.ubertask.maxreduces 1 Threshold for number of reduces, beyond which job is considered too big for the ubertasking optimization. CURRENTLY THE CODE CANNOT SUPPORT MORE THAN ONE REDUCE and will ignore larger values. (Zero is a valid max, however.) Users may override this value, but only downward. mapreduce.job.ubertask.maxbytes Threshold for number of input bytes, beyond which job is considered too big for the ubertasking optimization. If no value is specified, dfs.block.size is used as a default. Be sure to specify a default value in mapred-site.xml if the underlying filesystem is not HDFS. Users may override this value, but only downward. mapreduce.job.emit-timeline-data false Specifies if the Application Master should emit timeline data to the timeline server. Individual jobs can override this value. mapreduce.input.fileinputformat.split.minsize 0 The minimum size chunk that map input should be split into. Note that some file formats may have minimum split sizes that take priority over this setting. mapreduce.input.fileinputformat.list-status.num-threads 1 The number of threads to use to list and fetch block locations for the specified input paths. Note: multiple threads should not be used if a custom non thread-safe path filter is used. mapreduce.jobtracker.maxtasks.perjob -1 The maximum number of tasks for a single job. A value of -1 indicates that there is no maximum. mapreduce.input.lineinputformat.linespermap 1 When using NLineInputFormat, the number of lines of input data to include in each split. mapreduce.client.submit.file.replication 10 The replication level for submitted job files. This should be around the square root of the number of nodes. mapreduce.tasktracker.dns.interface default The name of the Network Interface from which a task tracker should report its IP address. mapreduce.tasktracker.dns.nameserver default The host name or IP address of the name server (DNS) which a TaskTracker should use to determine the host name used by the JobTracker for communication and display purposes. mapreduce.tasktracker.http.threads 40 The number of worker threads that for the http server. This is used for map output fetching mapreduce.tasktracker.http.address 0.0.0.0:50060 The task tracker http server address and port. If the port is 0 then the server will start on a free port. mapreduce.task.files.preserve.failedtasks false Should the files for failed tasks be kept. This should only be used on jobs that are failing, because the storage is never reclaimed. It also prevents the map outputs from being erased from the reduce directory as they are consumed. mapreduce.output.fileoutputformat.compress false Should the job outputs be compressed? mapreduce.output.fileoutputformat.compress.type RECORD If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK. mapreduce.output.fileoutputformat.compress.codec org.apache.hadoop.io.compress.DefaultCodec If the job outputs are compressed, how should they be compressed? mapreduce.map.output.compress false Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression. mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.DefaultCodec If the map outputs are compressed, how should they be compressed? map.sort.class org.apache.hadoop.util.QuickSort The default sort class for sorting keys. mapreduce.task.userlog.limit.kb 0 The maximum size of user-logs of each task in KB. 0 disables the cap. yarn.app.mapreduce.am.container.log.limit.kb 0 The maximum size of the MRAppMaster attempt container logs in KB. 0 disables the cap. yarn.app.mapreduce.task.container.log.backups 0 Number of backup files for task logs when using ContainerRollingLogAppender (CRLA). See org.apache.log4j.RollingFileAppender.maxBackupIndex. By default, ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA is enabled for tasks when both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.task.container.log.backups are greater than zero. yarn.app.mapreduce.am.container.log.backups 0 Number of backup files for the ApplicationMaster logs when using ContainerRollingLogAppender (CRLA). See org.apache.log4j.RollingFileAppender.maxBackupIndex. By default, ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA is enabled for the ApplicationMaster when both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.am.container.log.backups are greater than zero. mapreduce.job.userlog.retain.hours 24 The maximum time, in hours, for which the user-logs are to be retained after the job completion. mapreduce.jobtracker.hosts.filename Names a file that contains the list of nodes that may connect to the jobtracker. If the value is empty, all hosts are permitted. mapreduce.jobtracker.hosts.exclude.filename Names a file that contains the list of hosts that should be excluded by the jobtracker. If the value is empty, no hosts are excluded. mapreduce.jobtracker.heartbeats.in.second 100 Expert: Approximate number of heart-beats that could arrive at JobTracker in a second. Assuming each RPC can be processed in 10msec, the default value is made 100 RPCs in a second. mapreduce.jobtracker.tasktracker.maxblacklists 4 The number of blacklists for a taskTracker by various jobs after which the task tracker could be blacklisted across all jobs. The tracker will be given a tasks later (after a day). The tracker will become a healthy tracker after a restart. mapreduce.job.maxtaskfailures.per.tracker 3 The number of task-failures on a tasktracker of a given job after which new tasks of that job aren't assigned to it. It MUST be less than mapreduce.map.maxattempts and mapreduce.reduce.maxattempts otherwise the failed task will never be tried on a different node. mapreduce.client.output.filter FAILED The filter for controlling the output of the task's userlogs sent to the console of the JobClient. The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and ALL. mapreduce.client.completion.pollinterval 5000 The interval (in milliseconds) between which the JobClient polls the JobTracker for updates about job status. You may want to set this to a lower value to make tests run faster on a single node system. Adjusting this value in production may lead to unwanted client-server traffic. mapreduce.client.progressmonitor.pollinterval 1000 The interval (in milliseconds) between which the JobClient reports status to the console and checks for job completion. You may want to set this to a lower value to make tests run faster on a single node system. Adjusting this value in production may lead to unwanted client-server traffic. mapreduce.jobtracker.persist.jobstatus.active true Indicates if persistency of job status information is active or not. mapreduce.jobtracker.persist.jobstatus.hours 1 The number of hours job status information is persisted in DFS. The job status information will be available after it drops of the memory queue and between jobtracker restarts. With a zero value the job status information is not persisted at all in DFS. mapreduce.jobtracker.persist.jobstatus.dir /jobtracker/jobsInfo The directory where the job status information is persisted in a file system to be available after it drops of the memory queue and between jobtracker restarts. mapreduce.task.profile false To set whether the system should collect profiler information for some of the tasks in this job? The information is stored in the user log directory. The value is "true" if task profiling is enabled. mapreduce.task.profile.maps 0-2 To set the ranges of map tasks to profile. mapreduce.task.profile has to be set to true for the value to be accounted. mapreduce.task.profile.reduces 0-2 To set the ranges of reduce tasks to profile. mapreduce.task.profile has to be set to true for the value to be accounted. mapreduce.task.profile.params -agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s JVM profiler parameters used to profile map and reduce task attempts. This string may contain a single format specifier %s that will be replaced by the path to profile.out in the task attempt log directory. To specify different profiling options for map tasks and reduce tasks, more specific parameters mapreduce.task.profile.map.params and mapreduce.task.profile.reduce.params should be used. mapreduce.task.profile.map.params ${mapreduce.task.profile.params} Map-task-specific JVM profiler parameters. See mapreduce.task.profile.params mapreduce.task.profile.reduce.params ${mapreduce.task.profile.params} Reduce-task-specific JVM profiler parameters. See mapreduce.task.profile.params mapreduce.task.skip.start.attempts 2 The number of Task attempts AFTER which skip mode will be kicked off. When skip mode is kicked off, the tasks reports the range of records which it will process next, to the TaskTracker. So that on failures, TT knows which ones are possibly the bad records. On further executions, those are skipped. mapreduce.map.skip.proc.count.autoincr true The flag which if set to true, SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented by MapRunner after invoking the map function. This value must be set to false for applications which process the records asynchronously or buffer the input records. For example streaming. In such cases applications should increment this counter on their own. mapreduce.reduce.skip.proc.count.autoincr true The flag which if set to true, SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented by framework after invoking the reduce function. This value must be set to false for applications which process the records asynchronously or buffer the input records. For example streaming. In such cases applications should increment this counter on their own. mapreduce.job.skip.outdir If no value is specified here, the skipped records are written to the output directory at _logs/skip. User can stop writing skipped records by giving the value "none". mapreduce.map.skip.maxrecords 0 The number of acceptable skip records surrounding the bad record PER bad record in mapper. The number includes the bad record as well. To turn the feature of detection/skipping of bad records off, set the value to 0. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to indicate that framework need not try to narrow down. Whatever records(depends on application) get skipped are acceptable. mapreduce.reduce.skip.maxgroups 0 The number of acceptable skip groups surrounding the bad group PER bad group in reducer. The number includes the bad group as well. To turn the feature of detection/skipping of bad groups off, set the value to 0. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to indicate that framework need not try to narrow down. Whatever groups(depends on application) get skipped are acceptable. mapreduce.ifile.readahead true Configuration key to enable/disable IFile readahead. mapreduce.ifile.readahead.bytes 4194304 Configuration key to set the IFile readahead length in bytes. mapreduce.jobtracker.taskcache.levels 2 This is the max level of the task cache. For example, if the level is 2, the tasks cached are at the host level and at the rack level. mapreduce.job.queuename default Queue to which a job is submitted. This must match one of the queues defined in mapred-queues.xml for the system. Also, the ACL setup for the queue must allow the current user to submit a job to the queue. Before specifying a queue, ensure that the system is configured with the queue, and access is allowed for submitting jobs to the queue. mapreduce.job.tags Tags for the job that will be passed to YARN at submission time. Queries to YARN for applications can filter on these tags. mapreduce.cluster.acls.enabled false Specifies whether ACLs should be checked for authorization of users for doing various queue and job level operations. ACLs are disabled by default. If enabled, access control checks are made by JobTracker and TaskTracker when requests are made by users for queue operations like submit job to a queue and kill a job in the queue and job operations like viewing the job-details (See mapreduce.job.acl-view-job) or for modifying the job (See mapreduce.job.acl-modify-job) using Map/Reduce APIs, RPCs or via the console and web user interfaces. For enabling this flag(mapreduce.cluster.acls.enabled), this is to be set to true in mapred-site.xml on JobTracker node and on all TaskTracker nodes. mapreduce.job.acl-modify-job Job specific access-control list for 'modifying' the job. It is only used if authorization is enabled in Map/Reduce by setting the configuration property mapreduce.cluster.acls.enabled to true. This specifies the list of users and/or groups who can do modification operations on the job. For specifying a list of users and groups the format to use is "user1,user2 group1,group". If set to '*', it allows all users/groups to modify this job. If set to ' '(i.e. space), it allows none. This configuration is used to guard all the modifications with respect to this job and takes care of all the following operations: o killing this job o killing a task of this job, failing a task of this job o setting the priority of this job Each of these operations are also protected by the per-queue level ACL "acl-administer-jobs" configured via mapred-queues.xml. So a caller should have the authorization to satisfy either the queue-level ACL or the job-level ACL. Irrespective of this ACL configuration, (a) job-owner, (b) the user who started the cluster, (c) members of an admin configured supergroup configured via mapreduce.cluster.permissions.supergroup and (d) queue administrators of the queue to which this job was submitted to configured via acl-administer-jobs for the specific queue in mapred-queues.xml can do all the modification operations on a job. By default, nobody else besides job-owner, the user who started the cluster, members of supergroup and queue administrators can perform modification operations on a job. mapreduce.job.acl-view-job Job specific access-control list for 'viewing' the job. It is only used if authorization is enabled in Map/Reduce by setting the configuration property mapreduce.cluster.acls.enabled to true. This specifies the list of users and/or groups who can view private details about the job. For specifying a list of users and groups the format to use is "user1,user2 group1,group". If set to '*', it allows all users/groups to modify this job. If set to ' '(i.e. space), it allows none. This configuration is used to guard some of the job-views and at present only protects APIs that can return possibly sensitive information of the job-owner like o job-level counters o task-level counters o tasks' diagnostic information o task-logs displayed on the TaskTracker web-UI and o job.xml showed by the JobTracker's web-UI Every other piece of information of jobs is still accessible by any other user, for e.g., JobStatus, JobProfile, list of jobs in the queue, etc. Irrespective of this ACL configuration, (a) job-owner, (b) the user who started the cluster, (c) members of an admin configured supergroup configured via mapreduce.cluster.permissions.supergroup and (d) queue administrators of the queue to which this job was submitted to configured via acl-administer-jobs for the specific queue in mapred-queues.xml can do all the view operations on a job. By default, nobody else besides job-owner, the user who started the cluster, memebers of supergroup and queue administrators can perform view operations on a job. mapreduce.tasktracker.indexcache.mb 10 The maximum memory that a task tracker allows for the index cache that is used when serving map outputs to reducers. mapreduce.job.token.tracking.ids.enabled false Whether to write tracking ids of tokens to job-conf. When true, the configuration property "mapreduce.job.token.tracking.ids" is set to the token-tracking-ids of the job mapreduce.job.token.tracking.ids When mapreduce.job.token.tracking.ids.enabled is set to true, this is set by the framework to the token-tracking-ids used by the job. mapreduce.task.merge.progress.records 10000 The number of records to process during merge before sending a progress notification to the TaskTracker. mapreduce.task.combine.progress.records 10000 The number of records to process during combine output collection before sending a progress notification. mapreduce.job.reduce.slowstart.completedmaps 0.05 Fraction of the number of maps in the job which should be complete before reduces are scheduled for the job. mapreduce.job.complete.cancel.delegation.tokens true if false - do not unregister/cancel delegation tokens from renewal, because same tokens may be used by spawned jobs mapreduce.tasktracker.taskcontroller org.apache.hadoop.mapred.DefaultTaskController TaskController which is used to launch and manage task execution mapreduce.tasktracker.group Expert: Group to which TaskTracker belongs. If LinuxTaskController is configured via mapreduce.tasktracker.taskcontroller, the group owner of the task-controller binary should be same as this group. mapreduce.shuffle.port 13562 Default port that the ShuffleHandler will run on. ShuffleHandler is a service run at the NodeManager to facilitate transfers of intermediate Map outputs to requesting Reducers. mapreduce.job.reduce.shuffle.consumer.plugin.class org.apache.hadoop.mapreduce.task.reduce.Shuffle Name of the class whose instance will be used to send shuffle requests by reducetasks of this job. The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin. mapreduce.tasktracker.healthchecker.script.path Absolute path to the script which is periodicallyrun by the node health monitoring service to determine if the node is healthy or not. If the value of this key is empty or the file does not exist in the location configured here, the node health monitoring service is not started. mapreduce.tasktracker.healthchecker.interval 60000 Frequency of the node health script to be run, in milliseconds mapreduce.tasktracker.healthchecker.script.timeout 600000 Time after node health script should be killed if unresponsive and considered that the script has failed. mapreduce.tasktracker.healthchecker.script.args List of arguments which are to be passed to node health script when it is being launched comma seperated. mapreduce.job.counters.limit 120 Limit on the number of user counters allowed per job. mapreduce.framework.name local The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn. yarn.app.mapreduce.am.staging-dir /tmp/hadoop-yarn/staging The staging dir used while submitting jobs. mapreduce.am.max-attempts 2 The maximum number of application attempts. It is a application-specific setting. It should not be larger than the global number set by resourcemanager. Otherwise, it will be override. The default number is set to 2, to allow at least one retry for AM. mapreduce.job.end-notification.url Indicates url which will be called on completion of job to inform end status of job. User can give at most 2 variables with URI : $jobId and $jobStatus. If they are present in URI, then they will be replaced by their respective values. mapreduce.job.end-notification.retry.attempts 0 The number of times the submitter of the job wants to retry job end notification if it fails. This is capped by mapreduce.job.end-notification.max.attempts mapreduce.job.end-notification.retry.interval 1000 The number of milliseconds the submitter of the job wants to wait before job end notification is retried if it fails. This is capped by mapreduce.job.end-notification.max.retry.interval mapreduce.job.end-notification.max.attempts 5 true The maximum number of times a URL will be read for providing job end notification. Cluster administrators can set this to limit how long after end of a job, the Application Master waits before exiting. Must be marked as final to prevent users from overriding this. mapreduce.job.end-notification.max.retry.interval 5000 true The maximum amount of time (in milliseconds) to wait before retrying job end notification. Cluster administrators can set this to limit how long the Application Master waits before exiting. Must be marked as final to prevent users from overriding this. yarn.app.mapreduce.am.env User added environment variables for the MR App Master processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit tasktracker's B env variable. yarn.app.mapreduce.am.admin.user.env Environment variables for the MR App Master processes for admin purposes. These values are set first and can be overridden by the user env (yarn.app.mapreduce.am.env) Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit app master's B env variable. yarn.app.mapreduce.am.command-opts -Xmx1024m Java opts for the MR App Master processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. yarn.app.mapreduce.am.admin-command-opts Java opts for the MR App Master processes for admin purposes. It will appears before the opts set by yarn.app.mapreduce.am.command-opts and thus its options can be overridden user. Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. yarn.app.mapreduce.am.job.task.listener.thread-count 30 The number of threads used to handle RPC calls in the MR AppMaster from remote tasks yarn.app.mapreduce.am.job.client.port-range Range of ports that the MapReduce AM can use when binding. Leave blank if you want all possible ports. For example 50000-50050,50100-50200 yarn.app.mapreduce.am.job.committer.cancel-timeout 60000 The amount of time in milliseconds to wait for the output committer to cancel an operation if the job is killed yarn.app.mapreduce.am.job.committer.commit-window 10000 Defines a time window in milliseconds for output commit operations. If contact with the RM has occurred within this window then commits are allowed, otherwise the AM will not allow output commits until contact with the RM has been re-established. yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms 1000 The interval in ms at which the MR AppMaster should send heartbeats to the ResourceManager yarn.app.mapreduce.client-am.ipc.max-retries 3 The number of client retries to the AM - before reconnecting to the RM to fetch Application Status. yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts 3 The number of client retries on socket timeouts to the AM - before reconnecting to the RM to fetch Application Status. yarn.app.mapreduce.client.max-retries 3 The number of client retries to the RM/HS before throwing exception. This is a layer above the ipc. yarn.app.mapreduce.am.resource.mb 1536 The amount of memory the MR AppMaster needs. yarn.app.mapreduce.am.resource.cpu-vcores 1 The number of virtual CPU cores the MR AppMaster needs. CLASSPATH for MR applications. A comma-separated list of CLASSPATH entries. If mapreduce.application.framework is set then this must specify the appropriate classpath for that archive, and the name of the archive must be present in the classpath. If mapreduce.app-submission.cross-platform is false, platform-specific environment vairable expansion syntax would be used to construct the default CLASSPATH entries. For Linux: $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*. For Windows: %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*, %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*. If mapreduce.app-submission.cross-platform is true, platform-agnostic default CLASSPATH for MR applications would be used: {{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*, {{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/* Parameter expansion marker will be replaced by NodeManager on container launch based on the underlying OS accordingly. mapreduce.application.classpath If enabled, user can submit an application cross-platform i.e. submit an application from a Windows client to a Linux/Unix server or vice versa. mapreduce.app-submission.cross-platform false Path to the MapReduce framework archive. If set, the framework archive will automatically be distributed along with the job, and this path would normally reside in a public location in an HDFS filesystem. As with distributed cache files, this can be a URL with a fragment specifying the alias to use for the archive name. For example, hdfs:/mapred/framework/hadoop-mapreduce-2.1.1.tar.gz#mrframework would alias the localized archive as "mrframework". Note that mapreduce.application.classpath must include the appropriate classpath for the specified framework. The base name of the archive, or alias of the archive if an alias is used, must appear in the specified classpath. mapreduce.application.framework.path mapreduce.job.classloader false Whether to use a separate (isolated) classloader for user classes in the task JVM. mapreduce.job.classloader.system.classes Used to override the default definition of the system classes for the job classloader. The system classes are a comma-separated list of classes that should be loaded from the system classpath, not the user-supplied JARs, when mapreduce.job.classloader is enabled. Names ending in '.' (period) are treated as package names, and names starting with a '-' are treated as negative matches. mapreduce.jobhistory.address 0.0.0.0:10020 MapReduce JobHistory Server IPC host:port mapreduce.jobhistory.webapp.address 0.0.0.0:19888 MapReduce JobHistory Server Web UI host:port mapreduce.jobhistory.keytab Location of the kerberos keytab file for the MapReduce JobHistory Server. /etc/security/keytab/jhs.service.keytab mapreduce.jobhistory.principal Kerberos principal name for the MapReduce JobHistory Server. jhs/_HOST@REALM.TLD mapreduce.jobhistory.intermediate-done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate mapreduce.jobhistory.done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done mapreduce.jobhistory.cleaner.enable true mapreduce.jobhistory.cleaner.interval-ms 86400000 How often the job history cleaner checks for files to delete, in milliseconds. Defaults to 86400000 (one day). Files are only deleted if they are older than mapreduce.jobhistory.max-age-ms. mapreduce.jobhistory.max-age-ms 604800000 Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week). mapreduce.jobhistory.client.thread-count 10 The number of threads to handle client API requests mapreduce.jobhistory.datestring.cache.size 200000 Size of the date string cache. Effects the number of directories which will be scanned to find a job. mapreduce.jobhistory.joblist.cache.size 20000 Size of the job list cache mapreduce.jobhistory.loadedjobs.cache.size 5 Size of the loaded job cache mapreduce.jobhistory.move.interval-ms 180000 Scan for history files to more from intermediate done dir to done dir at this frequency. mapreduce.jobhistory.move.thread-count 3 The number of threads used to move files. mapreduce.jobhistory.store.class The HistoryStorage class to use to cache history data. mapreduce.jobhistory.minicluster.fixed.ports false Whether to use fixed ports with the minicluster mapreduce.jobhistory.admin.address 0.0.0.0:10033 The address of the History server admin interface. mapreduce.jobhistory.admin.acl * ACL of who can be admin of the History server. mapreduce.jobhistory.recovery.enable false Enable the history server to store server state and recover server state upon startup. If enabled then mapreduce.jobhistory.recovery.store.class must be specified. mapreduce.jobhistory.recovery.store.class org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService The HistoryServerStateStoreService class to store history server state for recovery. mapreduce.jobhistory.recovery.store.fs.uri ${hadoop.tmp.dir}/mapred/history/recoverystore The URI where history server state will be stored if HistoryServerFileSystemStateStoreService is configured as the recovery storage class. mapreduce.jobhistory.http.policy HTTP_ONLY This configures the HTTP endpoint for JobHistoryServer web UI. The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https sahara-8.0.0/sahara/service/edp/resources/mapred-job-config.xml0000666000175100017510000000311613245514472024502 0ustar zuulzuul00000000000000 mapred.reducer.new-api true mapred.mapper.new-api true mapred.input.dir mapred.output.dir mapred.mapoutput.key.class mapred.mapoutput.value.class mapred.output.key.class mapred.output.value.class mapreduce.map.class mapreduce.reduce.class mapred.mapper.class mapred.reducer.class mapred.jar mapred.job.name sahara-8.0.0/sahara/service/edp/resources/workflow.xml0000666000175100017510000000045413245514472023073 0ustar zuulzuul00000000000000 Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}] sahara-8.0.0/sahara/service/engine.py0000666000175100017510000002064013245514472017533 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc import datetime import string from oslo_config import cfg from oslo_log import log as logging import six from sahara import conductor as c from sahara import context from sahara.i18n import _ from sahara.service import networks from sahara.utils import cluster as cluster_utils from sahara.utils import cluster_progress_ops as cpo from sahara.utils import edp from sahara.utils.openstack import base as b from sahara.utils.openstack import images as sahara_images from sahara.utils import poll_utils from sahara.utils import remote LOG = logging.getLogger(__name__) conductor = c.API CONF = cfg.CONF @six.add_metaclass(abc.ABCMeta) class Engine(object): @abc.abstractmethod def create_cluster(self, cluster): pass @abc.abstractmethod def scale_cluster(self, cluster, node_group_id_map): pass @abc.abstractmethod def shutdown_cluster(self, cluster, force): pass @abc.abstractmethod def rollback_cluster(self, cluster, reason): pass @abc.abstractmethod def get_type_and_version(self): """Returns engine type and version Result should be in the form 'type.major.minor'. """ def get_node_group_image_username(self, node_group): image_id = node_group.get_image_id() return b.execute_with_retries( sahara_images.image_manager().get, image_id).username @poll_utils.poll_status('ips_assign_timeout', _("Assign IPs"), sleep=1) def _ips_assign(self, ips_assigned, cluster, instances): if not cluster_utils.check_cluster_exists(cluster): return True for instance in instances: if instance.id not in ips_assigned: with context.set_current_instance_id(instance.instance_id): if networks.init_instances_ips(instance): ips_assigned.add(instance.id) cpo.add_successful_event(instance) return len(ips_assigned) == len(instances) def _await_networks(self, cluster, instances): if not instances: return cpo.add_provisioning_step(cluster.id, _("Assign IPs"), len(instances)) ips_assigned = set() self._ips_assign(ips_assigned, cluster, instances) LOG.info("All instances have IPs assigned") cluster = conductor.cluster_get(context.ctx(), cluster) instances = cluster_utils.get_instances(cluster, ips_assigned) cpo.add_provisioning_step( cluster.id, _("Wait for instance accessibility"), len(instances)) with context.ThreadGroup() as tg: for instance in instances: with context.set_current_instance_id(instance.instance_id): tg.spawn("wait-for-ssh-%s" % instance.instance_name, self._wait_until_accessible, instance) LOG.info("All instances are accessible") @poll_utils.poll_status( 'wait_until_accessible', _("Wait for instance accessibility"), sleep=5) def _is_accessible(self, instance): if not cluster_utils.check_cluster_exists(instance.cluster): return True try: # check if ssh is accessible and cloud-init # script is finished generating authorized_keys exit_code, stdout = instance.remote().execute_command( "ls .ssh/authorized_keys", raise_when_error=False) if exit_code == 0: LOG.debug('Instance is accessible') return True except Exception as ex: ip_used = "internal_ip" if CONF.proxy_command and \ CONF.proxy_command_use_internal_ip else "management_ip" LOG.debug("Can't login to node, IP: {ip}, reason {reason}" .format(ip=getattr(instance, ip_used), reason=ex)) return False return False @cpo.event_wrapper(mark_successful_on_exit=True) def _wait_until_accessible(self, instance): self._is_accessible(instance) def _configure_instances(self, cluster): """Configure active instances. * generate /etc/hosts * change /etc/resolv.conf * setup passwordless login * etc. """ cpo.add_provisioning_step( cluster.id, _("Configure instances"), cluster_utils.count_instances(cluster)) with context.ThreadGroup() as tg: for node_group in cluster.node_groups: for instance in node_group.instances: with context.set_current_instance_id(instance.instance_id): tg.spawn("configure-instance-{}".format( instance.instance_name), self._configure_instance, instance, cluster ) @cpo.event_wrapper(mark_successful_on_exit=True) def _configure_instance(self, instance, cluster): self._configure_instance_etc_hosts(instance, cluster) if cluster.use_designate_feature(): self._configure_instance_resolve_conf(instance) def _configure_instance_etc_hosts(self, instance, cluster): LOG.debug('Configuring "/etc/hosts" of instance.') hosts_file = cluster_utils.generate_etc_hosts(cluster) with instance.remote() as r: r.write_file_to('etc-hosts', hosts_file) r.execute_command('sudo hostname %s' % instance.fqdn()) r.execute_command('sudo mv etc-hosts /etc/hosts') r.execute_command('sudo usermod -s /bin/bash $USER') def _configure_instance_resolve_conf(self, instance): LOG.debug('Setting up those name servers from sahara.conf ' 'which are lacked in the /etc/resolv.conf.') with instance.remote() as r: code, curr_resolv_conf = r.execute_command('cat /etc/resolv.conf') diff = cluster_utils.generate_resolv_conf_diff(curr_resolv_conf) if diff.strip(): position = curr_resolv_conf.find('nameserver') if position == -1: position = 0 new_resolv_conf = "{}\n{}{}".format( curr_resolv_conf[:position], diff, curr_resolv_conf[position:]) r.write_file_to('resolv-conf', new_resolv_conf) r.execute_command('sudo mv resolv-conf /etc/resolv.conf') def _generate_user_data_script(self, node_group, instance_name): script = """#!/bin/bash echo "${public_key}" >> ${user_home}/.ssh/authorized_keys\n # ====== COMMENT OUT Defaults requiretty in /etc/sudoers ======== sed '/^Defaults requiretty*/ s/^/#/' -i /etc/sudoers\n """ script += remote.get_userdata_template() username = node_group.image_username if username == "root": user_home = "/root/" else: user_home = "/home/%s/" % username script_template = string.Template(script) return script_template.safe_substitute( public_key=node_group.cluster.management_public_key, user_home=user_home, instance_name=instance_name) # Deletion ops def _clean_job_executions(self, cluster): ctx = context.ctx() for je in conductor.job_execution_get_all(ctx, cluster_id=cluster.id): update = {"cluster_id": None} if not je.end_time: info = je.info.copy() if je.info else {} info['status'] = edp.JOB_STATUS_KILLED update.update({"info": info, "end_time": datetime.datetime.now()}) conductor.job_execution_update(ctx, je, update) def _remove_db_objects(self, cluster): ctx = context.ctx() cluster = conductor.cluster_get(ctx, cluster) instances = cluster_utils.get_instances(cluster) for inst in instances: conductor.instance_remove(ctx, inst) sahara-8.0.0/sahara/service/networks.py0000666000175100017510000000436313245514472020146 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six from sahara import conductor as c from sahara import context from sahara.utils.openstack import nova conductor = c.API CONF = cfg.CONF def init_instances_ips(instance): """Extracts internal and management ips. As internal ip will be used the first ip from the nova networks CIDRs. If use_floating_ip flag is set than management ip will be the first non-internal ip. """ server = nova.get_instance_info(instance) management_ip = None internal_ip = None for addresses in six.itervalues(server.addresses): # selects IPv4 preferentially for address in sorted(addresses, key=lambda addr: addr['version']): if address['OS-EXT-IPS:type'] == 'fixed': internal_ip = internal_ip or address['addr'] else: management_ip = management_ip or address['addr'] # tmckay-fp okay # conf.use_floating_ips becomes # "use a floating ip for the management ip if one is defined" # assignment comes from nova conf setting, or from floating_ip_pool value # tmckay-fp log an extra warning here in the neutron # case that the node group has a floating ip pool but # we don't have a management ip yet ... cluster = instance.cluster if (not CONF.use_floating_ips or not management_ip or (cluster.has_proxy_gateway() and not instance.node_group.is_proxy_gateway)): management_ip = internal_ip conductor.instance_update(context.ctx(), instance, {"management_ip": management_ip, "internal_ip": internal_ip}) return internal_ip and management_ip sahara-8.0.0/sahara/service/volumes.py0000666000175100017510000002256013245514472017763 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import re import threading from oslo_config import cfg from oslo_log import log as logging from sahara import conductor as c from sahara import context from sahara.i18n import _ from sahara.plugins import provisioning as plugin_base from sahara.utils import cluster_progress_ops as cpo from sahara.utils.openstack import base as b from sahara.utils.openstack import cinder from sahara.utils.openstack import nova from sahara.utils import poll_utils conductor = c.API LOG = logging.getLogger(__name__) CONF = cfg.CONF def _get_timeout_for_disk_preparing(cluster): configs = cluster.cluster_configs.to_dict() option_name = plugin_base.DISKS_PREPARING_TIMEOUT.name option_target = plugin_base.DISKS_PREPARING_TIMEOUT.applicable_target try: return int(configs[option_target][option_name]) except Exception: return int(plugin_base.DISKS_PREPARING_TIMEOUT.default_value) def _is_xfs_enabled(cluster): configs = cluster.cluster_configs.to_dict() option_name = plugin_base.XFS_ENABLED.name option_target = plugin_base.XFS_ENABLED.applicable_target try: return bool(configs[option_target][option_name]) except Exception: return bool(plugin_base.XFS_ENABLED.default_value) def _get_os_distrib(remote): return remote.get_os_distrib() def _check_installed_xfs(instance): redhat = "rpm -q xfsprogs || yum install -y xfsprogs" debian = "dpkg -s xfsprogs || apt-get -y install xfsprogs" cmd_map = { "centos": redhat, "fedora": redhat, "redhatenterpriseserver": redhat, "redhat": redhat, "ubuntu": debian, 'debian': debian } with instance.remote() as r: distro = _get_os_distrib(r) if not cmd_map.get(distro): LOG.warning("Cannot verify installation of XFS tools for " "unknown distro {distro}.".format(distro=distro)) return False try: r.execute_command(cmd_map.get(distro), run_as_root=True) return True except Exception as e: LOG.warning("Cannot install xfsprogs: {reason}".format(reason=e)) return False def _can_use_xfs(instances): cluster = instances[0].cluster if not _is_xfs_enabled(cluster): return False for instance in instances: if not _check_installed_xfs(instance): return False return True def mount_to_instances(instances): if len(instances) == 0: return use_xfs = _can_use_xfs(instances) for instance in instances: with context.set_current_instance_id(instance.instance_id): devices = _find_instance_devices(instance) if devices: cpo.add_provisioning_step( instance.cluster_id, _("Mount volumes to {inst_name} instance").format( inst_name=instance.instance_name), len(devices)) formatted_devices = [] lock = threading.Lock() with context.ThreadGroup() as tg: # Since formating can take several minutes (for large # disks) and can be done in parallel, launch one thread # per disk. for device in devices: tg.spawn('format-device-%s' % device, _format_device, instance, device, use_xfs, formatted_devices, lock) conductor.instance_update( context.current(), instance, {"storage_devices_number": len(formatted_devices)}) for idx, dev in enumerate(formatted_devices): _mount_volume_to_node(instance, idx+1, dev, use_xfs) def _find_instance_devices(instance): with instance.remote() as r: code, attached_info = r.execute_command( "lsblk -r | awk '$6 ~ /disk/ || /part/ {print \"/dev/\" $1}'") attached_dev = attached_info.split() code, mounted_info = r.execute_command( "mount | awk '$1 ~ /^\/dev/ {print $1}'") mounted_dev = mounted_info.split() # find and ignore Nova config drive for label in ("config-2", "CONFIG-2"): code, nova_config_drive = r.execute_command( "/sbin/blkid -t LABEL=\"%s\" -odevice" % label, raise_when_error=False, run_as_root=True ) drive_name = nova_config_drive.strip() if code == 0 and drive_name in attached_dev: attached_dev.remove(drive_name) break # filtering attached devices, that should not be mounted for dev in attached_dev[:]: idx = re.sub("\D", "", dev) if idx: if dev in mounted_dev: if re.sub("\d", "", dev) in attached_dev: attached_dev.remove(re.sub("\d", "", dev)) attached_dev.remove(dev) for dev in attached_dev[:]: if re.sub("\D", "", dev): if re.sub("\d", "", dev) in attached_dev: attached_dev.remove(dev) attached_dev = [dev for dev in attached_dev if dev not in mounted_dev] return attached_dev @cpo.event_wrapper(mark_successful_on_exit=True) def _mount_volume_to_node(instance, index, device, use_xfs): LOG.debug("Mounting volume {device} to instance".format(device=device)) mount_point = instance.node_group.volume_mount_prefix + str(index) _mount_volume(instance, device, mount_point, use_xfs) LOG.debug("Mounted volume to instance") def _format_device( instance, device, use_xfs, formatted_devices=None, lock=None): with instance.remote() as r: try: timeout = _get_timeout_for_disk_preparing(instance.cluster) # Format devices with better performance options: # - reduce number of blocks reserved for root to 1% # - use 'dir_index' for faster directory listings # - use 'extents' to work faster with large files # - disable journaling fs_opts = '-F -m 1 -O dir_index,extents,^has_journal' command = 'sudo mkfs.ext4 %s %s' % (fs_opts, device) if use_xfs: command = 'sudo mkfs.xfs -f %s' % device r.execute_command(command, timeout=timeout) if lock: with lock: formatted_devices.append(device) except Exception as e: LOG.warning("Device {dev} cannot be formatted: {reason}".format( dev=device, reason=e)) cpo.add_fail_event(instance, e) def _mount_volume(instance, device_path, mount_point, use_xfs): with instance.remote() as r: try: timeout = _get_timeout_for_disk_preparing(instance.cluster) # Mount volumes with better performance options: # - enable write-back for ext4 # - do not store access time # - disable barrier for xfs r.execute_command('sudo mkdir -p %s' % mount_point) mount_opts = '-o data=writeback,noatime,nodiratime' if use_xfs: mount_opts = "-t xfs -o noatime,nodiratime,nobarrier" r.execute_command('sudo mount %s %s %s' % (mount_opts, device_path, mount_point), timeout=timeout) r.execute_command( 'sudo sh -c "grep %s /etc/mtab >> /etc/fstab"' % device_path) except Exception: LOG.error("Error mounting volume to instance") raise def detach_from_instance(instance): for volume_id in instance.volumes: _detach_volume(instance, volume_id) _delete_volume(volume_id) @poll_utils.poll_status( 'detach_volume_timeout', _("Await for volume become detached"), sleep=2) def _await_detach(volume_id): volume = cinder.get_volume(volume_id) if volume.status not in ['available', 'error']: return False return True def _detach_volume(instance, volume_id): volume = cinder.get_volume(volume_id) try: LOG.debug("Detaching volume {id} from instance".format(id=volume_id)) b.execute_with_retries(nova.client().volumes.delete_server_volume, instance.instance_id, volume_id) except Exception: LOG.error("Can't detach volume {id}".format(id=volume.id)) detach_timeout = CONF.timeouts.detach_volume_timeout LOG.debug("Waiting {timeout} seconds to detach {id} volume".format( timeout=detach_timeout, id=volume_id)) _await_detach(volume_id) def _delete_volume(volume_id): LOG.debug("Deleting volume {volume}".format(volume=volume_id)) volume = cinder.get_volume(volume_id) try: b.execute_with_retries(volume.delete) except Exception: LOG.error("Can't delete volume {volume}".format(volume=volume.id)) sahara-8.0.0/sahara/service/coordinator.py0000666000175100017510000001076413245514472020617 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import bisect import hashlib from oslo_config import cfg from oslo_log import log from oslo_utils import uuidutils from tooz import coordination LOG = log.getLogger(__name__) coordinator_opts = [ cfg.IntOpt('coordinator_heartbeat_interval', default=1, help='Interval size between heartbeat execution in seconds. ' 'Heartbeats are executed to make sure that connection to ' 'the coordination server is active.'), cfg.IntOpt('hash_ring_replicas_count', default=40, help='Number of points that belongs to each member on a hash ' 'ring. The larger number leads to a better distribution.') ] CONF = cfg.CONF CONF.register_opts(coordinator_opts) class Coordinator(object): def __init__(self, backend_url): self.coordinator = None self.member_id = uuidutils.generate_uuid() if backend_url: try: self.coordinator = coordination.get_coordinator( backend_url, self.member_id) self.coordinator.start() LOG.info('Coordination backend loaded successfully.') except coordination.ToozError: LOG.error('Error connecting to coordination backend.') raise def is_started(self): if self.coordinator: return self.coordinator.is_started return False def heartbeat(self): if self.coordinator: self.coordinator.heartbeat() def join_group(self, group_id): if self.coordinator: try: self.coordinator.join_group(group_id).get() except coordination.GroupNotCreated: try: self.coordinator.create_group(group_id).get() except coordination.GroupAlreadyExist: pass self.coordinator.join_group(group_id).get() def get_members(self, group_id): if self.coordinator: for i in range(2): try: members = self.coordinator.get_members(group_id).get() if self.member_id in members: return members self.join_group(group_id) except coordination.GroupNotCreated: self.join_group(group_id) except coordination.ToozError as e: LOG.error("Couldn't get members of {group} group. " "Reason: {ex}".format( group=group_id, ex=str(e))) return [] class HashRing(Coordinator): def __init__(self, backend_url, group_id): self.group_id = group_id self.replicas = CONF.hash_ring_replicas_count super(HashRing, self).__init__(backend_url) self.join_group(group_id) @staticmethod def _hash(key): return int(hashlib.md5(str(key)).hexdigest(), 16) # nosec def _build_ring(self): ring = {} members = self.get_members(self.group_id) for member in members: for r in range(self.replicas): hashed_key = self._hash('%s:%s' % (member, r)) ring[hashed_key] = member return ring, sorted(ring.keys()) def _check_object(self, object, ring, sorted_keys): """Checks if this object belongs to this member or not""" hashed_key = self._hash(object.id) position = bisect.bisect(sorted_keys, hashed_key) position = position if position < len(sorted_keys) else 0 return ring[sorted_keys[position]] == self.member_id def get_subset(self, objects): """Returns subset that belongs to this member""" if self.coordinator: ring, keys = self._build_ring() if ring: return [obj for obj in objects if self._check_object( obj, ring, keys)] return [] return objects sahara-8.0.0/sahara/service/castellan/0000775000175100017510000000000013245515027017653 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/castellan/sahara_key_manager.py0000666000175100017510000000553313245514472024041 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from castellan.common.objects import passphrase as key from castellan.key_manager import key_manager as km """sahara.service.castellan.sahara_key_manager This module contains the KeyManager class that will be used by the castellan library, it is not meant for direct usage within sahara. """ class SaharaKeyManager(km.KeyManager): """Sahara specific key manager This manager is a thin wrapper around the secret being stored. It is intended for backward compatible use only. It will not store keys or generate UUIDs but instead return the secret that is being stored. This behavior allows Sahara to continue storing secrets in its database while using the Castellan key manager abstraction. """ def __init__(self, configuration=None): pass def create_key(self, context, algorithm=None, length=0, expiration=None, **kwargs): """creates a key algorithm, length, and expiration are unused by sahara keys. """ return key.Passphrase(passphrase=kwargs.get('passphrase', '')) def create_key_pair(self, *args, **kwargs): pass def store(self, context, key, expiration=None, **kwargs): """store a key in normal usage a store_key will return the UUID of the key as dictated by the key manager. Sahara would then store this UUID in its database to use for retrieval. As sahara is not actually using a key manager in this context it will return the key's payload for storage. """ return key.get_encoded() def get(self, context, key_id, **kwargs): """get a key since sahara is not actually storing key UUIDs the key_id to this function should actually be the key payload. this function will simply return a new SaharaKey based on that value. """ return key.Passphrase(passphrase=key_id) def delete(self, context, key_id, **kwargs): """delete a key as there is no external key manager, this function will not perform any external actions. therefore, it won't change anything. """ pass def list(self, *args, **kwargs): """list all managed keys current implementation of the key manager does not utilize this """ pass sahara-8.0.0/sahara/service/castellan/config.py0000666000175100017510000000364413245514472021506 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from castellan import options as castellan from oslo_config import cfg from sahara.utils.openstack import base as utils opts = [ cfg.BoolOpt('use_barbican_key_manager', default=False, help='Enable the usage of the OpenStack Key Management ' 'service provided by barbican.'), ] castellan_opts = [ cfg.StrOpt('barbican_api_endpoint', help='The endpoint to use for connecting to the barbican ' 'api controller. By default, castellan will use the ' 'URL from the service catalog.'), cfg.StrOpt('barbican_api_version', default='v1', help='Version of the barbican API, for example: "v1"'), ] castellan_group = cfg.OptGroup(name='castellan', title='castellan key manager options') CONF = cfg.CONF CONF.register_group(castellan_group) CONF.register_opts(opts) CONF.register_opts(castellan_opts, group=castellan_group) def validate_config(): if CONF.use_barbican_key_manager: # NOTE (elmiko) there is no need to set the api_class as castellan # uses barbican by default. castellan.set_defaults(CONF, auth_endpoint=utils.retrieve_auth_url()) else: castellan.set_defaults(CONF, api_class='sahara.service.castellan.' 'sahara_key_manager.SaharaKeyManager') sahara-8.0.0/sahara/service/castellan/__init__.py0000666000175100017510000000000013245514472021757 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/service/castellan/utils.py0000666000175100017510000000355613245514472021403 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from castellan.common.objects import passphrase from castellan import key_manager from sahara import context def delete_secret(id, ctx=None): """delete a secret from the external key manager :param id: The identifier of the secret to delete :param ctx: The context, and associated authentication, to use with this operation (defaults to the current context) """ if ctx is None: ctx = context.current() key_manager.API().delete(ctx, id) def get_secret(id, ctx=None): """get a secret associated with an id :param id: The identifier of the secret to retrieve :param ctx: The context, and associated authentication, to use with this operation (defaults to the current context) """ if ctx is None: ctx = context.current() key = key_manager.API().get(ctx, id) return key.get_encoded() def store_secret(secret, ctx=None): """store a secret and return its identifier :param secret: The secret to store, this should be a string :param ctx: The context, and associated authentication, to use with this operation (defaults to the current context) """ if ctx is None: ctx = context.current() key = passphrase.Passphrase(secret) return key_manager.API().store(ctx, key) sahara-8.0.0/sahara/swift/0000775000175100017510000000000013245515027015401 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/swift/swift_helper.py0000666000175100017510000000737513245514472020467 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_log import log as logging from sahara import context from sahara.swift import utils as su from sahara.utils import xmlutils as x LOG = logging.getLogger(__name__) CONF = cfg.CONF HADOOP_SWIFT_AUTH_URL = 'fs.swift.service.sahara.auth.url' HADOOP_SWIFT_TENANT = 'fs.swift.service.sahara.tenant' HADOOP_SWIFT_USERNAME = 'fs.swift.service.sahara.username' HADOOP_SWIFT_PASSWORD = 'fs.swift.service.sahara.password' HADOOP_SWIFT_REGION = 'fs.swift.service.sahara.region' HADOOP_SWIFT_TRUST_ID = 'fs.swift.service.sahara.trust.id' HADOOP_SWIFT_DOMAIN_NAME = 'fs.swift.service.sahara.domain.name' opts = [ cfg.StrOpt("public_identity_ca_file", help=("Location of ca certificate file to use for identity " "client requests via public endpoint")), cfg.StrOpt("public_object_store_ca_file", help=("Location of ca certificate file to use for object-store " "client requests via public endpoint")) ] public_endpoint_cert_group = cfg.OptGroup( name="object_store_access", title="Auth options for Swift access from VM") CONF.register_group(public_endpoint_cert_group) CONF.register_opts(opts, group=public_endpoint_cert_group) def retrieve_tenant(): return context.current().tenant_name def get_swift_configs(): configs = x.load_hadoop_xml_defaults('swift/resources/conf-template.xml') for conf in configs: if conf['name'] == HADOOP_SWIFT_AUTH_URL: conf['value'] = su.retrieve_auth_url() + "auth/tokens/" if conf['name'] == HADOOP_SWIFT_TENANT: conf['value'] = retrieve_tenant() if CONF.os_region_name and conf['name'] == HADOOP_SWIFT_REGION: conf['value'] = CONF.os_region_name result = [cfg for cfg in configs if cfg['value']] LOG.info("Swift would be integrated with the following " "params: {result}".format(result=result)) return result def read_default_swift_configs(): return x.load_hadoop_xml_defaults('swift/resources/conf-template.xml') def install_ssl_certs(instances): certs = [] if CONF.object_store_access.public_identity_ca_file: certs.append(CONF.object_store_access.public_identity_ca_file) if CONF.object_store_access.public_object_store_ca_file: certs.append(CONF.object_store_access.public_object_store_ca_file) if not certs: return with context.ThreadGroup() as tg: for inst in instances: tg.spawn("configure-ssl-cert-%s" % inst.instance_id, _install_ssl_certs, inst, certs) def _install_ssl_certs(instance, certs): register_cmd = ( "sudo su - -c \"keytool -import -alias sahara-%d -keystore " "`cut -f2 -d \\\"=\\\" /etc/profile.d/99-java.sh | head -1`" "/lib/security/cacerts -file /tmp/cert.pem -noprompt -storepass " "changeit\"") with instance.remote() as r: for idx, cert in enumerate(certs): with open(cert) as cert_fd: data = cert_fd.read() r.write_file_to("/tmp/cert.pem", data) try: r.execute_command(register_cmd % idx) finally: r.execute_command("rm /tmp/cert.pem") sahara-8.0.0/sahara/swift/__init__.py0000666000175100017510000000000013245514472017505 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/swift/utils.py0000666000175100017510000000267013245514472017125 0ustar zuulzuul00000000000000# Copyright (c) 2013 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg import six from six.moves.urllib import parse as urlparse from sahara.utils.openstack import base as clients_base CONF = cfg.CONF SWIFT_INTERNAL_PREFIX = "swift://" SWIFT_URL_SUFFIX_START = '.' SWIFT_URL_SUFFIX = SWIFT_URL_SUFFIX_START + 'sahara' def retrieve_auth_url(endpoint_type="publicURL"): """This function returns auth url v3 api. """ version_suffix = 'v3' # return auth url with trailing slash return clients_base.retrieve_auth_url( endpoint_type=endpoint_type, version=version_suffix) + "/" def inject_swift_url_suffix(url): if isinstance(url, six.string_types) and url.startswith("swift://"): u = urlparse.urlparse(url) if not u.netloc.endswith(SWIFT_URL_SUFFIX): return url.replace(u.netloc, u.netloc + SWIFT_URL_SUFFIX, 1) return url sahara-8.0.0/sahara/swift/resources/0000775000175100017510000000000013245515027017413 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/swift/resources/conf-template.xml0000666000175100017510000000447013245514472022705 0ustar zuulzuul00000000000000 fs.swift.service.sahara.auth.url fs.swift.service.sahara.tenant fs.swift.service.sahara.username fs.swift.service.sahara.password fs.swift.service.sahara.http.port 8080 fs.swift.service.sahara.https.port 443 fs.swift.service.sahara.public true fs.swift.service.sahara.auth.endpoint.prefix /endpoints/AUTH_ fs.swift.service.sahara.region fs.swift.service.sahara.apikey fs.swift.impl org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem fs.swift.connect.timeout fs.swift.socket.timeout fs.swift.connect.retry.count fs.swift.connect.throttle.delay fs.swift.blocksize fs.swift.partsize fs.swift.requestsize sahara-8.0.0/sahara/exceptions.py0000666000175100017510000003064613245514472017016 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import string from oslo_utils import uuidutils import six from sahara.i18n import _ class SaharaException(Exception): """Base Exception for the project To correctly use this class, inherit from it and define a 'message' and 'code' properties. """ code = "UNKNOWN_EXCEPTION" message = _("An unknown exception occurred") def __str__(self): return self.message def __init__(self, message=None, code=None, inject_error_id=True): self.uuid = uuidutils.generate_uuid() if code: self.code = code if message: self.message = message if inject_error_id: # Add Error UUID to the message if required self.message = (_('%(message)s\nError ID: %(id)s') % {'message': self.message, 'id': self.uuid}) super(SaharaException, self).__init__( '%s: %s' % (self.code, self.message)) class NotFoundException(SaharaException): code = "NOT_FOUND" message_template = _("Object '%s' is not found") # It could be a various property of object which was not found def __init__(self, value, message_template=None): self.value = value if message_template: formatted_message = message_template % value else: formatted_message = self.message_template % value super(NotFoundException, self).__init__(formatted_message) class NoUniqueMatchException(SaharaException): code = "NO_UNIQUE_MATCH" message_template = _( "Response {response} is not unique for this query {query}.") def __init__(self, response, query, message_template=None): template = message_template or self.message_template formatted_message = template.format(response=response, query=query) super(NoUniqueMatchException, self).__init__(formatted_message) class NameAlreadyExistsException(SaharaException): code = "NAME_ALREADY_EXISTS" message = _("Name already exists") class InvalidCredentials(SaharaException): message = _("Invalid credentials") code = "INVALID_CREDENTIALS" class InvalidReferenceException(SaharaException): code = "INVALID_REFERENCE" message = _("Invalid object reference") class RemoteCommandException(SaharaException): code = "REMOTE_COMMAND_FAILED" message_template = _("Error during command execution: \"%s\"") def __init__(self, cmd, ret_code=None, stdout=None, stderr=None): self.cmd = cmd self.ret_code = ret_code self.stdout = stdout self.stderr = stderr formatted_message = self.message_template % cmd def to_printable(s): return "".join(filter(lambda x: x in string.printable, s)) if ret_code: formatted_message = '%s\nReturn code: %s' % ( formatted_message, six.text_type(ret_code)) if stderr: formatted_message = '%s\nSTDERR:\n%s' % ( formatted_message, to_printable(stderr)) if stdout: formatted_message = '%s\nSTDOUT:\n%s' % ( formatted_message, to_printable(stdout)) super(RemoteCommandException, self).__init__(formatted_message) class InvalidDataException(SaharaException): """General exception to use for invalid data A more useful message should be passed to __init__ which tells the user more about why the data is invalid. """ code = "INVALID_DATA" message = _("Data is invalid") class BadJobBinaryInternalException(SaharaException): code = "BAD_JOB_BINARY" message = _("Job binary internal data must be a string of length " "greater than zero") class BadJobBinaryException(SaharaException): code = "BAD_JOB_BINARY" message = _("To work with JobBinary located in internal swift add 'user'" " and 'password' to extra") class DBDuplicateEntry(SaharaException): code = "DB_DUPLICATE_ENTRY" message = _("Database object already exists") class CreationFailed(SaharaException): message = _("Object was not created") code = "CREATION_FAILED" class CancelingFailed(SaharaException): message = _("Operation was not canceled") code = "CANCELING_FAILED" class SuspendingFailed(SaharaException): message = _("Operation was not suspended") code = "SUSPENDING_FAILED" class InvalidJobStatus(SaharaException): message = _("Invalid Job Status") code = "INVALID_JOB_STATUS" class DeletionFailed(SaharaException): code = "DELETION_FAILED" message = _("Object was not deleted") class MissingFloatingNetworkException(SaharaException): code = "MISSING_FLOATING_NETWORK" message_template = _("Node Group %s is missing 'floating_ip_pool' " "field") def __init__(self, ng_name): formatted_message = self.message_template % ng_name super(MissingFloatingNetworkException, self).__init__( formatted_message) class SwiftClientException(SaharaException): '''General wrapper object for swift client exceptions This exception is intended for wrapping the message from a swiftclient.ClientException in a SaharaException. The ClientException should be caught and an instance of SwiftClientException raised instead. ''' code = "SWIFT_CLIENT_EXCEPTION" message = _("An error has occurred while performing a request to Swift") class S3ClientException(SaharaException): '''General wrapper object for boto exceptions Intended to replace any errors raised by the botocore client. ''' code = "S3_CLIENT_EXCEPTION" message = _("An error has occurred while performing a request to S3") class DataTooBigException(SaharaException): code = "DATA_TOO_BIG" message_template = _("Size of data (%(size)s) is greater than maximum " "(%(maximum)s)") def __init__(self, size, maximum, message_template=None): if message_template: self.message_template = message_template formatted_message = self.message_template % ( {'size': size, 'maximum': maximum}) super(DataTooBigException, self).__init__(formatted_message) class ThreadException(SaharaException): code = "THREAD_EXCEPTION" message_template = _("An error occurred in thread '%(thread)s': %(e)s" "\n%(stacktrace)s") def __init__(self, thread_description, e, stacktrace): formatted_message = self.message_template % { 'thread': thread_description, 'e': six.text_type(e), 'stacktrace': stacktrace} super(ThreadException, self).__init__(formatted_message) class SubprocessException(SaharaException): code = "SUBPROCESS_EXCEPTION" message = _("Subprocess execution has failed") class NotImplementedException(SaharaException): code = "NOT_IMPLEMENTED" message_template = _("Feature '%s' is not implemented") def __init__(self, feature, message_template=None): if message_template: self.message_template = message_template formatted_message = self.message_template % feature super(NotImplementedException, self).__init__(formatted_message) class HeatStackException(SaharaException): code = "HEAT_STACK_EXCEPTION" message_template = _("Heat stack failed with status %s") def __init__(self, heat_stack_status=None, message=None): if message: formatted_message = message elif heat_stack_status: formatted_message = self.message_template % heat_stack_status else: formatted_message = _("Heat stack failed") super(HeatStackException, self).__init__(formatted_message) class ConfigurationError(SaharaException): code = "CONFIGURATION_ERROR" message = _("The configuration has failed") class IncorrectStateError(SaharaException): message = _("The object is in an incorrect state") code = "INCORRECT_STATE_ERROR" class FrozenClassError(SaharaException): code = "FROZEN_CLASS_ERROR" message_template = _("Class %s is immutable!") def __init__(self, instance): formatted_message = self.message_template % type(instance).__name__ super(FrozenClassError, self).__init__(formatted_message) class SystemError(SaharaException): code = "SYSTEM_ERROR" message = _("System error has occurred") class EDPError(SaharaException): code = "EDP_ERROR" message = _("Failed to complete EDP operation") class OozieException(SaharaException): code = "OOZIE_EXCEPTION" message = _("Failed to perform Oozie request") class TimeoutException(SaharaException): code = "TIMEOUT" message_template = _("'%(operation)s' timed out after %(timeout)i " "second(s)") def __init__(self, timeout, op_name=None, timeout_name=None): if op_name: op_name = _("Operation with name '%s'") % op_name else: op_name = _("Operation") formatted_message = self.message_template % { 'operation': op_name, 'timeout': timeout} if timeout_name: desc = _("%(message)s and following timeout was violated: " "%(timeout_name)s") formatted_message = desc % { 'message': formatted_message, 'timeout_name': timeout_name} super(TimeoutException, self).__init__(formatted_message) class DeprecatedException(SaharaException): code = "DEPRECATED" message = _("The version you are trying to use is deprecated") class Forbidden(SaharaException): code = "FORBIDDEN" message = _("You are not authorized to complete this action") class ImageNotRegistered(SaharaException): code = "IMAGE_NOT_REGISTERED" message_template = _("Image %s is not registered in Sahara") def __init__(self, image): formatted_message = self.message_template % image super(ImageNotRegistered, self).__init__(formatted_message) class MalformedRequestBody(SaharaException): code = "MALFORMED_REQUEST_BODY" message_template = _("Malformed message body: %(reason)s") def __init__(self, reason): formatted_message = self.message_template % {"reason": reason} super(MalformedRequestBody, self).__init__(formatted_message) class QuotaException(SaharaException): code = "QUOTA_ERROR" message_template = _("Quota exceeded for %(resource)s: " "Requested %(requested)s, " "but available %(available)s") def __init__(self, resource, requested, available): formatted_message = self.message_template % { 'resource': resource, 'requested': requested, 'available': available} super(QuotaException, self).__init__(formatted_message) class UpdateFailedException(SaharaException): code = "UPDATE_FAILED" message_template = _("Object '%s' could not be updated") # Object was unable to be updated def __init__(self, value, message_template=None): if message_template: self.message_template = message_template formatted_message = self.message_template % value super(UpdateFailedException, self).__init__(formatted_message) class MaxRetriesExceeded(SaharaException): code = "MAX_RETRIES_EXCEEDED" message_template = _("Operation %(operation)s wasn't executed correctly " "after %(attempts)d attempts") def __init__(self, attempts, operation): formatted_message = self.message_template % {'operation': operation, 'attempts': attempts} super(MaxRetriesExceeded, self).__init__(formatted_message) class InvalidJobExecutionInfoException(SaharaException): message = _("Job execution information is invalid") def __init__(self, message=None): if message: self.message = message self.code = "INVALID_JOB_EXECUTION_INFO" super(InvalidJobExecutionInfoException, self).__init__() sahara-8.0.0/sahara/i18n.py0000666000175100017510000000161213245514472015403 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # It's based on oslo.i18n usage in OpenStack Keystone project and # recommendations from https://docs.openstack.org/oslo.i18n/latest/ # user/usage.html import oslo_i18n _translators = oslo_i18n.TranslatorFactory(domain='sahara') # The primary translation function using the well-known name "_" _ = _translators.primary sahara-8.0.0/sahara/__init__.py0000666000175100017510000000000013245514472016351 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/0000775000175100017510000000000013245515027015407 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/0000775000175100017510000000000013245515027016366 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/conductor/0000775000175100017510000000000013245515027020366 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/conductor/base.py0000666000175100017510000000344113245514472021661 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara.conductor import manager from sahara.tests.unit import base class ConductorManagerTestCase(base.SaharaWithDbTestCase): def __init__(self, *args, **kwargs): """List of check callables could be specified. All return values from callables will be stored in setUp and checked in tearDown. """ self._checks = kwargs.pop("checks", []) super(ConductorManagerTestCase, self).__init__(*args, **kwargs) def setUp(self): super(ConductorManagerTestCase, self).setUp() self.api = manager.ConductorManager() self._results = [] for check in self._checks: self._results.append(copy.deepcopy(check())) def tearDown(self): for idx, check in enumerate(self._checks): check_val = check() self.assertEqual(self._results[idx], check_val, message="Check '%s' failed" % idx) super(ConductorManagerTestCase, self).tearDown() def assert_protected_resource_exception(self, ex): self.assertIn("marked as protected", str(ex)) def assert_created_in_another_tenant_exception(self, ex): self.assertIn("wasn't created in this tenant", str(ex)) sahara-8.0.0/sahara/tests/unit/conductor/__init__.py0000666000175100017510000000000013245514472022472 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/conductor/test_resource.py0000666000175100017510000002006013245514472023631 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import testtools from sahara.conductor import resource as r from sahara import exceptions as ex from sahara.swift import swift_helper from sahara.utils import edp SAMPLE_DICT = { 'first': [1, 2], 'second': {'a': 1, 'b': 2} } SAMPLE_NESTED_LISTS_DICT = { 'a': [[{'b': 123}]] } SAMPLE_CLUSTER_DICT = { 'name': 'test-cluster', 'cluster_configs': { 'general': { 'some_overridden_config': 'somevalue' } }, 'node_groups': [ { 'name': 'master', 'id': 'some_id' }, { 'id': 'some_id', 'name': 'worker', 'node_processes': ['tasktracker', 'datanode'], 'node_configs': {}, 'instances': [ { 'name': 'test-cluster-001', 'ip': '1.1.1.1' } ] } ] } SAMPLE_JOB_BINARY_DICT = { "created_at": "2014-02-14 16:26:08.895897", "description": "a job", "extra": { "password": "password", "user": "user" }, "id": "c0caf119-f380-4fab-a46e-0f28ebd23b5c", "name": "bob", "tenant_id": "6b859fb8d1f44e8eafdfb91f21309b5f", "updated_at": "null", "url": "swift://bob.sahara/job" } SAMPLE_JOB_BINARY_DICT2 = copy.copy(SAMPLE_JOB_BINARY_DICT) SAMPLE_JOB_BINARY_DICT2["name"] = "bill" SAMPLE_JOB_BINARY_DICT2["id"] = "c0caf119-1111-2222-a46e-0f28ebd23b5c" SAMPLE_JOB_BINARY_DICT2["url"] = "swift://bill.sahara/job" SAMPLE_JOB_DICT = { "tenant_id": "test_tenant", "name": "job_test", "description": "test_desc", "type": "Pig", "mains": [SAMPLE_JOB_BINARY_DICT], "libs": [SAMPLE_JOB_BINARY_DICT2] } SAMPLE_DATA_SOURCE = { 'name': 'input', 'description': 'some input', 'type': 'swift', 'url': 'swift://tmckay.sahara', 'credentials': { 'username': 'me', 'password': 'password' } } SAMPLE_JOB_EXECUTION = { "cluster_id": "7ed1c016-a8a3-4209-9931-6e80f58eea80", "created_at": "2014-02-14 17:46:56.631209", "extra": {}, "id": "1b0b1874-a261-4d1f-971a-a2cebadeba6c", "info": { "actions": [{"conf": "some stuff"}, {"conf": "more stuff"}], "status": edp.JOB_STATUS_PENDING }, "input_id": "b5ddde55-594e-428f-9040-028be81eb3c2", "job_configs": { "args": [ "bob", "bill" ], "configs": { swift_helper.HADOOP_SWIFT_PASSWORD: "openstack", swift_helper.HADOOP_SWIFT_USERNAME: "admin", "myfavoriteconfig": 1 }, "proxy_configs": { "proxy_username": "admin", "proxy_password": "openstack" }, "trusts": { "input_id": "9c528755099149b8b7166f3d0fa3bf10", "output_id": "3f2bde9d43ec440381dc9f736481e2b0" } }, "job_id": "d0f3e397-7bef-42f9-a4db-e5a96059246e", "output_id": "f4993830-aa97-4b0b-914a-ab6430f742b6", "tenant_id": "6b859fb8d1f44e8eafdfb91f21309b5f" } class TestResource(testtools.TestCase): def test_resource_creation(self): res = r.Resource(SAMPLE_DICT) self.assertIsInstance(res.first, list) self.assertEqual([1, 2], res.first) self.assertIsInstance(res.second, r.Resource) self.assertEqual(1, res.second.a) self.assertEqual(2, res.second.b) def test_resource_immutability(self): res = r.Resource(SAMPLE_DICT) with testtools.ExpectedException(ex.FrozenClassError): res.first.append(123) with testtools.ExpectedException(ex.FrozenClassError): res.first = 123 with testtools.ExpectedException(ex.FrozenClassError): res.second.a = 123 def test_nested_lists(self): res = r.Resource(SAMPLE_NESTED_LISTS_DICT) self.assertEqual(123, res.a[0][0].b) def test_cluster_resource(self): cluster = r.ClusterResource(SAMPLE_CLUSTER_DICT) self.assertEqual('test-cluster', cluster.name) self.assertEqual('master', cluster.node_groups[0].name) self.assertIsInstance(cluster.node_groups[0], r.NodeGroupResource) self.assertEqual('test-cluster', cluster.node_groups[0].cluster.name) self.assertEqual('test-cluster-001', cluster.node_groups[1].instances[0].name) self.assertIsInstance(cluster.node_groups[1].instances[0], r.InstanceResource) self.assertEqual('worker', cluster.node_groups[1].instances[0].node_group.name) def test_to_dict(self): cluster = r.ClusterResource(SAMPLE_CLUSTER_DICT) self.assertEqual(SAMPLE_CLUSTER_DICT, cluster.to_dict()) def test_to_dict_filtering(self): cluster_dict = copy.deepcopy(SAMPLE_CLUSTER_DICT) cluster_dict['management_private_key'] = 'abacaba' cluster_dict['node_groups'][0]['id'] = 'some_id' cluster = r.ClusterResource(cluster_dict) self.assertEqual(SAMPLE_CLUSTER_DICT, cluster.to_dict()) def test_to_wrapped_dict(self): cluster = r.ClusterResource(SAMPLE_CLUSTER_DICT) wrapped_dict = cluster.to_wrapped_dict() self.assertEqual(1, len(wrapped_dict)) self.assertEqual(SAMPLE_CLUSTER_DICT, wrapped_dict['cluster']) def test_job_binary_filter_extra(self): job_binary = r.JobBinary(SAMPLE_JOB_BINARY_DICT) wrapped_dict = job_binary.to_wrapped_dict() self.assertNotIn('extra', wrapped_dict) def test_data_source_filter_credentials(self): data_source = r.DataSource(SAMPLE_DATA_SOURCE) wrapped_dict = data_source.to_wrapped_dict() self.assertNotIn('credentials', wrapped_dict) def test_job_filter_job_binary(self): job = r.Job(SAMPLE_JOB_DICT) wrapped_dict = job.to_wrapped_dict() self.assertIn('mains', wrapped_dict["job"]) self.assertIn('libs', wrapped_dict["job"]) self.assertNotIn('extra', wrapped_dict["job"]['mains']) self.assertNotIn('extra', wrapped_dict["job"]['libs']) def test_job_execution_filter_credentials(self): job_exec = r.JobExecution(SAMPLE_JOB_EXECUTION) self.assertIn('extra', job_exec) self.assertIn(swift_helper.HADOOP_SWIFT_PASSWORD, job_exec['job_configs']['configs']) self.assertIn(swift_helper.HADOOP_SWIFT_USERNAME, job_exec['job_configs']['configs']) for a in job_exec['info']['actions']: self.assertIn('conf', a) self.assertIn('trusts', job_exec['job_configs']) self.assertIn('input_id', job_exec['job_configs']['trusts']) self.assertIn('output_id', job_exec['job_configs']['trusts']) self.assertIn('proxy_configs', job_exec['job_configs']) self.assertIn('proxy_username', job_exec['job_configs']['proxy_configs']) self.assertIn('proxy_password', job_exec['job_configs']['proxy_configs']) wrapped_dict = job_exec.to_wrapped_dict()['job_execution'] self.assertNotIn('extra', wrapped_dict) configs = wrapped_dict['job_configs']['configs'] self.assertEqual("", configs[swift_helper.HADOOP_SWIFT_PASSWORD]) self.assertEqual("", configs[swift_helper.HADOOP_SWIFT_USERNAME]) for a in wrapped_dict['info']['actions']: self.assertNotIn('conf', a) self.assertNotIn('trusts', wrapped_dict['job_configs']) self.assertNotIn('proxy_configs', wrapped_dict['job_configs']) sahara-8.0.0/sahara/tests/unit/conductor/manager/0000775000175100017510000000000013245515027022000 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/conductor/manager/test_from_template.py0000666000175100017510000000757113245514472026266 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara.conductor import manager from sahara import context import sahara.tests.unit.conductor.base as test_base from sahara.tests.unit.conductor.manager import test_clusters from sahara.tests.unit.conductor.manager import test_templates CORRECT_CONF = { u'service_1': {u'config_2': u'value_2', u'config_1': u'value_1'}, u'service_2': {u'config_1': u'value_1'} } class ObjectsFromTemplatesTest(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(ObjectsFromTemplatesTest, self).__init__( checks=[ lambda: CORRECT_CONF, lambda: test_clusters.SAMPLE_CLUSTER, lambda: test_templates.SAMPLE_CLT, lambda: test_templates.SAMPLE_NGT, lambda: manager.CLUSTER_DEFAULTS, lambda: manager.NODE_GROUP_DEFAULTS, lambda: manager.INSTANCE_DEFAULTS, ], *args, **kwargs) def test_cluster_create_from_templates(self): ctx = context.ctx() # create node_group_template ng_tmpl = copy.deepcopy(test_templates.SAMPLE_NGT) ng_tmpl['volumes_size'] = 10 ng_tmpl['node_configs']['service_1']['config_2'] = 'value_2' ng_tmpl = self.api.node_group_template_create(ctx, ng_tmpl) # create cluster template cl_tmpl = self.api.cluster_template_create(ctx, test_templates.SAMPLE_CLT) # create cluster cluster_val = copy.deepcopy(test_clusters.SAMPLE_CLUSTER) cluster_val['cluster_template_id'] = cl_tmpl['id'] cluster_val['node_groups'][0]['node_group_template_id'] = ng_tmpl['id'] cluster = self.api.cluster_create(ctx, cluster_val) self.assertEqual(CORRECT_CONF, cluster['cluster_configs']) for node_group in cluster['node_groups']: if node_group['name'] == 'ng_1': self.assertEqual(['p1', 'p2'], node_group['node_processes']) self.assertEqual(10, node_group['volumes_size']) self.assertEqual(CORRECT_CONF, node_group['node_configs']) def test_node_group_add_from_template(self): ctx = context.ctx() # create cluster sample_copy = copy.deepcopy(test_clusters.SAMPLE_CLUSTER) cluster = self.api.cluster_create(ctx, sample_copy) # create node_group_template ng_tmpl = copy.deepcopy(test_templates.SAMPLE_NGT) ng_tmpl['volumes_size'] = 10 ng_tmpl['node_configs']['service_1']['config_2'] = 'value_2' ng_tmpl = self.api.node_group_template_create(ctx, ng_tmpl) # add node group to cluster ng = copy.deepcopy(test_clusters.SAMPLE_CLUSTER['node_groups'][0]) ng['node_group_template_id'] = ng_tmpl['id'] ng['count'] = 5 ng['name'] = 'ng_3' self.api.node_group_add(ctx, cluster['id'], ng) # refetch cluster cluster = self.api.cluster_get(ctx, cluster['id']) for node_group in cluster['node_groups']: if node_group['name'] == 'ng_3': self.assertEqual(['p1', 'p2'], node_group['node_processes']) self.assertEqual(10, node_group['volumes_size']) self.assertEqual(CORRECT_CONF, node_group['node_configs']) self.assertEqual(5, node_group['count']) sahara-8.0.0/sahara/tests/unit/conductor/manager/test_templates.py0000666000175100017510000007177713245514472025437 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import mock from oslo_utils import uuidutils import six from sqlalchemy import exc as sa_ex import testtools from sahara.conductor import manager from sahara import context from sahara.db.sqlalchemy import models as m from sahara import exceptions as ex from sahara.service.validations import cluster_template_schema as cl_schema from sahara.service.validations import node_group_template_schema as ngt_schema import sahara.tests.unit.conductor.base as test_base import sahara.tests.unit.conductor.manager.test_clusters as cluster_tests SAMPLE_NGT = { "name": "ngt_test", "flavor_id": "42", "plugin_name": "test_plugin", "hadoop_version": "test_version", "node_processes": ["p1", "p2"], "image_id": uuidutils.generate_uuid(), "node_configs": { "service_1": { "config_1": "value_1" }, "service_2": { "config_1": "value_1" } }, "volumes_per_node": 1, "volumes_size": 1, "volume_type": "big", "volumes_availability_zone": "here", "volume_mount_prefix": "/tmp", "description": "my template", "floating_ip_pool": "public", "security_groups": ["cat", "dog"], "auto_security_group": False, "availability_zone": "here", "is_proxy_gateway": False, "volume_local_to_instance": False, 'use_autoconfig': True, "is_public": False, "is_protected": False } SAMPLE_CLT = { "name": "clt_test", "plugin_name": "test_plugin", "hadoop_version": "test_version", "default_image_id": uuidutils.generate_uuid(), "cluster_configs": { "service_1": { "config_1": "value_1" }, "service_2": { "config_1": "value_1" } }, "node_groups": [ { "name": "ng_1", "flavor_id": "42", "node_processes": ["p1", "p2"], "count": 1, "floating_ip_pool": None, "security_groups": None, "availability_zone": None, 'use_autoconfig': True, "shares": None }, { "name": "ng_2", "flavor_id": "42", "node_processes": ["p3", "p4"], "count": 3, "floating_ip_pool": None, "security_groups": ["group1", "group2"], "availability_zone": None, 'use_autoconfig': True, "shares": None } ], "anti_affinity": ["datanode"], "description": "my template", "neutron_management_network": uuidutils.generate_uuid(), "shares": None, "is_public": False, "is_protected": False } class NodeGroupTemplates(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(NodeGroupTemplates, self).__init__( checks=[ lambda: SAMPLE_CLT, lambda: SAMPLE_NGT, lambda: manager.CLUSTER_DEFAULTS, lambda: manager.NODE_GROUP_DEFAULTS, lambda: manager.INSTANCE_DEFAULTS, ], *args, **kwargs) def test_minimal_ngt_create_list_delete(self): ctx = context.ctx() self.api.node_group_template_create(ctx, SAMPLE_NGT) lst = self.api.node_group_template_get_all(ctx) self.assertEqual(1, len(lst)) ngt_id = lst[0]['id'] self.api.node_group_template_destroy(ctx, ngt_id) lst = self.api.node_group_template_get_all(ctx) self.assertEqual(0, len(lst)) def test_duplicate_ngt_create(self): ctx = context.ctx() self.api.node_group_template_create(ctx, SAMPLE_NGT) with testtools.ExpectedException(ex.DBDuplicateEntry): self.api.node_group_template_create(ctx, SAMPLE_NGT) def test_ngt_fields(self): ctx = context.ctx() ngt_db_obj_id = self.api.node_group_template_create( ctx, SAMPLE_NGT)['id'] ngt_db_obj = self.api.node_group_template_get(ctx, ngt_db_obj_id) self.assertIsInstance(ngt_db_obj, dict) for key, val in SAMPLE_NGT.items(): self.assertEqual(val, ngt_db_obj.get(key), "Key not found %s" % key) def test_ngt_delete(self): ctx = context.ctx() db_obj_ngt = self.api.node_group_template_create(ctx, SAMPLE_NGT) _id = db_obj_ngt['id'] self.api.node_group_template_destroy(ctx, _id) with testtools.ExpectedException(ex.NotFoundException): self.api.node_group_template_destroy(ctx, _id) def test_ngt_delete_default(self): ctx = context.ctx() vals = copy.copy(SAMPLE_NGT) vals["name"] = "protected" vals["is_protected"] = True ngt_prot = self.api.node_group_template_create(ctx, vals) ngt_prot_id = ngt_prot['id'] vals["name"] = "protected_default" vals["is_protected"] = True vals["is_default"] = True ngt_prot_def = self.api.node_group_template_create(ctx, vals) ngt_prot_def_id = ngt_prot_def['id'] # We should not be able to delete ngt_prot until we remove # the protected flag, even if we pass ignore_prot_on_def with testtools.ExpectedException(ex.DeletionFailed): self.api.node_group_template_destroy(ctx, ngt_prot_id) with testtools.ExpectedException(ex.DeletionFailed): self.api.node_group_template_destroy(ctx, ngt_prot_id, ignore_prot_on_def=True) update_values = {"is_protected": False} self.api.node_group_template_update(ctx, ngt_prot_id, update_values) self.api.node_group_template_destroy(ctx, ngt_prot_id) with testtools.ExpectedException(ex.NotFoundException): self.api.node_group_template_destroy(ctx, ngt_prot_id) # However, for the protected_default we should be able to # override the protected check by passing ignore_prot_on_def with testtools.ExpectedException(ex.DeletionFailed): self.api.node_group_template_destroy(ctx, ngt_prot_def_id) self.api.node_group_template_destroy(ctx, ngt_prot_def_id, ignore_prot_on_def=True) with testtools.ExpectedException(ex.NotFoundException): self.api.node_group_template_destroy(ctx, ngt_prot_def_id) def test_ngt_search(self): ctx = context.ctx() ngt = copy.deepcopy(SAMPLE_NGT) ngt["name"] = "frederica" ngt["plugin_name"] = "test plugin" self.api.node_group_template_create(ctx, ngt) lst = self.api.node_group_template_get_all(ctx) self.assertEqual(1, len(lst)) # Exact match kwargs = {'name': ngt['name'], 'plugin_name': ngt['plugin_name']} lst = self.api.node_group_template_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Valid field but no matching value kwargs = {'name': ngt['name']+"foo"} lst = self.api.node_group_template_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Valid field with substrings kwargs = {'name': "red", 'plugin_name': "test"} lst = self.api.node_group_template_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Invalid field self.assertRaises(sa_ex.InvalidRequestError, self.api.node_group_template_get_all, ctx, **{'badfield': 'junk'}) @mock.patch('sahara.db.sqlalchemy.api.regex_filter') def test_ngt_search_regex(self, regex_filter): # do this so we can return the correct value def _regex_filter(query, cls, regex_cols, search_opts): return query, search_opts regex_filter.side_effect = _regex_filter ctx = context.ctx() self.api.node_group_template_get_all(ctx) self.assertEqual(0, regex_filter.call_count) self.api.node_group_template_get_all(ctx, regex_search=True, name="fox") self.assertEqual(1, regex_filter.call_count) args, kwargs = regex_filter.call_args self.assertIs(args[1], m.NodeGroupTemplate) self.assertEqual(args[2], ["name", "description", "plugin_name", "tenant_id"]) self.assertEqual(args[3], {"name": "fox"}) def test_ngt_update(self): ctx = context.ctx() ngt = self.api.node_group_template_create(ctx, SAMPLE_NGT) ngt_id = ngt["id"] UPDATE_NAME = "UpdatedSampleNGTName" update_values = {"name": UPDATE_NAME} updated_ngt = self.api.node_group_template_update(ctx, ngt_id, update_values) self.assertEqual(UPDATE_NAME, updated_ngt["name"]) updated_ngt = self.api.node_group_template_get(ctx, ngt_id) self.assertEqual(UPDATE_NAME, updated_ngt["name"]) with testtools.ExpectedException(ex.NotFoundException): self.api.node_group_template_update(ctx, -1, update_values) ngt = self.api.node_group_template_create(ctx, SAMPLE_NGT) ngt_id = ngt['id'] with testtools.ExpectedException(ex.DBDuplicateEntry): self.api.node_group_template_update(ctx, ngt_id, update_values) def test_ngt_update_default(self): ctx = context.ctx() vals = copy.copy(SAMPLE_NGT) vals["name"] = "protected" vals["is_protected"] = True ngt_prot = self.api.node_group_template_create(ctx, vals) ngt_prot_id = ngt_prot["id"] vals["name"] = "protected_default" vals["is_protected"] = True vals["is_default"] = True ngt_prot_def = self.api.node_group_template_create(ctx, vals) ngt_prot_def_id = ngt_prot_def["id"] # We should not be able to update ngt_prot until we remove # the is_protected flag, even if we pass ignore_prot_on_def UPDATE_NAME = "UpdatedSampleNGTName" update_values = {"name": UPDATE_NAME} with testtools.ExpectedException(ex.UpdateFailedException): self.api.node_group_template_update(ctx, ngt_prot_id, update_values) with testtools.ExpectedException(ex.UpdateFailedException): self.api.node_group_template_update(ctx, ngt_prot_id, update_values, ignore_prot_on_def=True) update_values["is_protected"] = False updated_ngt = self.api.node_group_template_update(ctx, ngt_prot_id, update_values) self.assertEqual(UPDATE_NAME, updated_ngt["name"]) # However, for the ngt_prot_def we should be able to # override the is_protected check by passing ignore_prot_on_def update_values = {"name": UPDATE_NAME+"default"} with testtools.ExpectedException(ex.UpdateFailedException): self.api.node_group_template_update(ctx, ngt_prot_def_id, update_values) updated_ngt = self.api.node_group_template_update( ctx, ngt_prot_def_id, update_values, ignore_prot_on_def=True) self.assertEqual(UPDATE_NAME+"default", updated_ngt["name"]) self.assertTrue(updated_ngt["is_protected"]) self.assertTrue(updated_ngt["is_default"]) def test_ngt_update_with_nulls(self): ctx = context.ctx() ngt = self.api.node_group_template_create(ctx, SAMPLE_NGT) ngt_id = ngt["id"] updated_values = copy.deepcopy(SAMPLE_NGT) for prop, value in six.iteritems( ngt_schema.NODE_GROUP_TEMPLATE_SCHEMA["properties"]): if type(value["type"]) is list and "null" in value["type"]: updated_values[prop] = None # Prove that we can call update on these fields with null values # without an exception self.api.node_group_template_update(ctx, ngt_id, updated_values) updated_ngt = self.api.node_group_template_get(ctx, ngt_id) for prop, value in six.iteritems(updated_values): if value is None: self.assertIsNone(updated_ngt[prop]) def test_ngt_update_delete_when_protected(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_NGT) sample['is_protected'] = True ngt = self.api.node_group_template_create(ctx, sample) ngt_id = ngt["id"] with testtools.ExpectedException(ex.UpdateFailedException): self.api.node_group_template_update(ctx, ngt_id, {"name": "tmpl"}) with testtools.ExpectedException(ex.DeletionFailed): self.api.node_group_template_destroy(ctx, ngt_id) self.api.node_group_template_update(ctx, ngt_id, {"name": "tmpl", "is_protected": False}) def test_public_ngt_update_from_another_tenant(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_NGT) sample['is_public'] = True ngt = self.api.node_group_template_create(ctx, sample) ngt_id = ngt["id"] ctx.tenant_id = 'tenant_2' with testtools.ExpectedException(ex.UpdateFailedException): self.api.node_group_template_update(ctx, ngt_id, {"name": "tmpl"}) def test_public_ngt_delete_from_another_tenant(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_NGT) sample['is_public'] = True ngt = self.api.node_group_template_create(ctx, sample) ngt_id = ngt["id"] ctx.tenant_id = 'tenant_2' with testtools.ExpectedException(ex.DeletionFailed): self.api.node_group_template_destroy(ctx, ngt_id) class ClusterTemplates(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(ClusterTemplates, self).__init__( checks=[ lambda: SAMPLE_CLT, lambda: SAMPLE_NGT, lambda: manager.CLUSTER_DEFAULTS, lambda: manager.NODE_GROUP_DEFAULTS, lambda: manager.INSTANCE_DEFAULTS, ], *args, **kwargs) def test_minimal_clt_create_list_delete(self): ctx = context.ctx() self.api.cluster_template_create(ctx, SAMPLE_CLT) lst = self.api.cluster_template_get_all(ctx) self.assertEqual(1, len(lst)) clt_id = lst[0]['id'] self.api.cluster_template_destroy(ctx, clt_id) lst = self.api.cluster_template_get_all(ctx) self.assertEqual(0, len(lst)) with testtools.ExpectedException(ex.NotFoundException): self.api.cluster_template_destroy(ctx, clt_id) def test_duplicate_clt_create(self): ctx = context.ctx() self.api.cluster_template_create(ctx, SAMPLE_CLT) with testtools.ExpectedException(ex.DBDuplicateEntry): self.api.cluster_template_create(ctx, SAMPLE_CLT) def test_clt_fields(self): ctx = context.ctx() clt_db_obj_id = self.api.cluster_template_create(ctx, SAMPLE_CLT)['id'] clt_db_obj = self.api.cluster_template_get(ctx, clt_db_obj_id) self.assertIsInstance(clt_db_obj, dict) for key, val in SAMPLE_CLT.items(): if key == 'node_groups': # this will be checked separately continue self.assertEqual(val, clt_db_obj.get(key), "Key not found %s" % key) for ng in clt_db_obj["node_groups"]: ng.pop("created_at") ng.pop("updated_at") ng.pop("id") ng.pop("tenant_id") self.assertEqual(clt_db_obj_id, ng.pop("cluster_template_id")) ng.pop("image_id") ng.pop("node_configs") ng.pop("node_group_template_id") ng.pop("volume_mount_prefix") ng.pop("volumes_size") ng.pop("volumes_per_node") ng.pop("volumes_availability_zone") ng.pop("volume_type") ng.pop("auto_security_group") ng.pop("is_proxy_gateway") ng.pop('volume_local_to_instance') self.assertEqual(SAMPLE_CLT["node_groups"], clt_db_obj["node_groups"]) def test_clt_delete(self): ctx = context.ctx() db_obj_clt = self.api.cluster_template_create(ctx, SAMPLE_CLT) _id = db_obj_clt['id'] self.api.cluster_template_destroy(ctx, _id) with testtools.ExpectedException(ex.NotFoundException): self.api.cluster_template_destroy(ctx, _id) def test_clt_delete_default(self): ctx = context.ctx() vals = copy.copy(SAMPLE_CLT) vals["name"] = "protected" vals["is_protected"] = True clt_prot = self.api.cluster_template_create(ctx, vals) clt_prot_id = clt_prot['id'] vals["name"] = "protected_default" vals["is_protected"] = True vals["is_default"] = True clt_prot_def = self.api.cluster_template_create(ctx, vals) clt_prot_def_id = clt_prot_def['id'] # We should not be able to delete clt_prot until we remove # the is_protected flag, even if we pass ignore_prot_on_def with testtools.ExpectedException(ex.DeletionFailed): self.api.cluster_template_destroy(ctx, clt_prot_id) with testtools.ExpectedException(ex.DeletionFailed): self.api.cluster_template_destroy(ctx, clt_prot_id, ignore_prot_on_def=True) update_values = {"is_protected": False} self.api.cluster_template_update(ctx, clt_prot_id, update_values) self.api.cluster_template_destroy(ctx, clt_prot_id) with testtools.ExpectedException(ex.NotFoundException): self.api.cluster_template_destroy(ctx, clt_prot_id) # However, for clt_prot_def we should be able to override # the is_protected check by passing ignore_prot_on_def with testtools.ExpectedException(ex.DeletionFailed): self.api.cluster_template_destroy(ctx, clt_prot_def_id) self.api.cluster_template_destroy(ctx, clt_prot_def_id, ignore_prot_on_def=True) with testtools.ExpectedException(ex.NotFoundException): self.api.cluster_template_destroy(ctx, clt_prot_def_id) def test_clt_search(self): ctx = context.ctx() clt = copy.deepcopy(SAMPLE_CLT) clt["name"] = "frederica" clt["plugin_name"] = "test_plugin" self.api.cluster_template_create(ctx, clt) lst = self.api.cluster_template_get_all(ctx) self.assertEqual(1, len(lst)) # Exact match kwargs = {'name': clt['name'], 'plugin_name': clt['plugin_name']} lst = self.api.cluster_template_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Valid field but no matching value kwargs = {'name': clt['name']+"foo"} lst = self.api.cluster_template_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Valid field with substrings kwargs = {'name': "red", 'plugin_name': "test"} lst = self.api.cluster_template_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Invalid field self.assertRaises(sa_ex.InvalidRequestError, self.api.cluster_template_get_all, ctx, **{'badfield': 'junk'}) @mock.patch('sahara.db.sqlalchemy.api.regex_filter') def test_clt_search_regex(self, regex_filter): # do this so we can return the correct value def _regex_filter(query, cls, regex_cols, search_opts): return query, search_opts regex_filter.side_effect = _regex_filter ctx = context.ctx() self.api.cluster_template_get_all(ctx) self.assertEqual(0, regex_filter.call_count) self.api.cluster_template_get_all(ctx, regex_search=True, name="fox") self.assertEqual(1, regex_filter.call_count) args, kwargs = regex_filter.call_args self.assertIs(args[1], m.ClusterTemplate) self.assertEqual(args[2], ["name", "description", "plugin_name", "tenant_id"]) self.assertEqual(args[3], {"name": "fox"}) def test_clt_update(self): ctx = context.ctx() clt = self.api.cluster_template_create(ctx, SAMPLE_CLT) clt_id = clt["id"] UPDATE_NAME = "UpdatedClusterTemplate" update_values = {"name": UPDATE_NAME} updated_clt = self.api.cluster_template_update(ctx, clt_id, update_values) self.assertEqual(UPDATE_NAME, updated_clt["name"]) updated_clt = self.api.cluster_template_get(ctx, clt_id) self.assertEqual(UPDATE_NAME, updated_clt["name"]) self.assertEqual(clt["node_groups"], updated_clt["node_groups"]) # check duplicate name handling clt = self.api.cluster_template_create(ctx, SAMPLE_CLT) clt_id = clt["id"] with testtools.ExpectedException(ex.DBDuplicateEntry): self.api.cluster_template_update(ctx, clt_id, update_values) with testtools.ExpectedException(ex.NotFoundException): self.api.cluster_template_update(ctx, -1, update_values) # create a cluster and try updating the referenced cluster template cluster_val = copy.deepcopy(cluster_tests.SAMPLE_CLUSTER) cluster_val['name'] = "ClusterTemplateUpdateTestCluster" cluster_val['cluster_template_id'] = clt['id'] self.api.cluster_create(ctx, cluster_val) update_values = {"name": "noUpdateInUseName"} with testtools.ExpectedException(ex.UpdateFailedException): self.api.cluster_template_update(ctx, clt['id'], update_values) def test_clt_update_default(self): ctx = context.ctx() vals = copy.copy(SAMPLE_CLT) vals["name"] = "protected" vals["is_protected"] = True clt_prot = self.api.cluster_template_create(ctx, vals) clt_prot_id = clt_prot["id"] vals["name"] = "protected_default" vals["is_protected"] = True vals["is_default"] = True clt_prot_def = self.api.cluster_template_create(ctx, vals) clt_prot_def_id = clt_prot_def["id"] # We should not be able to update clt_prot until we remove # the is_protected flag, even if we pass ignore_prot_on_def UPDATE_NAME = "UpdatedClusterTemplate" update_values = {"name": UPDATE_NAME} with testtools.ExpectedException(ex.UpdateFailedException): self.api.cluster_template_update(ctx, clt_prot_id, update_values) with testtools.ExpectedException(ex.UpdateFailedException): self.api.cluster_template_update(ctx, clt_prot_id, update_values, ignore_prot_on_def=True) update_values["is_protected"] = False updated_clt = self.api.cluster_template_update(ctx, clt_prot_id, update_values) self.assertEqual(UPDATE_NAME, updated_clt["name"]) # However, for the clt_prot_def we should be able to # override the is_protected check by passing ignore_prot_on_def update_values = {"name": UPDATE_NAME+"default"} with testtools.ExpectedException(ex.UpdateFailedException): self.api.cluster_template_update(ctx, clt_prot_def_id, update_values) updated_clt = self.api.cluster_template_update(ctx, clt_prot_def_id, update_values, ignore_prot_on_def=True) self.assertEqual(UPDATE_NAME+"default", updated_clt["name"]) self.assertTrue(updated_clt["is_default"]) self.assertTrue(updated_clt["is_protected"]) def test_clt_update_with_nulls(self): ctx = context.ctx() clt = self.api.cluster_template_create(ctx, SAMPLE_CLT) clt_id = clt["id"] updated_values = copy.deepcopy(SAMPLE_CLT) for prop, value in six.iteritems( cl_schema.CLUSTER_TEMPLATE_SCHEMA["properties"]): if type(value["type"]) is list and "null" in value["type"]: updated_values[prop] = None # Prove that we can call update on these fields with null values # without an exception self.api.cluster_template_update(ctx, clt_id, updated_values) updated_clt = self.api.cluster_template_get(ctx, clt_id) for prop, value in six.iteritems(updated_values): if value is None: # Conductor populates node groups with [] when # the value given is null if prop == "node_groups": self.assertEqual([], updated_clt[prop]) else: self.assertIsNone(updated_clt[prop]) def test_clt_update_delete_when_protected(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_CLT) sample['is_protected'] = True clt = self.api.cluster_template_create(ctx, sample) clt_id = clt["id"] with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.cluster_template_update(ctx, clt_id, {"name": "tmpl"}) except ex.UpdateFailedException as e: self.assert_protected_resource_exception(e) raise with testtools.ExpectedException(ex.DeletionFailed): try: self.api.cluster_template_destroy(ctx, clt_id) except ex.DeletionFailed as e: self.assert_protected_resource_exception(e) raise self.api.cluster_template_update(ctx, clt_id, {"name": "tmpl", "is_protected": False}) def test_public_clt_update_delete_from_another_tenant(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_CLT) sample['is_public'] = True clt = self.api.cluster_template_create(ctx, sample) clt_id = clt["id"] ctx.tenant_id = 'tenant_2' with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.cluster_template_update(ctx, clt_id, {"name": "tmpl"}) except ex.UpdateFailedException as e: self.assert_created_in_another_tenant_exception(e) raise with testtools.ExpectedException(ex.DeletionFailed): try: self.api.cluster_template_destroy(ctx, clt_id) except ex.DeletionFailed as e: self.assert_created_in_another_tenant_exception(e) raise def test_update_clt_on_ngt_update(self): # Prove that cluster templates get updated with proper values # after a referenced node group template is updated ctx = context.ctx() ngt = self.api.node_group_template_create(ctx, SAMPLE_NGT) sample = copy.deepcopy(SAMPLE_CLT) sample["node_groups"] = [ {"node_group_template_id": ngt['id'], "count": 1} ] ct = self.api.cluster_template_create(ctx, sample) UPDATE_FLAVOR = "41" update_values = {"flavor_id": UPDATE_FLAVOR} self.api.node_group_template_update(ctx, ngt["id"], update_values) updated_ct = self.api.cluster_template_get(ctx, ct["id"]) self.assertEqual(UPDATE_FLAVOR, updated_ct["node_groups"][0]["flavor_id"]) sahara-8.0.0/sahara/tests/unit/conductor/manager/test_defaults.py0000666000175100017510000000564213245514472025234 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six from sahara.conductor import manager from sahara import context import sahara.tests.unit.conductor.base as test_base from sahara.tests.unit.conductor.manager import test_clusters from sahara.utils import general class DefaultsTest(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(DefaultsTest, self).__init__( checks=[ lambda: test_clusters.SAMPLE_CLUSTER, lambda: manager.CLUSTER_DEFAULTS, lambda: manager.NODE_GROUP_DEFAULTS, lambda: manager.INSTANCE_DEFAULTS, ], *args, **kwargs) def _assert_props(self, obj, **rules): for k, v in six.iteritems(rules): self.assertIn(k, obj) self.assertEqual(v, obj[k]) def test_apply_defaults(self): self.assertEqual( {"a": 1, "b": 2, "c": 0}, manager._apply_defaults({"c": 0, "b": 2}, {"a": 1, "b": 2, "c": 3})) def _create_sample_cluster(self): ctx = context.ctx() cluster = self.api.cluster_create(ctx, test_clusters.SAMPLE_CLUSTER) self.assertIsInstance(cluster, dict) return cluster def test_cluster_defaults(self): cluster = self._create_sample_cluster() self._assert_props(cluster, status="undefined", status_description="", info={}) def test_node_group_defaults(self): cluster = self._create_sample_cluster() for ng in cluster['node_groups']: self._assert_props(ng, node_configs={}, volumes_per_node=0, volumes_size=0, volume_mount_prefix="/volumes/disk") def test_instance_defaults(self): ctx = context.ctx() cluster = self._create_sample_cluster() cluster_id = cluster["id"] ng_id = cluster["node_groups"][-1]["id"] self.api.instance_add(ctx, ng_id, { "instance_name": "vm123" }) cluster = self.api.cluster_get(ctx, cluster_id) ng = general.find_dict(cluster['node_groups'], id=ng_id) instance = general.find_dict(ng['instances'], instance_name="vm123") self._assert_props(instance, volumes=[]) sahara-8.0.0/sahara/tests/unit/conductor/manager/test_edp.py0000666000175100017510000012045413245514472024174 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import datetime import mock from sqlalchemy import exc as sa_exc import testtools from sahara import context from sahara.db.sqlalchemy import models as m from sahara import exceptions as ex from sahara.service.castellan import config as castellan import sahara.tests.unit.conductor.base as test_base from sahara.tests.unit.conductor.manager import test_clusters from sahara.utils import edp SAMPLE_DATA_SOURCE = { "tenant_id": "tenant_1", "name": "ngt_test", "description": "test_desc", "type": "Cassandra", "url": "localhost:1080", "credentials": { "user": "test", "password": "123" }, "is_public": False, "is_protected": False } SAMPLE_JOB = { "tenant_id": "tenant_1", "name": "job_test", "description": "test_desc", "type": edp.JOB_TYPE_PIG, "mains": [], "is_public": False, "is_protected": False } SAMPLE_JOB_EXECUTION = { "tenant_id": "tenant_1", "return_code": "1", "job_id": "undefined", "input_id": "undefined", "output_id": "undefined", "start_time": datetime.datetime.now(), "cluster_id": None, "is_public": False, "is_protected": False } SAMPLE_CONF_JOB_EXECUTION = { "tenant_id": "tenant_1", "progress": "0.1", "return_code": "1", "job_id": "undefined", "input_id": "undefined", "output_id": "undefined", "cluster_id": None, "job_configs": { "conf2": "value_je", "conf3": "value_je" } } BINARY_DATA = b"vU}\x97\x1c\xdf\xa686\x08\xf2\tf\x0b\xb1}" SAMPLE_JOB_BINARY_INTERNAL = { "tenant_id": "test_tenant", "name": "job_test", "data": BINARY_DATA, "is_public": False, "is_protected": False } SAMPLE_JOB_BINARY = { "tenant_id": "test_tenant", "name": "job_binary_test", "description": "test_dec", "url": "internal-db://test_binary", "is_public": False, "is_protected": False } SAMPLE_JOB_BINARY_UPDATE = { "name": "updatedName", "url": "internal-db://updated-fake-url" } SAMPLE_JOB_BINARY_SWIFT = { "tenant_id": "test_tenant", "name": "job_binary_test_swift", "description": "the description", "url": "swift://test_swift_url", } SAMPLE_JOB_BINARY_SWIFT_UPDATE = { "name": "SwifterName", "url": "swift://updated-swift" } class DataSourceTest(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(DataSourceTest, self).__init__( checks=[ lambda: SAMPLE_DATA_SOURCE ], *args, **kwargs) def setUp(self): super(DataSourceTest, self).setUp() castellan.validate_config() def test_crud_operation_create_list_delete(self): ctx = context.ctx() self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) lst = self.api.data_source_get_all(ctx) self.assertEqual(1, len(lst)) ds_id = lst[0]['id'] self.api.data_source_destroy(ctx, ds_id) lst = self.api.data_source_get_all(ctx) self.assertEqual(0, len(lst)) def test_duplicate_data_source_create(self): ctx = context.ctx() self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) with testtools.ExpectedException(ex.DBDuplicateEntry): self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) def test_data_source_fields(self): ctx = context.ctx() ctx.tenant_id = SAMPLE_DATA_SOURCE['tenant_id'] ds_db_obj_id = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE)['id'] ds_db_obj = self.api.data_source_get(ctx, ds_db_obj_id) self.assertIsInstance(ds_db_obj, dict) for key, val in SAMPLE_DATA_SOURCE.items(): self.assertEqual(val, ds_db_obj.get(key), "Key not found %s" % key) def test_data_source_delete(self): ctx = context.ctx() db_obj_ds = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) _id = db_obj_ds['id'] self.api.data_source_destroy(ctx, _id) with testtools.ExpectedException(ex.NotFoundException): self.api.data_source_destroy(ctx, _id) def test_data_source_search(self): ctx = context.ctx() ctx.tenant_id = SAMPLE_DATA_SOURCE['tenant_id'] self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) lst = self.api.data_source_get_all(ctx) self.assertEqual(1, len(lst)) kwargs = {'name': SAMPLE_DATA_SOURCE['name'], 'tenant_id': SAMPLE_DATA_SOURCE['tenant_id']} lst = self.api.data_source_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Valid field but no matching value kwargs = {'name': SAMPLE_DATA_SOURCE['name']+"foo"} lst = self.api.data_source_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Valid field with substrings kwargs = {'name': 'ngt', 'tenant_id': SAMPLE_DATA_SOURCE['tenant_id']} lst = self.api.data_source_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Invalid field self.assertRaises(sa_exc.InvalidRequestError, self.api.data_source_get_all, ctx, **{'badfield': 'somevalue'}) @mock.patch('sahara.db.sqlalchemy.api.regex_filter') def test_data_source_search_regex(self, regex_filter): # do this so we can return the correct value def _regex_filter(query, cls, regex_cols, search_opts): return query, search_opts regex_filter.side_effect = _regex_filter ctx = context.ctx() self.api.data_source_get_all(ctx) self.assertEqual(0, regex_filter.call_count) self.api.data_source_get_all(ctx, regex_search=True, name="fox") self.assertEqual(1, regex_filter.call_count) args, kwargs = regex_filter.call_args self.assertIs(args[1], m.DataSource) self.assertEqual(args[2], ["name", "description", "url"]) self.assertEqual(args[3], {"name": "fox"}) def test_data_source_count_in(self): ctx = context.ctx() ctx.tenant_id = SAMPLE_DATA_SOURCE['tenant_id'] src = copy.copy(SAMPLE_DATA_SOURCE) self.api.data_source_create(ctx, src) cnt = self.api.data_source_count(ctx, name='ngt_test') self.assertEqual(1, cnt) cnt = self.api.data_source_count(ctx, name=('ngt_test', 'test2', 'test3')) self.assertEqual(1, cnt) cnt = self.api.data_source_count(ctx, name=('test1', 'test2', 'test3')) self.assertEqual(0, cnt) lst = self.api.data_source_get_all(ctx, name='ngt_test') myid = lst[0]['id'] cnt = self.api.data_source_count(ctx, name=('ngt_test', 'test2', 'test3'), id=myid) self.assertEqual(1, cnt) cnt = self.api.data_source_count(ctx, name=('ngt_test', 'test2', 'test3'), id=(myid, '2')) self.assertEqual(1, cnt) def test_data_source_count_like(self): ctx = context.ctx() ctx.tenant_id = SAMPLE_DATA_SOURCE['tenant_id'] src = copy.copy(SAMPLE_DATA_SOURCE) self.api.data_source_create(ctx, src) cnt = self.api.data_source_count(ctx, name='ngt_test') self.assertEqual(1, cnt) cnt = self.api.data_source_count(ctx, name='ngt%') self.assertEqual(1, cnt) cnt = self.api.data_source_count(ctx, name=('ngt_test',), url='localhost%') self.assertEqual(1, cnt) cnt = self.api.data_source_count(ctx, name=('ngt_test',), url='localhost') self.assertEqual(0, cnt) def test_data_source_update(self): ctx = context.ctx() orig = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) update_json = {"name": "updatedName", "url": "swift://updatedFakeUrl"} updated = self.api.data_source_update(ctx, orig["id"], update_json) self.assertEqual("updatedName", updated["name"]) self.assertEqual("swift://updatedFakeUrl", updated["url"]) def test_ds_update_delete_when_protected(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_DATA_SOURCE) sample['is_protected'] = True ds = self.api.data_source_create(ctx, sample) ds_id = ds["id"] with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.data_source_update(ctx, ds_id, {"name": "ds"}) except ex.UpdateFailedException as e: self.assert_protected_resource_exception(e) raise e with testtools.ExpectedException(ex.DeletionFailed): try: self.api.data_source_destroy(ctx, ds_id) except ex.DeletionFailed as e: self.assert_protected_resource_exception(e) raise e self.api.data_source_update(ctx, ds_id, {"name": "ds", "is_protected": False}) def test_public_ds_update_delete_from_another_tenant(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_DATA_SOURCE) sample['is_public'] = True ds = self.api.data_source_create(ctx, sample) ds_id = ds["id"] ctx.tenant_id = 'tenant_2' with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.data_source_update(ctx, ds_id, {"name": "ds"}) except ex.UpdateFailedException as e: self.assert_created_in_another_tenant_exception(e) raise e with testtools.ExpectedException(ex.DeletionFailed): try: self.api.data_source_destroy(ctx, ds_id) except ex.DeletionFailed as e: self.assert_created_in_another_tenant_exception(e) raise e class JobExecutionTest(test_base.ConductorManagerTestCase): def setUp(self): super(JobExecutionTest, self).setUp() castellan.validate_config() def test_crud_operation_create_list_delete_update(self): ctx = context.ctx() job = self.api.job_create(ctx, SAMPLE_JOB) ds_input = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT = copy.copy(SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT['name'] = 'output' ds_output = self.api.data_source_create(ctx, SAMPLE_DATA_OUTPUT) SAMPLE_JOB_EXECUTION['job_id'] = job['id'] SAMPLE_JOB_EXECUTION['input_id'] = ds_input['id'] SAMPLE_JOB_EXECUTION['output_id'] = ds_output['id'] self.api.job_execution_create(ctx, SAMPLE_JOB_EXECUTION) lst = self.api.job_execution_get_all(ctx) self.assertEqual(1, len(lst)) count = self.api.job_execution_count(ctx) self.assertEqual(1, count) job_ex_id = lst[0]['id'] self.assertIsNone(lst[0]['info']) new_info = {"status": edp.JOB_STATUS_PENDING} self.api.job_execution_update(ctx, job_ex_id, {'info': new_info}) updated_job = self.api.job_execution_get(ctx, job_ex_id) self.assertEqual(new_info, updated_job['info']) self.assertEqual(SAMPLE_JOB_EXECUTION['start_time'], updated_job['start_time']) self.api.job_execution_destroy(ctx, job_ex_id) with testtools.ExpectedException(ex.NotFoundException): self.api.job_execution_update(ctx, job_ex_id, {'info': new_info}) with testtools.ExpectedException(ex.NotFoundException): self.api.job_execution_destroy(ctx, job_ex_id) lst = self.api.job_execution_get_all(ctx) self.assertEqual(0, len(lst)) def test_crud_operation_on_configured_jobs(self): ctx = context.ctx() job = self.api.job_create(ctx, SAMPLE_JOB) ds_input = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT = copy.copy(SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT['name'] = 'output' ds_output = self.api.data_source_create(ctx, SAMPLE_DATA_OUTPUT) SAMPLE_CONF_JOB_EXECUTION['job_id'] = job['id'] SAMPLE_CONF_JOB_EXECUTION['input_id'] = ds_input['id'] SAMPLE_CONF_JOB_EXECUTION['output_id'] = ds_output['id'] self.api.job_execution_create(ctx, SAMPLE_CONF_JOB_EXECUTION) lst = self.api.job_execution_get_all(ctx) self.assertEqual(1, len(lst)) job_ex = lst[0] configs = { 'conf2': 'value_je', 'conf3': 'value_je' } self.assertEqual(configs, job_ex['job_configs']) def test_null_data_sources(self): ctx = context.ctx() job = self.api.job_create(ctx, SAMPLE_JOB) SAMPLE_CONF_JOB_EXECUTION['job_id'] = job['id'] SAMPLE_CONF_JOB_EXECUTION['input_id'] = None SAMPLE_CONF_JOB_EXECUTION['output_id'] = None id = self.api.job_execution_create(ctx, SAMPLE_CONF_JOB_EXECUTION)['id'] job_exec = self.api.job_execution_get(ctx, id) self.assertIsNone(job_exec['input_id']) self.assertIsNone(job_exec['output_id']) def test_deletion_constraints_on_data_and_jobs(self): ctx = context.ctx() job = self.api.job_create(ctx, SAMPLE_JOB) ds_input = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT = copy.copy(SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT['name'] = 'output' ds_output = self.api.data_source_create(ctx, SAMPLE_DATA_OUTPUT) SAMPLE_CONF_JOB_EXECUTION['job_id'] = job['id'] SAMPLE_CONF_JOB_EXECUTION['input_id'] = ds_input['id'] SAMPLE_CONF_JOB_EXECUTION['output_id'] = ds_output['id'] self.api.job_execution_create(ctx, SAMPLE_CONF_JOB_EXECUTION) with testtools.ExpectedException(ex.DeletionFailed): self.api.data_source_destroy(ctx, ds_input['id']) with testtools.ExpectedException(ex.DeletionFailed): self.api.data_source_destroy(ctx, ds_output['id']) with testtools.ExpectedException(ex.DeletionFailed): self.api.job_destroy(ctx, job['id']) def test_job_execution_search(self): ctx = context.ctx() jvals = copy.copy(SAMPLE_JOB) jvals["name"] = "frederica" job = self.api.job_create(ctx, jvals) ds_input = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT = copy.copy(SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT['name'] = 'output' ds_output = self.api.data_source_create(ctx, SAMPLE_DATA_OUTPUT) job_exec = copy.copy(SAMPLE_JOB_EXECUTION) job_exec['job_id'] = job['id'] job_exec['input_id'] = ds_input['id'] job_exec['output_id'] = ds_output['id'] ctx.tenant_id = job_exec['tenant_id'] self.api.job_execution_create(ctx, job_exec) lst = self.api.job_execution_get_all(ctx) self.assertEqual(1, len(lst)) kwargs = {'tenant_id': job_exec['tenant_id']} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Valid field but no matching value kwargs = {'job_id': job_exec['job_id']+"foo"} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Valid field with substrings kwargs = {'job.name': "red"} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Invalid field self.assertRaises(sa_exc.InvalidRequestError, self.api.job_execution_get_all, ctx, **{'badfield': 'somevalue'}) @mock.patch('sahara.db.sqlalchemy.api.regex_filter') def test_job_execution_search_regex(self, regex_filter): # do this so we can return the correct value def _regex_filter(query, cls, regex_cols, search_opts): return query, search_opts regex_filter.side_effect = _regex_filter ctx = context.ctx() self.api.job_execution_get_all(ctx) self.assertEqual(0, regex_filter.call_count) self.api.job_execution_get_all(ctx, regex_search=True, **{"job.name": "fox", "cluster.name": "jack", "id": "124"}) self.assertEqual(3, regex_filter.call_count) # First call, after externals were removed args, kwargs = regex_filter.call_args_list[0] self.assertIs(args[1], m.JobExecution) self.assertEqual(args[2], ["job.name", "cluster.name"]) self.assertEqual(args[3], {"id": "124"}) # Second call, looking for cluster.name args, kwargs = regex_filter.call_args_list[1] self.assertIs(args[1], m.Cluster) self.assertEqual(args[2], ["name"]) self.assertEqual(args[3], {"name": "jack"}) # Third call, looking for job.name args, kwargs = regex_filter.call_args_list[2] self.assertIs(args[1], m.Job) self.assertEqual(args[2], ["name"]) self.assertEqual(args[3], {"name": "fox"}) def test_job_execution_advanced_search(self): ctx = context.ctx() job = self.api.job_create(ctx, SAMPLE_JOB) ds_input = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT = copy.copy(SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT['name'] = 'output' ds_output = self.api.data_source_create(ctx, SAMPLE_DATA_OUTPUT) # Create a cluster cl1 = self.api.cluster_create(ctx, test_clusters.SAMPLE_CLUSTER) # Create a second cluster with a different name cl2_vals = copy.copy(test_clusters.SAMPLE_CLUSTER) cl2_vals['name'] = 'test_cluster2' cl2 = self.api.cluster_create(ctx, cl2_vals) my_sample_job_exec = copy.copy(SAMPLE_JOB_EXECUTION) my_sample_job_exec['job_id'] = job['id'] my_sample_job_exec['input_id'] = ds_input['id'] my_sample_job_exec['output_id'] = ds_output['id'] my_sample_job_exec['cluster_id'] = cl1['id'] # Run job on cluster 1 self.api.job_execution_create(ctx, my_sample_job_exec) # Run the same job on cluster 2 and set status my_sample_job_exec['cluster_id'] = cl2['id'] my_sample_job_exec['info'] = {'status': 'KiLLeD'} self.api.job_execution_create(ctx, my_sample_job_exec) # Search only with job execution fields (finds both) lst = self.api.job_execution_get_all(ctx, **{'return_code': 1}) self.assertEqual(2, len(lst)) # Search on cluster name kwargs = {'cluster.name': cl1['name'], 'return_code': 1} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Search on cluster name and job name kwargs = {'cluster.name': cl1['name'], 'job.name': SAMPLE_JOB['name'], 'return_code': 1} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Search on cluster name, job name, and status kwargs = {'cluster.name': cl2['name'], 'job.name': SAMPLE_JOB['name'], 'status': 'killed', 'return_code': 1} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Search on job name (finds both) kwargs = {'job.name': SAMPLE_JOB['name'], 'return_code': 1} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(2, len(lst)) # invalid cluster name value kwargs = {'cluster.name': cl1['name']+'foo', 'job.name': SAMPLE_JOB['name']} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # invalid job name value kwargs = {'cluster.name': cl1['name'], 'job.name': SAMPLE_JOB['name']+'foo'} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # invalid status value kwargs = {'cluster.name': cl1['name'], 'status': 'PENDING'} lst = self.api.job_execution_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) class JobTest(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(JobTest, self).__init__( checks=[ lambda: SAMPLE_JOB ], *args, **kwargs) def test_crud_operation_create_list_delete_update(self): ctx = context.ctx() self.api.job_create(ctx, SAMPLE_JOB) lst = self.api.job_get_all(ctx) self.assertEqual(1, len(lst)) jo_id = lst[0]['id'] update_jo = self.api.job_update(ctx, jo_id, {'description': 'update'}) self.assertEqual('update', update_jo['description']) self.api.job_destroy(ctx, jo_id) lst = self.api.job_get_all(ctx) self.assertEqual(0, len(lst)) with testtools.ExpectedException(ex.NotFoundException): self.api.job_destroy(ctx, jo_id) def test_job_fields(self): ctx = context.ctx() ctx.tenant_id = SAMPLE_JOB['tenant_id'] job_id = self.api.job_create(ctx, SAMPLE_JOB)['id'] job = self.api.job_get(ctx, job_id) self.assertIsInstance(job, dict) for key, val in SAMPLE_JOB.items(): self.assertEqual(val, job.get(key), "Key not found %s" % key) def test_job_search(self): ctx = context.ctx() job = copy.copy(SAMPLE_JOB) job["name"] = "frederica" job["description"] = "thebestjob" ctx.tenant_id = job['tenant_id'] self.api.job_create(ctx, job) lst = self.api.job_get_all(ctx) self.assertEqual(1, len(lst)) kwargs = {'name': job['name'], 'tenant_id': job['tenant_id']} lst = self.api.job_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Valid field but no matching value lst = self.api.job_get_all(ctx, **{'name': job['name']+"foo"}) self.assertEqual(0, len(lst)) # Valid field with substrings kwargs = {'name': "red", 'description': "best"} lst = self.api.job_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Invalid field self.assertRaises(sa_exc.InvalidRequestError, self.api.job_get_all, ctx, **{'badfield': 'somevalue'}) @mock.patch('sahara.db.sqlalchemy.api.regex_filter') def test_job_search_regex(self, regex_filter): # do this so we can return the correct value def _regex_filter(query, cls, regex_cols, search_opts): return query, search_opts regex_filter.side_effect = _regex_filter ctx = context.ctx() self.api.job_get_all(ctx) self.assertEqual(0, regex_filter.call_count) self.api.job_get_all(ctx, regex_search=True, name="fox") self.assertEqual(1, regex_filter.call_count) args, kwargs = regex_filter.call_args self.assertIs(args[1], m.Job) self.assertEqual(args[2], ["name", "description"]) self.assertEqual(args[3], {"name": "fox"}) def test_job_update_delete_when_protected(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_JOB) sample['is_protected'] = True job = self.api.job_create(ctx, sample) job_id = job["id"] with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.job_update(ctx, job_id, {"name": "job"}) except ex.UpdateFailedException as e: self.assert_protected_resource_exception(e) raise e with testtools.ExpectedException(ex.DeletionFailed): try: self.api.job_destroy(ctx, job_id) except ex.DeletionFailed as e: self.assert_protected_resource_exception(e) raise e self.api.job_update(ctx, job_id, {"name": "job", "is_protected": False}) def test_public_job_update_delete_from_another_tenant(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_JOB) sample['is_public'] = True job = self.api.job_create(ctx, sample) job_id = job["id"] ctx.tenant_id = 'tenant_2' with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.job_update(ctx, job_id, {"name": "job"}) except ex.UpdateFailedException as e: self.assert_created_in_another_tenant_exception(e) raise e with testtools.ExpectedException(ex.DeletionFailed): try: self.api.job_destroy(ctx, job_id) except ex.DeletionFailed as e: self.assert_created_in_another_tenant_exception(e) raise e class JobBinaryInternalTest(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(JobBinaryInternalTest, self).__init__( checks=[ lambda: SAMPLE_JOB_BINARY_INTERNAL ], *args, **kwargs) def test_crud_operation_create_list_delete_update(self): ctx = context.ctx() self.api.job_binary_internal_create(ctx, SAMPLE_JOB_BINARY_INTERNAL) lst = self.api.job_binary_internal_get_all(ctx) self.assertEqual(1, len(lst)) job_bin_int_id = lst[0]['id'] update_jbi = self.api.job_binary_internal_update( ctx, job_bin_int_id, {'name': 'newname'}) self.assertEqual('newname', update_jbi['name']) self.api.job_binary_internal_destroy(ctx, job_bin_int_id) lst = self.api.job_binary_internal_get_all(ctx) self.assertEqual(0, len(lst)) with testtools.ExpectedException(ex.NotFoundException): self.api.job_binary_internal_destroy(ctx, job_bin_int_id) def test_duplicate_job_binary_internal_create(self): ctx = context.ctx() self.api.job_binary_internal_create(ctx, SAMPLE_JOB_BINARY_INTERNAL) with testtools.ExpectedException(ex.DBDuplicateEntry): self.api.job_binary_internal_create(ctx, SAMPLE_JOB_BINARY_INTERNAL) def test_job_binary_internal_get_raw(self): ctx = context.ctx() id = self.api.job_binary_internal_create(ctx, SAMPLE_JOB_BINARY_INTERNAL )['id'] data = self.api.job_binary_internal_get_raw_data(ctx, id) self.assertEqual(SAMPLE_JOB_BINARY_INTERNAL["data"], data) self.api.job_binary_internal_destroy(ctx, id) data = self.api.job_binary_internal_get_raw_data(ctx, id) self.assertIsNone(data) def test_job_binary_internal_fields(self): ctx = context.ctx() ctx.tenant_id = SAMPLE_JOB_BINARY_INTERNAL['tenant_id'] id = self.api.job_binary_internal_create( ctx, SAMPLE_JOB_BINARY_INTERNAL)['id'] internal = self.api.job_binary_internal_get(ctx, id) self.assertIsInstance(internal, dict) with testtools.ExpectedException(KeyError): internal["data"] internal["data"] = self.api.job_binary_internal_get_raw_data(ctx, id) for key, val in SAMPLE_JOB_BINARY_INTERNAL.items(): if key == "datasize": self.assertEqual(len(BINARY_DATA), internal["datasize"]) else: self.assertEqual(val, internal.get(key), "Key not found %s" % key) def test_job_binary_internal_search(self): ctx = context.ctx() jbi = copy.copy(SAMPLE_JOB_BINARY_INTERNAL) jbi["name"] = "frederica" ctx.tenant_id = jbi['tenant_id'] self.api.job_binary_internal_create(ctx, jbi) lst = self.api.job_binary_internal_get_all(ctx) self.assertEqual(1, len(lst)) kwargs = {'name': jbi['name'], 'tenant_id': jbi['tenant_id']} lst = self.api.job_binary_internal_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Valid field but no matching value kwargs = {'name': jbi['name']+"foo"} lst = self.api.job_binary_internal_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Valid field with substrings kwargs = {'name': "red"} lst = self.api.job_binary_internal_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Invalid field self.assertRaises(sa_exc.InvalidRequestError, self.api.job_binary_internal_get_all, ctx, **{'badfield': 'junk'}) @mock.patch('sahara.db.sqlalchemy.api.regex_filter') def test_job_binary_internal_search_regex(self, regex_filter): # do this so we can return the correct value def _regex_filter(query, cls, regex_cols, search_opts): return query, search_opts regex_filter.side_effect = _regex_filter ctx = context.ctx() self.api.job_binary_internal_get_all(ctx) self.assertEqual(0, regex_filter.call_count) self.api.job_binary_internal_get_all(ctx, regex_search=True, name="fox") self.assertEqual(1, regex_filter.call_count) args, kwargs = regex_filter.call_args self.assertIs(args[1], m.JobBinaryInternal) self.assertEqual(args[2], ["name"]) self.assertEqual(args[3], {"name": "fox"}) def test_jbi_update_delete_when_protected(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_JOB_BINARY_INTERNAL) sample['is_protected'] = True jbi = self.api.job_binary_internal_create(ctx, sample) jbi_id = jbi["id"] with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.job_binary_internal_update(ctx, jbi_id, {"name": "jbi"}) except ex.UpdateFailedException as e: self.assert_protected_resource_exception(e) raise e with testtools.ExpectedException(ex.DeletionFailed): try: self.api.job_binary_internal_destroy(ctx, jbi_id) except ex.DeletionFailed as e: self.assert_protected_resource_exception(e) raise e self.api.job_binary_internal_update(ctx, jbi_id, {"name": "jbi", "is_protected": False}) def test_public_jbi_update_delete_from_another_tenant(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_JOB_BINARY_INTERNAL) sample['is_public'] = True jbi = self.api.job_binary_internal_create(ctx, sample) jbi_id = jbi["id"] ctx.tenant_id = 'tenant_2' with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.job_binary_internal_update(ctx, jbi_id, {"name": "jbi"}) except ex.UpdateFailedException as e: self.assert_created_in_another_tenant_exception(e) raise e with testtools.ExpectedException(ex.DeletionFailed): try: self.api.job_binary_internal_destroy(ctx, jbi_id) except ex.DeletionFailed as e: self.assert_created_in_another_tenant_exception(e) raise e class JobBinaryTest(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(JobBinaryTest, self).__init__( checks=[ lambda: SAMPLE_JOB_BINARY ], *args, **kwargs) def setUp(self): super(JobBinaryTest, self).setUp() castellan.validate_config() def test_crud_operation_create_list_delete(self): ctx = context.ctx() self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY) lst = self.api.job_binary_get_all(ctx) self.assertEqual(1, len(lst)) job_binary_id = lst[0]['id'] self.api.job_binary_destroy(ctx, job_binary_id) lst = self.api.job_binary_get_all(ctx) self.assertEqual(0, len(lst)) with testtools.ExpectedException(ex.NotFoundException): self.api.job_binary_destroy(ctx, job_binary_id) def test_job_binary_fields(self): ctx = context.ctx() ctx.tenant_id = SAMPLE_JOB_BINARY['tenant_id'] job_binary_id = self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY)['id'] job_binary = self.api.job_binary_get(ctx, job_binary_id) self.assertIsInstance(job_binary, dict) for key, val in SAMPLE_JOB_BINARY.items(): self.assertEqual(val, job_binary.get(key), "Key not found %s" % key) def _test_job_binary_referenced(self, reference): ctx = context.ctx() job_binary_id = self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY)['id'] job_values = copy.copy(SAMPLE_JOB) job_values[reference] = [job_binary_id] job_id = self.api.job_create(ctx, job_values)['id'] # Delete while referenced, fails with testtools.ExpectedException(ex.DeletionFailed): self.api.job_binary_destroy(ctx, job_binary_id) # Delete while not referenced self.api.job_destroy(ctx, job_id) self.api.job_binary_destroy(ctx, job_binary_id) lst = self.api.job_binary_get_all(ctx) self.assertEqual(0, len(lst)) def test_job_binary_referenced_mains(self): self._test_job_binary_referenced("mains") def test_job_binary_referenced_libs(self): self._test_job_binary_referenced("libs") def test_duplicate_job_binary_create(self): ctx = context.ctx() self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY) with testtools.ExpectedException(ex.DBDuplicateEntry): self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY) def test_job_binary_search(self): ctx = context.ctx() jb = copy.copy(SAMPLE_JOB_BINARY) jb["name"] = "frederica" jb["url"] = "http://thebestbinary" ctx.tenant_id = jb['tenant_id'] self.api.job_binary_create(ctx, jb) lst = self.api.job_binary_get_all(ctx) self.assertEqual(1, len(lst)) kwargs = {'name': jb['name'], 'tenant_id': jb['tenant_id']} lst = self.api.job_binary_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Valid field but no matching value kwargs = {'name': jb['name']+"foo"} lst = self.api.job_binary_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Valid field with substrings kwargs = {'name': "red", 'url': "best"} lst = self.api.job_binary_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Invalid field self.assertRaises(sa_exc.InvalidRequestError, self.api.job_binary_get_all, ctx, **{'badfield': 'somevalue'}) @mock.patch('sahara.db.sqlalchemy.api.regex_filter') def test_job_binary_search_regex(self, regex_filter): # do this so we can return the correct value def _regex_filter(query, cls, regex_cols, search_opts): return query, search_opts regex_filter.side_effect = _regex_filter ctx = context.ctx() self.api.job_binary_get_all(ctx) self.assertEqual(0, regex_filter.call_count) self.api.job_binary_get_all(ctx, regex_search=True, name="fox") self.assertEqual(1, regex_filter.call_count) args, kwargs = regex_filter.call_args self.assertIs(args[1], m.JobBinary) self.assertEqual(args[2], ["name", "description", "url"]) self.assertEqual(args[3], {"name": "fox"}) def test_job_binary_update(self): ctx = context.ctx() original = self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY_SWIFT) updated = self.api.job_binary_update( ctx, original["id"], SAMPLE_JOB_BINARY_SWIFT_UPDATE) # Make sure that the update did indeed succeed self.assertEqual( SAMPLE_JOB_BINARY_SWIFT_UPDATE["name"], updated["name"]) self.assertEqual(SAMPLE_JOB_BINARY_SWIFT_UPDATE["url"], updated["url"]) # Make sure we do NOT update a binary in use by a PENDING job self._create_job_execution_ref_job_binary(ctx, original["id"]) with testtools.ExpectedException(ex.UpdateFailedException): self.api.job_binary_update( ctx, original["id"], SAMPLE_JOB_BINARY_SWIFT_UPDATE) original = self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY) # Make sure that internal URL update fails with testtools.ExpectedException(ex.UpdateFailedException): self.api.job_binary_update( ctx, original["id"], SAMPLE_JOB_BINARY_UPDATE) def _create_job_execution_ref_job_binary(self, ctx, jb_id): JOB_REF_BINARY = copy.copy(SAMPLE_JOB) JOB_REF_BINARY["mains"] = [jb_id] job = self.api.job_create(ctx, JOB_REF_BINARY) ds_input = self.api.data_source_create(ctx, SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT = copy.copy(SAMPLE_DATA_SOURCE) SAMPLE_DATA_OUTPUT['name'] = 'output' ds_output = self.api.data_source_create(ctx, SAMPLE_DATA_OUTPUT) SAMPLE_JOB_EXECUTION['job_id'] = job['id'] SAMPLE_JOB_EXECUTION['input_id'] = ds_input['id'] SAMPLE_JOB_EXECUTION['output_id'] = ds_output['id'] self.api.job_execution_create(ctx, SAMPLE_JOB_EXECUTION) lst = self.api.job_execution_get_all(ctx) job_ex_id = lst[0]["id"] new_info = {"status": edp.JOB_STATUS_PENDING} self.api.job_execution_update(ctx, job_ex_id, {"info": new_info}) def test_jb_update_delete_when_protected(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_JOB_BINARY) sample['is_protected'] = True jb = self.api.job_binary_create(ctx, sample) jb_id = jb["id"] with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.job_binary_update(ctx, jb_id, {"name": "jb"}) except ex.UpdateFailedException as e: self.assert_protected_resource_exception(e) raise e with testtools.ExpectedException(ex.DeletionFailed): try: self.api.job_binary_destroy(ctx, jb_id) except ex.DeletionFailed as e: self.assert_protected_resource_exception(e) raise e self.api.job_binary_update(ctx, jb_id, {"name": "jb", "is_protected": False}) def test_public_jb_update_delete_from_another_tenant(self): ctx = context.ctx() sample = copy.deepcopy(SAMPLE_JOB_BINARY) sample['is_public'] = True jb = self.api.job_binary_create(ctx, sample) jb_id = jb["id"] ctx.tenant_id = 'tenant_2' with testtools.ExpectedException(ex.UpdateFailedException): try: self.api.job_binary_update(ctx, jb_id, {"name": "jb"}) except ex.UpdateFailedException as e: self.assert_created_in_another_tenant_exception(e) raise e with testtools.ExpectedException(ex.DeletionFailed): try: self.api.job_binary_destroy(ctx, jb_id) except ex.DeletionFailed as e: self.assert_created_in_another_tenant_exception(e) raise e sahara-8.0.0/sahara/tests/unit/conductor/manager/__init__.py0000666000175100017510000000000013245514472024104 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/conductor/manager/test_clusters.py0000666000175100017510000003452013245514472025266 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import mock from sqlalchemy import exc as sa_exc import testtools from sahara.conductor import manager from sahara import context from sahara.db.sqlalchemy import models as m from sahara import exceptions as ex import sahara.tests.unit.conductor.base as test_base from sahara.utils import cluster as c_u SAMPLE_CLUSTER = { "plugin_name": "test_plugin", "hadoop_version": "test_version", "tenant_id": "tenant_1", "name": "test_cluster", "user_keypair_id": "my_keypair", "node_groups": [ { "name": "ng_1", "flavor_id": "42", "node_processes": ["p1", "p2"], "count": 1, "security_groups": None, 'use_autoconfig': True, "shares": None }, { "name": "ng_2", "flavor_id": "42", "node_processes": ["p3", "p4"], "count": 3, "security_groups": ["group1", "group2"], 'use_autoconfig': True, "shares": None } ], "cluster_configs": { "service_1": { "config_2": "value_2" }, "service_2": { "config_1": "value_1" } }, "shares": [], "is_public": False, "is_protected": False } class ClusterTest(test_base.ConductorManagerTestCase): def __init__(self, *args, **kwargs): super(ClusterTest, self).__init__( checks=[ lambda: SAMPLE_CLUSTER, lambda: manager.CLUSTER_DEFAULTS, lambda: manager.NODE_GROUP_DEFAULTS, lambda: manager.INSTANCE_DEFAULTS, ], *args, **kwargs) def test_cluster_create_list_update_delete(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) self.assertIsInstance(cluster_db_obj, dict) lst = self.api.cluster_get_all(ctx) self.assertEqual(1, len(lst)) cl_id = lst[0]["id"] updated_cl = self.api.cluster_update( ctx, cl_id, {"is_public": True}) self.assertIsInstance(updated_cl, dict) self.assertEqual(True, updated_cl["is_public"]) self.api.cluster_destroy(ctx, cl_id) lst = self.api.cluster_get_all(ctx) self.assertEqual(0, len(lst)) with testtools.ExpectedException(ex.NotFoundException): self.api.cluster_destroy(ctx, cl_id) def test_duplicate_cluster_create(self): ctx = context.ctx() self.api.cluster_create(ctx, SAMPLE_CLUSTER) with testtools.ExpectedException(ex.DBDuplicateEntry): self.api.cluster_create(ctx, SAMPLE_CLUSTER) def test_cluster_fields(self): ctx = context.ctx() cl_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) self.assertIsInstance(cl_db_obj, dict) for key, val in SAMPLE_CLUSTER.items(): if key == 'node_groups': # this will be checked separately continue self.assertEqual(val, cl_db_obj.get(key), "Key not found %s" % key) for ng in cl_db_obj["node_groups"]: ng.pop("created_at") ng.pop("updated_at") ng.pop("id") self.assertEqual(cl_db_obj["id"], ng.pop("cluster_id")) ng.pop("image_id") self.assertEqual([], ng.pop("instances")) ng.pop("node_configs") ng.pop("node_group_template_id") ng.pop("volume_mount_prefix") ng.pop("volumes_size") ng.pop("volumes_per_node") ng.pop("volumes_availability_zone") ng.pop("volume_type") ng.pop("floating_ip_pool") ng.pop("image_username") ng.pop("open_ports") ng.pop("auto_security_group") ng.pop("is_proxy_gateway") ng.pop("tenant_id") ng.pop("availability_zone") ng.pop('volume_local_to_instance') self.assertEqual(SAMPLE_CLUSTER["node_groups"], cl_db_obj["node_groups"]) def test_cluster_no_ng(self): ctx = context.ctx() cluster_schema = copy.deepcopy(SAMPLE_CLUSTER) cluster_schema.pop('node_groups') cl_db_obj = self.api.cluster_create(ctx, cluster_schema) self.assertIsInstance(cl_db_obj, dict) for key, val in cluster_schema.items(): self.assertEqual(val, cl_db_obj.get(key), "Key not found %s" % key) self.assertEqual([], cl_db_obj["node_groups"]) def test_cluster_update_status(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] updated_cl = self.api.cluster_update( ctx, _id, {"status": c_u.CLUSTER_STATUS_ACTIVE}) self.assertIsInstance(updated_cl, dict) self.assertEqual(c_u.CLUSTER_STATUS_ACTIVE, updated_cl["status"]) get_cl_obj = self.api.cluster_get(ctx, _id) self.assertEqual(updated_cl, get_cl_obj) with testtools.ExpectedException(ex.NotFoundException): self.api.cluster_update( ctx, "bad_id", {"status": c_u.CLUSTER_STATUS_ACTIVE}) def _ng_in_cluster(self, cluster_db_obj, ng_id): for ng in cluster_db_obj["node_groups"]: if ng["id"] == ng_id: return ng return None def test_add_node_group(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] node_group = { "name": "ng_3", "flavor_id": "42", "node_processes": ["p3", "p4"], "count": 5 } ng_id = self.api.node_group_add(ctx, _id, node_group) cluster_db_obj = self.api.cluster_get(ctx, _id) found_ng = self._ng_in_cluster(cluster_db_obj, ng_id) self.assertTrue(found_ng, "New Node Group not found") def test_update_node_group(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] self.assertEqual(2, len(cluster_db_obj["node_groups"])) ng_id = cluster_db_obj["node_groups"][-1]["id"] self.api.node_group_update(ctx, ng_id, {"image_id": "test_image"}) cluster_db_obj = self.api.cluster_get(ctx, _id) found_ng = self._ng_in_cluster(cluster_db_obj, ng_id) self.assertTrue(found_ng, "Updated Node Group not found") for ng in cluster_db_obj["node_groups"]: if ng["id"] != ng_id: continue self.assertEqual("test_image", ng["image_id"]) def test_delete_node_group(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] ng_id = cluster_db_obj["node_groups"][-1]["id"] self.api.node_group_remove(ctx, ng_id) cluster_db_obj = self.api.cluster_get(ctx, _id) found_ng = self._ng_in_cluster(cluster_db_obj, ng_id) self.assertFalse(found_ng, "Node Group is still in a CLuster") with testtools.ExpectedException(ex.NotFoundException): self.api.node_group_remove(ctx, ng_id) def _add_instance(self, ctx, ng_id): instance = { "instance_name": "additional_vm" } return self.api.instance_add(ctx, ng_id, instance) def _add_instance_ipv6(self, ctx, ng_id, instance_name): instance = { "instance_name": instance_name, "internal_ip": "FE80:0000:0000:0000:0202:B3FF:FE1E:8329", "management_ip": "FE80:0000:0000:0000:0202:B3FF:FE1E:8329" } return self.api.instance_add(ctx, ng_id, instance) def test_add_instance(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] ng_id = cluster_db_obj["node_groups"][-1]["id"] count = cluster_db_obj["node_groups"][-1]["count"] self._add_instance(ctx, ng_id) cluster_db_obj = self.api.cluster_get(ctx, _id) for ng in cluster_db_obj["node_groups"]: if ng["id"] != ng_id: continue ng.pop('tenant_id') self.assertEqual(count + 1, ng["count"]) self.assertEqual("additional_vm", ng["instances"][0]["instance_name"]) def test_add_instance_ipv6(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] ng_id = cluster_db_obj["node_groups"][-1]["id"] count = cluster_db_obj["node_groups"][-1]["count"] instance_name = "additional_vm_ipv6" self._add_instance_ipv6(ctx, ng_id, instance_name) cluster_db_obj = self.api.cluster_get(ctx, _id) for ng in cluster_db_obj["node_groups"]: if ng["id"] != ng_id: continue ng.pop('tenant_id') self.assertEqual(count + 1, ng["count"]) self.assertEqual(instance_name, ng["instances"][0]["instance_name"]) def test_update_instance(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] ng_id = cluster_db_obj["node_groups"][-1]["id"] instance_id = self._add_instance(ctx, ng_id) self.api.instance_update(ctx, instance_id, {"management_ip": "1.1.1.1"}) cluster_db_obj = self.api.cluster_get(ctx, _id) for ng in cluster_db_obj["node_groups"]: if ng["id"] != ng_id: continue self.assertEqual("1.1.1.1", ng["instances"][0]["management_ip"]) def test_update_instance_ipv6(self): ctx = context.ctx() ip = "FE80:0000:0000:0000:0202:B3FF:FE1E:8329" cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] ng_id = cluster_db_obj["node_groups"][-1]["id"] instance_id = self._add_instance(ctx, ng_id) self.api.instance_update(ctx, instance_id, {"management_ip": ip}) cluster_db_obj = self.api.cluster_get(ctx, _id) for ng in cluster_db_obj["node_groups"]: if ng["id"] != ng_id: continue self.assertEqual(ip, ng["instances"][0]["management_ip"]) def test_remove_instance(self): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] ng_id = cluster_db_obj["node_groups"][-1]["id"] count = cluster_db_obj["node_groups"][-1]["count"] instance_id = self._add_instance(ctx, ng_id) cluster_db_obj = self.api.cluster_get(ctx, _id) for ng in cluster_db_obj["node_groups"]: if ng["id"] != ng_id: continue self.assertEqual(count + 1, ng["count"]) self.api.instance_remove(ctx, instance_id) cluster_db_obj = self.api.cluster_get(ctx, _id) for ng in cluster_db_obj["node_groups"]: if ng["id"] != ng_id: continue self.assertEqual(count, ng["count"]) with testtools.ExpectedException(ex.NotFoundException): self.api.instance_remove(ctx, instance_id) def test_cluster_search(self): ctx = context.ctx() vals = copy.deepcopy(SAMPLE_CLUSTER) vals['name'] = "test_name" self.api.cluster_create(ctx, vals) lst = self.api.cluster_get_all(ctx) self.assertEqual(1, len(lst)) kwargs = {'name': vals['name'], 'plugin_name': vals['plugin_name']} lst = self.api.cluster_get_all(ctx, **kwargs) self.assertEqual(1, len(lst)) # Valid field but no matching value kwargs = {'name': vals['name']+'foo'} lst = self.api.cluster_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Valid field with substrings kwargs = {'name': 'test'} lst = self.api.cluster_get_all(ctx, **kwargs) self.assertEqual(0, len(lst)) # Invalid field self.assertRaises(sa_exc.InvalidRequestError, self.api.cluster_get_all, ctx, **{'badfield': 'somevalue'}) @mock.patch('sahara.db.sqlalchemy.api.regex_filter') def test_cluster_search_regex(self, regex_filter): # do this so we can return the correct value def _regex_filter(query, cls, regex_cols, search_opts): return query, search_opts regex_filter.side_effect = _regex_filter ctx = context.ctx() self.api.cluster_get_all(ctx) self.assertEqual(0, regex_filter.call_count) self.api.cluster_get_all(ctx, regex_search=True, name="fox") self.assertEqual(1, regex_filter.call_count) args, kwargs = regex_filter.call_args self.assertIs(args[1], m.Cluster) self.assertEqual(args[2], ["name", "description", "plugin_name", "tenant_id"]) self.assertEqual(args[3], {"name": "fox"}) @mock.patch("sahara.service.edp.utils.shares.mount_shares") def test_cluster_update_shares(self, mount_shares): ctx = context.ctx() cluster_db_obj = self.api.cluster_create(ctx, SAMPLE_CLUSTER) _id = cluster_db_obj["id"] test_shares = [ { "id": "bd71d2d5-60a0-4ed9-a3d2-ad312c368880", "path": "/mnt/manila", "access_level": "rw" } ] updated_cl = self.api.cluster_update(ctx, _id, {"shares": test_shares}) self.assertIsInstance(updated_cl, dict) self.assertEqual(test_shares, updated_cl["shares"]) get_cl_obj = self.api.cluster_get(ctx, _id) self.assertEqual(updated_cl, get_cl_obj) sahara-8.0.0/sahara/tests/unit/conductor/manager/test_edp_interface.py0000666000175100017510000000703513245514472026213 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara import context import sahara.tests.unit.conductor.base as test_base from sahara.tests.unit.conductor.manager import test_edp def _merge_dict(original, update): new = copy.deepcopy(original) new.update(update) return new SAMPLE_JOB = _merge_dict(test_edp.SAMPLE_JOB, { "interface": [ { "name": "Reducer Count", "mapping_type": "configs", "location": "mapred.reduce.tasks", "value_type": "number", "required": True, "default": "1" }, { "name": "Input Path", "mapping_type": "params", "location": "INPUT", "value_type": "data_source", "required": False, "default": "hdfs://path" }, { "name": "Positional Argument 2", "mapping_type": "args", "location": "1", "value_type": "string", "required": False, "default": "default" }, { "name": "Positional Argument 1", "mapping_type": "args", "location": "0", "value_type": "string", "required": False, "default": "arg_1" }, ] }) SAMPLE_JOB_EXECUTION = _merge_dict(test_edp.SAMPLE_JOB, { "interface": { "Reducer Count": "2", "Positional Argument 2": "arg_2" }, "job_configs": {"args": ["arg_3"], "configs": {"mapred.map.tasks": "3"}} }) class JobExecutionTest(test_base.ConductorManagerTestCase): def test_interface_flows(self): ctx = context.ctx() job = self.api.job_create(ctx, SAMPLE_JOB) arg_names = [arg['name'] for arg in job['interface']] self.assertEqual(arg_names, ["Reducer Count", "Input Path", "Positional Argument 2", "Positional Argument 1"]) job_ex_input = copy.deepcopy(SAMPLE_JOB_EXECUTION) job_ex_input['job_id'] = job['id'] self.api.job_execution_create(ctx, job_ex_input) lst = self.api.job_execution_get_all(ctx) self.assertEqual(1, len(lst)) job_ex_result = lst[0] configs = { 'configs': {'mapred.reduce.tasks': '2', 'mapred.map.tasks': '3'}, 'args': ['arg_1', 'arg_2', 'arg_3'], 'params': {'INPUT': 'hdfs://path'} } self.assertEqual(configs, job_ex_result['job_configs']) self.api.job_execution_destroy(ctx, job_ex_result['id']) del job_ex_input['job_configs'] self.api.job_execution_create(ctx, job_ex_input) lst = self.api.job_execution_get_all(ctx) self.assertEqual(1, len(lst)) job_ex_result = lst[0] configs = { 'configs': {'mapred.reduce.tasks': '2'}, 'args': ['arg_1', 'arg_2'], 'params': {'INPUT': 'hdfs://path'} } self.assertEqual(configs, job_ex_result['job_configs']) sahara-8.0.0/sahara/tests/unit/conductor/test_api.py0000666000175100017510000001775313245514472022572 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import testtools from sahara import conductor from sahara import context from sahara import exceptions from sahara.tests.unit import base from sahara.utils import general as gu SAMPLE_CLUSTER = { 'plugin_name': 'test_plugin', 'hadoop_version': 'test_version', 'tenant_id': 'tenant_1', 'name': 'test_cluster', 'user_keypair_id': 'my_keypair', 'node_groups': [ { 'name': 'ng_1', 'flavor_id': '42', 'node_processes': ['p1', 'p2'], 'count': 1 }, { 'name': 'ng_2', 'flavor_id': '42', 'node_processes': ['p3', 'p4'], 'count': 3 } ], 'cluster_configs': { 'service_1': { 'config_2': 'value_2' }, 'service_2': { 'config_1': 'value_1' } }, } SAMPLE_NODE_GROUP = { 'name': 'ng_3', 'flavor_id': '42', 'node_processes': ['p5', 'p6'], 'count': 5 } SAMPLE_INSTANCE = { 'instance_name': 'test-name', 'instance_id': '123456', 'management_ip': '1.2.3.1' } SAMPLE_JOB = { "tenant_id": "test_tenant", "name": "job_test", "description": "test_desc", "type": "pig" } SAMPLE_JOB_BINARY = { "tenant_id": "test_tenant", "name": "job_binary_test", "description": "test_dec", "url": "internal-db://test_binary", } class TestConductorApi(base.SaharaWithDbTestCase): def setUp(self): super(TestConductorApi, self).setUp() self.api = conductor.API def _make_sample(self): ctx = context.ctx() cluster = self.api.cluster_create(ctx, SAMPLE_CLUSTER) return ctx, cluster def test_update_by_id(self): ctx, cluster = self._make_sample() self.api.cluster_update(ctx, cluster.id, {'name': 'changed'}) updated_cluster = self.api.cluster_get(ctx, cluster.id) self.assertEqual('changed', updated_cluster['name']) self.api.cluster_destroy(ctx, updated_cluster.id) cluster_list = self.api.cluster_get_all(ctx) self.assertEqual(0, len(cluster_list)) def test_add_node_group_to_cluster_id(self): ctx, cluster = self._make_sample() ng_id = self.api.node_group_add(ctx, cluster.id, SAMPLE_NODE_GROUP) self.assertTrue(ng_id) def test_update_node_group_by_id(self): ctx, cluster = self._make_sample() ng_id = cluster.node_groups[0].id self.api.node_group_update(ctx, ng_id, {'name': 'changed_ng'}) cluster = self.api.cluster_get(ctx, cluster.id) ng = gu.get_by_id(cluster.node_groups, ng_id) self.assertEqual('changed_ng', ng.name) def test_remove_node_group(self): ctx, cluster = self._make_sample() ng = cluster.node_groups[0] self.api.node_group_remove(ctx, ng) cluster = self.api.cluster_get(ctx, cluster.id) self.assertNotIn(ng, cluster.node_groups) def test_add_instance_to_node_group_id(self): ctx, cluster = self._make_sample() inst_id = self.api.instance_add(ctx, cluster.node_groups[0].id, SAMPLE_INSTANCE) self.assertTrue(inst_id) def test_update_instance_by_id(self): ctx, cluster = self._make_sample() ng_id = cluster.node_groups[0].id inst_id = self.api.instance_add(ctx, ng_id, SAMPLE_INSTANCE) self.api.instance_update(ctx, inst_id, {'instance_name': 'tst123'}) cluster = self.api.cluster_get(ctx, cluster.id) ng = gu.get_by_id(cluster.node_groups, ng_id) self.assertEqual('tst123', ng.instances[0].instance_name) def test_instance_volume_ops(self): ctx, cluster = self._make_sample() ng_id = cluster.node_groups[0].id inst_id = self.api.instance_add(ctx, ng_id, SAMPLE_INSTANCE) self.api.append_volume(ctx, inst_id, 0) self.api.append_volume(ctx, inst_id, 1) cluster = self.api.cluster_get(ctx, cluster.id) ng = gu.get_by_id(cluster.node_groups, ng_id) self.assertEqual(2, len(gu.get_by_id(ng.instances, inst_id).volumes)) self.api.remove_volume(ctx, inst_id, 0) cluster = self.api.cluster_get(ctx, cluster.id) ng = gu.get_by_id(cluster.node_groups, ng_id) self.assertEqual(1, len(gu.get_by_id(ng.instances, inst_id).volumes)) def _get_events(self, ctx, cluster_id, step_id=None): cluster = self.api.cluster_get(ctx, cluster_id, show_progress=True) events = [] for step in cluster.provision_progress: if step_id == step['id']: return step['events'] else: events += step['events'] if step_id: return events else: return [] def test_events_ops(self): ctx, cluster = self._make_sample() st_name = "some_name" st_type = "some_type" st_info = "some_info" # test provision step creation step_id = self.api.cluster_provision_step_add(ctx, cluster.id, { 'step_name': st_name, 'step_type': st_type, }) ncluster = self.api.cluster_get(ctx, cluster.id) self.assertEqual(1, len(ncluster['provision_progress'])) provision_step = ncluster['provision_progress'][0] self.assertEqual(st_name, provision_step['step_name']) self.assertEqual(st_type, provision_step['step_type']) self.assertEqual(cluster.id, provision_step['cluster_id']) # test adding event to step and getting events from step self.api.cluster_event_add(ctx, step_id, { 'node_group_id': 'node_group_id', 'instance_id': 'instance_id', 'instance_name': st_name, 'event_info': st_info, 'successful': True }) events = self._get_events(ctx, cluster.id, step_id) self.assertEqual(1, len(events)) self.assertEqual(st_name, events[0].instance_name) self.assertTrue(events[0].successful) self.assertEqual(st_info, events[0].event_info) self.api.cluster_destroy(ctx, cluster.id) with testtools.ExpectedException(exceptions.NotFoundException): self._get_events(ctx, cluster.id, step_id) def test_job_main_name(self): ctx = context.ctx() job_binary = self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY) job_binary_id = job_binary["id"] job_values = copy.copy(SAMPLE_JOB) job_values["mains"] = [job_binary_id] job = self.api.job_create(ctx, job_values) name = self.api.job_main_name(ctx, job) self.assertEqual(SAMPLE_JOB_BINARY["name"], name) def test_job_no_main_name(self): ctx = context.ctx() job = self.api.job_create(ctx, SAMPLE_JOB) name = self.api.job_main_name(ctx, job) self.assertIsNone(name) def test_job_libs_names(self): ctx = context.ctx() job_binary = self.api.job_binary_create(ctx, SAMPLE_JOB_BINARY) job_binary_id_0 = job_binary["id"] jb_1_values = copy.copy(SAMPLE_JOB_BINARY) jb_1_values["name"] = "some_other_name" job_binary = self.api.job_binary_create(ctx, jb_1_values) job_binary_id_1 = job_binary["id"] job_values = copy.copy(SAMPLE_JOB) job_values["libs"] = [job_binary_id_0, job_binary_id_1] job = self.api.job_create(ctx, job_values) names = self.api.job_lib_names(ctx, job) self.assertEqual(["job_binary_test", "some_other_name"], names) sahara-8.0.0/sahara/tests/unit/api/0000775000175100017510000000000013245515027017137 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/api/test_acl.py0000666000175100017510000000261613245514472021321 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_policy import policy as cpolicy from sahara.api import acl from sahara import exceptions as ex from sahara.tests.unit import base class TestAcl(base.SaharaTestCase): def _set_policy(self, json): acl.setup_policy() rules = cpolicy.Rules.load_json(json) acl.ENFORCER.set_rules(rules, use_conf=False) def test_policy_allow(self): @acl.enforce("data-processing:clusters:get_all") def test(): pass json = '{"data-processing:clusters:get_all": ""}' self._set_policy(json) test() def test_policy_deny(self): @acl.enforce("data-processing:clusters:get_all") def test(): pass json = '{"data-processing:clusters:get_all": "!"}' self._set_policy(json) self.assertRaises(ex.Forbidden, test) sahara-8.0.0/sahara/tests/unit/api/__init__.py0000666000175100017510000000000013245514472021243 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/api/middleware/0000775000175100017510000000000013245515027021254 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/api/middleware/__init__.py0000666000175100017510000000000013245514472023360 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/api/middleware/test_auth_valid.py0000666000175100017510000000677313245514472025027 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import webob.dec from sahara.api.middleware import auth_valid from sahara.tests.unit import base as test_base class AuthValidatorTest(test_base.SaharaTestCase): tid1 = '8f9f0c8c4c634d6280e785b44a10b8ab' tid2 = '431c8ab1f14f4607bdfc17e05b3924d1' @staticmethod @webob.dec.wsgify def application(req): return "Banana" def setUp(self): super(AuthValidatorTest, self).setUp() self.app = auth_valid.AuthValidator(self.application) def test_auth_ok_project_id_in_url(self): req = webob.Request.blank("/v1.1/%s/clusters" % self.tid1, accept="text/plain", method="GET", environ={"HTTP_X_TENANT_ID": self.tid1}) res = req.get_response(self.app) self.assertEqual(200, res.status_code) def test_auth_ok_no_project_id_in_url(self): req = webob.Request.blank("/v1.1/clusters", accept="text/plain", method="GET", environ={"HTTP_X_TENANT_ID": self.tid1}) res = req.get_response(self.app) self.assertEqual(200, res.status_code) def test_auth_ok_without_path(self): req = webob.Request.blank("/", accept="text/plain", method="GET", environ={"HTTP_X_TENANT_ID": self.tid1}) res = req.get_response(self.app) self.assertEqual(200, res.status_code) def test_auth_without_environ(self): req = webob.Request.blank("/v1.1/%s/clusters" % self.tid1, accept="text/plain", method="GET") res = req.get_response(self.app) self.assertEqual(503, res.status_code) def test_auth_with_wrong_url(self): req = webob.Request.blank("/v1.1", accept="text/plain", method="GET", environ={"HTTP_X_TENANT_ID": self.tid1}) res = req.get_response(self.app) self.assertEqual(404, res.status_code) def test_auth_different_tenant(self): req = webob.Request.blank("/v1.1/%s/clusters" % self.tid1, accept="text/plain", method="GET", environ={"HTTP_X_TENANT_ID": self.tid2}) res = req.get_response(self.app) self.assertEqual(401, res.status_code) def test_auth_tenant_id_in_url_v2(self): # NOTE(jfreud): we expect AuthValidator to let this case pass through # although Flask will reject it with a 404 further down the pipeline req = webob.Request.blank("/v2/%s/clusters" % self.tid1, accept="text/plain", method="GET", environ={"HTTP_X_TENANT_ID": self.tid1}) res = req.get_response(self.app) self.assertEqual(200, res.status_code) sahara-8.0.0/sahara/tests/unit/db/0000775000175100017510000000000013245515027016753 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/db/templates/0000775000175100017510000000000013245515027020751 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/db/templates/test_delete.py0000666000175100017510000003544513245514472023644 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara import context from sahara.db.templates import api as template_api from sahara.db.templates import utils as u from sahara.tests.unit.conductor import base from sahara.tests.unit.db.templates import common as c class Config(c.Config): def __init__(self, option_values=None): option_values = option_values or {} if "name" not in option_values: option_values["name"] = "delete" super(Config, self).__init__(option_values) class TemplateDeleteTestCase(base.ConductorManagerTestCase): def setUp(self): super(TemplateDeleteTestCase, self).setUp() self.logger = c.Logger() template_api.set_logger(self.logger) def test_node_group_template_delete_by_id(self): self.logger.clear_log() self.setup_context(tenant_id=None) ctx = context.ctx() t = self.api.node_group_template_create(ctx, c.SAMPLE_NGT) option_values = {"tenant_id": t["tenant_id"], "id": t["id"]} template_api.set_conf(Config(option_values)) template_api.do_node_group_template_delete_by_id() msg = 'Deleted node group template {info}'.format( info=u.name_and_id(t)) self.assertIn(msg, self.logger.infos) t = self.api.node_group_template_get(ctx, t["id"]) self.assertIsNone(t) def test_node_group_template_delete_by_id_skipped(self): self.logger.clear_log() self.setup_context(tenant_id=None) ctx = context.ctx() template_values = copy.copy(c.SAMPLE_NGT) template_values["is_default"] = False t = self.api.node_group_template_create(ctx, template_values) option_values = {"tenant_id": t["tenant_id"], "id": t["id"]} template_api.set_conf(Config(option_values)) template_api.do_node_group_template_delete_by_id() msg = ("Deletion of node group template {info} skipped, " "not a default template".format(info=u.name_and_id(t))) self.assertIn(msg, self.logger.warnings) t = self.api.node_group_template_get(ctx, t["id"]) self.assertIsNotNone(t) def test_node_group_template_delete_bad_id(self): self.logger.clear_log() option_values = {"tenant_id": 1, "id": "badid"} template_api.set_conf(Config(option_values)) template_api.do_node_group_template_delete_by_id() msg = ("Deletion of node group template {id} failed, " "no such template".format(id=option_values["id"])) self.assertIn(msg, self.logger.warnings) def test_node_group_template_delete_by_name(self): self.logger.clear_log() ctx = context.ctx() t = self.api.node_group_template_create(ctx, c.SAMPLE_NGT) option_values = {"tenant_id": t["tenant_id"], "template_name": t["name"]} template_api.set_conf(Config(option_values)) template_api.do_node_group_template_delete() msg = 'Deleted node group template {info}'.format( info=u.name_and_id(t)) self.assertIn(msg, self.logger.infos) t = self.api.node_group_template_get(ctx, t["id"]) self.assertIsNone(t) def test_node_group_template_delete_by_name_skipped(self): self.logger.clear_log() ctx = context.ctx() template_values = copy.copy(c.SAMPLE_NGT) template_values["is_default"] = False t = self.api.node_group_template_create(ctx, template_values) option_values = {"tenant_id": t["tenant_id"], "template_name": t["name"]} template_api.set_conf(Config(option_values)) template_api.do_node_group_template_delete() msg = ("Deletion of node group template {name} failed, " "no such template".format(name=t["name"])) self.assertIn(msg, self.logger.warnings) t = self.api.node_group_template_get(ctx, t["id"]) self.assertIsNotNone(t) def test_node_group_template_delete_in_use(self): self.logger.clear_log() ctx = context.ctx() t = self.api.node_group_template_create(ctx, c.SAMPLE_NGT) # Make a cluster that references the node group template cluster_values = copy.deepcopy(c.SAMPLE_CLUSTER) cluster_values["node_groups"][0]["node_group_template_id"] = t["id"] cl = self.api.cluster_create(ctx, cluster_values) # Make a cluster template that references the node group template cluster_temp_values = copy.deepcopy(c.SAMPLE_CLT) cluster_temp_values["node_groups"] = cluster_values["node_groups"] clt = self.api.cluster_template_create(ctx, cluster_temp_values) # Set up the expected messages msgs = ["Node group template {info} in use " "by clusters {clusters}".format( info=u.name_and_id(t), clusters=[cl["name"]])] msgs += ["Node group template {info} in use " "by cluster templates {cluster_temps}".format( info=u.name_and_id(t), cluster_temps=[clt["name"]])] msgs += ["Deletion of node group template {info} failed".format( info=u.name_and_id(t))] # Check delete by name option_values = {"tenant_id": t["tenant_id"], "template_name": t["name"]} template_api.set_conf(Config(option_values)) template_api.do_node_group_template_delete() for msg in msgs: self.assertIn(msg, self.logger.warnings) self.logger.clear_log() # Check again with delete by id option_values = {"tenant_id": t["tenant_id"], "id": t["id"]} template_api.set_conf(Config(option_values)) template_api.do_node_group_template_delete_by_id() for msg in msgs: self.assertIn(msg, self.logger.warnings) self.logger.clear_log() def test_cluster_template_delete_by_id(self): self.logger.clear_log() self.setup_context(tenant_id=None) ctx = context.ctx() t = self.api.cluster_template_create(ctx, c.SAMPLE_CLT) option_values = {"tenant_id": t["tenant_id"], "id": t["id"]} template_api.set_conf(Config(option_values)) template_api.do_cluster_template_delete_by_id() msg = 'Deleted cluster template {info}'.format( info=u.name_and_id(t)) self.assertIn(msg, self.logger.infos) t = self.api.cluster_template_get(ctx, t["id"]) self.assertIsNone(t) def test_cluster_template_delete_by_id_skipped(self): self.logger.clear_log() ctx = context.ctx() template_values = copy.copy(c.SAMPLE_CLT) template_values["is_default"] = False t = self.api.cluster_template_create(ctx, template_values) option_values = {"tenant_id": t["tenant_id"], "id": t["id"]} template_api.set_conf(Config(option_values)) template_api.do_cluster_template_delete_by_id() msg = ("Deletion of cluster template {info} skipped, " "not a default template".format(info=u.name_and_id(t))) self.assertIn(msg, self.logger.warnings) t = self.api.cluster_template_get(ctx, t["id"]) self.assertIsNotNone(t) def test_cluster_template_delete_bad_id(self): self.logger.clear_log() option_values = {"tenant_id": 1, "id": "badid"} template_api.set_conf(Config(option_values)) template_api.do_cluster_template_delete_by_id() msg = ("Deletion of cluster template {id} failed, " "no such template".format(id=option_values["id"])) self.assertIn(msg, self.logger.warnings) def test_cluster_template_delete_by_name(self): self.logger.clear_log() ctx = context.ctx() t = self.api.cluster_template_create(ctx, c.SAMPLE_NGT) option_values = {"tenant_id": t["tenant_id"], "template_name": t["name"]} template_api.set_conf(Config(option_values)) template_api.do_cluster_template_delete() msg = 'Deleted cluster template {info}'.format( info=u.name_and_id(t)) self.assertIn(msg, self.logger.infos) t = self.api.cluster_template_get(ctx, t["id"]) self.assertIsNone(t) def test_cluster_template_delete_by_name_skipped(self): self.logger.clear_log() ctx = context.ctx() template_values = copy.copy(c.SAMPLE_NGT) template_values["is_default"] = False t = self.api.cluster_template_create(ctx, template_values) option_values = {"tenant_id": t["tenant_id"], "template_name": t["name"]} template_api.set_conf(Config(option_values)) template_api.do_cluster_template_delete() msg = ("Deletion of cluster template {name} failed, " "no such template".format(name=t["name"])) self.assertIn(msg, self.logger.warnings) t = self.api.cluster_template_get(ctx, t["id"]) self.assertIsNotNone(t) def test_cluster_template_delete_in_use(self): self.logger.clear_log() ctx = context.ctx() t = self.api.cluster_template_create(ctx, c.SAMPLE_CLT) # Make a cluster that references the cluster template cluster_values = copy.deepcopy(c.SAMPLE_CLUSTER) cluster_values["cluster_template_id"] = t["id"] del cluster_values["node_groups"] cl = self.api.cluster_create(ctx, cluster_values) # Set up the expected messages msgs = ["Cluster template {info} in use " "by clusters {clusters}".format( info=u.name_and_id(t), clusters=[cl["name"]])] msgs += ["Deletion of cluster template {info} failed".format( info=u.name_and_id(t))] # Check delete by name option_values = {"tenant_id": t["tenant_id"], "template_name": t["name"]} template_api.set_conf(Config(option_values)) template_api.do_cluster_template_delete() for msg in msgs: self.assertIn(msg, self.logger.warnings) self.logger.clear_log() # Check again with delete by id option_values = {"tenant_id": t["tenant_id"], "id": t["id"]} template_api.set_conf(Config(option_values)) template_api.do_cluster_template_delete_by_id() for msg in msgs: self.assertIn(msg, self.logger.warnings) self.logger.clear_log() def _make_templates(self, ctx, name, plugin_name, plugin_version): # Make a node group template values = copy.copy(c.SAMPLE_NGT) values["name"] = "ngt_" + name values["plugin_name"] = plugin_name values["hadoop_version"] = plugin_version ngt = self.api.node_group_template_create(ctx, values) # Make a cluster template that references the node group template values = copy.deepcopy(c.SAMPLE_CLT) values["name"] = "clt_" + name values["plugin_name"] = plugin_name values["hadoop_version"] = plugin_version values["node_groups"][0]["node_group_template_id"] = ngt["id"] clt = self.api.cluster_template_create(ctx, values) return ngt, clt def test_do_delete(self): self.logger.clear_log() ctx = context.ctx() # Make some plugins to delete ngt, clt = self._make_templates(ctx, "first", "plugin", "v1") # Make some more for the same plugin, different version ngt2, clt2 = self._make_templates(ctx, "second", "plugin", "v2") # Make another set for a different plugin, overlapping version safe_ngt, safe_clt = self._make_templates(ctx, "third", "plugin2", "v1") # Run a delete by plugin name/version for the first set option_values = {"tenant_id": ngt["tenant_id"], "plugin_name": [ngt["plugin_name"]], "plugin_version": [ngt["hadoop_version"]]} template_api.set_conf(Config(option_values)) # Should delete clt and then ngt, check for messages in order template_api.do_delete() msgs = ["Deleted cluster template {info}".format( info=u.name_and_id(clt))] msgs += ["Deleted node group template {info}".format( info=u.name_and_id(ngt))] self.assertEqual(msgs, self.logger.infos) self.assertIsNone(self.api.node_group_template_get(ctx, ngt["id"])) self.assertIsNone(self.api.cluster_template_get(ctx, clt["id"])) # Make sure the other templates are still there self.assertIsNotNone(self.api.node_group_template_get(ctx, ngt2["id"])) self.assertIsNotNone(self.api.cluster_template_get(ctx, clt2["id"])) self.assertIsNotNone(self.api.node_group_template_get(ctx, safe_ngt["id"])) self.assertIsNotNone(self.api.cluster_template_get(ctx, safe_clt["id"])) # Run delete again for the plugin but with no version specified self.logger.clear_log() option_values = {"tenant_id": ngt2["tenant_id"], "plugin_name": [ngt2["plugin_name"]], "plugin_version": None} template_api.set_conf(Config(option_values)) # Should delete clt2 and then ngt2, check for messages in order template_api.do_delete() msgs = ["Deleted cluster template {info}".format( info=u.name_and_id(clt2))] msgs += ["Deleted node group template {info}".format( info=u.name_and_id(ngt2))] self.assertEqual(msgs, self.logger.infos) self.assertIsNone(self.api.node_group_template_get(ctx, ngt2["id"])) self.assertIsNone(self.api.cluster_template_get(ctx, clt2["id"])) # Make sure the other templates are still there self.assertIsNotNone(self.api.node_group_template_get(ctx, safe_ngt["id"])) self.assertIsNotNone(self.api.cluster_template_get(ctx, safe_clt["id"])) sahara-8.0.0/sahara/tests/unit/db/templates/common.py0000666000175100017510000000477513245514472022635 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six class Command(object): def __init__(self, option_values): for k, v in six.iteritems(option_values): setattr(self, k, v) class Config(object): def __init__(self, option_values=None): self.command = Command(option_values or {}) class Logger(object): def __init__(self): self.clear_log() def warning(self, message): self.warnings.append(message) def info(self, message): self.infos.append(message) def debug(self, message): pass def clear_log(self): self.warnings = [] self.infos = [] SAMPLE_NGT = { "plugin_name": "test_plugin", "flavor_id": "42", "tenant_id": "tenant_1", "hadoop_version": "test_version", "name": "ngt_test", "node_processes": ["p1", "p2"], "floating_ip_pool": None, "availability_zone": None, "node_configs": { "service_1": { "config_1": "value_1" }, "service_2": { "config_1": "value_1" }, }, "is_default": True } SAMPLE_CLT = { "plugin_name": "test_plugin", "tenant_id": "tenant_1", "hadoop_version": "test_version", "name": "clt-test", "cluster_configs": { "service_1": { "config_1": "value_1" }, "service_2": { "config_1": "value_1" } }, "node_groups": [ { "name": "ng_1", "flavor_id": "42", "node_processes": ["p1", "p2"], "count": 1, "floating_ip_pool": None, "security_groups": None, "availability_zone": None, } ], "is_default": True } SAMPLE_CLUSTER = { "name": "test_cluster", "plugin_name": "test_plugin", "hadoop_version": "test_version", "tenant_id": "tenant_1", "node_groups": [ { "name": "ng_1", "node_group_template_id": "ng_1_id", "count": 1, } ] } sahara-8.0.0/sahara/tests/unit/db/templates/test_utils.py0000666000175100017510000001227713245514472023540 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy from sahara import context from sahara.db.templates import utils from sahara.tests.unit.conductor import base from sahara.tests.unit.db.templates import common as c class FakeNGT(object): def __init__(self, id): self.node_group_template_id = id class FakeCluster(object): def __init__(self, name, node_groups=None, cluster_template_id=None): self.name = name self.node_groups = node_groups or [] self.cluster_template_id = cluster_template_id class TemplateUtilsTestCase(base.ConductorManagerTestCase): def test_substitute_ng_ids(self): cl = {"node_groups": [{"name": "worker", "node_group_template_id": "{vanilla-worker}", "count": 3}, {"name": "master", "node_group_template_id": "{vanilla-master}", "count": 1}, {"name": "secondary-name", "node_group_template_id": "some_id"}]} ng_dict = {"vanilla-worker": 1, "vanilla-master": 2} utils.substitute_ng_ids(cl, ng_dict) self.assertEqual("1", cl["node_groups"][0]["node_group_template_id"]) self.assertEqual("2", cl["node_groups"][1]["node_group_template_id"]) self.assertEqual("some_id", cl["node_groups"][2]["node_group_template_id"]) def test_check_plugin_version(self): template = {"plugin_name": "vanilla", "hadoop_version": "2.7.1"} self.assertTrue(utils.check_plugin_version(template, None)) self.assertTrue(utils.check_plugin_version(template, ["2.7.1"])) self.assertTrue(utils.check_plugin_version(template, ["vanilla.2.7.1"])) self.assertFalse(utils.check_plugin_version(template, ["1.2.1"])) def test_check_plugin_name_and_version(self): template = {"plugin_name": "vanilla", "hadoop_version": "2.7.1"} self.assertTrue(utils.check_plugin_name_and_version( template, None, ["2.7.1"])) self.assertTrue(utils.check_plugin_name_and_version( template, ["vanilla"], None)) self.assertTrue(utils.check_plugin_name_and_version( template, ["vanilla"], ["2.7.1"])) self.assertTrue(utils.check_plugin_name_and_version( template, ["vanilla"], ["vanilla.2.7.1"])) self.assertFalse(utils.check_plugin_name_and_version( template, ["hdp"], ["2.7.1"])) def test_check_node_group_template_usage(self): ng1 = FakeNGT(1) ng2 = FakeNGT(2) cluster = FakeCluster("cluster", [ng1]) template = FakeCluster("template", [ng2]) cluster_users, template_users = utils.check_node_group_template_usage( 1, [cluster], [template]) self.assertEqual([cluster.name], cluster_users) self.assertEqual([], template_users) cluster_users, template_users = utils.check_node_group_template_usage( 2, [cluster], [template]) self.assertEqual([], cluster_users) self.assertEqual([template.name], template_users) def test_check_cluster_template_usage(self): cluster = FakeCluster("cluster", cluster_template_id=1) cluster_users = utils.check_cluster_template_usage(1, [cluster]) self.assertEqual([cluster.name], cluster_users) def test_find_node_group_template_by_name(self): ctx = context.ctx() t = self.api.node_group_template_create(ctx, c.SAMPLE_NGT) found = utils.find_node_group_template_by_name(ctx, c.SAMPLE_NGT["name"]) self.assertEqual(t["id"], found["id"]) found = utils.find_node_group_template_by_name(ctx, "fred") self.assertIsNone(found) def test_find_cluster_template_by_name(self): ctx = context.ctx() t = self.api.cluster_template_create(ctx, c.SAMPLE_CLT) found = utils.find_cluster_template_by_name(ctx, c.SAMPLE_CLT["name"]) self.assertEqual(t["id"], found["id"]) found = utils.find_cluster_template_by_name(ctx, "fred") self.assertIsNone(found) def test_value_diff(self): current = {"cat": "meow", "dog": "woof", "horse": ["neigh", "whinny"]} new_values = {"dog": "bark", "horse": "snort"} original = copy.deepcopy(current) backup = utils.value_diff(current, new_values) self.assertEqual({"dog": "woof", "horse": ["neigh", "whinny"]}, backup) # current is unchanged self.assertEqual(original, current) sahara-8.0.0/sahara/tests/unit/db/templates/__init__.py0000666000175100017510000000000013245514472023055 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/db/templates/test_update.py0000666000175100017510000007567613245514472023676 0ustar zuulzuul00000000000000# Copyright 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import tempfile import jsonschema import mock from oslo_serialization import jsonutils as json from oslo_utils import uuidutils from sahara import context from sahara.db.templates import api as template_api from sahara.db.templates import utils as u from sahara.tests.unit.conductor import base from sahara.tests.unit.db.templates import common as c cluster_json = { "plugin_name": "vanilla", "hadoop_version": "2.7.1", "node_groups": [ { "name": "worker", "count": 3, "node_group_template_id": "{vanilla-260-default-worker}" }, { "name": "master", "count": 1, "node_group_template_id": "{vanilla-260-default-master}" } ], "name": "vanilla-260-default-cluster", "neutron_management_network": "{neutron_management_network}", "cluster_configs": {} } master_json = { "plugin_name": "vanilla", "hadoop_version": "2.7.1", "node_processes": [ "namenode", "resourcemanager", "hiveserver" ], "name": "vanilla-260-default-master", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", 'security_groups': "{security_groups}" } worker_json = { "plugin_name": "vanilla", "hadoop_version": "2.7.1", "node_processes": [ "nodemanager", "datanode" ], "name": "vanilla-260-default-worker", "floating_ip_pool": "{floating_ip_pool}", "flavor_id": "{flavor_id}", "auto_security_group": "{auto_security_group}", 'security_groups': "{security_groups}" } class Config(c.Config): def __init__(self, option_values=None): option_values = option_values or {} if "name" not in option_values: option_values["name"] = "update" super(Config, self).__init__(option_values) class TemplateUpdateTestCase(base.ConductorManagerTestCase): def setUp(self): super(TemplateUpdateTestCase, self).setUp() self.logger = c.Logger() template_api.set_logger(self.logger) @mock.patch("sahara.utils.api_validator.ApiValidator.validate") def test_check_cluster_templates_valid(self, validate): self.logger.clear_log() ng_templates = [{"template": c.SAMPLE_NGT, "path": "/foo"}] # Reference the node group template by name clt = copy.copy(c.SAMPLE_CLT) clt["node_groups"] = [ {"name": "test", "count": 1, "node_group_template_id": "{%s}" % c.SAMPLE_NGT["name"]} ] cl_templates = [{"template": clt, "path": "/bar"}] # Test failed validation validate.side_effect = jsonschema.ValidationError("mistake") res = template_api.check_cluster_templates_valid(ng_templates, cl_templates) self.assertTrue(res) msg = "Validation for /bar failed, mistake" self.assertIn(msg, self.logger.warnings) # Validation passes, name replaced validate.side_effect = None self.logger.clear_log() res = template_api.check_cluster_templates_valid(ng_templates, cl_templates) self.assertFalse(res) node_groups = validate.call_args[0][0]["node_groups"] self.assertTrue(uuidutils.is_uuid_like( node_groups[0]["node_group_template_id"])) def test_add_config_section(self): # conf here can't be a mock.Mock() because hasattr will # return true conf = Config() conf.register_group = mock.Mock() conf.register_opts = mock.Mock() template_api.set_conf(conf) opts = ["option"] # Named config section template_api.add_config_section("section", opts) self.assertEqual(1, conf.register_group.call_count) config_group = conf.register_group.call_args[0][0] self.assertEqual("section", config_group.name) self.assertEqual([ mock.call(opts, config_group)], conf.register_opts.call_args_list) conf.register_group.reset_mock() conf.register_opts.reset_mock() # No config section, opts should be registered against # the default section template_api.add_config_section(None, opts) conf.register_group.assert_not_called() conf.register_opts.assert_called_with(opts) @mock.patch("sahara.db.templates.api.add_config_section") def test_add_config_section_for_template(self, add_config_section): conf = mock.Mock() conf.list_all_sections = mock.Mock() template_api.set_conf(conf) # No config sections conf.list_all_sections.return_value = [] ngt = c.SAMPLE_NGT template_api.add_config_section_for_template(ngt) add_config_section.assert_called_with(None, template_api.all_template_opts) add_config_section.reset_mock() # Add config section matching plugin conf.list_all_sections.return_value += [ngt["plugin_name"]] template_api.add_config_section_for_template(ngt) add_config_section.assert_called_with(ngt["plugin_name"], template_api.all_template_opts) add_config_section.reset_mock() # Add config section matching plugin and version section = "{plugin_name}_{hadoop_version}".format(**ngt) conf.list_all_sections.return_value += [section] template_api.add_config_section_for_template(ngt) add_config_section.assert_called_with(section, template_api.all_template_opts) add_config_section.reset_mock() # Add config section matching plugin, version and name section = "{plugin_name}_{hadoop_version}_{name}".format(**ngt) conf.list_all_sections.return_value += [section] template_api.add_config_section_for_template(ngt) add_config_section.assert_called_with( section, template_api.node_group_template_opts) add_config_section.reset_mock() # Add config section matching name section = "{name}".format(**ngt) conf.list_all_sections.return_value += [section] template_api.add_config_section_for_template(ngt) add_config_section.assert_called_with( section, template_api.node_group_template_opts) add_config_section.reset_mock() def test_substitute_config_values_ngt(self): ngt = copy.copy(c.SAMPLE_NGT) ngt["flavor_id"] = "{flavor_id}" ngt["floating_ip_pool"] = "{floating_ip_pool}" configs = {"flavor_id": "2", "floating_ip_pool": None} template_api.substitute_config_values(configs, ngt, "/path") self.assertEqual("2", ngt["flavor_id"]) self.assertIsNone(ngt["floating_ip_pool"]) def test_substitute_config_values_clt(self): clt = copy.copy(c.SAMPLE_CLT) clt["neutron_management_network"] = "{neutron_management_network}" clt["default_image_id"] = "{default_image_id}" netid = uuidutils.generate_uuid() configs = {"neutron_management_network": netid, "default_image_id": None} template_api.substitute_config_values(configs, clt, "/path") self.assertEqual(netid, clt["neutron_management_network"]) self.assertIsNone(clt["default_image_id"]) def _write_files(self, tempdir, templates): files = [] for template in templates: fp = tempfile.NamedTemporaryFile(suffix=".json", mode="w", dir=tempdir, delete=False) json.dump(template, fp) files.append(fp.name) fp.close() return files @mock.patch("sahara.db.templates.api.get_configs") @mock.patch("sahara.db.templates.api.add_config_section_for_template") def test_process_files(self, add_config_section, get_configs): self.logger.clear_log() tempdir = tempfile.mkdtemp() # This should be ignored by process files some_other_json = {"name": "fred", "description": "not a template"} files = self._write_files( tempdir, [cluster_json, master_json, worker_json, some_other_json]) get_configs.return_value = {"flavor_id": '2', 'security_groups': [], 'auto_security_group': False} option_values = {"plugin_name": None, "plugin_version": None} template_api.set_conf(Config(option_values)) # Check that cluster and ng templates are read and returned ng_templates, cl_templates = template_api.process_files(tempdir, files) cl_temp_names = [f["template"]["name"] for f in cl_templates] ng_temp_names = [f["template"]["name"] for f in ng_templates] self.assertEqual([cluster_json["name"]], cl_temp_names) self.assertEqual([master_json["name"], worker_json["name"]], ng_temp_names) # Plugin name/version filtering applied option_values = {"plugin_name": "vanilla", "plugin_version": "2.7.1"} template_api.set_conf(Config(option_values)) ng_templates, cl_templates = template_api.process_files(tempdir, files) self.assertEqual(1, len(cl_templates)) self.assertEqual(2, len(ng_templates)) option_values = {"plugin_name": "hdp", "plugin_version": "2.7.1"} template_api.set_conf(Config(option_values)) ng_templates, cl_templates = template_api.process_files(tempdir, files) self.assertEqual(0, len(cl_templates)) self.assertEqual(0, len(ng_templates)) @mock.patch("sahara.db.templates.api.get_configs") @mock.patch("sahara.db.templates.api.add_config_section_for_template") def test_process_files_validation_error(self, add_config_section, get_configs): self.logger.clear_log() tempdir = tempfile.mkdtemp() files = self._write_files( tempdir, [cluster_json, master_json, worker_json]) get_configs.return_value = { "flavor_id": '2', 'security_groups': [], 'auto_security_group': False } option_values = {"plugin_name": None, "plugin_version": None} template_api.set_conf(Config(option_values)) # Bad JSON validation for ng should cause all files to be skipped bad_worker = copy.copy(worker_json) bad_worker["my_dog"] = ["fido"] new_file = self._write_files(tempdir, [bad_worker])[0] ng_templates, cl_templates = template_api.process_files( tempdir, files + [new_file]) self.assertEqual(0, len(ng_templates)) self.assertEqual(0, len(cl_templates)) msg = ("Validation for {path} failed, " "Additional properties are not allowed".format(path=new_file)) self.assertTrue(self.logger.warnings[0].startswith(msg)) @mock.patch("sahara.db.templates.api.get_configs") @mock.patch("sahara.db.templates.api.add_config_section_for_template") def test_process_files_bad_json(self, add_config_section, get_configs): self.logger.clear_log() tempdir = tempfile.mkdtemp() files = self._write_files( tempdir, [cluster_json, master_json, worker_json]) get_configs.return_value = {"flavor_id": '2', 'security_groups': [], 'auto_security_group': False} option_values = {"plugin_name": None, "plugin_version": None} template_api.set_conf(Config(option_values)) # Invalid JSON should cause all files to be skipped fp = tempfile.NamedTemporaryFile(suffix=".json", dir=tempdir, delete=False) fp.write(b"not json") files += [fp.name] fp.close() ng_templates, cl_templates = template_api.process_files(tempdir, files) self.assertEqual(0, len(ng_templates)) self.assertEqual(0, len(cl_templates)) msg = ("Error processing {name}".format(name=files[-1])) self.assertTrue(self.logger.warnings[0].startswith(msg)) msg = ("Skipping processing for {dir}, " "error processing files".format(dir=tempdir)) self.assertEqual(msg, self.logger.warnings[1]) def test_add_node_group_templates(self): self.logger.clear_log() ctx = context.ctx() # Create a record that will be updated in the db existing = copy.copy(c.SAMPLE_NGT) existing = self.api.node_group_template_create(ctx, existing) # Create the update update = copy.copy(c.SAMPLE_NGT) update["flavor_id"] = "6" # Create a record that will be new in the db new = copy.copy(c.SAMPLE_NGT) new["name"] = "new_name" ngts = [{"template": update, "path": "foo"}, {"template": new, "path": "bar"}] ng_info, error = template_api.add_node_group_templates(ctx, ngts) self.assertFalse(error) new = self.api.node_group_template_get_all(ctx, name=new["name"])[0] self.assertIsNotNone(new) # ng_info["created"] is a list of templates that were created self.assertEqual(1, len(ng_info["created"])) self.assertEqual(new["id"], ng_info["created"][0]["id"]) # ng_info["updated"] is a list of tuples for templates that # were updated. First element in the tuple is the template, # second is a dictionary of fields that were updated. self.assertEqual(1, len(ng_info["updated"])) self.assertEqual(existing["id"], ng_info["updated"][0][0]["id"]) self.assertEqual({"flavor_id": "42"}, ng_info["updated"][0][1]) # ng_info["dict"] is a dictionary of name/id pairs self.assertEqual({new["name"]: new["id"], existing["name"]: existing["id"]}, ng_info["ids"]) msg = ("Created node group template {info} from bar".format( info=u.name_and_id(new))) self.assertIn(msg, self.logger.infos) msg = ("Updated node group template {info} from foo".format( info=u.name_and_id(existing))) self.assertIn(msg, self.logger.infos) self.api.node_group_template_destroy(ctx, new["id"], ignore_prot_on_def=True) self.api.node_group_template_destroy(ctx, existing["id"], ignore_prot_on_def=True) @mock.patch("sahara.conductor.API.node_group_template_update") @mock.patch("sahara.db.templates.api.reverse_node_group_template_creates") @mock.patch("sahara.db.templates.api.reverse_node_group_template_updates") def test_add_node_group_templates_update_failed(self, reverse_updates, reverse_creates, ng_update): self.logger.clear_log() ctx = context.ctx() ng_update.side_effect = Exception("mistake") # Create a record that will be updated in the db existing = copy.copy(c.SAMPLE_NGT) existing = self.api.node_group_template_create(ctx, existing) # Create the update update = copy.copy(c.SAMPLE_NGT) update["flavor_id"] = "6" # Create a record that will be new in the db new = copy.copy(c.SAMPLE_NGT) new["name"] = "new_name" ngts = [{"template": new, "path": "bar"}, {"template": update, "path": "foo"}] ng_info, error = template_api.add_node_group_templates(ctx, ngts) new = self.api.node_group_template_get_all(ctx, name=new["name"])[0] self.assertTrue(error) self.assertEqual(1, reverse_creates.call_count) # call should have been (ctx, [new]) self.assertEqual(new["id"], reverse_creates.call_args[0][1][0]["id"]) self.assertEqual(1, reverse_updates.call_count) msg = ("Update of node group template {info} failed, mistake".format( info=u.name_and_id(existing))) self.assertIn(msg, self.logger.warnings) self.api.node_group_template_destroy(ctx, new["id"], ignore_prot_on_def=True) self.api.node_group_template_destroy(ctx, existing["id"], ignore_prot_on_def=True) @mock.patch("sahara.conductor.API.node_group_template_create") @mock.patch("sahara.db.templates.api.reverse_node_group_template_creates") @mock.patch("sahara.db.templates.api.reverse_node_group_template_updates") def test_add_node_group_templates_create_failed(self, reverse_updates, reverse_creates, ng_create): self.logger.clear_log() ctx = context.ctx() ng_create.side_effect = Exception("mistake") # Create a record that will be updated in the db existing = copy.copy(c.SAMPLE_NGT) existing = self.api.node_group_template_create(ctx, existing) # Create the update update = copy.copy(c.SAMPLE_NGT) update["flavor_id"] = "6" # Create a record that will be new in the db new = copy.copy(c.SAMPLE_NGT) new["name"] = "new_name" ngts = [{"template": update, "path": "foo"}, {"template": new, "path": "bar"}] ng_info, error = template_api.add_node_group_templates(ctx, ngts) self.assertTrue(error) self.assertEqual(1, reverse_creates.call_count) self.assertEqual(1, reverse_updates.call_count) # call should have been (ctx, [(existing, updated_fields)]) self.assertEqual({"flavor_id": existing["flavor_id"]}, reverse_updates.call_args[0][1][0][1]) msg = "Creation of node group template from bar failed, mistake" self.assertIn(msg, self.logger.warnings) self.api.node_group_template_destroy(ctx, existing["id"], ignore_prot_on_def=True) def test_add_cluster_templates(self): self.logger.clear_log() ctx = context.ctx() # Create a record that will be updated in the db existing = copy.copy(c.SAMPLE_CLT) existing = self.api.cluster_template_create(ctx, existing) # Create the update update = copy.copy(c.SAMPLE_CLT) update["hadoop_version"] = "1" # Create a record that will be new in the db new = copy.copy(c.SAMPLE_CLT) new["name"] = "new_name" clts = [{"template": update, "path": "foo"}, {"template": new, "path": "bar"}] error = template_api.add_cluster_templates(ctx, clts, {}) self.assertFalse(error) new = self.api.cluster_template_get_all(ctx, name=new["name"])[0] self.assertIsNotNone(new) msg = ("Created cluster template {info} from bar".format( info=u.name_and_id(new))) self.assertIn(msg, self.logger.infos) msg = ("Updated cluster template {info} from foo".format( info=u.name_and_id(existing))) self.assertIn(msg, self.logger.infos) self.api.cluster_template_destroy(ctx, new["id"], ignore_prot_on_def=True) self.api.cluster_template_destroy(ctx, existing["id"], ignore_prot_on_def=True) @mock.patch("sahara.conductor.API.cluster_template_update") @mock.patch("sahara.db.templates.api.reverse_cluster_template_creates") @mock.patch("sahara.db.templates.api.reverse_cluster_template_updates") def test_add_cluster_templates_update_failed(self, reverse_updates, reverse_creates, cl_update): self.logger.clear_log() ctx = context.ctx() cl_update.side_effect = Exception("mistake") # Create a record that will be updated in the db existing = copy.copy(c.SAMPLE_CLT) existing = self.api.cluster_template_create(ctx, existing) # Create the update update = copy.copy(c.SAMPLE_CLT) update["hadoop_version"] = "1" # Create a record that will be new in the db new = copy.copy(c.SAMPLE_CLT) new["name"] = "new_name" clts = [{"template": new, "path": "bar"}, {"template": update, "path": "foo"}] error = template_api.add_cluster_templates(ctx, clts, {}) new = self.api.cluster_template_get_all(ctx, name=new["name"])[0] self.assertTrue(error) self.assertEqual(1, reverse_creates.call_count) # call should have been (ctx, [new]) self.assertEqual(new["id"], reverse_creates.call_args[0][1][0]["id"]) self.assertEqual(1, reverse_updates.call_count) msg = ("Update of cluster template {info} failed, mistake".format( info=u.name_and_id(existing))) self.assertIn(msg, self.logger.warnings) self.api.cluster_template_destroy(ctx, new["id"], ignore_prot_on_def=True) self.api.cluster_template_destroy(ctx, existing["id"], ignore_prot_on_def=True) @mock.patch("sahara.conductor.API.cluster_template_create") @mock.patch("sahara.db.templates.api.reverse_cluster_template_creates") @mock.patch("sahara.db.templates.api.reverse_cluster_template_updates") def test_add_cluster_templates_create_failed(self, reverse_updates, reverse_creates, cl_create): self.logger.clear_log() ctx = context.ctx() cl_create.side_effect = Exception("mistake") # Create a record that will be updated in the db existing = copy.copy(c.SAMPLE_CLT) existing = self.api.cluster_template_create(ctx, existing) # Create the update update = copy.copy(c.SAMPLE_CLT) update["hadoop_version"] = "1" # Create a record that will be new in the db new = copy.copy(c.SAMPLE_CLT) new["name"] = "new_name" clts = [{"template": update, "path": "foo"}, {"template": new, "path": "bar"}] error = template_api.add_cluster_templates(ctx, clts, {}) self.assertTrue(error) self.assertEqual(1, reverse_creates.call_count) self.assertEqual(1, reverse_updates.call_count) # call should have been (ctx, [(existing, updated_fields)]) # updated fields will contain hadoop_version and node_groups, # since node_groups is modified by the conductor updated_fields = reverse_updates.call_args[0][1][0][1] self.assertEqual(updated_fields["hadoop_version"], existing["hadoop_version"]) self.assertIn("node_groups", updated_fields) msg = "Creation of cluster template from bar failed, mistake" self.assertIn(msg, self.logger.warnings) self.api.cluster_template_destroy(ctx, existing["id"], ignore_prot_on_def=True) @mock.patch("sahara.db.templates.api.get_configs") @mock.patch("sahara.db.templates.api.add_config_section_for_template") def test_do_update_trash(self, add_config, get_configs): self.logger.clear_log() ctx = context.ctx() tempdir = tempfile.mkdtemp() self._write_files(tempdir, [cluster_json, master_json, worker_json]) get_configs.return_value = { "flavor_id": '2', "neutron_management_network": uuidutils.generate_uuid(), 'auto_security_group': True, 'security_groups': [], } option_values = {"tenant_id": ctx.tenant_id, "directory": tempdir, "norecurse": None, "plugin_name": None, "plugin_version": None} template_api.set_conf(Config(option_values)) template_api.do_update() ngs = self.api.node_group_template_get_all(ctx) ng_names = sorted([ng["name"] for ng in ngs]) self.assertEqual(sorted([master_json["name"], worker_json["name"]]), ng_names) clts = self.api.cluster_template_get_all(ctx) clt_names = sorted([clt["name"] for clt in clts]) clts = self.api.cluster_template_get_all(ctx) self.assertEqual([cluster_json["name"]], clt_names) @mock.patch("sahara.db.templates.api.check_cluster_templates_valid") @mock.patch("sahara.db.templates.api.get_configs") @mock.patch("sahara.db.templates.api.add_config_section_for_template") def test_do_update_cluster_invalid(self, add_config, get_configs, clt_valid): self.logger.clear_log() ctx = context.ctx() tempdir = tempfile.mkdtemp() self._write_files(tempdir, [cluster_json, master_json, worker_json]) get_configs.return_value = { "flavor_id": '2', "neutron_management_network": uuidutils.generate_uuid() } option_values = {"tenant_id": ctx.tenant_id, "directory": tempdir, "norecurse": None, "plugin_name": None, "plugin_version": None} template_api.set_conf(Config(option_values)) clt_valid.return_value = True template_api.do_update() ngs = self.api.node_group_template_get_all(ctx) self.assertEqual([], ngs) clts = self.api.cluster_template_get_all(ctx) self.assertEqual([], clts) msg = ("Skipping processing for {dir}, " "error processing cluster templates".format(dir=tempdir)) self.assertIn(msg, self.logger.warnings) @mock.patch("sahara.db.templates.api.check_usage_of_existing") @mock.patch("sahara.db.templates.api.get_configs") @mock.patch("sahara.db.templates.api.add_config_section_for_template") def test_do_update_existing_fails(self, add_config, get_configs, check_existing): self.logger.clear_log() ctx = context.ctx() tempdir = tempfile.mkdtemp() self._write_files(tempdir, [cluster_json, master_json, worker_json]) get_configs.return_value = { "flavor_id": '2', "neutron_management_network": uuidutils.generate_uuid() } option_values = {"tenant_id": ctx.tenant_id, "directory": tempdir, "norecurse": None, "plugin_name": None, "plugin_version": None} template_api.set_conf(Config(option_values)) check_existing.return_value = True template_api.do_update() ngs = self.api.node_group_template_get_all(ctx) self.assertEqual([], ngs) clts = self.api.cluster_template_get_all(ctx) self.assertEqual([], clts) msg = ("Skipping processing for {dir}, " "templates in use".format(dir=tempdir)) self.assertIn(msg, self.logger.warnings) @mock.patch("sahara.db.templates.api.add_node_group_templates") @mock.patch("sahara.db.templates.api.get_configs") @mock.patch("sahara.db.templates.api.add_config_section_for_template") def test_do_update_add_ngts_fails(self, add_config, get_configs, add_ngts): self.logger.clear_log() ctx = context.ctx() tempdir = tempfile.mkdtemp() self._write_files(tempdir, [cluster_json, master_json, worker_json]) get_configs.return_value = { "flavor_id": '2', "neutron_management_network": uuidutils.generate_uuid() } option_values = {"tenant_id": ctx.tenant_id, "directory": tempdir, "norecurse": None, "plugin_name": None, "plugin_version": None} template_api.set_conf(Config(option_values)) add_ngts.return_value = ({}, True) template_api.do_update() ngs = self.api.node_group_template_get_all(ctx) self.assertEqual([], ngs) clts = self.api.cluster_template_get_all(ctx) self.assertEqual([], clts) msg = ("Skipping processing for {dir}, " "error processing node group templates".format(dir=tempdir)) self.assertIn(msg, self.logger.warnings) @mock.patch("sahara.db.templates.api.reverse_node_group_template_creates") @mock.patch("sahara.db.templates.api.reverse_node_group_template_updates") @mock.patch("sahara.db.templates.api.add_cluster_templates") @mock.patch("sahara.db.templates.api.get_configs") @mock.patch("sahara.db.templates.api.add_config_section_for_template") def test_do_update_add_clts_fails(self, add_config, get_configs, add_clts, reverse_ng_updates, reverse_ng_creates): self.logger.clear_log() ctx = context.ctx() tempdir = tempfile.mkdtemp() self._write_files(tempdir, [cluster_json, master_json, worker_json]) get_configs.return_value = { "flavor_id": '2', "neutron_management_network": uuidutils.generate_uuid() } option_values = {"tenant_id": ctx.tenant_id, "directory": tempdir, "norecurse": None, "plugin_name": None, "plugin_version": None} template_api.set_conf(Config(option_values)) add_clts.return_value = True template_api.do_update() self.assertEqual(1, reverse_ng_creates.call_count) self.assertEqual(1, reverse_ng_updates.call_count) clts = self.api.cluster_template_get_all(ctx) self.assertEqual([], clts) msg = ("Skipping processing for {dir}, " "error processing cluster templates".format(dir=tempdir)) self.assertIn(msg, self.logger.warnings) sahara-8.0.0/sahara/tests/unit/db/sqlalchemy/0000775000175100017510000000000013245515027021115 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/db/sqlalchemy/test_types.py0000666000175100017510000001075013245514472023702 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import sqlalchemy as sa import testtools from sahara.db.sqlalchemy import types class JsonEncodedTest(testtools.TestCase): def test_impl(self): impl = types.JsonEncoded.impl self.assertEqual(sa.Text, impl) def test_process_bind_param(self): t = types.JsonEncoded() self.assertEqual('{"a": 1}', t.process_bind_param({"a": 1}, None)) def test_process_bind_param_none(self): t = types.JsonEncoded() self.assertIsNone(t.process_bind_param(None, None)) def test_process_result_value(self): t = types.JsonEncoded() self.assertEqual({"a": 1}, t.process_result_value('{"a": 1}', None)) def test_process_result_value_none(self): t = types.JsonEncoded() self.assertIsNone(t.process_result_value(None, None)) class MutableDictTest(testtools.TestCase): def test_creation(self): sample = {"a": 1, "b": 2} d = types.MutableDict(sample) self.assertEqual(sample, d) def test_coerce_dict(self): sample = {"a": 1, "b": 2} md = types.MutableDict.coerce("test", sample) self.assertEqual(sample, md) self.assertIsInstance(md, types.MutableDict) def test_coerce_mutable_dict(self): sample = {"a": 1, "b": 2} sample_md = types.MutableDict(sample) md = types.MutableDict.coerce("test", sample_md) self.assertEqual(sample, md) self.assertIs(sample_md, md) def test_coerce_unsupported(self): with testtools.ExpectedException(ValueError): types.MutableDict.coerce("test", list()) @mock.patch.object(types.MutableDict, 'changed') def test_changed_on_update(self, m): sample = {"a": 1, "b": 2} d = types.MutableDict(sample) d.update({"b": 3}) self.assertEqual({"a": 1, "b": 3}, d) self.assertEqual(1, m.call_count) @mock.patch.object(types.MutableDict, 'changed') def test_changed_on_setitem(self, m): sample = {"a": 1, "b": 2} d = types.MutableDict(sample) d["b"] = 3 self.assertEqual({"a": 1, "b": 3}, d) self.assertEqual(1, m.call_count) @mock.patch.object(types.MutableDict, 'changed') def test_changed_on_delitem(self, m): sample = {"a": 1, "b": 2} d = types.MutableDict(sample) del d["b"] self.assertEqual({"a": 1}, d) self.assertEqual(1, m.call_count) class MutableListTest(testtools.TestCase): def test_creation(self): sample = [1, 2, 3] d = types.MutableList(sample) self.assertEqual(sample, d) def test_coerce_list(self): sample = [1, 2, 3] md = types.MutableList.coerce("test", sample) self.assertEqual(sample, md) self.assertIsInstance(md, types.MutableList) def test_coerce_mutable_list(self): sample = [1, 2, 3] sample_md = types.MutableList(sample) md = types.MutableList.coerce("test", sample_md) self.assertEqual(sample, md) self.assertIs(sample_md, md) def test_coerce_unsupported(self): with testtools.ExpectedException(ValueError): types.MutableList.coerce("test", dict()) @mock.patch.object(types.MutableList, 'changed') def test_changed_on_append(self, m): sample = [1, 2, 3] lst = types.MutableList(sample) lst.append(4) self.assertEqual([1, 2, 3, 4], lst) self.assertEqual(1, m.call_count) @mock.patch.object(types.MutableList, 'changed') def test_changed_on_setitem(self, m): sample = [1, 2, 3] lst = types.MutableList(sample) lst[2] = 4 self.assertEqual([1, 2, 4], lst) self.assertEqual(1, m.call_count) @mock.patch.object(types.MutableList, 'changed') def test_changed_on_delitem(self, m): sample = [1, 2, 3] lst = types.MutableList(sample) del lst[2] self.assertEqual([1, 2], lst) self.assertEqual(1, m.call_count) sahara-8.0.0/sahara/tests/unit/db/sqlalchemy/__init__.py0000666000175100017510000000000013245514472023221 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/db/test_utils.py0000666000175100017510000001047013245514472021533 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara import context from sahara.db.sqlalchemy import api from sahara.db.sqlalchemy import models as m import sahara.tests.unit.base as base class TestPaginationUtils(testtools.TestCase): def test_get_prev_and_next_objects(self): query = [mock.MagicMock(id=i) for i in range(100)] res = api._get_prev_and_next_objects(query, 5, None) self.assertEqual((None, 4), res) res = api._get_prev_and_next_objects(query, None, None) self.assertEqual((None, None), res) res = api._get_prev_and_next_objects(query, 5, mock.MagicMock(id=42)) self.assertEqual((37, 47), res) res = api._get_prev_and_next_objects(query, 5, mock.MagicMock(id=4)) self.assertEqual((None, 9), res) res = api._get_prev_and_next_objects(query, 5, mock.MagicMock(id=100)) self.assertEqual((None, None), res) def test_parse_sorting_args(self): self.assertEqual(("name", "desc"), api._parse_sorting_args("-name")) self.assertEqual(("name", "asc"), api._parse_sorting_args("name")) class TestRegex(testtools.TestCase): def test_get_regex_op(self): regex_op = api._get_regex_op("mysql://user:passw@localhost/sahara") self.assertEqual("REGEXP", regex_op) regex_op = api._get_regex_op("postgresql://localhost/sahara") self.assertEqual("~", regex_op) regex_op = api._get_regex_op("sqlite://user:passw@localhost/sahara") self.assertIsNone(regex_op) class TestRegexFilter(base.SaharaWithDbTestCase): @mock.patch("sahara.db.sqlalchemy.api._get_regex_op") def test_regex_filter(self, get_regex_op): query = api.model_query(m.ClusterTemplate, context.ctx()) regex_cols = ["name", "description", "plugin_name"] search_opts = {"name": "fred", "hadoop_version": "2", "bogus": "jack", "plugin_name": "vanilla"} # Since regex_op is None remaining_opts should be a copy of search_opts get_regex_op.return_value = None query, remaining_opts = api.regex_filter( query, m.ClusterTemplate, regex_cols, search_opts) self.assertEqual(search_opts, remaining_opts) self.assertIsNot(search_opts, remaining_opts) # Since regex_cols is [] remaining_opts should be a copy of search_opts get_regex_op.return_value = "REGEXP" query, remaining_opts = api.regex_filter( query, m.ClusterTemplate, [], search_opts) self.assertEqual(search_opts, remaining_opts) self.assertIsNot(search_opts, remaining_opts) # Remaining should be search_opts with name and plugin_name removed # These are the only fields that are in regex_cols and also in # the model. get_regex_op.return_value = "REGEXP" query, remaining_opts = api.regex_filter( query, m.ClusterTemplate, regex_cols, search_opts) self.assertEqual({"hadoop_version": "2", "bogus": "jack"}, remaining_opts) # bogus is not in the model so it should be left in remaining # even though regex_cols lists it regex_cols.append("bogus") query, remaining_opts = api.regex_filter( query, m.ClusterTemplate, regex_cols, search_opts) self.assertEqual({"hadoop_version": "2", "bogus": "jack"}, remaining_opts) # name will not be removed because the value is not a string search_opts["name"] = 5 query, remaining_opts = api.regex_filter( query, m.ClusterTemplate, regex_cols, search_opts) self.assertEqual({"hadoop_version": "2", "bogus": "jack", "name": 5}, remaining_opts) sahara-8.0.0/sahara/tests/unit/db/__init__.py0000666000175100017510000000000013245514472021057 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/db/migration/0000775000175100017510000000000013245515027020744 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/db/migration/test_migrations_base.py0000666000175100017510000001432213245514472025532 0ustar zuulzuul00000000000000# Copyright 2010-2011 OpenStack Foundation # Copyright 2012-2013 IBM Corp. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # # # Ripped off from Nova's test_migrations.py # The only difference between Nova and this code is usage of alembic instead # of sqlalchemy migrations. # # There is an ongoing work to extact similar code to oslo incubator. Once it is # extracted we'll be able to remove this file and use oslo. import io import os import alembic from alembic import command from alembic import config as alembic_config from alembic import migration from alembic import script as alembic_script from oslo_config import cfg from oslo_db.sqlalchemy import test_migrations as t_m from oslo_log import log as logging import sahara.db.migration from sahara.db.sqlalchemy import api as sa from sahara.db.sqlalchemy import model_base LOG = logging.getLogger(__name__) CONF = cfg.CONF class BaseWalkMigrationTestCase(object): ALEMBIC_CONFIG = alembic_config.Config( os.path.join(os.path.dirname(sahara.db.migration.__file__), 'alembic.ini') ) ALEMBIC_CONFIG.sahara_config = CONF def _configure(self, engine): """For each type of repository we should do some of configure steps. For migrate_repo we should set under version control our database. For alembic we should configure database settings. For this goal we should use oslo_config and openstack.commom.db.sqlalchemy.session with database functionality (reset default settings and session cleanup). """ CONF.set_override('connection', str(engine.url), group='database') sa.cleanup() def _alembic_command(self, alembic_command, engine, *args, **kwargs): """Most of alembic command return data into output. We should redefine this setting for getting info. """ self.ALEMBIC_CONFIG.stdout = buf = io.StringIO() CONF.set_override('connection', str(engine.url), group='database') sa.cleanup() getattr(command, alembic_command)(*args, **kwargs) res = buf.getvalue().strip() LOG.debug('Alembic command {command} returns: {result}'.format( command=alembic_command, result=res)) sa.cleanup() return res def _get_versions(self): """Stores a list of versions. Since alembic version has a random algorithm of generation (SA-migrate has an ordered autoincrement naming) we should store a list of versions (version for upgrade) for successful testing of migrations in up mode. """ env = alembic_script.ScriptDirectory.from_config(self.ALEMBIC_CONFIG) versions = [] for rev in env.walk_revisions(): versions.append(rev.revision) versions.reverse() return versions def walk_versions(self, engine=None): # Determine latest version script from the repo, then # upgrade from 1 through to the latest, with no data # in the databases. This just checks that the schema itself # upgrades successfully. self._configure(engine) versions = self._get_versions() for ver in versions: self._migrate_up(engine, ver, with_data=True) def _get_version_from_db(self, engine): """Returns latest version from db for each type of migrate repo.""" conn = engine.connect() try: context = migration.MigrationContext.configure(conn) version = context.get_current_revision() or '-1' finally: conn.close() return version def _migrate(self, engine, version, cmd): """Base method for manipulation with migrate repo. It will upgrade or downgrade the actual database. """ self._alembic_command(cmd, engine, self.ALEMBIC_CONFIG, version) def _migrate_up(self, engine, version, with_data=False): """migrate up to a new version of the db. We allow for data insertion and post checks at every migration version with special _pre_upgrade_### and _check_### functions in the main test. """ # NOTE(sdague): try block is here because it's impossible to debug # where a failed data migration happens otherwise check_version = version try: if with_data: data = None pre_upgrade = getattr( self, "_pre_upgrade_%s" % check_version, None) if pre_upgrade: data = pre_upgrade(engine) self._migrate(engine, version, 'upgrade') self.assertEqual(version, self._get_version_from_db(engine)) if with_data: check = getattr(self, "_check_%s" % check_version, None) if check: check(engine, data) except Exception: LOG.error("Failed to migrate to version {version} on engine " "{engine}".format(version=version, engine=engine)) raise class TestModelsMigrationsSync(t_m.ModelsMigrationsSync): """Class for comparison of DB migration scripts and models. Allows to check if the DB schema obtained by applying of migration scripts is equal to the one produced from models definitions. """ ALEMBIC_CONFIG = alembic_config.Config( os.path.join(os.path.dirname(sahara.db.migration.__file__), 'alembic.ini') ) ALEMBIC_CONFIG.sahara_config = CONF def get_engine(self): return self.engine def db_sync(self, engine): CONF.set_override('connection', str(engine.url), group='database') alembic.command.upgrade(self.ALEMBIC_CONFIG, 'head') def get_metadata(self): return model_base.SaharaBase.metadata sahara-8.0.0/sahara/tests/unit/db/migration/test_db_manage_cli.py0000666000175100017510000000614613245514472025115 0ustar zuulzuul00000000000000# Copyright 2012 New Dream Network, LLC (DreamHost) # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sys import mock import testscenarios import testtools from sahara.db.migration import cli class TestCli(testtools.TestCase): func_name = '' exp_args = () exp_kwargs = {} scenarios = [ ('stamp', dict(argv=['prog', 'stamp', 'foo'], func_name='stamp', exp_args=('foo',), exp_kwargs={'sql': False})), ('stamp-sql', dict(argv=['prog', 'stamp', 'foo', '--sql'], func_name='stamp', exp_args=('foo',), exp_kwargs={'sql': True})), ('current', dict(argv=['prog', 'current'], func_name='current', exp_args=[], exp_kwargs=dict())), ('history', dict(argv=['prog', 'history'], func_name='history', exp_args=[], exp_kwargs=dict())), ('check_migration', dict(argv=['prog', 'check_migration'], func_name='branches', exp_args=[], exp_kwargs=dict())), ('sync_revision_autogenerate', dict(argv=['prog', 'revision', '--autogenerate', '-m', 'message'], func_name='revision', exp_args=(), exp_kwargs={ 'message': 'message', 'sql': False, 'autogenerate': True})), ('sync_revision_sql', dict(argv=['prog', 'revision', '--sql', '-m', 'message'], func_name='revision', exp_args=(), exp_kwargs={ 'message': 'message', 'sql': True, 'autogenerate': False})), ('upgrade-sql', dict(argv=['prog', 'upgrade', '--sql', 'head'], func_name='upgrade', exp_args=('head',), exp_kwargs={'sql': True})), ('upgrade-delta', dict(argv=['prog', 'upgrade', '--delta', '3'], func_name='upgrade', exp_args=('+3',), exp_kwargs={'sql': False})) ] def setUp(self): super(TestCli, self).setUp() do_alembic_cmd_p = mock.patch.object(cli, 'do_alembic_command') self.addCleanup(do_alembic_cmd_p.stop) self.do_alembic_cmd = do_alembic_cmd_p.start() self.addCleanup(cli.CONF.reset) def test_cli(self): with mock.patch.object(sys, 'argv', self.argv): cli.main() self.do_alembic_cmd.assert_has_calls( [mock.call( mock.ANY, self.func_name, *self.exp_args, **self.exp_kwargs)] ) def load_tests(loader, in_tests, pattern): return testscenarios.load_tests_apply_scenarios(loader, in_tests, pattern) sahara-8.0.0/sahara/tests/unit/db/migration/test_migrations.py0000666000175100017510000005333213245514472024544 0ustar zuulzuul00000000000000# Copyright 2014 OpenStack Foundation # Copyright 2014 Mirantis Inc # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. """ Tests for database migrations. For the opportunistic testing you need to set up a db named 'openstack_citest' with user 'openstack_citest' and password 'openstack_citest' on localhost. The test will then use that db and u/p combo to run the tests. For postgres on Ubuntu this can be done with the following commands: sudo -u postgres psql postgres=# create user openstack_citest with createdb login password 'openstack_citest'; postgres=# create database openstack_citest with owner openstack_citest; """ import os from oslo_db.sqlalchemy import test_base from oslo_db.sqlalchemy import utils as db_utils from sahara.tests.unit.db.migration import test_migrations_base as base class SaharaMigrationsCheckers(object): def assertColumnExists(self, engine, table, column): t = db_utils.get_table(engine, table) self.assertIn(column, t.c) def assertColumnsExist(self, engine, table, columns): for column in columns: self.assertColumnExists(engine, table, column) def assertColumnType(self, engine, table, column, column_type): t = db_utils.get_table(engine, table) column_ref_type = str(t.c[column].type) self.assertEqual(column_ref_type, column_type) def assertColumnCount(self, engine, table, columns): t = db_utils.get_table(engine, table) self.assertEqual(len(columns), len(t.columns)) def assertColumnNotExists(self, engine, table, column): t = db_utils.get_table(engine, table) self.assertNotIn(column, t.c) def assertIndexExists(self, engine, table, index): t = db_utils.get_table(engine, table) index_names = [idx.name for idx in t.indexes] self.assertIn(index, index_names) def assertIndexMembers(self, engine, table, index, members): self.assertIndexExists(engine, table, index) t = db_utils.get_table(engine, table) index_columns = None for idx in t.indexes: if idx.name == index: index_columns = idx.columns.keys() break self.assertEqual(sorted(members), sorted(index_columns)) def test_walk_versions(self): self.walk_versions(self.engine) def _pre_upgrade_001(self, engine): # Anything returned from this method will be # passed to corresponding _check_xxx method as 'data'. pass def _check_001(self, engine, data): job_binary_internal_columns = [ 'created_at', 'updated_at', 'id', 'tenant_id', 'name', 'data', 'datasize' ] self.assertColumnsExist( engine, 'job_binary_internal', job_binary_internal_columns) self.assertColumnCount( engine, 'job_binary_internal', job_binary_internal_columns) node_group_templates_columns = [ 'created_at', 'updated_at', 'id', 'name', 'description', 'tenant_id', 'flavor_id', 'image_id', 'plugin_name', 'hadoop_version', 'node_processes', 'node_configs', 'volumes_per_node', 'volumes_size', 'volume_mount_prefix', 'floating_ip_pool' ] self.assertColumnsExist( engine, 'node_group_templates', node_group_templates_columns) self.assertColumnCount( engine, 'node_group_templates', node_group_templates_columns) data_sources_columns = [ 'created_at', 'updated_at', 'id', 'tenant_id', 'name', 'description', 'type', 'url', 'credentials' ] self.assertColumnsExist( engine, 'data_sources', data_sources_columns) self.assertColumnCount( engine, 'data_sources', data_sources_columns) cluster_templates_columns = [ 'created_at', 'updated_at', 'id', 'name', 'description', 'cluster_configs', 'default_image_id', 'anti_affinity', 'tenant_id', 'neutron_management_network', 'plugin_name', 'hadoop_version' ] self.assertColumnsExist( engine, 'cluster_templates', cluster_templates_columns) self.assertColumnCount( engine, 'cluster_templates', cluster_templates_columns) job_binaries_columns = [ 'created_at', 'updated_at', 'id', 'tenant_id', 'name', 'description', 'url', 'extra' ] self.assertColumnsExist( engine, 'job_binaries', job_binaries_columns) self.assertColumnCount( engine, 'job_binaries', job_binaries_columns) jobs_columns = [ 'created_at', 'updated_at', 'id', 'tenant_id', 'name', 'description', 'type' ] self.assertColumnsExist(engine, 'jobs', jobs_columns) self.assertColumnCount(engine, 'jobs', jobs_columns) templates_relations_columns = [ 'created_at', 'updated_at', 'id', 'tenant_id', 'name', 'flavor_id', 'image_id', 'node_processes', 'node_configs', 'volumes_per_node', 'volumes_size', 'volume_mount_prefix', 'count', 'cluster_template_id', 'node_group_template_id', 'floating_ip_pool' ] self.assertColumnsExist( engine, 'templates_relations', templates_relations_columns) self.assertColumnCount( engine, 'templates_relations', templates_relations_columns) mains_association_columns = [ 'Job_id', 'JobBinary_id' ] self.assertColumnsExist( engine, 'mains_association', mains_association_columns) self.assertColumnCount( engine, 'mains_association', mains_association_columns) libs_association_columns = [ 'Job_id', 'JobBinary_id' ] self.assertColumnsExist( engine, 'libs_association', libs_association_columns) self.assertColumnCount( engine, 'libs_association', libs_association_columns) clusters_columns = [ 'created_at', 'updated_at', 'id', 'name', 'description', 'tenant_id', 'trust_id', 'is_transient', 'plugin_name', 'hadoop_version', 'cluster_configs', 'default_image_id', 'neutron_management_network', 'anti_affinity', 'management_private_key', 'management_public_key', 'user_keypair_id', 'status', 'status_description', 'info', 'extra', 'cluster_template_id' ] self.assertColumnsExist(engine, 'clusters', clusters_columns) self.assertColumnCount(engine, 'clusters', clusters_columns) node_groups_columns = [ 'created_at', 'updated_at', 'id', 'name', 'tenant_id', 'flavor_id', 'image_id', 'image_username', 'node_processes', 'node_configs', 'volumes_per_node', 'volumes_size', 'volume_mount_prefix', 'count', 'cluster_id', 'node_group_template_id', 'floating_ip_pool' ] self.assertColumnsExist(engine, 'node_groups', node_groups_columns) self.assertColumnCount(engine, 'node_groups', node_groups_columns) job_executions_columns = [ 'created_at', 'updated_at', 'id', 'tenant_id', 'job_id', 'input_id', 'output_id', 'start_time', 'end_time', 'cluster_id', 'info', 'progress', 'oozie_job_id', 'return_code', 'job_configs', 'extra' ] self.assertColumnsExist( engine, 'job_executions', job_executions_columns) self.assertColumnCount( engine, 'job_executions', job_executions_columns) instances_columns = [ 'created_at', 'updated_at', 'id', 'tenant_id', 'node_group_id', 'instance_id', 'instance_name', 'internal_ip', 'management_ip', 'volumes' ] self.assertColumnsExist(engine, 'instances', instances_columns) self.assertColumnCount(engine, 'instances', instances_columns) self._data_001(engine, data) def _data_001(self, engine, data): datasize = 512 * 1024 # 512kB data = os.urandom(datasize) t = db_utils.get_table(engine, 'job_binary_internal') engine.execute(t.insert(), data=data, id='123', name='name') new_data = engine.execute(t.select()).fetchone().data self.assertEqual(data, new_data) engine.execute(t.delete()) def _check_002(self, engine, data): # currently, 002 is just a placeholder pass def _check_003(self, engine, data): # currently, 003 is just a placeholder pass def _check_004(self, engine, data): # currently, 004 is just a placeholder pass def _check_005(self, engine, data): # currently, 005 is just a placeholder pass def _check_006(self, engine, data): # currently, 006 is just a placeholder pass def _pre_upgrade_007(self, engine): desc = 'magic' t = db_utils.get_table(engine, 'clusters') engine.execute(t.insert(), id='123', name='name', plugin_name='pname', hadoop_version='1', management_private_key='2', management_public_key='3', status_description=desc) def _check_007(self, engine, data): t = db_utils.get_table(engine, 'clusters') res = engine.execute(t.select().where(t.c.id == '123')).first() self.assertEqual('magic', res['status_description']) engine.execute(t.delete()) # check that status_description can keep 128kb. # MySQL varchar can not keep more then 64kb desc = 'a' * 128 * 1024 # 128kb t = db_utils.get_table(engine, 'clusters') engine.execute(t.insert(), id='123', name='name', plugin_name='plname', hadoop_version='hversion', management_private_key='1', management_public_key='2', status_description=desc) new_desc = engine.execute(t.select()).fetchone().status_description self.assertEqual(desc, new_desc) engine.execute(t.delete()) def _check_008(self, engine, data): self.assertColumnExists(engine, 'node_group_templates', 'security_groups') self.assertColumnExists(engine, 'node_groups', 'security_groups') self.assertColumnExists(engine, 'templates_relations', 'security_groups') def _check_009(self, engine, data): self.assertColumnExists(engine, 'clusters', 'rollback_info') def _check_010(self, engine, data): self.assertColumnExists(engine, 'node_group_templates', 'auto_security_group') self.assertColumnExists(engine, 'node_groups', 'auto_security_group') self.assertColumnExists(engine, 'templates_relations', 'auto_security_group') self.assertColumnExists(engine, 'node_groups', 'open_ports') def _check_011(self, engine, data): self.assertColumnExists(engine, 'clusters', 'sahara_info') def _check_012(self, engine, data): self.assertColumnExists(engine, 'node_group_templates', 'availability_zone') self.assertColumnExists(engine, 'node_groups', 'availability_zone') self.assertColumnExists(engine, 'templates_relations', 'availability_zone') def _check_014(self, engine, data): self.assertColumnExists(engine, 'node_group_templates', 'volume_type') self.assertColumnExists(engine, 'node_groups', 'volume_type') self.assertColumnExists(engine, 'templates_relations', 'volume_type') def _check_015(self, engine, data): provision_steps_columns = [ 'created_at', 'updated_at', 'id', 'cluster_id', 'tenant_id', 'step_name', 'step_type', 'completed', 'total', 'successful', 'started_at', 'completed_at', ] events_columns = [ 'created_at', 'updated_at', 'id', 'node_group_id', 'instance_id', 'instance_name', 'event_info', 'successful', 'step_id', ] self.assertColumnCount(engine, 'cluster_provision_steps', provision_steps_columns) self.assertColumnsExist(engine, 'cluster_provision_steps', provision_steps_columns) self.assertColumnCount(engine, 'cluster_events', events_columns) self.assertColumnsExist(engine, 'cluster_events', events_columns) def _check_016(self, engine, data): self.assertColumnExists(engine, 'node_group_templates', 'is_proxy_gateway') self.assertColumnExists(engine, 'node_groups', 'is_proxy_gateway') self.assertColumnExists(engine, 'templates_relations', 'is_proxy_gateway') def _check_017(self, engine, data): self.assertColumnNotExists(engine, 'job_executions', 'progress') def _check_018(self, engine, data): self.assertColumnExists(engine, 'node_group_templates', 'volume_local_to_instance') self.assertColumnExists(engine, 'node_groups', 'volume_local_to_instance') self.assertColumnExists(engine, 'templates_relations', 'volume_local_to_instance') def _check_019(self, engine, data): self.assertColumnExists(engine, 'node_group_templates', 'is_default') self.assertColumnExists(engine, 'cluster_templates', 'is_default') def _check_020(self, engine, data): self.assertColumnNotExists(engine, 'cluster_provision_steps', 'completed') self.assertColumnNotExists(engine, 'cluster_provision_steps', 'completed_at') self.assertColumnNotExists(engine, 'cluster_provision_steps', 'started_at') def _check_021(self, engine, data): self.assertColumnExists(engine, 'job_executions', 'data_source_urls') def _check_022(self, engine, data): columns = [ 'created_at', 'updated_at', 'id', 'job_id', 'tenant_id', 'name', 'description', 'mapping_type', 'location', 'value_type', 'required', 'order', 'default' ] self.assertColumnCount(engine, 'job_interface_arguments', columns) self.assertColumnsExist(engine, 'job_interface_arguments', columns) def _check_023(self, engine, data): self.assertColumnExists(engine, 'clusters', 'use_autoconfig') self.assertColumnExists(engine, 'cluster_templates', 'use_autoconfig') self.assertColumnExists(engine, 'node_group_templates', 'use_autoconfig') self.assertColumnExists(engine, 'node_groups', 'use_autoconfig') self.assertColumnExists(engine, 'templates_relations', 'use_autoconfig') def _check_024(self, engine, data): tables = [ 'node_group_templates', 'node_groups', 'templates_relations', 'clusters', 'cluster_templates' ] for table in tables: self.assertColumnExists(engine, table, 'shares') def _check_025(self, engine, data): self.assertColumnType(engine, 'instances', 'internal_ip', 'VARCHAR(45)') self.assertColumnType(engine, 'instances', 'management_ip', 'VARCHAR(45)') def _check_026(self, engine, data): tables = [ 'clusters', 'cluster_templates', 'node_group_templates', 'data_sources', 'job_executions', 'jobs', 'job_binary_internal', 'job_binaries', ] for table in tables: self.assertColumnExists(engine, table, 'is_public') self.assertColumnExists(engine, table, 'is_protected') def _check_027(self, engine, data): self.assertColumnNotExists(engine, 'job_executions', 'oozie_job_id') self.assertColumnExists(engine, 'job_executions', 'engine_job_id') def _check_028(self, engine, data): self.assertColumnExists(engine, 'instances', 'storage_devices_number') def _pre_upgrade_029(self, engine): t = db_utils.get_table(engine, 'node_group_templates') engine.execute(t.insert(), id='123', name='first', plugin_name='plg', hadoop_version='1', flavor_id='1', volumes_per_node=0, is_default=True, is_protected=False) engine.execute(t.insert(), id='124', name='second', plugin_name='plg', hadoop_version='1', flavor_id='1', volumes_per_node=0, is_default=False, is_protected=False) t = db_utils.get_table(engine, 'cluster_templates') engine.execute(t.insert(), id='123', name='name', plugin_name='plg', hadoop_version='1', is_default=True, is_protected=False) engine.execute(t.insert(), id='124', name='name', plugin_name='plg', hadoop_version='1', is_default=False, is_protected=False) def _check_029(self, engine, data): t = db_utils.get_table(engine, 'node_group_templates') res = engine.execute(t.select().where(t.c.id == '123')).first() self.assertTrue(res['is_protected']) res = engine.execute(t.select().where(t.c.id == '124')).first() self.assertFalse(res['is_protected']) engine.execute(t.delete()) t = db_utils.get_table(engine, 'cluster_templates') res = engine.execute(t.select().where(t.c.id == '123')).first() self.assertTrue(res['is_protected']) res = engine.execute(t.select().where(t.c.id == '124')).first() self.assertFalse(res['is_protected']) engine.execute(t.delete()) def _check_030(self, engine, data): health_check_columns = [ 'status', 'name', 'description', 'id', 'verification_id', 'created_at', 'updated_at' ] verification_columns = [ 'status', 'id', 'cluster_id', 'created_at', 'updated_at' ] self.assertColumnCount(engine, 'cluster_verifications', verification_columns) self.assertColumnsExist(engine, 'cluster_verifications', verification_columns) self.assertColumnCount(engine, 'cluster_health_checks', health_check_columns) self.assertColumnsExist(engine, 'cluster_health_checks', health_check_columns) def _check_031(self, engine, data): plugins_data_columns = [ 'name', 'id', 'tenant_id', 'version_labels', 'plugin_labels', 'updated_at', 'created_at' ] self.assertColumnCount(engine, 'plugin_data', plugins_data_columns) self.assertColumnsExist(engine, 'plugin_data', plugins_data_columns) def _check_033(self, engine, data): self.assertColumnExists(engine, 'clusters', 'anti_affinity_ratio') class TestMigrationsMySQL(SaharaMigrationsCheckers, base.BaseWalkMigrationTestCase, base.TestModelsMigrationsSync, test_base.MySQLOpportunisticTestCase): pass class TestMigrationsPostgresql(SaharaMigrationsCheckers, base.BaseWalkMigrationTestCase, base.TestModelsMigrationsSync, test_base.PostgreSQLOpportunisticTestCase): pass sahara-8.0.0/sahara/tests/unit/db/migration/__init__.py0000666000175100017510000000000013245514472023050 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/testutils.py0000666000175100017510000000410613245514472021006 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_utils import uuidutils from sahara.conductor import resource as r def create_cluster(name, tenant, plugin, version, node_groups, **kwargs): dct = {'id': uuidutils.generate_uuid(), 'name': name, 'tenant_id': tenant, 'plugin_name': plugin, 'hadoop_version': version, 'node_groups': node_groups, 'cluster_configs': {}, "sahara_info": {}, 'user_keypair_id': None, 'default_image_id': None, 'is_protected': False} dct.update(kwargs) return r.ClusterResource(dct) def make_ng_dict(name, flavor, processes, count, instances=None, volumes_size=None, node_configs=None, resource=False, **kwargs): node_configs = node_configs or {} instances = instances or [] dct = {'id': uuidutils.generate_uuid(), 'name': name, 'volumes_size': volumes_size, 'flavor_id': flavor, 'node_processes': processes, 'count': count, 'instances': instances, 'node_configs': node_configs, 'security_groups': None, 'auto_security_group': False, 'availability_zone': None, 'volumes_availability_zone': None, 'open_ports': [], 'is_proxy_gateway': False, 'volume_local_to_instance': False} dct.update(kwargs) if resource: return r.NodeGroupTemplateResource(dct) return dct def make_inst_dict(inst_id, inst_name, management_ip='1.2.3.4'): return {'instance_id': inst_id, 'instance_name': inst_name, 'management_ip': management_ip} sahara-8.0.0/sahara/tests/unit/test_exceptions.py0000666000175100017510000001422613245514472022172 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import six from sahara import exceptions as exc from sahara.tests.unit import base class TestExceptions(base.SaharaTestCase): def _validate_exc(self, exc, expected_message, *args, **kwargs): message = "" try: raise exc(*args, **kwargs) except exc as e: message = six.text_type(e) if message.find("\nError ID") != -1: message = message.split("\nError ID")[0] self.assertEqual(expected_message, message) def test_not_found(self): self._validate_exc( exc.NotFoundException, "Object 'sample' is not found", "sample") self._validate_exc( exc.NotFoundException, "Not found 'sample'", "sample", "Not found '%s'") def test_name_already_exists(self): self._validate_exc( exc.NameAlreadyExistsException, "Name already exists") self._validate_exc( exc.NameAlreadyExistsException, "message", "message") def test_invalid_credentials(self): self._validate_exc( exc.InvalidCredentials, "Invalid credentials") self._validate_exc( exc.InvalidCredentials, "message", "message") def test_invalid_reference(self): self._validate_exc( exc.InvalidReferenceException, "Invalid object reference") self._validate_exc( exc.InvalidReferenceException, "message", "message") def test_remote_command(self): exc_message = ('Error during command execution: "cmd"\n' 'Return code: 0\n' 'STDERR:\n' 'stderr\n' 'STDOUT:\n' 'stdout') self._validate_exc( exc.RemoteCommandException, exc_message, "cmd", "0", "stdout", "stderr") def test_invalid_data(self): self._validate_exc( exc.InvalidDataException, "Data is invalid") self._validate_exc( exc.InvalidDataException, "message", "message") def test_bad_job_binary_internal(self): ex_message = ("Job binary internal data must be a string of length " "greater than zero") self._validate_exc( exc.BadJobBinaryInternalException, ex_message) self._validate_exc( exc.BadJobBinaryInternalException, "message", "message") def test_bad_job_binary(self): ex_message = ("To work with JobBinary located in internal swift" " add 'user' and 'password' to extra") self._validate_exc( exc.BadJobBinaryException, ex_message) self._validate_exc( exc.BadJobBinaryException, "message", "message") def test_db_duplicate(self): ex_message = "Database object already exists" self._validate_exc( exc.DBDuplicateEntry, ex_message) self._validate_exc( exc.DBDuplicateEntry, "message", "message") def test_creation_failed(self): ex_message = "Object was not created" self._validate_exc( exc.CreationFailed, ex_message) self._validate_exc( exc.CreationFailed, "message", "message") def test_canceling_failed(self): ex_message = "Operation was not canceled" self._validate_exc( exc.CancelingFailed, ex_message) self._validate_exc( exc.CancelingFailed, "message", "message") def test_deletion_failed(self): ex_message = "Object was not deleted" self._validate_exc( exc.DeletionFailed, ex_message) self._validate_exc( exc.DeletionFailed, "message", "message") def test_missing_floating_network(self): message = ("Node Group name is missing 'floating_ip_pool' " "field") self._validate_exc( exc.MissingFloatingNetworkException, message, "name") def test_swift_client(self): self._validate_exc(exc.SwiftClientException, "message", "message") def test_data_too_big(self): exc_message = ("Size of data (size) is greater than maximum " "(maximum)") self._validate_exc( exc.DataTooBigException, "size, maximum", "size", "maximum", "%(size)s, %(maximum)s") self._validate_exc( exc.DataTooBigException, exc_message, "size", "maximum") def test_not_implemented(self): self._validate_exc( exc.NotImplementedException, "Feature 'bond' is not implemented", "bond") self._validate_exc( exc.NotImplementedException, "feature", "feature", "%s") def test_incorrect_state(self): self._validate_exc(exc.IncorrectStateError, "message", "message") def test_deprecated(self): self._validate_exc(exc.DeprecatedException, "message", "message") def test_forbidden(self): self._validate_exc(exc.Forbidden, "message", "message") self._validate_exc(exc.Forbidden, "You are not authorized " "to complete this action") def test_image_not_registered(self): self._validate_exc( exc.ImageNotRegistered, 'Image image is not registered in Sahara', 'image') def test_malformed_request_body(self): self._validate_exc( exc.MalformedRequestBody, "Malformed message body: reason", "reason") def test_update_failed(self): self._validate_exc( exc.UpdateFailedException, "Object 'value' could not be updated", "value") self._validate_exc( exc.UpdateFailedException, "value", "value", "%s") sahara-8.0.0/sahara/tests/unit/utils/0000775000175100017510000000000013245515027017526 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/utils/test_api_validator.py0000666000175100017510000002472213245514472023771 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import jsonschema from oslo_utils import uuidutils import testtools from sahara.utils import api_validator def _validate(schema, data): validator = api_validator.ApiValidator(schema) validator.validate(data) class ApiValidatorTest(testtools.TestCase): def _validate_success(self, schema, data): return _validate(schema, data) def _validate_failure(self, schema, data): self.assertRaises(jsonschema.ValidationError, _validate, schema, data) def test_validate_required(self): schema = { "type": "object", "properties": { "prop-1": { "type": "string", }, }, } self._validate_success(schema, { "prop-1": "asd", }) self._validate_success(schema, { "prop-2": "asd", }) schema["required"] = ["prop-1"] self._validate_success(schema, { "prop-1": "asd", }) self._validate_failure(schema, { "prop-2": "asd", }) def test_validate_additionalProperties(self): schema = { "type": "object", "properties": { "prop-1": { "type": "string", }, }, "required": ["prop-1"] } self._validate_success(schema, { "prop-1": "asd", }) self._validate_success(schema, { "prop-1": "asd", "prop-2": "asd", }) schema["additionalProperties"] = True self._validate_success(schema, { "prop-1": "asd", }) self._validate_success(schema, { "prop-1": "asd", "prop-2": "asd", }) schema["additionalProperties"] = False self._validate_success(schema, { "prop-1": "asd", }) self._validate_failure(schema, { "prop-1": "asd", "prop-2": "asd", }) def test_validate_string(self): schema = { "type": "string", } self._validate_success(schema, "asd") self._validate_success(schema, "") self._validate_failure(schema, 1) self._validate_failure(schema, 1.5) self._validate_failure(schema, True) def test_validate_string_with_length(self): schema = { "type": "string", "minLength": 1, "maxLength": 10, } self._validate_success(schema, "a") self._validate_success(schema, "a" * 10) self._validate_failure(schema, "") self._validate_failure(schema, "a" * 11) def test_validate_integer(self): schema = { 'type': 'integer', } self._validate_success(schema, 0) self._validate_success(schema, 1) self._validate_failure(schema, "1") self._validate_failure(schema, "a") self._validate_failure(schema, True) def test_validate_integer_w_range(self): schema = { 'type': 'integer', 'minimum': 1, 'maximum': 10, } self._validate_success(schema, 1) self._validate_success(schema, 10) self._validate_failure(schema, 0) self._validate_failure(schema, 11) def test_validate_uuid(self): schema = { "type": "string", "format": "uuid", } id = uuidutils.generate_uuid() self._validate_success(schema, id) self._validate_success(schema, id.replace("-", "")) def test_validate_valid_name(self): schema = { "type": "string", "format": "valid_name", } self._validate_success(schema, "abcd") self._validate_success(schema, "abcd123") self._validate_success(schema, "abcd-123") self._validate_success(schema, "abcd_123") self._validate_failure(schema, "_123") self._validate_success(schema, "a" * 64) self._validate_failure(schema, "") self._validate_success(schema, "hadoop-examples-2.6.0.jar") self._validate_success(schema, "hadoop-examples-2.6.0") self._validate_success(schema, "hadoop-examples-2.6.0.") self._validate_success(schema, "1") self._validate_success(schema, "1a") self._validate_success(schema, "a1") self._validate_success(schema, "A1") self._validate_success(schema, "A1B") self._validate_success(schema, "a.b") self._validate_success(schema, "a..b") self._validate_success(schema, "a._.b") self._validate_success(schema, "a_") self._validate_success(schema, "a-b-001") self._validate_failure(schema, "-aaaa-bbbb") self._validate_failure(schema, ".aaaa-bbbb") self._validate_failure(schema, None) self._validate_failure(schema, 1) self._validate_failure(schema, ["1"]) def test_validate_valid_keypair_name(self): schema = { "type": "string", "format": "valid_keypair_name", } self._validate_success(schema, "abcd") self._validate_success(schema, "abcd123") self._validate_success(schema, "abcd-123") self._validate_success(schema, "abcd_123") self._validate_success(schema, "_123") self._validate_success(schema, "a" * 64) self._validate_failure(schema, "") self._validate_failure(schema, "hadoop-examples-2.6.0.jar") self._validate_failure(schema, "hadoop-examples-2.6.0") self._validate_failure(schema, "hadoop-examples-2.6.0.") self._validate_success(schema, "1") self._validate_success(schema, "1a") self._validate_success(schema, "a1") self._validate_success(schema, "A1") self._validate_success(schema, "A1B") self._validate_failure(schema, "a.b") self._validate_failure(schema, "a..b") self._validate_failure(schema, "a._.b") self._validate_success(schema, "a_") self._validate_success(schema, "a-b-001") self._validate_success(schema, "-aaaa-bbbb") self._validate_success(schema, "-aaaa bbbb") self._validate_success(schema, " -aaaa bbbb") self._validate_failure(schema, ".aaaa-bbbb") self._validate_failure(schema, None) self._validate_failure(schema, 1) self._validate_failure(schema, ["1"]) def test_validate_valid_name_hostname(self): schema = { "type": "string", "format": "valid_name_hostname", "minLength": 1, } self._validate_success(schema, "abcd") self._validate_success(schema, "abcd123") self._validate_success(schema, "abcd-123") self._validate_failure(schema, "abcd_123") self._validate_failure(schema, "_123") self._validate_success(schema, "a" * 64) self._validate_failure(schema, "") self._validate_failure(schema, "hadoop-examples-2.6.0.jar") self._validate_failure(schema, "hadoop-examples-2.6.0") self._validate_failure(schema, "hadoop-examples-2.6.0.") self._validate_failure(schema, "1") self._validate_failure(schema, "1a") self._validate_success(schema, "a1") self._validate_success(schema, "A1") self._validate_success(schema, "A1B") self._validate_success(schema, "aB") self._validate_success(schema, "a.b") self._validate_failure(schema, "a..b") self._validate_failure(schema, "a._.b") self._validate_failure(schema, "a_") self._validate_success(schema, "a-b-001") self._validate_failure(schema, None) self._validate_failure(schema, 1) self._validate_failure(schema, ["1"]) def test_validate_hostname(self): schema = { "type": "string", "format": "hostname", } self._validate_success(schema, "abcd") self._validate_success(schema, "abcd123") self._validate_success(schema, "abcd-123") self._validate_failure(schema, "abcd_123") self._validate_failure(schema, "_123") self._validate_failure(schema, "a" * 64) self._validate_failure(schema, "") def test_validate_configs(self): schema = { "type": "object", "properties": { "configs": { "type": "configs", } }, "additionalProperties": False } self._validate_success(schema, { "configs": { "at-1": { "c-1": "c", "c-2": 1, "c-3": True, }, "at-2": { "c-4": "c", "c-5": 1, "c-6": True, }, }, }) self._validate_failure(schema, { "configs": { "at-1": { "c-1": 1.5 }, } }) self._validate_failure(schema, { "configs": { 1: { "c-1": "c" }, } }) self._validate_failure(schema, { "configs": { "at-1": { 1: "asd", }, } }) self._validate_failure(schema, { "configs": { "at-1": [ "a", "b", "c", ], } }) def test_validate_flavor(self): schema = { 'type': "flavor", } self._validate_success(schema, 0) self._validate_success(schema, 1) self._validate_success(schema, "0") self._validate_success(schema, "1") self._validate_success(schema, uuidutils.generate_uuid()) self._validate_failure(schema, True) self._validate_failure(schema, 0.1) self._validate_failure(schema, "0.1") self._validate_failure(schema, "asd") sahara-8.0.0/sahara/tests/unit/utils/test_neutron.py0000666000175100017510000000614313245514472022642 0ustar zuulzuul00000000000000# Copyright (c) 2013 Hortonworks, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.tests.unit import base from sahara.utils.openstack import neutron as neutron_client class NeutronClientTest(base.SaharaTestCase): @mock.patch("sahara.utils.openstack.keystone.token_auth") @mock.patch("neutronclient.neutron.client.Client") def test_get_router(self, patched, token_auth): patched.side_effect = _test_get_neutron_client neutron = neutron_client.NeutronClient( '33b47310-b7a8-4559-bf95-45ba669a448e', None, None) self.assertEqual('6c4d4e32-3667-4cd4-84ea-4cc1e98d18be', neutron.get_router()) def _test_get_neutron_client(api_version, *args, **kwargs): return FakeNeutronClient() class FakeNeutronClient(object): def list_routers(self): return {"routers": [{"status": "ACTIVE", "external_gateway_info": { "network_id": "61f95d3f-495e-4409-8c29-0b806283c81e"}, "name": "router1", "admin_state_up": True, "tenant_id": "903809ded3434f8d89948ee71ca9f5bb", "routes": [], "id": "6c4d4e32-3667-4cd4-84ea-4cc1e98d18be"}]} def list_ports(self, device_id=None): return {"ports": [ {"status": "ACTIVE", "name": "", "admin_state_up": True, "network_id": "33b47310-b7a8-4559-bf95-45ba669a448e", "tenant_id": "903809ded3434f8d89948ee71ca9f5bb", "binding:vif_type": "ovs", "device_owner": "compute:None", "binding:capabilities": {"port_filter": True}, "mac_address": "fa:16:3e:69:25:1c", "fixed_ips": [ {"subnet_id": "bfa9d0a1-9efb-4bff-bd2b-c103c053560f", "ip_address": "10.0.0.8"}], "id": "0f3df685-bc55-4314-9b76-835e1767b78f", "security_groups": ["f9fee2a2-bb0b-44e4-8092-93a43dc45cda"], "device_id": "c2129c18-6707-4f07-94cf-00b2fef8eea7"}, {"status": "ACTIVE", "name": "", "admin_state_up": True, "network_id": "33b47310-b7a8-4559-bf95-45ba669a448e", "tenant_id": "903809ded3434f8d89948ee71ca9f5bb", "binding:vif_type": "ovs", "device_owner": "network:router_interface", "binding:capabilities": {"port_filter": True}, "mac_address": "fa:16:3e:c5:b0:cb", "fixed_ips": [ {"subnet_id": "bfa9d0a1-9efb-4bff-bd2b-c103c053560f", "ip_address": "10.0.0.1"}], "id": "27193ae1-142a-436c-ab41-c77b1df032a1", "security_groups": [], "device_id": "6c4d4e32-3667-4cd4-84ea-4cc1e98d18be"}]} sahara-8.0.0/sahara/tests/unit/utils/test_patches.py0000666000175100017510000000444213245514472022577 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import xml.dom.minidom as xml import six import testtools from sahara.utils import patches class MinidomPatchesTest(testtools.TestCase): def setUp(self): super(MinidomPatchesTest, self).setUp() patches.patch_minidom_writexml() def _generate_n_prettify_xml(self): doc = xml.Document() pi = doc.createProcessingInstruction('xml-smth', 'type="text/smth" ' 'href="test.smth"') doc.insertBefore(pi, doc.firstChild) configuration = doc.createElement("root") doc.appendChild(configuration) for idx in six.moves.xrange(0, 5): elem = doc.createElement("element") configuration.appendChild(elem) name = doc.createElement("name") elem.appendChild(name) name_text = doc.createTextNode("key-%s" % idx) name.appendChild(name_text) value = doc.createElement("value") elem.appendChild(value) value_text = doc.createTextNode("value-%s" % idx) value.appendChild(value_text) return doc.toprettyxml(indent=" ") def test_minidom_toprettyxml(self): self.assertEqual(""" key-0 value-0 key-1 value-1 key-2 value-2 key-3 value-3 key-4 value-4 """, self._generate_n_prettify_xml()) sahara-8.0.0/sahara/tests/unit/utils/test_poll_utils.py0000666000175100017510000001237213245514472023337 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import six import testtools from sahara import context from sahara.tests.unit import base from sahara.utils import poll_utils class FakeCluster(object): def __init__(self, cluster_configs): self.cluster_configs = cluster_configs class FakeOption(object): def __init__(self, default_value, section, name): self.default_value = default_value self.name = name self.applicable_target = section class TestPollUtils(base.SaharaTestCase): def setUp(self): super(TestPollUtils, self).setUp() context.sleep = mock.Mock() @mock.patch('sahara.utils.poll_utils.LOG.debug') def test_poll_success(self, logger): poll_utils.poll(**{'get_status': lambda: True, 'kwargs': {}, 'timeout': 5, 'sleep': 3}) expected_call = mock.call( 'Operation was executed successfully in timeout 5') self.assertEqual(1, logger.call_count) self.assertEqual([expected_call], logger.call_args_list) @mock.patch('sahara.utils.poll_utils._get_consumed') def test_poll_failed_first_scenario(self, get_consumed): get_consumed.return_value = 0 message = "" try: poll_utils.poll( **{'get_status': lambda: False, 'kwargs': {}, 'timeout': 0, 'sleep': 3}) except Exception as e: message = six.text_type(e) if message.find('Error ID') != -1: message = message.split("\n")[0] expected_message = "'Operation' timed out after 0 second(s)" self.assertEqual(expected_message, message) @mock.patch('sahara.utils.poll_utils._get_consumed') def test_poll_failed_second_scenario(self, get_consumed): get_consumed.return_value = 0 message = "" try: poll_utils.poll( **{'get_status': lambda: False, 'kwargs': {}, 'timeout': 0, 'sleep': 3, 'timeout_name': "some timeout"}) except Exception as e: message = six.text_type(e) if message.find('Error ID') != -1: message = message.split("\n")[0] expected_message = ("'Operation' timed out after 0 second(s) and " "following timeout was violated: some timeout") self.assertEqual(expected_message, message) @mock.patch('sahara.utils.poll_utils.LOG.debug') @mock.patch('sahara.utils.cluster.check_cluster_exists') def test_plugin_poll_first_scenario(self, cluster_exists, logger): cluster_exists.return_value = True fake_get_status = mock.Mock() fake_get_status.side_effect = [False, False, True] fake_kwargs = {'kwargs': {'cat': 'tom', 'bond': 'james bond'}} poll_utils.plugin_option_poll( FakeCluster({}), fake_get_status, FakeOption(5, 'target', 'name'), 'fake_operation', 5, **fake_kwargs) expected_call = mock.call('Operation with name fake_operation was ' 'executed successfully in timeout 5') self.assertEqual([expected_call], logger.call_args_list) @mock.patch('sahara.utils.poll_utils.LOG.debug') @mock.patch('sahara.utils.cluster.check_cluster_exists') def test_plugin_poll_second_scenario(self, cluster_exists, logger): cluster_exists.return_value = False fake_get_status = mock.Mock() fake_get_status.side_effect = [False, False, True] fake_kwargs = {'kwargs': {'cat': 'tom', 'bond': 'james bond'}} poll_utils.plugin_option_poll( FakeCluster({'target': {'name': 7}}), fake_get_status, FakeOption(5, 'target', 'name'), 'fake_operation', 5, **fake_kwargs) expected_call = mock.call('Operation with name fake_operation was ' 'executed successfully in timeout 7') self.assertEqual([expected_call], logger.call_args_list) def test_poll_exception_strategy_first_scenario(self): fake_get_status = mock.Mock() fake_get_status.side_effect = [False, ValueError()] with testtools.ExpectedException(ValueError): poll_utils.poll(fake_get_status) def test_poll_exception_strategy_second_scenario(self): fake_get_status = mock.Mock() fake_get_status.side_effect = [False, ValueError()] poll_utils.poll(fake_get_status, exception_strategy='mark_as_true') self.assertEqual(2, fake_get_status.call_count) def test_poll_exception_strategy_third_scenario(self): fake_get_status = mock.Mock() fake_get_status.side_effect = [False, ValueError(), True] poll_utils.poll(fake_get_status, exception_strategy='mark_as_false') self.assertEqual(3, fake_get_status.call_count) sahara-8.0.0/sahara/tests/unit/utils/openstack/0000775000175100017510000000000013245515027021515 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/utils/openstack/test_heat.py0000666000175100017510000000247413245514472024063 0ustar zuulzuul00000000000000# Copyright (c) 2017 Massachusetts Open Cloud # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara.utils.openstack import heat as heat_u class HeatClientTest(testtools.TestCase): @mock.patch('heatclient.client.Client') @mock.patch('sahara.utils.openstack.base.url_for') @mock.patch('sahara.service.sessions.cache') @mock.patch('sahara.context.ctx') def test_abandon(self, ctx, cache, url_for, heat): class _FakeHeatStacks(object): def delete(self, stack): call_list.append("delete") def abandon(self, stack): call_list.append("abandon") heat.return_value.stacks = _FakeHeatStacks() call_list = [] heat_u.abandon_stack(mock.Mock()) self.assertEqual(call_list, ["delete", "abandon"]) sahara-8.0.0/sahara/tests/unit/utils/openstack/test_images.py0000666000175100017510000000701313245514472024401 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.tests.unit import base from sahara.utils.openstack import images as sahara_images class FakeImage(object): def __init__(self, name, tags, username): self.name = name self.tags = tags self.username = username class TestImages(base.SaharaTestCase): def setUp(self): super(TestImages, self).setUp() self.override_config('auth_url', 'https://127.0.0.1:8080/v3/', 'trustee') @mock.patch('sahara.utils.openstack.base.url_for', return_value='') def test_list_registered_images(self, url_for_mock): some_images = [ FakeImage('foo', ['bar', 'baz'], 'test'), FakeImage('baz', [], 'test'), FakeImage('spam', [], "")] with mock.patch( 'sahara.utils.openstack.images.SaharaImageManager.list', return_value=some_images): manager = sahara_images.image_manager() images = manager.list_registered() self.assertEqual(2, len(images)) images = manager.list_registered(name='foo') self.assertEqual(1, len(images)) self.assertEqual('foo', images[0].name) self.assertEqual('test', images[0].username) images = manager.list_registered(name='eggs') self.assertEqual(0, len(images)) images = manager.list_registered(tags=['bar']) self.assertEqual(1, len(images)) self.assertEqual('foo', images[0].name) images = manager.list_registered(tags=['bar', 'eggs']) self.assertEqual(0, len(images)) @mock.patch('sahara.utils.openstack.images.SaharaImageManager.set_meta') def test_set_image_info(self, set_meta): with mock.patch('sahara.utils.openstack.base.url_for'): manager = sahara_images.image_manager() manager.set_image_info('id', 'ubuntu') self.assertEqual( ('id', {'_sahara_username': 'ubuntu'}), set_meta.call_args[0]) manager.set_image_info('id', 'ubuntu', 'descr') self.assertEqual( ('id', {'_sahara_description': 'descr', '_sahara_username': 'ubuntu'}), set_meta.call_args[0]) @mock.patch('sahara.utils.openstack.images.SaharaImageManager.get') @mock.patch('sahara.utils.openstack.images.SaharaImageManager.delete_meta') def test_unset_image_info(self, delete_meta, get_image): manager = sahara_images.image_manager() image = mock.MagicMock() image.tags = ['fake', 'fake_2.0'] image.username = 'ubuntu' image.description = 'some description' get_image.return_value = image manager.unset_image_info('id') self.assertEqual( ('id', ['_sahara_tag_fake', '_sahara_tag_fake_2.0', '_sahara_description', '_sahara_username']), delete_meta.call_args[0]) sahara-8.0.0/sahara/tests/unit/utils/openstack/test_swift.py0000666000175100017510000000321613245514472024271 0ustar zuulzuul00000000000000# Copyright (c) 2015 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.tests.unit import base as testbase from sahara.utils.openstack import swift class SwiftClientTest(testbase.SaharaTestCase): @mock.patch('sahara.swift.swift_helper.retrieve_tenant') @mock.patch('sahara.swift.utils.retrieve_auth_url') @mock.patch('swiftclient.Connection') def test_client(self, swift_connection, retrieve_auth_url, retrieve_tenant): swift.client('testuser', '12345') self.assertEqual(1, swift_connection.call_count) @mock.patch('sahara.utils.openstack.base.url_for') @mock.patch('swiftclient.Connection') @mock.patch('sahara.utils.openstack.keystone.token_from_auth') @mock.patch('sahara.utils.openstack.keystone.auth_for_proxy') def test_client_with_trust(self, auth_for_proxy, token_from_auth, swift_connection, url_for): swift.client('testuser', '12345', 'test_trust') self.assertEqual(1, auth_for_proxy.call_count) self.assertEqual(1, token_from_auth.call_count) self.assertEqual(1, swift_connection.call_count) sahara-8.0.0/sahara/tests/unit/utils/openstack/__init__.py0000666000175100017510000000000013245514472023621 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/utils/openstack/test_base.py0000666000175100017510000002511213245514472024046 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from cinderclient import exceptions as cinder_exc from heatclient import exc as heat_exc from keystoneauth1 import exceptions as keystone_exc import mock from neutronclient.common import exceptions as neutron_exc from novaclient import exceptions as nova_exc from sahara import exceptions as sahara_exc from sahara.tests.unit import base as testbase from sahara.utils.openstack import base class TestBase(testbase.SaharaTestCase): def test_url_for_regions(self): service_catalog = ( '[{"endpoints": ' ' [{"adminURL": "http://192.168.0.5:8774/v2", ' ' "region": "RegionOne", ' ' "id": "83d12c9ad2d647ecab7cbe91adb8666b", ' ' "internalURL": "http://192.168.0.5:8774/v2", ' ' "publicURL": "http://172.18.184.5:8774/v2"}, ' ' {"adminURL": "http://192.168.0.6:8774/v2", ' ' "region": "RegionTwo", ' ' "id": "07c5a555176246c783d8f0497c98537b", ' ' "internalURL": "http://192.168.0.6:8774/v2", ' ' "publicURL": "http://172.18.184.6:8774/v2"}], ' ' "endpoints_links": [], ' ' "type": "compute", ' ' "name": "nova"}]') self.override_config("os_region_name", "RegionOne") self.assertEqual("http://192.168.0.5:8774/v2", base.url_for(service_catalog, "compute")) self.override_config("os_region_name", "RegionTwo") self.assertEqual("http://192.168.0.6:8774/v2", base.url_for(service_catalog, "compute")) class AuthUrlTest(testbase.SaharaTestCase): def test_retrieve_auth_url_api_v3(self): self.override_config('use_identity_api_v3', True) correct = "https://127.0.0.1:8080/v3" def _assert(uri): self.override_config('auth_url', uri, 'trustee') self.assertEqual(correct, base.retrieve_auth_url()) _assert("%s/" % correct) _assert("https://127.0.0.1:8080") _assert("https://127.0.0.1:8080/") _assert("https://127.0.0.1:8080/v2.0") _assert("https://127.0.0.1:8080/v2.0/") _assert("https://127.0.0.1:8080/v3") _assert("https://127.0.0.1:8080/v3/") @mock.patch("sahara.utils.openstack.base.url_for") def test_retrieve_auth_url_api_v3_without_port(self, mock_url_for): self.override_config('use_identity_api_v3', True) self.setup_context(service_catalog=True) correct = "https://127.0.0.1/v3" def _assert(uri): mock_url_for.return_value = uri self.assertEqual(correct, base.retrieve_auth_url()) _assert("%s/" % correct) _assert("https://127.0.0.1") _assert("https://127.0.0.1/") _assert("https://127.0.0.1/v2.0") _assert("https://127.0.0.1/v2.0/") _assert("https://127.0.0.1/v3") _assert("https://127.0.0.1/v3/") @mock.patch("sahara.utils.openstack.base.url_for") def test_retrieve_auth_url_api_v3_path_present(self, mock_url_for): self.override_config('use_identity_api_v3', True) self.setup_context(service_catalog=True) correct = "https://127.0.0.1/identity/v3" def _assert(uri): mock_url_for.return_value = uri self.assertEqual(correct, base.retrieve_auth_url()) _assert("%s" % correct) _assert("%s/" % correct) _assert("https://127.0.0.1/identity") _assert("https://127.0.0.1/identity/") def test_retrieve_auth_url_api_v20(self): self.override_config('use_identity_api_v3', False) correct = "https://127.0.0.1:8080/v2.0" def _assert(uri): self.override_config('auth_url', uri, 'trustee') self.assertEqual(correct, base.retrieve_auth_url()) _assert("%s/" % correct) _assert("https://127.0.0.1:8080") _assert("https://127.0.0.1:8080/") _assert("https://127.0.0.1:8080/v2.0") _assert("https://127.0.0.1:8080/v2.0/") _assert("https://127.0.0.1:8080/v3") _assert("https://127.0.0.1:8080/v3/") @mock.patch("sahara.utils.openstack.base.url_for") def test_retrieve_auth_url_api_v20_without_port(self, mock_url_for): self.override_config('use_identity_api_v3', False) self.setup_context(service_catalog=True) correct = "https://127.0.0.1/v2.0" def _assert(uri): mock_url_for.return_value = uri self.assertEqual(correct, base.retrieve_auth_url()) _assert("%s/" % correct) _assert("https://127.0.0.1") _assert("https://127.0.0.1/") _assert("https://127.0.0.1/v2.0") _assert("https://127.0.0.1/v2.0/") _assert("https://127.0.0.1/v3") _assert("https://127.0.0.1/v3/") class ExecuteWithRetryTest(testbase.SaharaTestCase): def setUp(self): super(ExecuteWithRetryTest, self).setUp() self.fake_client_call = mock.MagicMock() self.fake_client_call.__name__ = 'fake_client_call' self.override_config('retries_number', 2, 'retries') @mock.patch('sahara.context.sleep') def _check_error_without_retry(self, error, code, m_sleep): self.fake_client_call.side_effect = error(code) self.assertRaises(error, base.execute_with_retries, self.fake_client_call) self.assertEqual(1, self.fake_client_call.call_count) self.fake_client_call.reset_mock() @mock.patch('sahara.context.sleep') def _check_error_with_retry(self, error, code, m_sleep): self.fake_client_call.side_effect = error(code) self.assertRaises(sahara_exc.MaxRetriesExceeded, base.execute_with_retries, self.fake_client_call) self.assertEqual(3, self.fake_client_call.call_count) self.fake_client_call.reset_mock() def test_novaclient_calls_without_retry(self): # check that following errors will not be retried self._check_error_without_retry(nova_exc.BadRequest, 400) self._check_error_without_retry(nova_exc.Unauthorized, 401) self._check_error_without_retry(nova_exc.Forbidden, 403) self._check_error_without_retry(nova_exc.NotFound, 404) self._check_error_without_retry(nova_exc.MethodNotAllowed, 405) self._check_error_without_retry(nova_exc.Conflict, 409) self._check_error_without_retry(nova_exc.HTTPNotImplemented, 501) def test_novaclient_calls_with_retry(self): # check that following errors will be retried self._check_error_with_retry(nova_exc.OverLimit, 413) self._check_error_with_retry(nova_exc.RateLimit, 429) def test_cinderclient_calls_without_retry(self): # check that following errors will not be retried self._check_error_without_retry(cinder_exc.BadRequest, 400) self._check_error_without_retry(cinder_exc.Unauthorized, 401) self._check_error_without_retry(cinder_exc.Forbidden, 403) self._check_error_without_retry(cinder_exc.NotFound, 404) self._check_error_without_retry(nova_exc.HTTPNotImplemented, 501) def test_cinderclient_calls_with_retry(self): # check that following error will be retried self._check_error_with_retry(cinder_exc.OverLimit, 413) def test_neutronclient_calls_without_retry(self): # check that following errors will not be retried # neutron exception expects string in constructor self._check_error_without_retry(neutron_exc.BadRequest, "400") self._check_error_without_retry(neutron_exc.Forbidden, "403") self._check_error_without_retry(neutron_exc.NotFound, "404") self._check_error_without_retry(neutron_exc.Conflict, "409") def test_neutronclient_calls_with_retry(self): # check that following errors will be retried # neutron exception expects string in constructor self._check_error_with_retry(neutron_exc.InternalServerError, "500") self._check_error_with_retry(neutron_exc.ServiceUnavailable, "503") def test_heatclient_calls_without_retry(self): # check that following errors will not be retried self._check_error_without_retry(heat_exc.HTTPBadRequest, 400) self._check_error_without_retry(heat_exc.HTTPUnauthorized, 401) self._check_error_without_retry(heat_exc.HTTPForbidden, 403) self._check_error_without_retry(heat_exc.HTTPNotFound, 404) self._check_error_without_retry(heat_exc.HTTPMethodNotAllowed, 405) self._check_error_without_retry(heat_exc.HTTPConflict, 409) self._check_error_without_retry(heat_exc.HTTPUnsupported, 415) self._check_error_without_retry(heat_exc.HTTPNotImplemented, 501) def test_heatclient_calls_with_retry(self): # check that following errors will be retried self._check_error_with_retry(heat_exc.HTTPInternalServerError, 500) self._check_error_with_retry(heat_exc.HTTPBadGateway, 502) self._check_error_with_retry(heat_exc.HTTPServiceUnavailable, 503) def test_keystoneclient_calls_without_retry(self): # check that following errors will not be retried self._check_error_without_retry(keystone_exc.BadRequest, 400) self._check_error_without_retry(keystone_exc.Unauthorized, 401) self._check_error_without_retry(keystone_exc.Forbidden, 403) self._check_error_without_retry(keystone_exc.NotFound, 404) self._check_error_without_retry(keystone_exc.MethodNotAllowed, 405) self._check_error_without_retry(keystone_exc.Conflict, 409) self._check_error_without_retry(keystone_exc.UnsupportedMediaType, 415) self._check_error_without_retry(keystone_exc.HttpNotImplemented, 501) def test_keystoneclient_calls_with_retry(self): # check that following errors will be retried self._check_error_with_retry(keystone_exc.RequestTimeout, 408) self._check_error_with_retry(keystone_exc.InternalServerError, 500) self._check_error_with_retry(keystone_exc.BadGateway, 502) self._check_error_with_retry(keystone_exc.ServiceUnavailable, 503) self._check_error_with_retry(keystone_exc.GatewayTimeout, 504) sahara-8.0.0/sahara/tests/unit/utils/test_heat.py0000666000175100017510000000476613245514472022102 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara import exceptions as ex from sahara.utils.openstack import heat as h def stack(status, upd_time=None): status_reason = status status = status[status.index('_') + 1:] return mock.Mock(status=status, updated_time=upd_time, stack_status_reason=status_reason) class TestClusterStack(testtools.TestCase): @mock.patch('sahara.utils.openstack.heat.get_stack') @mock.patch("sahara.context.sleep", return_value=None) def test_wait_completion(self, sleep, client): cl = mock.Mock(stack_name='cluster') client.side_effect = [stack( 'CREATE_IN_PROGRESS'), stack('CREATE_COMPLETE')] h.wait_stack_completion(cl) self.assertEqual(2, client.call_count) client.side_effect = [ stack('UPDATE_IN_PROGRESS'), stack('UPDATE_COMPLETE')] h.wait_stack_completion(cl) self.assertEqual(4, client.call_count) client.side_effect = [ stack('DELETE_IN_PROGRESS'), stack('DELETE_COMPLETE')] h.wait_stack_completion(cl) self.assertEqual(6, client.call_count) client.side_effect = [ stack('CREATE_COMPLETE'), stack('CREATE_COMPLETE'), stack('UPDATE_IN_PROGRESS'), stack('UPDATE_COMPLETE', 1)] h.wait_stack_completion(cl, is_update=True) self.assertEqual(10, client.call_count) client.side_effect = [stack('UPDATE_COMPLETE'), stack( 'UPDATE_IN_PROGRESS'), stack('UPDATE_COMPLETE', 1)] h.wait_stack_completion(cl, is_update=True) self.assertEqual(13, client.call_count) client.side_effect = [ stack('CREATE_IN_PROGRESS'), stack('CREATE_FAILED')] with testtools.ExpectedException( ex.HeatStackException, value_re=("Heat stack failed with status " "CREATE_FAILED\nError ID: .*")): h.wait_stack_completion(cl) sahara-8.0.0/sahara/tests/unit/utils/test_types.py0000666000175100017510000000170713245514472022315 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from sahara.utils import types class TypesTestCase(testtools.TestCase): def test_is_int(self): self.assertTrue(types.is_int('1')) self.assertTrue(types.is_int('0')) self.assertTrue(types.is_int('-1')) self.assertFalse(types.is_int('1.1')) self.assertFalse(types.is_int('ab')) self.assertFalse(types.is_int('')) sahara-8.0.0/sahara/tests/unit/utils/test_ssh_remote.py0000666000175100017510000004663413245514472023331 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import shlex import mock import testtools from sahara import exceptions as ex from sahara.tests.unit import base from sahara.utils import ssh_remote class TestEscapeQuotes(testtools.TestCase): def test_escape_quotes(self): s = ssh_remote._escape_quotes('echo "\\"Hello, world!\\""') self.assertEqual(r'echo \"\\\"Hello, world!\\\"\"', s) class TestGetOsDistrib(testtools.TestCase): @mock.patch('sahara.utils.ssh_remote._execute_command', return_value=[1, 'Ubuntu']) @mock.patch('sahara.utils.ssh_remote._get_python_to_execute', return_value='python3') def test_get_os_distrib(self, python, p_execute_command): d = ssh_remote._get_os_distrib() p_execute_command.assert_called_once_with( ('printf "import platform\nprint(platform.linux_distribution(' 'full_distribution_name=0)[0])" | python3'), run_as_root=False) self.assertEqual('ubuntu', d) class TestInstallPackages(testtools.TestCase): @mock.patch('sahara.utils.ssh_remote._get_os_version') @mock.patch('sahara.utils.ssh_remote._get_os_distrib') @mock.patch('sahara.utils.ssh_remote._execute_command') def test_install_packages(self, p_execute_command, p_get_os_distrib, p_get_os_version): packages = ('git', 'emacs', 'tree') # test ubuntu p_get_os_distrib.return_value = 'ubuntu' ssh_remote._install_packages(packages) p_execute_command.assert_called_with( 'RUNLEVEL=1 apt-get install -y git emacs tree', run_as_root=True) # test centos p_get_os_distrib.return_value = 'centos' ssh_remote._install_packages(packages) p_execute_command.assert_called_with( 'yum install -y git emacs tree', run_as_root=True) # test fedora < 22 p_get_os_distrib.return_value = 'fedora' p_get_os_version.return_value = 20 ssh_remote._install_packages(packages) p_execute_command.assert_called_with( 'yum install -y git emacs tree', run_as_root=True) # test fedora >=22 p_get_os_distrib.return_value = 'fedora' p_get_os_version.return_value = 23 ssh_remote._install_packages(packages) p_execute_command.assert_called_with( 'dnf install -y git emacs tree', run_as_root=True) # test redhat p_get_os_distrib.return_value = 'redhat' ssh_remote._install_packages(packages) p_execute_command.assert_called_with( 'yum install -y git emacs tree', run_as_root=True) @mock.patch('sahara.utils.ssh_remote._get_os_distrib', return_value='windows me') def test_install_packages_bad(self, p_get_os_distrib): with testtools.ExpectedException( ex.NotImplementedException, 'Package Installation is not implemented for OS windows me.*'): ssh_remote._install_packages(('git', 'emacs', 'tree')) class TestUpdateRepository(testtools.TestCase): @mock.patch('sahara.utils.ssh_remote._get_os_version') @mock.patch('sahara.utils.ssh_remote._get_os_distrib') @mock.patch('sahara.utils.ssh_remote._execute_command') def test_update_repository(self, p_execute_command, p_get_os_distrib, p_get_os_version): # test ubuntu p_get_os_distrib.return_value = 'ubuntu' ssh_remote._update_repository() p_execute_command.assert_called_with( 'apt-get update', run_as_root=True) # test centos p_get_os_distrib.return_value = 'centos' ssh_remote._update_repository() p_execute_command.assert_called_with( 'yum clean all', run_as_root=True) # test fedora < 22 p_get_os_distrib.return_value = 'fedora' p_get_os_version.return_value = 20 ssh_remote._update_repository() p_execute_command.assert_called_with( 'yum clean all', run_as_root=True) # test fedora >=22 p_get_os_distrib.return_value = 'fedora' p_get_os_version.return_value = 23 ssh_remote._update_repository() p_execute_command.assert_called_with( 'dnf clean all', run_as_root=True) # test redhat p_get_os_distrib.return_value = 'redhat' ssh_remote._update_repository() p_execute_command.assert_called_with( 'yum clean all', run_as_root=True) @mock.patch('sahara.utils.ssh_remote._get_os_distrib', return_value='windows me') def test_update_repository_bad(self, p_get_os_distrib): with testtools.ExpectedException( ex.NotImplementedException, 'Repository Update is not implemented for OS windows me.*'): ssh_remote._update_repository() class FakeCluster(object): def __init__(self, priv_key): self.management_private_key = priv_key self.neutron_management_network = 'network1' def has_proxy_gateway(self): return False def get_proxy_gateway_node(self): return None class FakeNodeGroup(object): def __init__(self, user, priv_key): self.image_username = user self.cluster = FakeCluster(priv_key) self.floating_ip_pool = 'public' class FakeInstance(object): def __init__(self, inst_name, inst_id, management_ip, internal_ip, user, priv_key): self.instance_name = inst_name self.instance_id = inst_id self.management_ip = management_ip self.internal_ip = internal_ip self.node_group = FakeNodeGroup(user, priv_key) @property def cluster(self): return self.node_group.cluster class TestInstanceInteropHelper(base.SaharaTestCase): def setUp(self): super(TestInstanceInteropHelper, self).setUp() p_sma = mock.patch('sahara.utils.ssh_remote._acquire_remote_semaphore') p_sma.start() p_smr = mock.patch('sahara.utils.ssh_remote._release_remote_semaphore') p_smr.start() p_neutron_router = mock.patch( 'sahara.utils.openstack.neutron.NeutronClient.get_router', return_value='fakerouter') p_neutron_router.start() # During tests subprocesses are not used (because _sahara-subprocess # is not installed in /bin and Mock objects cannot be pickled). p_start_subp = mock.patch('sahara.utils.procutils.start_subprocess', return_value=42) p_start_subp.start() p_run_subp = mock.patch('sahara.utils.procutils.run_in_subprocess') self.run_in_subprocess = p_run_subp.start() p_shut_subp = mock.patch('sahara.utils.procutils.shutdown_subprocess') p_shut_subp.start() self.patchers = [p_sma, p_smr, p_neutron_router, p_start_subp, p_run_subp, p_shut_subp] def tearDown(self): for patcher in self.patchers: patcher.stop() super(TestInstanceInteropHelper, self).tearDown() def setup_context(self, username="test_user", tenant_id="tenant_1", token="test_auth_token", tenant_name='test_tenant', **kwargs): service_catalog = '''[ { "type": "network", "endpoints": [ { "region": "RegionOne", "publicURL": "http://localhost/" } ] } ]''' super(TestInstanceInteropHelper, self).setup_context( username=username, tenant_id=tenant_id, token=token, tenant_name=tenant_name, service_catalog=service_catalog, **kwargs) # When use_floating_ips=True, no proxy should be used: _connect is called # with proxy=None and ProxiedHTTPAdapter is not used. @mock.patch('sahara.utils.ssh_remote.ProxiedHTTPAdapter') def test_use_floating_ips(self, p_adapter): self.override_config('use_floating_ips', True) instance = FakeInstance('inst1', '123', '10.0.0.1', '10.0.0.1', 'user1', 'key1') remote = ssh_remote.InstanceInteropHelper(instance) # Test SSH remote.execute_command('/bin/true') self.run_in_subprocess.assert_any_call( 42, ssh_remote._connect, ('10.0.0.1', 'user1', 'key1', None, None, None)) # Test HTTP remote.get_http_client(8080) self.assertFalse(p_adapter.called) # When use_floating_ips=False and use_namespaces=True, a netcat socket # created with 'ip netns exec qrouter-...' should be used to access # instances. @mock.patch("sahara.service.trusts.get_os_admin_auth_plugin") @mock.patch("sahara.utils.openstack.keystone.token_auth") @mock.patch('sahara.utils.ssh_remote._simple_exec_func') @mock.patch('sahara.utils.ssh_remote.ProxiedHTTPAdapter') def test_use_namespaces(self, p_adapter, p_simple_exec_func, token_auth, use_os_admin): self.override_config('use_floating_ips', False) self.override_config('use_namespaces', True) instance = FakeInstance('inst2', '123', '10.0.0.2', '10.0.0.2', 'user2', 'key2') remote = ssh_remote.InstanceInteropHelper(instance) # Test SSH remote.execute_command('/bin/true') self.run_in_subprocess.assert_any_call( 42, ssh_remote._connect, ('10.0.0.2', 'user2', 'key2', 'ip netns exec qrouter-fakerouter nc 10.0.0.2 22', None, None)) # Test HTTP remote.get_http_client(8080) p_adapter.assert_called_once_with( p_simple_exec_func(), '10.0.0.2', 8080) p_simple_exec_func.assert_any_call( shlex.split('ip netns exec qrouter-fakerouter nc 10.0.0.2 8080')) # When proxy_command is set, a user-defined netcat socket should be used to # access instances. @mock.patch('sahara.utils.ssh_remote._simple_exec_func') @mock.patch('sahara.utils.ssh_remote.ProxiedHTTPAdapter') def test_proxy_command(self, p_adapter, p_simple_exec_func): self.override_config('proxy_command', 'ssh fakerelay nc {host} {port}') instance = FakeInstance('inst3', '123', '10.0.0.3', '10.0.0.3', 'user3', 'key3') remote = ssh_remote.InstanceInteropHelper(instance) # Test SSH remote.execute_command('/bin/true') self.run_in_subprocess.assert_any_call( 42, ssh_remote._connect, ('10.0.0.3', 'user3', 'key3', 'ssh fakerelay nc 10.0.0.3 22', None, None)) # Test HTTP remote.get_http_client(8080) p_adapter.assert_called_once_with( p_simple_exec_func(), '10.0.0.3', 8080) p_simple_exec_func.assert_any_call( shlex.split('ssh fakerelay nc 10.0.0.3 8080')) @mock.patch('sahara.utils.ssh_remote._simple_exec_func') @mock.patch('sahara.utils.ssh_remote.ProxiedHTTPAdapter') def test_proxy_command_internal_ip(self, p_adapter, p_simple_exec_func): self.override_config('proxy_command', 'ssh fakerelay nc {host} {port}') self.override_config('proxy_command_use_internal_ip', True) instance = FakeInstance('inst3', '123', '10.0.0.3', '10.0.0.4', 'user3', 'key3') remote = ssh_remote.InstanceInteropHelper(instance) # Test SSH remote.execute_command('/bin/true') self.run_in_subprocess.assert_any_call( 42, ssh_remote._connect, ('10.0.0.4', 'user3', 'key3', 'ssh fakerelay nc 10.0.0.4 22', None, None)) # Test HTTP remote.get_http_client(8080) p_adapter.assert_called_once_with( p_simple_exec_func(), '10.0.0.4', 8080) p_simple_exec_func.assert_any_call( shlex.split('ssh fakerelay nc 10.0.0.4 8080')) def test_proxy_command_bad(self): self.override_config('proxy_command', '{bad_kw} nc {host} {port}') instance = FakeInstance('inst4', '123', '10.0.0.4', '10.0.0.4', 'user4', 'key4') remote = ssh_remote.InstanceInteropHelper(instance) # Test SSH self.assertRaises(ex.SystemError, remote.execute_command, '/bin/true') # Test HTTP self.assertRaises(ex.SystemError, remote.get_http_client, 8080) @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') def test_get_os_distrib(self, p_run_s): instance = FakeInstance('inst4', '123', '10.0.0.4', '10.0.0.4', 'user4', 'key4') remote = ssh_remote.InstanceInteropHelper(instance) remote.get_os_distrib() p_run_s.assert_called_with(ssh_remote._get_os_distrib, None, "get_os_distrib") @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_install_packages(self, p_log_command, p_run_s): instance = FakeInstance('inst5', '123', '10.0.0.5', '10.0.0.5', 'user5', 'key5') remote = ssh_remote.InstanceInteropHelper(instance) packages = ['pkg1', 'pkg2'] remote.install_packages(packages) description = 'Installing packages "%s"' % list(packages) p_run_s.assert_called_once_with( ssh_remote._install_packages, None, description, packages) p_log_command.assert_called_with(description) @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_update_repository(self, p_log_command, p_run_s): instance = FakeInstance('inst6', '123', '10.0.0.6', '10.0.0.6', 'user6', 'key6') remote = ssh_remote.InstanceInteropHelper(instance) remote.update_repository() p_run_s.assert_called_once_with(ssh_remote._update_repository, None, 'Updating repository') p_log_command.assert_called_with('Updating repository') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_write_file_to(self, p_log_command, p_run_s): instance = FakeInstance('inst7', '123', '10.0.0.7', '10.0.0.7', 'user7', 'key7') remote = ssh_remote.InstanceInteropHelper(instance) description = 'Writing file "file"' remote.write_file_to("file", "data") p_run_s.assert_called_once_with(ssh_remote._write_file_to, None, description, "file", "data", False) p_log_command.assert_called_with(description) @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_write_files_to(self, p_log_command, p_run_s): instance = FakeInstance('inst8', '123', '10.0.0.8', '10.0.0.8', 'user8', 'key8') remote = ssh_remote.InstanceInteropHelper(instance) description = 'Writing files "[\'file\']"' remote.write_files_to({"file": "data"}) p_run_s.assert_called_once_with(ssh_remote._write_files_to, None, description, {"file": "data"}, False) p_log_command.assert_called_with(description) @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_append_to_file(self, p_log_command, p_run_s): instance = FakeInstance('inst9', '123', '10.0.0.9', '10.0.0.9', 'user9', 'key9') remote = ssh_remote.InstanceInteropHelper(instance) description = 'Appending to file "file"' remote.append_to_file("file", "data") p_run_s.assert_called_once_with(ssh_remote._append_to_file, None, description, "file", "data", False) p_log_command.assert_called_with(description) @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_append_to_files(self, p_log_command, p_run_s): instance = FakeInstance('inst10', '123', '10.0.0.10', '10.0.0.10', 'user10', 'key10') remote = ssh_remote.InstanceInteropHelper(instance) description = 'Appending to files "[\'file\']"' remote.append_to_files({"file": "data"}) p_run_s.assert_called_once_with(ssh_remote._append_to_files, None, description, {"file": "data"}, False) p_log_command.assert_called_with(description) @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_read_file_from(self, p_log_command, p_run_s): instance = FakeInstance('inst11', '123', '10.0.0.11', '10.0.0.11', 'user11', 'key11') remote = ssh_remote.InstanceInteropHelper(instance) description = 'Reading file "file"' remote.read_file_from("file") p_run_s.assert_called_once_with(ssh_remote._read_file_from, None, description, "file", False) p_log_command.assert_called_with(description) @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_replace_remote_string(self, p_log_command, p_run_s): instance = FakeInstance('inst12', '123', '10.0.0.12', '10.0.0.12', 'user12', 'key12') remote = ssh_remote.InstanceInteropHelper(instance) description = 'In file "file" replacing string "str1" with "str2"' remote.replace_remote_string("file", "str1", "str2") p_run_s.assert_called_once_with(ssh_remote._replace_remote_string, None, description, "file", "str1", "str2") p_log_command.assert_called_with(description) @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._run_s') @mock.patch('sahara.utils.ssh_remote.InstanceInteropHelper._log_command') def test_execute_on_vm_interactive(self, p_log_command, p_run_s): instance = FakeInstance('inst13', '123', '10.0.0.13', '10.0.0.13', 'user13', 'key13') remote = ssh_remote.InstanceInteropHelper(instance) description = 'Executing interactively "factor 42"' remote.execute_on_vm_interactive("factor 42", None) p_run_s.assert_called_once_with(ssh_remote._execute_on_vm_interactive, None, description, "factor 42", None) p_log_command(description) sahara-8.0.0/sahara/tests/unit/utils/test_xml_utils.py0000666000175100017510000001574713245514472023202 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import xml.dom.minidom as xml import pkg_resources as pkg import testtools from sahara.utils import patches as p from sahara.utils import xmlutils as x from sahara import version class XMLUtilsTestCase(testtools.TestCase): def setUp(self): super(XMLUtilsTestCase, self).setUp() p.patch_minidom_writexml() def test_load_xml_defaults(self): self.assertEqual( [{'name': u'name1', 'value': u'value1', 'description': 'descr1'}, {'name': u'name2', 'value': u'value2', 'description': 'descr2'}, {'name': u'name3', 'value': '', 'description': 'descr3'}, {'name': u'name4', 'value': '', 'description': 'descr4'}, {'name': u'name5', 'value': u'value5', 'description': ''}], x.load_hadoop_xml_defaults( 'tests/unit/resources/test-default.xml')) def test_parse_xml_with_name_and_value(self): file_name = 'tests/unit/resources/test-default.xml' fname = pkg.resource_filename( version.version_info.package, file_name) with open(fname, "r") as f: doc = "".join(line.strip() for line in f) self.assertEqual( [{'name': u'name1', 'value': u'value1'}, {'name': u'name2', 'value': u'value2'}, {'name': u'name3', 'value': ''}, {'name': u'name4', 'value': ''}, {'name': u'name5', 'value': u'value5'}], x.parse_hadoop_xml_with_name_and_value(doc) ) def test_adjust_description(self): self.assertEqual("", x._adjust_field("\n")) self.assertEqual("", x._adjust_field("\n ")) self.assertEqual("abcdef", x._adjust_field(u"abc\n def\n ")) self.assertEqual("abc de f", x._adjust_field("abc d\n e f\n")) self.assertEqual("abc", x._adjust_field("a\tb\t\nc")) def test_create_hadoop_xml(self): conf = x.load_hadoop_xml_defaults( 'tests/unit/resources/test-default.xml') self.assertEqual(""" name1 some_val1 name2 2 """, x.create_hadoop_xml({'name1': 'some_val1', 'name2': 2}, conf),) def test_add_property_to_configuration(self): doc = self.create_default_doc() x.add_properties_to_configuration(doc, 'test', {'': 'empty1', None: 'empty2'}) self.assertEqual(""" """, doc.toprettyxml(indent=" ")) test_conf = {'name1': 'value1', 'name2': 'value2'} x.add_properties_to_configuration(doc, 'test', test_conf) self.assertEqual(""" name1 value1 name2 value2 """, doc.toprettyxml(indent=" ")) x.add_property_to_configuration(doc, 'name3', 'value3') self.assertEqual(""" name1 value1 name2 value2 name3 value3 """, doc.toprettyxml(indent=" ")) def test_get_if_not_exist_and_add_text_element(self): doc = self.create_default_doc() x.get_and_create_if_not_exist(doc, 'test', 'tag_to_add') self.assertEqual(""" """, doc.toprettyxml(indent=" ")) x.add_text_element_to_tag(doc, 'tag_to_add', 'p', 'v') self.assertEqual("""

v

""", doc.toprettyxml(indent=" ")) def test_get_if_not_exist_and_add_to_element(self): doc = self.create_default_doc() elem = x.get_and_create_if_not_exist(doc, 'test', 'tag_to_add') x.add_text_element_to_element(doc, elem, 'p', 'v') self.assertEqual("""

v

""", doc.toprettyxml(indent=" ")) def test_add_tagged_list(self): doc = self.create_default_doc() x.add_tagged_list(doc, 'test', 'list_item', ['a', 'b']) self.assertEqual(""" a b """, doc.toprettyxml(indent=" ")) def test_add_equal_separated_dict(self): doc = self.create_default_doc() x.add_equal_separated_dict(doc, 'test', 'dict_item', {'': 'empty1', None: 'empty2'}) self.assertEqual(""" """, doc.toprettyxml(indent=" ")) x.add_equal_separated_dict(doc, 'test', 'dict_item', {'a': 'b', 'c': 'd'}) self.assertEqual(""" a=b c=d """, doc.toprettyxml(indent=" ")) def create_default_doc(self): doc = xml.Document() test = doc.createElement('test') doc.appendChild(test) return doc def _get_xml_text(self, strip): doc = x.load_xml_document("service/edp/resources/workflow.xml", strip) x.add_child(doc, 'action', 'java') x.add_text_element_to_tag(doc, 'java', 'sometag', 'somevalue') return doc.toprettyxml(indent=" ").split("\n") def test_load_xml_document_strip(self): # Get the lines from the xml docs stripped = set(self._get_xml_text(True)) unstripped = set(self._get_xml_text(False)) # Prove they're different diff = stripped.symmetric_difference(unstripped) self.assertGreater(len(diff), 0) # Prove the differences are only blank lines non_blank_diffs = [l for l in diff if not l.isspace()] self.assertEqual(0, len(non_blank_diffs)) sahara-8.0.0/sahara/tests/unit/utils/test_configs.py0000666000175100017510000000253713245514472022603 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corp. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from sahara.utils import configs class ConfigsTestCase(testtools.TestCase): def test_merge_configs(self): a = { 'HDFS': { 'param1': 'value1', 'param2': 'value2' } } b = { 'HDFS': { 'param1': 'value3', 'param3': 'value4' }, 'YARN': { 'param5': 'value5' } } res = configs.merge_configs(a, b) expected = { 'HDFS': { 'param1': 'value3', 'param2': 'value2', 'param3': 'value4' }, 'YARN': { 'param5': 'value5' } } self.assertEqual(expected, res) sahara-8.0.0/sahara/tests/unit/utils/test_proxy.py0000666000175100017510000001324413245514472022331 0ustar zuulzuul00000000000000# Copyright (c) 2014 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from oslo_utils import uuidutils from sahara.service.edp import job_utils from sahara.tests.unit import base from sahara.utils import proxy as p class TestProxyUtils(base.SaharaWithDbTestCase): def setUp(self): super(TestProxyUtils, self).setUp() @mock.patch('sahara.service.castellan.utils.store_secret') @mock.patch('sahara.context.ctx') @mock.patch('sahara.conductor.API.job_execution_update') @mock.patch('sahara.service.trusts.create_trust') @mock.patch('sahara.utils.openstack.keystone.auth_for_proxy') @mock.patch('sahara.utils.openstack.keystone.auth') @mock.patch('sahara.utils.proxy.proxy_user_create') def test_create_proxy_user_for_job_execution(self, proxy_user, trustor, trustee, trust, job_execution_update, context_current, passwd): job_execution = mock.Mock(id=1, output_id=2, job_id=3, job_configs=None) job_execution.job_configs = mock.Mock(to_dict=mock.Mock( return_value={} )) proxy_user.return_value = "proxy_user" passwd.return_value = "test_password" trustor.return_value = "test_trustor" trustee.return_value = "test_trustee" trust.return_value = "123456" ctx = mock.Mock() context_current.return_value = ctx p.create_proxy_user_for_job_execution(job_execution) update = {'job_configs': {'proxy_configs': None}} update['job_configs']['proxy_configs'] = { 'proxy_username': 'job_1', 'proxy_password': 'test_password', 'proxy_trust_id': '123456' } job_execution_update.assert_called_with(ctx, job_execution, update) @mock.patch('sahara.conductor.API.job_get') @mock.patch('sahara.conductor.API.data_source_get') @mock.patch('sahara.conductor.API.data_source_count') @mock.patch('sahara.context.ctx') def test_job_execution_requires_proxy_user(self, ctx, data_source_count, data_source, job): self.override_config('use_domain_for_proxy_users', True) job_execution = mock.Mock(input_id=1, output_id=2, job_id=3, job_configs={}) data_source.return_value = mock.Mock(url='swift://container/object') self.assertTrue(p.job_execution_requires_proxy_user(job_execution)) data_source.return_value = mock.Mock(url='') job.return_value = mock.Mock( mains=[mock.Mock(url='swift://container/object')]) self.assertTrue(p.job_execution_requires_proxy_user(job_execution)) job.return_value = mock.Mock( mains=[], libs=[mock.Mock(url='swift://container/object')]) self.assertTrue(p.job_execution_requires_proxy_user(job_execution)) job_execution.job_configs = {'args': ['swift://container/object']} job.return_value = mock.Mock( mains=[], libs=[]) self.assertTrue(p.job_execution_requires_proxy_user(job_execution)) job_execution.job_configs = { 'configs': {'key': 'swift://container/object'}} self.assertTrue(p.job_execution_requires_proxy_user(job_execution)) job_execution.job_configs = { 'params': {'key': 'swift://container/object'}} self.assertTrue(p.job_execution_requires_proxy_user(job_execution)) data_source_count.return_value = 0 job_execution.job_configs = { 'configs': {job_utils.DATA_SOURCE_SUBST_NAME: True}} job.return_value = mock.Mock( mains=[], libs=[]) self.assertFalse(p.job_execution_requires_proxy_user(job_execution)) ctx.return_value = 'dummy' data_source_count.return_value = 1 job_execution.job_configs = { 'configs': {job_utils.DATA_SOURCE_SUBST_NAME: True}, 'args': [job_utils.DATA_SOURCE_PREFIX+'somevalue']} self.assertTrue(p.job_execution_requires_proxy_user(job_execution)) data_source_count.assert_called_with('dummy', name=('somevalue',), url='swift://%') data_source_count.reset_mock() data_source_count.return_value = 1 myid = uuidutils.generate_uuid() job_execution.job_configs = { 'configs': {job_utils.DATA_SOURCE_SUBST_UUID: True}, 'args': [myid]} job.return_value = mock.Mock( mains=[], libs=[]) self.assertTrue(p.job_execution_requires_proxy_user(job_execution)) data_source_count.assert_called_with('dummy', id=(myid,), url='swift://%') sahara-8.0.0/sahara/tests/unit/utils/test_crypto.py0000666000175100017510000000247713245514472022476 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from sahara.utils import crypto as c class CryptoTest(testtools.TestCase): def test_generate_key_pair(self): kp = c.generate_key_pair() self.assertIsInstance(kp, tuple) self.assertIsNotNone(kp[0]) self.assertIsNotNone(kp[1]) self.assertIn('-----BEGIN RSA PRIVATE KEY-----', kp[0]) self.assertIn('-----END RSA PRIVATE KEY-----', kp[0]) self.assertIn('ssh-rsa ', kp[1]) self.assertIn('Generated-by-Sahara', kp[1]) def test_to_paramiko_private_key(self): pk_str = c.generate_key_pair()[0] pk = c.to_paramiko_private_key(pk_str) self.assertIsNotNone(pk) self.assertEqual(2048, pk.size) self.assertEqual('ssh-rsa', pk.get_name()) sahara-8.0.0/sahara/tests/unit/utils/test_rpc.py0000666000175100017510000000676613245514472021747 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara import main from sahara.tests.unit import base from sahara.utils import rpc as messaging class TestMessagingSetup(base.SaharaTestCase): def setUp(self): super(TestMessagingSetup, self).setUp() self.patchers = [] notifier_init_patch = mock.patch('oslo_messaging.Notifier') self.notifier_init = notifier_init_patch.start() self.patchers.append(notifier_init_patch) get_notif_transp_patch = mock.patch( 'oslo_messaging.get_notification_transport') self.get_notify_transport = get_notif_transp_patch.start() self.patchers.append(get_notif_transp_patch) get_transport_patch = mock.patch('oslo_messaging.get_rpc_transport') self.get_transport = get_transport_patch.start() self.patchers.append(get_transport_patch) set_def_patch = mock.patch('oslo_messaging.set_transport_defaults') self.set_transport_def = set_def_patch.start() self.patchers.append(set_def_patch) def tearDown(self): messaging.NOTIFICATION_TRANSPORT = None messaging.MESSAGING_TRANSPORT = None messaging.NOTIFIER = None for patch in reversed(self.patchers): patch.stop() super(TestMessagingSetup, self).tearDown() def test_set_defaults(self): messaging.setup('distributed') self.assertIsNotNone(messaging.MESSAGING_TRANSPORT) self.assertIsNotNone(messaging.NOTIFICATION_TRANSPORT) self.assertIsNotNone(messaging.NOTIFIER) expected = [ mock.call('sahara') ] self.assertEqual(expected, self.set_transport_def.call_args_list) self.assertEqual( [mock.call(main.CONF)], self.get_transport.call_args_list) self.assertEqual( [mock.call(main.CONF)], self.get_notify_transport.call_args_list) self.assertEqual(1, self.notifier_init.call_count) def test_fallback(self): self.get_notify_transport.side_effect = ValueError() messaging.setup('distributed') self.assertIsNotNone(messaging.MESSAGING_TRANSPORT) self.assertIsNotNone(messaging.NOTIFICATION_TRANSPORT) self.assertEqual( messaging.MESSAGING_TRANSPORT, messaging.NOTIFICATION_TRANSPORT) self.assertIsNotNone(messaging.NOTIFIER) expected = [ mock.call('sahara') ] self.assertEqual(expected, self.set_transport_def.call_args_list) self.assertEqual( [mock.call(main.CONF)], self.get_transport.call_args_list) self.assertEqual( [mock.call(main.CONF)], self.get_notify_transport.call_args_list) self.assertEqual(1, self.notifier_init.call_count) def test_only_notifications(self): messaging.setup('all-in-one') self.assertEqual(0, self.get_transport.call_count) self.assertEqual(1, self.get_notify_transport.call_count) sahara-8.0.0/sahara/tests/unit/utils/test_general.py0000666000175100017510000000466113245514472022570 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.tests.unit import base from sahara.utils import general class UtilsGeneralTest(base.SaharaWithDbTestCase): def setUp(self): super(UtilsGeneralTest, self).setUp() def test_find_dict(self): iterable = [ { "a": 1 }, { "a": 1, "b": 2, "c": 3 }, { "a": 2 }, { "c": 3 } ] self.assertEqual({"a": 1, "b": 2, "c": 3}, general.find_dict(iterable, a=1, b=2)) self.assertIsNone(general.find_dict(iterable, z=4)) def test_find(self): lst = [mock.Mock(a=5), mock.Mock(b=5), mock.Mock(a=7, b=7)] self.assertEqual(lst[0], general.find(lst, a=5)) self.assertEqual(lst[1], general.find(lst, b=5)) self.assertIsNone(general.find(lst, a=8)) self.assertEqual(lst[2], general.find(lst, a=7)) self.assertEqual(lst[2], general.find(lst, a=7, b=7)) def test_generate_instance_name(self): inst_name = "cluster-worker-001" self.assertEqual( inst_name, general.generate_instance_name("cluster", "worker", 1)) self.assertEqual( inst_name, general.generate_instance_name("CLUSTER", "WORKER", 1)) def test_get_by_id(self): lst = [mock.Mock(id=5), mock.Mock(id=7)] self.assertIsNone(general.get_by_id(lst, 9)) self.assertEqual(lst[0], general.get_by_id(lst, 5)) self.assertEqual(lst[1], general.get_by_id(lst, 7)) def test_natural_sort_key(self): str_test = "ABC123efg345DD" str_list = ['abc', 123, 'efg', 345, 'dd'] str_sort = general.natural_sort_key(str_test) self.assertEqual(len(str_list), len(str_sort)) self.assertEqual(str_list, str_sort) sahara-8.0.0/sahara/tests/unit/utils/test_cluster.py0000666000175100017510000001523513245514472022633 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara import conductor from sahara import context from sahara.tests.unit import base from sahara.tests.unit.conductor import test_api from sahara.utils import cluster as cluster_utils class UtilsClusterTest(base.SaharaWithDbTestCase): def setUp(self): super(UtilsClusterTest, self).setUp() self.api = conductor.API def _make_sample(self): ctx = context.ctx() cluster = self.api.cluster_create(ctx, test_api.SAMPLE_CLUSTER) return cluster def test_change_cluster_status(self): cluster = self._make_sample() cluster = cluster_utils.change_cluster_status( cluster, cluster_utils.CLUSTER_STATUS_DELETING, "desc") self.assertEqual(cluster_utils.CLUSTER_STATUS_DELETING, cluster.status) self.assertEqual("desc", cluster.status_description) cluster_utils.change_cluster_status( cluster, cluster_utils.CLUSTER_STATUS_SPAWNING) self.assertEqual(cluster_utils.CLUSTER_STATUS_DELETING, cluster.status) def test_change_status_description(self): ctx = context.ctx() cluster = self._make_sample() cluster_id = cluster.id cluster = cluster_utils.change_cluster_status_description( cluster, "desc") self.assertEqual('desc', cluster.status_description) self.api.cluster_destroy(ctx, cluster) cluster = cluster_utils.change_cluster_status_description( cluster_id, "desc") self.assertIsNone(cluster) def test_count_instances(self): cluster = self._make_sample() self.assertEqual(4, cluster_utils.count_instances(cluster)) def test_check_cluster_exists(self): ctx = context.ctx() cluster = self._make_sample() self.assertTrue(cluster_utils.check_cluster_exists(cluster)) self.api.cluster_destroy(ctx, cluster) self.assertFalse(cluster_utils.check_cluster_exists(cluster)) def test_get_instances(self): cluster = self._make_sample() ctx = context.ctx() idx = 0 ids = [] for ng in cluster.node_groups: for i in range(ng.count): idx += 1 ids.append(self.api.instance_add(ctx, ng, { 'instance_id': str(idx), 'instance_name': str(idx), })) cluster = self.api.cluster_get(ctx, cluster) instances = cluster_utils.get_instances(cluster, ids) ids = set() for inst in instances: ids.add(inst.instance_id) self.assertEqual(idx, len(ids)) for i in range(1, idx): self.assertIn(str(i), ids) instances = cluster_utils.get_instances(cluster) ids = set() for inst in instances: ids.add(inst.instance_id) self.assertEqual(idx, len(ids)) for i in range(1, idx): self.assertIn(str(i), ids) def test_clean_cluster_from_empty_ng(self): ctx = context.ctx() cluster = self._make_sample() ng = cluster.node_groups[0] ng_len = len(cluster.node_groups) self.api.node_group_update(ctx, ng, {'count': 0}) cluster = self.api.cluster_get(ctx, cluster.id) cluster_utils.clean_cluster_from_empty_ng(cluster) cluster = self.api.cluster_get(ctx, cluster.id) self.assertEqual(ng_len - 1, len(cluster.node_groups)) @mock.patch("sahara.conductor.objects.Cluster.use_designate_feature") @mock.patch("socket.gethostbyname") @mock.patch("sahara.utils.openstack.base.url_for") def test_generate_etc_hosts(self, mock_url, mock_get_host, mock_use_designate): cluster = self._make_sample() mock_use_designate.return_value = False ctx = context.ctx() idx = 0 for ng in cluster.node_groups: for i in range(ng.count): idx += 1 self.api.instance_add(ctx, ng, { 'instance_id': str(idx), 'instance_name': str(idx), 'internal_ip': str(idx), }) cluster = self.api.cluster_get(ctx, cluster) mock_url.side_effect = ["http://keystone.local:1234/v13", "http://swift.local:5678/v42"] mock_get_host.side_effect = ["1.2.3.4", "5.6.7.8"] value = cluster_utils.generate_etc_hosts(cluster) expected = ("127.0.0.1 localhost\n" "1 1.novalocal 1\n" "2 2.novalocal 2\n" "3 3.novalocal 3\n" "4 4.novalocal 4\n" "1.2.3.4 keystone.local\n" "5.6.7.8 swift.local\n") self.assertEqual(expected, value) @mock.patch("sahara.conductor.objects.Cluster.use_designate_feature") @mock.patch("socket.gethostbyname") @mock.patch("sahara.utils.openstack.base.url_for") def test_generate_etc_hosts_with_designate(self, mock_url, mock_get_host, mock_use_designate): cluster = self._make_sample() mock_use_designate.return_value = True mock_url.side_effect = ["http://keystone.local:1234/v13", "http://swift.local:5678/v42"] mock_get_host.side_effect = ["1.2.3.4", "5.6.7.8"] value = cluster_utils.generate_etc_hosts(cluster) expected = ("127.0.0.1 localhost\n" "1.2.3.4 keystone.local\n" "5.6.7.8 swift.local\n") self.assertEqual(expected, value) def test_generate_resolv_conf_diff(self): curr_resolv_conf = "search openstacklocal\nnameserver 8.8.8.8\n" self.override_config("nameservers", ['1.1.1.1']) value = cluster_utils.generate_resolv_conf_diff(curr_resolv_conf) expected = "nameserver 1.1.1.1\n" self.assertEqual(expected, value) self.override_config("nameservers", ['1.1.1.1', '8.8.8.8', '2.2.2.2']) value = cluster_utils.generate_resolv_conf_diff(curr_resolv_conf) expected = ("nameserver 1.1.1.1\n" "nameserver 2.2.2.2\n") self.assertEqual(expected, value) sahara-8.0.0/sahara/tests/unit/utils/test_edp.py0000666000175100017510000000462313245514472021721 0ustar zuulzuul00000000000000# Copyright (c) 2014 Red Hat Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara.utils import edp class EdpUtilTest(testtools.TestCase): def test_split_job_type(self): jtype, stype = edp.split_job_type(edp.JOB_TYPE_MAPREDUCE) self.assertEqual(edp.JOB_TYPE_MAPREDUCE, jtype) self.assertEqual(edp.JOB_SUBTYPE_NONE, stype) jtype, stype = edp.split_job_type(edp.JOB_TYPE_MAPREDUCE_STREAMING) self.assertEqual(edp.JOB_TYPE_MAPREDUCE, jtype) self.assertEqual(edp.JOB_SUBTYPE_STREAMING, stype) def test_compare_job_type(self): self.assertTrue(edp.compare_job_type( edp.JOB_TYPE_JAVA, edp.JOB_TYPE_JAVA, edp.JOB_TYPE_MAPREDUCE, strict=True)) self.assertFalse(edp.compare_job_type( edp.JOB_TYPE_MAPREDUCE_STREAMING, edp.JOB_TYPE_JAVA, edp.JOB_TYPE_MAPREDUCE, strict=True)) self.assertTrue(edp.compare_job_type( edp.JOB_TYPE_MAPREDUCE_STREAMING, edp.JOB_TYPE_JAVA, edp.JOB_TYPE_MAPREDUCE)) self.assertFalse(edp.compare_job_type( edp.JOB_TYPE_MAPREDUCE, edp.JOB_TYPE_JAVA, edp.JOB_TYPE_MAPREDUCE_STREAMING)) def test_get_builtin_binaries_java_available(self): job = mock.Mock(type=edp.JOB_TYPE_JAVA) configs = {edp.ADAPT_FOR_OOZIE: True} binaries = edp.get_builtin_binaries(job, configs) self.assertEqual(1, len(binaries)) binary = binaries[0] self.assertTrue(binary['name'].startswith('builtin-')) self.assertTrue(binary['name'].endswith('.jar')) self.assertIsNotNone(binary['raw']) def test_get_builtin_binaries_empty(self): for job_type in edp.JOB_TYPES_ALL: job = mock.Mock(type=job_type) self.assertEqual(0, len(edp.get_builtin_binaries(job, {}))) sahara-8.0.0/sahara/tests/unit/utils/__init__.py0000666000175100017510000000000013245514472021632 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/utils/test_cinder.py0000666000175100017510000000711113245514472022410 0ustar zuulzuul00000000000000# -*- coding: utf-8 -*- # Copyright (c) 2014 Adrien Vergé # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from keystoneauth1 import exceptions as keystone_exceptions import mock from oslo_config import cfg from sahara import main from sahara.tests.unit import base as test_base from sahara.utils.openstack import cinder CONF = cfg.CONF class TestCinder(test_base.SaharaTestCase): def setup_context(self, username="test_user", tenant_id="tenant_1", token="test_auth_token", tenant_name='test_tenant', **kwargs): self.override_config('os_region_name', 'RegionOne') # Fake service_catalog with both volumev2 # and volumev3 services available service_catalog = '''[ { "type": "volumev2", "endpoints": [ { "region": "RegionOne", "internalURL": "http://localhost/" } ] }, { "type": "volumev3", "endpoints": [ { "region": "RegionOne", "internalURL": "http://localhost/" } ] }]''' super(TestCinder, self).setup_context( username=username, tenant_id=tenant_id, token=token, tenant_name=tenant_name, service_catalog=service_catalog, **kwargs) @mock.patch('sahara.utils.openstack.keystone.auth') @mock.patch('cinderclient.v3.client.Client') @mock.patch('cinderclient.v2.client.Client') def test_get_cinder_client_api_v2(self, patched2, patched3, auth): self.override_config('api_version', 2, group='cinder') patched2.return_value = FakeCinderClient(2) patched3.return_value = FakeCinderClient(3) client = cinder.client() self.assertEqual(2, client.client.api_version) @mock.patch('sahara.utils.openstack.keystone.auth') @mock.patch('cinderclient.v3.client.Client') @mock.patch('cinderclient.v2.client.Client') def test_get_cinder_client_api_v3(self, patched2, patched3, auth): self.override_config('api_version', 3, group='cinder') patched2.return_value = FakeCinderClient(2) patched3.return_value = FakeCinderClient(3) client = cinder.client() self.assertEqual(3, client.client.api_version) def test_cinder_bad_api_version(self): self.override_config('api_version', 1, group='cinder') cinder.validate_config() # Check bad version falls back to latest supported version self.assertEqual(3, main.CONF.cinder.api_version) @mock.patch('sahara.utils.openstack.base.url_for') def test_check_cinder_exists(self, mock_url_for): mock_url_for.return_value = None self.assertTrue(cinder.check_cinder_exists()) mock_url_for.reset_mock() mock_url_for.side_effect = keystone_exceptions.EndpointNotFound() self.assertFalse(cinder.check_cinder_exists()) class FakeCinderClient(object): def __init__(self, api_version): class FakeCinderHTTPClient(object): def __init__(self, api_version): self.api_version = api_version self.client = FakeCinderHTTPClient(api_version) sahara-8.0.0/sahara/tests/unit/utils/test_hacking.py0000666000175100017510000000613013245514472022550 0ustar zuulzuul00000000000000# Copyright 2015 EasyStack Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import testtools from sahara.utils.hacking import checks class HackingTestCase(testtools.TestCase): def test_dict_constructor_with_list_copy(self): # Following checks for code-lines with pep8 error self.assertEqual(1, len(list(checks.dict_constructor_with_list_copy( " dict([(i, connect_info[i])")))) self.assertEqual(1, len(list(checks.dict_constructor_with_list_copy( " attrs = dict([(k, _from_json(v))")))) self.assertEqual(1, len(list(checks.dict_constructor_with_list_copy( " type_names = dict((value, key) for key, value in")))) self.assertEqual(1, len(list(checks.dict_constructor_with_list_copy( " dict((value, key) for key, value in")))) self.assertEqual(1, len(list(checks.dict_constructor_with_list_copy( "foo(param=dict((k, v) for k, v in bar.items()))")))) self.assertEqual(1, len(list(checks.dict_constructor_with_list_copy( " dict([[i,i] for i in range(3)])")))) self.assertEqual(1, len(list(checks.dict_constructor_with_list_copy( " dd = dict([i,i] for i in range(3))")))) # Following checks for ok code-lines self.assertEqual(0, len(list(checks.dict_constructor_with_list_copy( " dict()")))) self.assertEqual(0, len(list(checks.dict_constructor_with_list_copy( " create_kwargs = dict(snapshot=snapshot,")))) self.assertEqual(0, len(list(checks.dict_constructor_with_list_copy( " self._render_dict(xml, data_el, data.__dict__)")))) def test_use_jsonutils(self): self.assertEqual(0, len(list(checks.use_jsonutils( "import json # noqa", "path")))) self.assertEqual(0, len(list(checks.use_jsonutils( "from oslo_serialization import jsonutils as json", "path")))) self.assertEqual(0, len(list(checks.use_jsonutils( "import jsonschema", "path")))) self.assertEqual(1, len(list(checks.use_jsonutils( "import json", "path")))) self.assertEqual(1, len(list(checks.use_jsonutils( "import json as jsonutils", "path")))) def test_no_mutable_default_args(self): self.assertEqual(0, len(list(checks.no_mutable_default_args( "def foo (bar):")))) self.assertEqual(1, len(list(checks.no_mutable_default_args( "def foo (bar=[]):")))) self.assertEqual(1, len(list(checks.no_mutable_default_args( "def foo (bar={}):")))) sahara-8.0.0/sahara/tests/unit/utils/test_resources.py0000666000175100017510000000552213245514472023162 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.tests.unit import base from sahara.utils import resources class SimpleResourceTestCase(base.SaharaTestCase): def setUp(self): super(SimpleResourceTestCase, self).setUp() self.test_name = "test_res" self.test_info_0 = {"a": "a"} self.test_info_1 = {"b": "b"} def test_resource_init_attrs(self): r = resources.Resource(_name=self.test_name, _info=self.test_info_0) r.b = "b" self.assertEqual("a", r.a) self.assertEqual("b", r.__getattr__("b")) self.assertIn("b", r.__dict__) self.assertEqual(self.test_info_0, r._info) self.assertEqual(self.test_name, r._name) self.assertEqual(self.test_name, r.__resource_name__) def test_resource_to_dict(self): r = resources.Resource(_name=self.test_name, _info=self.test_info_0) self.assertEqual(self.test_info_0, r.to_dict()) self.assertEqual({self.test_name: self.test_info_0}, r.wrapped_dict) def test_resource_eq(self): r0 = resources.Resource(_name=self.test_name, _info=self.test_info_0) r1 = resources.Resource(_name=self.test_name, _info=self.test_info_1) self.assertNotEqual(r0, r1) def test_as_resource(self): r = resources.Resource(_name=self.test_name, _info=self.test_info_0) self.assertEqual(r, r.as_resource()) def test_repr(self): r = resources.Resource(_name=self.test_name, _info=self.test_info_0) dict_repr = self.test_info_0.__repr__() self.assertEqual("" % dict_repr, r.__repr__()) class InheritedBaseResourceTestCase(base.SaharaTestCase): def test_to_dict_no_filters(self): class A(resources.BaseResource): __filter_cols__ = [] test_a = A() test_a.some_attr = "some_value" a_dict = test_a.to_dict() self.assertEqual({"some_attr": "some_value"}, a_dict) def test_to_dict_with_filters_and_sa(self): class A(resources.BaseResource): __filter_cols__ = ["filtered"] test_a = A() test_a.some_attr = "some_value" test_a.filtered = "something_hidden" test_a._sa_instance_state = "some_sqlalchemy_magic" a_dict = test_a.to_dict() self.assertEqual({"some_attr": "some_value"}, a_dict) sahara-8.0.0/sahara/tests/unit/utils/test_cluster_progress_ops.py0000666000175100017510000002101613245514472025432 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from oslo_utils import uuidutils from sahara import conductor from sahara import context from sahara.tests.unit import base from sahara.tests.unit.conductor import test_api from sahara.utils import cluster_progress_ops as cpo class FakeInstance(object): def __init__(self): self.id = uuidutils.generate_uuid() self.name = uuidutils.generate_uuid() self.cluster_id = uuidutils.generate_uuid() self.node_group_id = uuidutils.generate_uuid() self.instance_id = uuidutils.generate_uuid() self.instance_name = uuidutils.generate_uuid() class ClusterProgressOpsTest(base.SaharaWithDbTestCase): def setUp(self): super(ClusterProgressOpsTest, self).setUp() self.api = conductor.API def _make_sample(self): ctx = context.ctx() cluster = self.api.cluster_create(ctx, test_api.SAMPLE_CLUSTER) return ctx, cluster def test_update_provisioning_steps(self): ctx, cluster = self._make_sample() step_id1 = self.api.cluster_provision_step_add(ctx, cluster.id, { "step_name": "some_name1", "total": 2, }) self.api.cluster_event_add(ctx, step_id1, { "event_info": "INFO", "successful": True }) self.api.cluster_provision_progress_update(ctx, cluster.id) # check that we have correct provision step result_cluster = self.api.cluster_get(ctx, cluster.id) result_step = result_cluster.provision_progress[0] self.assertIsNone(result_step.successful) # check updating in case of successful provision step self.api.cluster_event_add(ctx, step_id1, { "event_info": "INFO", "successful": True }) self.api.cluster_provision_progress_update(ctx, cluster.id) result_cluster = self.api.cluster_get(ctx, cluster.id) result_step = result_cluster.provision_progress[0] self.assertTrue(result_step.successful) # check updating in case of failed provision step step_id2 = self.api.cluster_provision_step_add(ctx, cluster.id, { "step_name": "some_name1", "total": 2, }) self.api.cluster_event_add(ctx, step_id2, { "event_info": "INFO", "successful": False, }) self.api.cluster_provision_progress_update(ctx, cluster.id) result_cluster = self.api.cluster_get(ctx, cluster.id) for step in result_cluster.provision_progress: if step.id == step_id2: self.assertFalse(step.successful) # check that it's possible to add provision step after failed step step_id3 = cpo.add_provisioning_step(cluster.id, "some_name", 2) self.assertEqual( step_id3, cpo.get_current_provisioning_step(cluster.id)) def test_get_cluster_events(self): ctx, cluster = self._make_sample() step_id1 = self.api.cluster_provision_step_add(ctx, cluster.id, { 'step_name': "some_name1", 'total': 3, }) step_id2 = self.api.cluster_provision_step_add(ctx, cluster.id, { 'step_name': "some_name", 'total': 2, }) self.api.cluster_event_add(ctx, step_id1, { "event_info": "INFO", 'successful': True, }) self.api.cluster_event_add(ctx, step_id2, { "event_info": "INFO", 'successful': True, }) cluster = self.api.cluster_get(context.ctx(), cluster.id, True) for step in cluster.provision_progress: self.assertEqual(1, len(step.events)) def _make_checks(self, instance_info, sleep=True): ctx = context.ctx() if sleep: context.sleep(2) current_instance_info = ctx.current_instance_info self.assertEqual(instance_info, current_instance_info) def test_instance_context_manager(self): fake_instances = [FakeInstance() for _ in range(50)] # check that InstanceContextManager works fine sequentially for instance in fake_instances: info = context.InstanceInfo( None, instance.id, instance.name, None) with context.InstanceInfoManager(info): self._make_checks(info, sleep=False) # check that InstanceContextManager works fine in parallel with context.ThreadGroup() as tg: for instance in fake_instances: info = context.InstanceInfo( None, instance.id, instance.name, None) with context.InstanceInfoManager(info): tg.spawn("make_checks", self._make_checks, info) @cpo.event_wrapper(True) def _do_nothing(self): pass @mock.patch('sahara.utils.cluster_progress_ops._find_in_args') @mock.patch('sahara.utils.cluster.check_cluster_exists') def test_event_wrapper(self, p_check_cluster_exists, p_find): self.override_config("disable_event_log", True) self._do_nothing() self.assertEqual(0, p_find.call_count) self.override_config("disable_event_log", False) p_find.return_value = FakeInstance() p_check_cluster_exists.return_value = False self._do_nothing() self.assertEqual(1, p_find.call_count) self.assertEqual(1, p_check_cluster_exists.call_count) def test_cluster_get_with_events(self): ctx, cluster = self._make_sample() step_id = cpo.add_provisioning_step(cluster.id, "Some name", 3) self.api.cluster_event_add(ctx, step_id, { 'event_info': "INFO", 'successful': True}) cluster = self.api.cluster_get(ctx, cluster.id, True) steps = cluster.provision_progress step = steps[0] self.assertEqual("Some name", step.step_name) self.assertEqual(3, step.total) self.assertEqual("INFO", step.events[0].event_info) @mock.patch('sahara.context.ctx') @mock.patch( 'sahara.utils.cluster_progress_ops.get_current_provisioning_step', return_value='step_id') @mock.patch('sahara.utils.cluster_progress_ops.conductor') def test_add_successful_event(self, conductor, get_step, ctx): instance = FakeInstance() self.override_config("disable_event_log", True) cpo.add_successful_event(instance) self.assertEqual(0, conductor.cluster_event_add.call_count) self.override_config("disable_event_log", False) cpo.add_successful_event(instance) self.assertEqual(1, conductor.cluster_event_add.call_count) args, kwargs = conductor.cluster_event_add.call_args self.assertEqual('step_id', args[1]) req_dict = { 'successful': True, 'node_group_id': instance.node_group_id, 'instance_id': instance.instance_id, 'instance_name': instance.instance_name, 'event_info': None, } self.assertEqual(req_dict, args[2]) @mock.patch('sahara.context.ctx') @mock.patch( 'sahara.utils.cluster_progress_ops.get_current_provisioning_step', return_value='step_id') @mock.patch('sahara.utils.cluster_progress_ops.conductor') def test_add_fail_event(self, conductor, get_step, ctx): instance = FakeInstance() self.override_config("disable_event_log", True) cpo.add_fail_event(instance, Exception('oops')) self.assertEqual(0, conductor.cluster_event_add.call_count) self.override_config("disable_event_log", False) cpo.add_fail_event(instance, Exception('oops')) self.assertEqual(1, conductor.cluster_event_add.call_count) args, kwargs = conductor.cluster_event_add.call_args self.assertEqual('step_id', args[1]) req_dict = { 'successful': False, 'node_group_id': instance.node_group_id, 'instance_id': instance.instance_id, 'instance_name': instance.instance_name, 'event_info': 'oops', } self.assertEqual(req_dict, args[2]) sahara-8.0.0/sahara/tests/unit/utils/notification/0000775000175100017510000000000013245515027022214 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/utils/notification/test_sender.py0000666000175100017510000000311113245514472025106 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara import context from sahara.tests.unit import base from sahara.utils.notification import sender class NotificationTest(base.SaharaTestCase): @mock.patch('sahara.utils.rpc.get_notifier') def test_update_cluster(self, mock_notify): class FakeNotifier(object): def info(self, *args): self.call = args notifier = FakeNotifier() mock_notify.return_value = notifier ctx = context.ctx() sender.status_notify('someId', 'someName', 'someStatus', "update") self.expected_args = (ctx, 'sahara.cluster.%s' % 'update', {'cluster_id': 'someId', 'cluster_name': 'someName', 'cluster_status': 'someStatus', 'project_id': ctx.tenant_id, 'user_id': ctx.user_id}) self.assertEqual(self.expected_args, notifier.call) sahara-8.0.0/sahara/tests/unit/utils/notification/__init__.py0000666000175100017510000000000013245514472024320 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/utils/test_api.py0000666000175100017510000000634313245514472021723 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import mock import testtools from sahara.utils import api from sahara.utils import types from sahara.utils import wsgi class APIUtilsTest(testtools.TestCase): class FakeCluster(object): def to_dict(self): return {"id": 42, "name": "myFirstCluster"} page = types.Page([FakeCluster()]) response = {"clusters": [ { "id": 42, "name": "myFirstCluster" } ] } @mock.patch('flask.request') @mock.patch('flask.Response') def test_render_pagination(self, flask, request): serializer = wsgi.JSONDictSerializer() request.status_code = 200 api.render(self.page, 'application/json', 200, name='clusters') body = serializer.serialize(self.response) flask.assert_called_with( response=body, status=200, mimetype='application/json') self.page.prev, self.page.next = 35, 49 api.render(self.page, 'application/json', 200, name='clusters') paginate_response = copy.copy(self.response) paginate_response["markers"] = \ {"prev": 35, "next": 49} body = serializer.serialize(paginate_response) flask.assert_called_with( response=body, status=200, mimetype='application/json') self.page.prev, self.page.next = 7, None api.render(self.page, 'application/json', 200, name='clusters') paginate_response = copy.copy(self.response) paginate_response["markers"] = {"prev": 7, "next": None} body = serializer.serialize(paginate_response) flask.assert_called_with( response=body, status=200, mimetype='application/json') self.page.prev, self.page.next = None, 14 api.render(self.page, 'application/json', 200, name='clusters') paginate_response = copy.copy(self.response) paginate_response["markers"] = {"prev": None, "next": 14} body = serializer.serialize(paginate_response) flask.assert_called_with( response=body, status=200, mimetype='application/json') self.page.prev, self.page.next = None, 11 api.render(self.page, 'application/json', 200, name='clusters') paginate_response = copy.copy(self.response) paginate_response["markers"] = \ {"prev": None, "next": 11} body = serializer.serialize(paginate_response) flask.assert_called_with( response=body, status=200, mimetype='application/json') self.page.prev, self.page.next = None, 11 api.render(self.page, 'application/json', 200, name='clusters') sahara-8.0.0/sahara/tests/unit/plugins/0000775000175100017510000000000013245515027020047 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/test_base_plugins_support.py0000666000175100017510000000203213245514472025731 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from sahara.plugins import base as pb class BasePluginsSupportTest(testtools.TestCase): def setUp(self): super(BasePluginsSupportTest, self).setUp() pb.setup_plugins() def test_plugins_loaded(self): plugins = [p.name for p in pb.PLUGINS.get_plugins()] self.assertIn('vanilla', plugins) self.assertIn('spark', plugins) self.assertIn('cdh', plugins) self.assertIn('ambari', plugins) sahara-8.0.0/sahara/tests/unit/plugins/ambari/0000775000175100017510000000000013245515027021302 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_configs.py0000666000175100017510000001505513245514472024356 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import collections import mock import six from sahara.plugins.ambari import configs from sahara.tests.unit import base class AmbariConfigsTestCase(base.SaharaTestCase): def setUp(self): super(AmbariConfigsTestCase, self).setUp() configs.load_configs("2.3") self.ng = mock.Mock() self.ng.node_configs = {} self.ng.cluster = mock.Mock() self.ng.cluster.hadoop_version = "2.3" self.instance = mock.Mock() self.instance.node_group = self.ng self.instance.storage_paths = mock.Mock() self.instance.storage_paths.return_value = ["/data1", "/data2"] def assertConfigEqual(self, expected, actual): self.assertEqual(len(expected), len(actual)) cnt_ex = collections.Counter() cnt_act = collections.Counter() for i, ex in enumerate(expected): for j, act in enumerate(actual): if ex == act: cnt_ex[i] += 1 cnt_act[j] += 1 self.assertEqual(len(expected), len(cnt_ex)) self.assertEqual(len(actual), len(cnt_act)) def test_get_service_to_configs_map(self): self.assertIsNone(configs.SERVICES_TO_CONFIGS_MAP) configs_map = configs.get_service_to_configs_map() configs_expected = { 'ZooKeeper': ['zoo.cfg', 'zookeeper-env'], 'Knox': ['knox-env', 'ranger-knox-plugin-properties', 'gateway-site'], 'YARN': ['yarn-site', 'mapred-env', 'yarn-env', 'capacity-scheduler', 'mapred-site'], 'general': ['cluster-env'], 'Flume': ['flume-env'], 'Ambari': ['ams-hbase-policy', 'ams-site', 'ams-env', 'ams-hbase-site', 'ams-hbase-env', 'ams-hbase-security-site'], 'HDFS': ['core-site', 'ranger-hdfs-plugin-properties', 'hadoop-policy', 'hdfs-site', 'hadoop-env'], 'Ranger': ['ranger-env', 'admin-properties', 'usersync-properties', 'ranger-site'], 'Spark': ['spark-defaults', 'spark-env'], 'Hive': ['hive-env', 'hive-site', 'hiveserver2-site', 'ranger-hive-plugin-properties'], 'Storm': ['ranger-storm-plugin-properties', 'storm-site', 'storm-env'], 'Oozie': ['oozie-env', 'oozie-site', 'tez-site'], 'HBase': ['ranger-hbase-plugin-properties', 'hbase-env', 'hbase-site', 'hbase-policy'], 'Sqoop': ['sqoop-env'], 'Kafka': ['kafka-broker', 'kafka-env'], 'Falcon': ['falcon-startup.properties', 'falcon-runtime.properties', 'falcon-env'] } for (key, item) in six.iteritems(configs_map): item.sort() for (key, item) in six.iteritems(configs_expected): item.sort() self.assertEqual(configs_map, configs_expected) self.assertIsNotNone(configs.SERVICES_TO_CONFIGS_MAP) def test_get_instance_params_default(self): instance_configs = configs.get_instance_params(self.instance) expected = [ { "hdfs-site": { "dfs.datanode.data.dir": "/data1/hdfs/data,/data2/hdfs/data", "dfs.journalnode.edits.dir": "/data1/hdfs/journalnode,/data2/hdfs/journalnode", "dfs.namenode.checkpoint.dir": "/data1/hdfs/namesecondary,/data2/hdfs/namesecondary", "dfs.namenode.name.dir": "/data1/hdfs/namenode,/data2/hdfs/namenode" } }, { "yarn-site": { "yarn.nodemanager.local-dirs": "/data1/yarn/local,/data2/yarn/local", "yarn.nodemanager.log-dirs": "/data1/yarn/log,/data2/yarn/log", "yarn.timeline-service.leveldb-timeline-store.path": "/data1/yarn/timeline,/data2/yarn/timeline" } }, { "oozie-site": { "oozie.service.AuthorizationService.security.enabled": "false" } } ] self.assertConfigEqual(expected, instance_configs) def test_get_instance_params(self): self.ng.node_configs = { "YARN": { "mapreduce.map.java.opts": "-Dk=v", "yarn.scheduler.minimum-allocation-mb": "256" } } instance_configs = configs.get_instance_params(self.instance) expected = [ { "hdfs-site": { "dfs.datanode.data.dir": "/data1/hdfs/data,/data2/hdfs/data", "dfs.journalnode.edits.dir": "/data1/hdfs/journalnode,/data2/hdfs/journalnode", "dfs.namenode.checkpoint.dir": "/data1/hdfs/namesecondary,/data2/hdfs/namesecondary", "dfs.namenode.name.dir": "/data1/hdfs/namenode,/data2/hdfs/namenode" } }, { "mapred-site": { "mapreduce.map.java.opts": "-Dk=v" } }, { "yarn-site": { "yarn.nodemanager.local-dirs": "/data1/yarn/local,/data2/yarn/local", "yarn.nodemanager.log-dirs": "/data1/yarn/log,/data2/yarn/log", "yarn.scheduler.minimum-allocation-mb": "256", "yarn.timeline-service.leveldb-timeline-store.path": "/data1/yarn/timeline,/data2/yarn/timeline" } }, { "oozie-site": { "oozie.service.AuthorizationService.security.enabled": "false" } } ] self.assertConfigEqual(expected, instance_configs) sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_validation.py0000666000175100017510000000464113245514472025057 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.ambari import common as p_common from sahara.plugins.ambari import plugin from sahara.plugins import exceptions from sahara.tests.unit import base def make_cluster(processes_map): m = mock.Mock() ngs = [] for count, processes in processes_map.items(): ng = mock.Mock() ng.count = count ng.node_processes = processes ngs.append(ng) m.node_groups = ngs return m class AmbariValidationTestCase(base.SaharaTestCase): def setUp(self): super(AmbariValidationTestCase, self).setUp() self.plugin = plugin.AmbariPluginProvider() def test_cluster_with_ambari(self): cluster = make_cluster({1: [p_common.AMBARI_SERVER, p_common.ZOOKEEPER_SERVER, p_common.NAMENODE, p_common.DATANODE, p_common.RESOURCEMANAGER, p_common.NODEMANAGER, p_common.HISTORYSERVER, p_common.APP_TIMELINE_SERVER, p_common.SECONDARY_NAMENODE]}) cluster.cluster_configs = {"general": {}} with mock.patch("sahara.plugins.ambari.validation.conductor") as p: p.cluster_get = mock.Mock() p.cluster_get.return_value = cluster self.assertIsNone(self.plugin.validate(cluster)) def test_cluster_without_ambari(self): cluster = make_cluster({1: ["spam"]}) with mock.patch("sahara.plugins.ambari.validation.conductor") as p: p.cluster_get = mock.Mock() p.cluster_get.return_value = cluster self.assertRaises(exceptions.InvalidComponentCountException, self.plugin.validate, cluster) sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_ha_helper.py0000666000175100017510000003007113245514472024650 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import mock from sahara.plugins.ambari import ha_helper as ha from sahara.tests.unit import base class HAHelperTestCase(base.SaharaTestCase): def setUp(self): super(HAHelperTestCase, self).setUp() self.cluster = mock.MagicMock() self.cluster.name = "clusterName" for i in range(1, 4): instance = mock.MagicMock() instance.fqdn.return_value = "in{}".format(i) instance.instance_name = "in{}name".format(i) setattr(self, "in{}".format(i), instance) self.bp = { "host_groups": [ { "components": [ {"name": "NAMENODE"} ] } ], "configurations": [ {"hdfs-site": {}}, {"yarn-site": {}}, {"core-site": {}}, {"hadoop-env": {}}, {"zoo.cfg": {}} ] } @mock.patch("sahara.plugins.ambari.ha_helper._set_high_zk_limits") @mock.patch("sahara.plugins.ambari.ha_helper._set_default_fs") def test_update_bp_ha_common(self, mock__set_default_fs, mock__set_high_zk_limits): ha.update_bp_ha_common(self.cluster, copy.deepcopy(self.bp)) self.assertTrue(mock__set_default_fs.called) self.assertTrue(mock__set_high_zk_limits.called) @mock.patch("sahara.plugins.ambari.ha_helper._configure_hdfs_site") @mock.patch("sahara.plugins.ambari.ha_helper._set_zk_quorum") @mock.patch("sahara.plugins.ambari.ha_helper._add_zkfc_to_namenodes") def test_update_bp_for_namenode_ha(self, mock__add_zkfc_to_namenodes, mock__set_zk_quorum, mock__configure_hdfs_site): ha.update_bp_for_namenode_ha(self.cluster, copy.deepcopy(self.bp)) self.assertTrue(mock__add_zkfc_to_namenodes.called) self.assertTrue(mock__set_zk_quorum.called) self.assertTrue(mock__configure_hdfs_site.called) @mock.patch("sahara.plugins.ambari.ha_helper._set_default_fs") @mock.patch("sahara.plugins.ambari.ha_helper._set_zk_quorum") @mock.patch("sahara.plugins.ambari.ha_helper._configure_yarn_site") def test_update_bp_for_resourcemanager_ha(self, mock__configure_yarn_site, mock__set_zk_quorum, mock__set_default_fs): ha.update_bp_for_resourcemanager_ha(self.cluster, copy.deepcopy(self.bp)) self.assertTrue(mock__configure_yarn_site.called) self.assertTrue(mock__set_zk_quorum.called) self.assertTrue(mock__set_default_fs.called) @mock.patch("sahara.plugins.ambari.ha_helper._confgure_hbase_site") def test_update_bp_for_hbase_ha(self, mock__confgure_hbase_site): ha.update_bp_for_hbase_ha(self.cluster, copy.deepcopy(self.bp)) self.assertTrue(mock__confgure_hbase_site.called) def test__add_zkfc_to_namenodes(self): bp = ha._add_zkfc_to_namenodes(copy.deepcopy(self.bp)) self.assertIn({"name": "ZKFC"}, bp["host_groups"][0]["components"]) @mock.patch("sahara.plugins.utils.get_instances") def test__set_default_fs(self, mock_get_instances): bp = ha._set_default_fs(self.cluster, copy.deepcopy(self.bp), ha.p_common.NAMENODE_HA) self.assertEqual("hdfs://hdfs-ha", ha._find_core_site(bp)["fs.defaultFS"]) mock_get_instances.return_value = [self.in1] bp = ha._set_default_fs(self.cluster, copy.deepcopy(self.bp), ha.p_common.RESOURCEMANAGER_HA) self.assertEqual("hdfs://{}:8020".format(self.in1.fqdn()), ha._find_core_site(bp)["fs.defaultFS"]) @mock.patch("sahara.plugins.utils.get_instances") def test__set_zk_quorum(self, mock_get_instances): mock_get_instances.return_value = [self.in1, self.in2, self.in3] bp = ha._set_zk_quorum(self.cluster, copy.deepcopy(self.bp), ha.CORE_SITE) self.assertEqual( "{}:2181,{}:2181,{}:2181".format( self.in1.fqdn(), self.in2.fqdn(), self.in3.fqdn()), ha._find_core_site(bp)['ha.zookeeper.quorum']) bp = ha._set_zk_quorum(self.cluster, copy.deepcopy(self.bp), ha.YARN_SITE) self.assertEqual( "{}:2181,{}:2181,{}:2181".format( self.in1.fqdn(), self.in2.fqdn(), self.in3.fqdn()), ha._find_yarn_site(bp)['hadoop.registry.zk.quorum']) def test__set_high_zk_limits(self): bp = ha._set_high_zk_limits(copy.deepcopy(self.bp)) self.assertEqual("10000", ha._find_zoo_cfg(bp)["tickTime"]) @mock.patch("sahara.plugins.utils.get_instances") def test__set_primary_and_standby_namenode(self, mock_get_instances): mock_get_instances.return_value = [self.in1, self.in2] bp = ha._set_primary_and_standby_namenode(self.cluster, copy.deepcopy(self.bp)) self.assertEqual( self.in1.fqdn(), ha._find_hadoop_env(bp)['dfs_ha_initial_namenode_active']) self.assertEqual( self.in2.fqdn(), ha._find_hadoop_env(bp)['dfs_ha_initial_namenode_standby']) @mock.patch("sahara.plugins.utils.get_instances") def test__configure_hdfs_site(self, mock_get_instances): mock_get_instances.return_value = [self.in1, self.in2] bp = ha._configure_hdfs_site(self.cluster, copy.deepcopy(self.bp)) j_nodes = ";".join( ["{}:8485".format(i.fqdn()) for i in mock_get_instances()]) nn_id_concat = ",".join( [i.instance_name for i in mock_get_instances()]) result = { "hdfs-site": { "dfs.client.failover.proxy.provider.hdfs-ha": "org.apache.hadoop.hdfs.server.namenode.ha." "ConfiguredFailoverProxyProvider", "dfs.ha.automatic-failover.enabled": "true", "dfs.ha.fencing.methods": "shell(/bin/true)", "dfs.nameservices": "hdfs-ha", "dfs.namenode.shared.edits.dir": "qjournal://{}/hdfs-ha".format(j_nodes), "dfs.ha.namenodes.hdfs-ha": nn_id_concat, "dfs.namenode.http-address": "{}:50070".format( self.in1.fqdn()), "dfs.namenode.https-address": "{}:50470".format( self.in1.fqdn()), } } prop = result["hdfs-site"] for i in mock_get_instances(): prop["dfs.namenode.http-address.hdfs-ha.%s" % i.instance_name] = ( "%s:50070" % i.fqdn()) prop["dfs.namenode.https-address.hdfs-ha.%s" % i.instance_name] = ( "%s:50470" % i.fqdn()) prop["dfs.namenode.rpc-address.hdfs-ha.%s" % i.instance_name] = ( "%s:8020" % i.fqdn()) self.assertEqual(result["hdfs-site"], ha._find_hdfs_site(bp)) @mock.patch("sahara.plugins.utils.get_instance") @mock.patch("sahara.plugins.utils.get_instances") def test__configure_yarn_site(self, mock_get_instances, mock_get_instance): mock_get_instances.return_value = [self.in1, self.in2, self.in3] mock_get_instance.return_value = self.in1 bp = ha._configure_yarn_site(self.cluster, copy.deepcopy(self.bp)) zks = ",".join(["%s:2181" % i.fqdn() for i in mock_get_instances()]) rm_ids = ",".join([i.instance_name for i in mock_get_instances()]) result = { "yarn-site": { "hadoop.registry.rm.enabled": "false", "yarn.resourcemanager.zk-address": zks, "yarn.log.server.url": "{}:19888/jobhistory/logs/".format( mock_get_instance().fqdn()), "yarn.resourcemanager.address": "{}:8050".format( mock_get_instances()[0].fqdn()), "yarn.resourcemanager.admin.address": "{}:8141".format( mock_get_instances()[0].fqdn()), "yarn.resourcemanager.cluster-id": self.cluster.name, "yarn.resourcemanager.ha.automatic-failover.zk-base-path": "/yarn-leader-election", "yarn.resourcemanager.ha.enabled": "true", "yarn.resourcemanager.ha.rm-ids": rm_ids, "yarn.resourcemanager.hostname": mock_get_instances()[0].fqdn(), "yarn.resourcemanager.recovery.enabled": "true", "yarn.resourcemanager.resource-tracker.address": "{}:8025".format(mock_get_instances()[0].fqdn()), "yarn.resourcemanager.scheduler.address": "{}:8030".format( mock_get_instances()[0].fqdn()), "yarn.resourcemanager.store.class": "org.apache.hadoop.yarn.server.resourcemanager.recovery." "ZKRMStateStore", "yarn.resourcemanager.webapp.address": "{}:8088".format( mock_get_instances()[0].fqdn()), "yarn.resourcemanager.webapp.https.address": "{}:8090".format( mock_get_instances()[0].fqdn()), "yarn.timeline-service.address": "{}:10200".format( mock_get_instance().fqdn()), "yarn.timeline-service.webapp.address": "{}:8188".format( mock_get_instance().fqdn()), "yarn.timeline-service.webapp.https.address": "{}:8190".format( mock_get_instance().fqdn()) } } props = result["yarn-site"] for i in mock_get_instances(): props["yarn.resourcemanager.hostname.{}".format( i.instance_name)] = i.fqdn() props["yarn.resourcemanager.webapp.address.{}".format( i.instance_name)] = "{}:8088".format(i.fqdn()) props["yarn.resourcemanager.webapp.https.address.{}".format( i.instance_name)] = "{}:8090".format(i.fqdn()) self.assertEqual(result["yarn-site"], ha._find_yarn_site(bp)) @mock.patch("sahara.plugins.utils.get_instances") def test__confgure_hbase_site(self, mock_get_instances): mock_get_instances.return_value = [self.in1, self.in2, self.in3] bp = ha._confgure_hbase_site(self.cluster, copy.deepcopy(self.bp)) result = { "hbase-site": { "hbase.regionserver.global.memstore.lowerLimit": "0.38", "hbase.regionserver.global.memstore.upperLimit": "0.4", "hbase.regionserver.handler.count": "60", "hbase.regionserver.info.port": "60030", "hbase.regionserver.storefile.refresh.period": "20", "hbase.rootdir": "hdfs://hdfs-ha/apps/hbase/data", "hbase.security.authentication": "simple", "hbase.security.authorization": "false", "hbase.superuser": "hbase", "hbase.tmp.dir": "/hadoop/hbase", "hbase.zookeeper.property.clientPort": "2181", "hbase.zookeeper.useMulti": "true", "hfile.block.cache.size": "0.40", "zookeeper.session.timeout": "30000", "zookeeper.znode.parent": "/hbase-unsecure", "hbase.zookeeper.quorum": ",".join([i.fqdn() for i in mock_get_instances()]) } } self.assertEqual(result["hbase-site"], ha._find_hbase_site(bp)) sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_health.py0000666000175100017510000001025313245514472024166 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import six import testtools from sahara.plugins.ambari import health from sahara.service.health import health_check_base from sahara.tests.unit import base class TestAmbariHealthCheck(base.SaharaTestCase): def _standard_negative_test(self, mockclient, return_value, col, count): mockclient.return_value = return_value pr = health.AlertsProvider(mock.Mock()) service = return_value[0].get('Alert').get('service_name') expected_exc = health_check_base.YellowHealthError if col == 'RED': expected_exc = health_check_base.RedHealthError with testtools.ExpectedException(expected_exc): try: health.AmbariServiceHealthCheck( mock.Mock(extra={}), pr, service).check_health() except Exception as e: self.assertEqual( "Cluster health is %s. Reason: " "Ambari Monitor has responded that cluster " "has %s alert(s)" % (col, count), six.text_type(e)) raise @mock.patch('sahara.plugins.ambari.client.AmbariClient.__init__') @mock.patch('sahara.plugins.ambari.client.AmbariClient.close') @mock.patch('sahara.plugins.ambari.client.AmbariClient.get_alerts_data') @mock.patch('sahara.plugins.utils.get_instance') def test_check_health(self, get_instance, alerts_response, close, init): init.return_value = None alerts_response.return_value = [ { 'Alert': { 'state': 'OK', 'service_name': 'ZOOKEEPER' } } ] result = health.AmbariServiceHealthCheck( mock.Mock(extra={}), health.AlertsProvider(mock.Mock()), 'ZOOKEEPER').check_health() self.assertEqual('No alerts found', result) self._standard_negative_test(alerts_response, [ { 'Alert': { 'state': 'WARNING', 'service_name': 'ZOOKEEPER' } } ], 'YELLOW', "1 warning") self._standard_negative_test(alerts_response, [ { 'Alert': { 'state': 'CRITICAL', 'service_name': 'ZOOKEEPER' } } ], 'RED', "1 critical") # not important service: only contribute as yellow self._standard_negative_test(alerts_response, [ { 'Alert': { 'state': 'CRITICAL', 'service_name': 'Kafka' } } ], 'YELLOW', "1 warning") self._standard_negative_test(alerts_response, [ { 'Alert': { 'state': 'CRITICAL', 'service_name': 'ZOOKEEPER' }, }, { 'Alert': { 'state': 'WARNING', 'service_name': 'ZOOKEEPER' } } ], 'RED', "1 critical and 1 warning") alerts_response.side_effect = [ValueError( "OOUCH!")] with testtools.ExpectedException(health_check_base.RedHealthError): try: health.AmbariHealthCheck( mock.Mock(extra={}), health.AlertsProvider(mock.Mock()) ).check_health() except Exception as e: self.assertEqual( "Cluster health is RED. Reason: " "Can't get response from Ambari Monitor: OOUCH!", six.text_type(e)) raise sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_open_ports.py0000666000175100017510000000210613245514472025107 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.ambari import common as p_common from sahara.plugins.ambari import plugin from sahara.tests.unit import base class GetPortsTestCase(base.SaharaTestCase): def setUp(self): super(GetPortsTestCase, self).setUp() self.plugin = plugin.AmbariPluginProvider() def test_get_ambari_port(self): ng = mock.Mock() ng.node_processes = [p_common.AMBARI_SERVER] ports = self.plugin.get_open_ports(ng) self.assertEqual([8080], ports) sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_common.py0000666000175100017510000000531613245514472024215 0ustar zuulzuul00000000000000# Copyright (c) 2017 EasyStack Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.ambari import common from sahara.tests.unit import base class AmbariCommonTestCase(base.SaharaTestCase): def setUp(self): super(AmbariCommonTestCase, self).setUp() self.master_ng = mock.Mock() self.master_ng.node_processes = ['Ambari', 'HiveServer'] self.worker_ng = mock.Mock() self.worker_ng.node_processes = ['DataNode', 'Oozie'] self.cluster = mock.Mock() self.cluster.node_groups = [self.master_ng, self.worker_ng] def test_get_ambari_proc_list(self): procs = common.get_ambari_proc_list(self.master_ng) expected = ['METRICS_COLLECTOR', 'HIVE_SERVER', 'MYSQL_SERVER', 'WEBHCAT_SERVER'] self.assertEqual(procs, expected) procs = common.get_ambari_proc_list(self.worker_ng) expected = ['DATANODE', 'OOZIE_SERVER', 'PIG'] self.assertEqual(procs, expected) @mock.patch('sahara.plugins.kerberos.is_kerberos_security_enabled') def test_get_clients(self, kerberos): kerberos.return_value = False clients = common.get_clients(self.cluster) expected = ['OOZIE_CLIENT', 'HIVE_CLIENT', 'HDFS_CLIENT', 'TEZ_CLIENT', 'METRICS_MONITOR'] for e in expected: self.assertIn(e, clients) kerberos.return_value = True clients = common.get_clients(self.cluster) expected = ['OOZIE_CLIENT', 'HIVE_CLIENT', 'HDFS_CLIENT', 'TEZ_CLIENT', 'METRICS_MONITOR', 'KERBEROS_CLIENT'] for e in expected: self.assertIn(e, clients) def test_instances_have_process(self): instance1 = mock.Mock() instance2 = mock.Mock() instance1.node_group = self.master_ng instance2.node_group = self.worker_ng self.assertTrue(common.instances_have_process([instance1], "Ambari")) self.assertTrue(common.instances_have_process([instance1, instance2], "DataNode")) self.assertFalse(common.instances_have_process([instance1, instance2], "DRPC Server")) sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_plugin.py0000666000175100017510000000334413245514472024222 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.ambari import plugin from sahara.tests.unit import base as test_base class TestPlugin(test_base.SaharaTestCase): def setUp(self): self.plugin = plugin.AmbariPluginProvider() super(TestPlugin, self).setUp() def test_job_types(self): self.assertEqual({ '2.3': [ 'Hive', 'Java', 'MapReduce', 'MapReduce.Streaming', 'Pig', 'Shell', 'Spark'], '2.4': [ 'Hive', 'Java', 'MapReduce', 'MapReduce.Streaming', 'Pig', 'Shell', 'Spark'], '2.5': [ 'Hive', 'Java', 'MapReduce', 'MapReduce.Streaming', 'Pig', 'Shell', 'Spark'], }, self.plugin.get_edp_job_types()) self.assertEqual({ '2.3': [ 'Hive', 'Java', 'MapReduce', 'MapReduce.Streaming', 'Pig', 'Shell', 'Spark'], }, self.plugin.get_edp_job_types(versions=['2.3'])) self.assertEqual({ '2.4': [ 'Hive', 'Java', 'MapReduce', 'MapReduce.Streaming', 'Pig', 'Shell', 'Spark'], }, self.plugin.get_edp_job_types(versions=['2.4'])) sahara-8.0.0/sahara/tests/unit/plugins/ambari/__init__.py0000666000175100017510000000000013245514472023406 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_requests_helper.py0000666000175100017510000000657313245514472026145 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.ambari import requests_helper from sahara.tests.unit import base class RequestsHelperTestCase(base.SaharaTestCase): def setUp(self): super(RequestsHelperTestCase, self).setUp() self.i1 = mock.MagicMock() self.i1.fqdn.return_value = "i1" self.i2 = mock.MagicMock() self.i2.fqdn.return_value = "i2" def test_build_datanode_decommission_request(self): c_name = "c1" instances = [self.i1, self.i2] res = requests_helper.build_datanode_decommission_request(c_name, instances) self.assertEqual("i1,i2", res["RequestInfo"]["parameters"]["excluded_hosts"]) self.assertEqual("c1", res["RequestInfo"]["operation_level"]["cluster_name"]) def test_build_nodemanager_decommission_request(self): c_name = "c1" instances = [self.i1, self.i2] res = requests_helper.build_nodemanager_decommission_request( c_name, instances) self.assertEqual("i1,i2", res["RequestInfo"]["parameters"]["excluded_hosts"]) self.assertEqual("c1", res["RequestInfo"]["operation_level"]["cluster_name"]) def test_build_namenode_restart_request(self): res = requests_helper.build_namenode_restart_request("c1", self.i1) self.assertEqual("i1", res["Requests/resource_filters"][0]["hosts"]) self.assertEqual("c1", res["RequestInfo"]["operation_level"]["cluster_name"]) def test_build_resourcemanager_restart_request(self): res = requests_helper.build_resourcemanager_restart_request("c1", self.i1) self.assertEqual("i1", res["Requests/resource_filters"][0]["hosts"]) self.assertEqual("c1", res["RequestInfo"]["operation_level"]["cluster_name"]) def test_build_stop_service_request(self): res = requests_helper.build_stop_service_request("HDFS") expected = { "RequestInfo": { "context": "Restart HDFS service (stopping)", }, "Body": { "ServiceInfo": { "state": "INSTALLED" } } } self.assertEqual(res, expected) def test_build_start_service_request(self): res = requests_helper.build_start_service_request("HDFS") expected = { "RequestInfo": { "context": "Restart HDFS service (starting)", }, "Body": { "ServiceInfo": { "state": "STARTED" } } } self.assertEqual(res, expected) sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_deploy.py0000666000175100017510000000571613245514472024225 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from oslo_serialization import jsonutils from sahara.plugins.ambari import deploy from sahara.tests.unit import base class TestDeploy(base.SaharaTestCase): @mock.patch('sahara.utils.cluster_progress_ops.add_provisioning_step') @mock.patch('sahara.utils.cluster.check_cluster_exists') @mock.patch('sahara.plugins.utils.get_instance') @mock.patch('sahara.plugins.ambari.client.AmbariClient.get') @mock.patch('sahara.plugins.ambari.client.AmbariClient.delete') def test_cleanup_config_groups(self, client_delete, client_get, get_instance, check_cluster_exists, add_provisioning_step): def response(data): fake = mock.Mock() fake.text = jsonutils.dumps(data) fake.raise_for_status.return_value = True return fake fake_config_groups = { 'items': [ {'ConfigGroup': {'id': "1"}}, {'ConfigGroup': {'id': "2"}} ] } config_group1 = { 'ConfigGroup': {'id': '1', 'group_name': "test:fakename"}} config_group2 = { 'ConfigGroup': {'id': '2', 'group_name': "test:toremove"}} fake_ambari = mock.Mock() fake_ambari.management_ip = "127.0.0.1" get_instance.return_value = fake_ambari inst1 = mock.Mock() inst1.instance_name = "toremove" cl = mock.Mock(extra={'ambari_password': "SUPER_STRONG"}) cl.name = "test" client_get.side_effect = [ response(fake_config_groups), response(config_group1), response(config_group2) ] client_delete.side_effect = [response({})] check_cluster_exists.return_value = True deploy.cleanup_config_groups(cl, [inst1]) get_calls = [ mock.call( 'http://127.0.0.1:8080/api/v1/clusters/test/config_groups'), mock.call( 'http://127.0.0.1:8080/api/v1/clusters/test/config_groups/1'), mock.call( 'http://127.0.0.1:8080/api/v1/clusters/test/config_groups/2') ] self.assertEqual(get_calls, client_get.call_args_list) delete_calls = [ mock.call( 'http://127.0.0.1:8080/api/v1/clusters/test/config_groups/2') ] self.assertEqual(delete_calls, client_delete.call_args_list) sahara-8.0.0/sahara/tests/unit/plugins/ambari/test_client.py0000666000175100017510000003417613245514472024211 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from oslo_serialization import jsonutils from sahara.plugins.ambari import client as ambari_client from sahara.plugins import exceptions as p_exc from sahara.tests.unit import base class AmbariClientTestCase(base.SaharaTestCase): def setUp(self): super(AmbariClientTestCase, self).setUp() self.http_client = mock.Mock() self.http_client.get = mock.Mock() self.http_client.post = mock.Mock() self.http_client.put = mock.Mock() self.http_client.delete = mock.Mock() self.headers = {"X-Requested-By": "sahara"} self.remote = mock.Mock() self.remote.get_http_client.return_value = self.http_client self.instance = mock.Mock() self.instance.remote.return_value = self.remote self.instance.management_ip = "1.2.3.4" self.good_pending_resp = mock.MagicMock() self.good_pending_resp.status_code = 200 self.good_pending_resp.text = ('{"Requests": ' '{"id": 1, "status": "PENDING"}}') def test_init_client_default(self): client = ambari_client.AmbariClient(self.instance) self.assertEqual(self.http_client, client._http_client) self.assertEqual("http://1.2.3.4:8080/api/v1", client._base_url) self.assertEqual("admin", client._auth.username) self.assertEqual("admin", client._auth.password) self.remote.get_http_client.assert_called_with("8080") def test_init_client_manual(self): client = ambari_client.AmbariClient(self.instance, port="1234", username="user", password="pass") self.assertEqual("http://1.2.3.4:1234/api/v1", client._base_url) self.assertEqual("user", client._auth.username) self.assertEqual("pass", client._auth.password) self.remote.get_http_client.assert_called_with("1234") def test_close_http_session(self): with ambari_client.AmbariClient(self.instance): pass self.remote.close_http_session.assert_called_with("8080") def test_get_method(self): client = ambari_client.AmbariClient(self.instance) client.get("http://spam") self.http_client.get.assert_called_with( "http://spam", verify=False, auth=client._auth, headers=self.headers) def test_post_method(self): client = ambari_client.AmbariClient(self.instance) client.post("http://spam", data="data") self.http_client.post.assert_called_with( "http://spam", data="data", verify=False, auth=client._auth, headers=self.headers) def test_put_method(self): client = ambari_client.AmbariClient(self.instance) client.put("http://spam", data="data") self.http_client.put.assert_called_with( "http://spam", data="data", verify=False, auth=client._auth, headers=self.headers) def test_delete_method(self): client = ambari_client.AmbariClient(self.instance) client.delete("http://spam") self.http_client.delete.assert_called_with( "http://spam", verify=False, auth=client._auth, headers=self.headers) def test_import_credential(self): resp = mock.Mock() resp.text = "" resp.status_code = 200 self.http_client.post.return_value = resp client = ambari_client.AmbariClient(self.instance) client.import_credential("test", alias="credential", data={"some": "data"}) self.http_client.post.assert_called_once_with( "http://1.2.3.4:8080/api/v1/clusters/test/credentials/credential", verify=False, data=jsonutils.dumps({"some": "data"}), auth=client._auth, headers=self.headers) def test_get_credential(self): resp = mock.Mock() resp.text = "" resp.status_code = 200 self.http_client.get.return_value = resp client = ambari_client.AmbariClient(self.instance) client.get_credential("test", alias="credential") self.http_client.get.assert_called_once_with( "http://1.2.3.4:8080/api/v1/clusters/test/credentials/credential", verify=False, auth=client._auth, headers=self.headers) resp.status_code = 404 self.assertRaises(ambari_client.AmbariNotFound, ambari_client.AmbariClient.check_response, resp, True) @mock.patch("sahara.plugins.ambari.client.AmbariClient.check_response") def test_get_alerts_data(self, mock_check_response): cluster = mock.Mock() cluster.name = "test_cluster" client = ambari_client.AmbariClient(self.instance) # check_response returning empty json mock_check_response.return_value = {} res = client.get_alerts_data(cluster) self.assertEqual(res, []) self.http_client.get.assert_called_once_with( "http://1.2.3.4:8080/api/v1/clusters/test_cluster/alerts?fields=*", verify=False, auth=client._auth, headers=self.headers) mock_check_response.assert_called_once() # check_response returning json with items as key mock_check_response.return_value = {'items': ['item1', 'item2']} res = client.get_alerts_data(cluster) self.assertEqual(res, ['item1', 'item2']) self.http_client.get.assert_called_with( "http://1.2.3.4:8080/api/v1/clusters/test_cluster/alerts?fields=*", verify=False, auth=client._auth, headers=self.headers) self.assertEqual(self.http_client.get.call_count, 2) self.assertEqual(mock_check_response.call_count, 2) def test_check_response(self): resp = mock.Mock() resp.status_code = 404 self.assertRaises(ambari_client.AmbariNotFound, ambari_client.AmbariClient.check_response, resp, True) resp.status_code = 200 resp.text = u'{"json": "example"}' resp.raise_for_status = mock.Mock() res = ambari_client.AmbariClient.check_response(resp) self.assertEqual(res, {"json": "example"}) resp.raise_for_status.assert_called_once() def test_req_id(self): resp = mock.Mock() resp.text = None self.assertRaises(p_exc.HadoopProvisionError, ambari_client.AmbariClient.req_id, resp) resp.text = u'{"text" : "example"}' self.assertRaises(p_exc.HadoopProvisionError, ambari_client.AmbariClient.req_id, resp) resp.text = u'{"Requests": {"example" : "text"}}' self.assertRaises(p_exc.HadoopProvisionError, ambari_client.AmbariClient.req_id, resp) resp.text = u'{"Requests" : {"id" : "test_id"}}' res = ambari_client.AmbariClient.req_id(resp) self.assertEqual(res, "test_id") def test_get_registered_hosts(self): client = ambari_client.AmbariClient(self.instance) resp_data = """{ "href" : "http://1.2.3.4:8080/api/v1/hosts", "items" : [ { "href" : "http://1.2.3.4:8080/api/v1/hosts/host1", "Hosts" : { "host_name" : "host1" } }, { "href" : "http://1.2.3.4:8080/api/v1/hosts/host2", "Hosts" : { "host_name" : "host2" } }, { "href" : "http://1.2.3.4:8080/api/v1/hosts/host3", "Hosts" : { "host_name" : "host3" } } ] }""" resp = mock.Mock() resp.text = resp_data resp.status_code = 200 self.http_client.get.return_value = resp hosts = client.get_registered_hosts() self.http_client.get.assert_called_with( "http://1.2.3.4:8080/api/v1/hosts", verify=False, auth=client._auth, headers=self.headers) self.assertEqual(3, len(hosts)) self.assertEqual("host1", hosts[0]["Hosts"]["host_name"]) self.assertEqual("host2", hosts[1]["Hosts"]["host_name"]) self.assertEqual("host3", hosts[2]["Hosts"]["host_name"]) def test_update_user_password(self): client = ambari_client.AmbariClient(self.instance) resp = mock.Mock() resp.text = "" resp.status_code = 200 self.http_client.put.return_value = resp client.update_user_password("bart", "old_pw", "new_pw") exp_req = jsonutils.dumps({ "Users": { "old_password": "old_pw", "password": "new_pw" } }) self.http_client.put.assert_called_with( "http://1.2.3.4:8080/api/v1/users/bart", data=exp_req, verify=False, auth=client._auth, headers=self.headers) def test_create_blueprint(self): client = ambari_client.AmbariClient(self.instance) resp = mock.Mock() resp.text = "" resp.status_code = 200 self.http_client.post.return_value = resp client.create_blueprint("cluster_name", {"some": "data"}) self.http_client.post.assert_called_with( "http://1.2.3.4:8080/api/v1/blueprints/cluster_name", data=jsonutils.dumps({"some": "data"}), verify=False, auth=client._auth, headers=self.headers) def test_create_cluster(self): client = ambari_client.AmbariClient(self.instance) resp = mock.Mock() resp.text = """{ "Requests": { "id": 1, "status": "InProgress" } }""" resp.status_code = 200 self.http_client.post.return_value = resp req_info = client.create_cluster("cluster_name", {"some": "data"}) self.assertEqual(1, req_info["id"]) self.http_client.post.assert_called_with( "http://1.2.3.4:8080/api/v1/clusters/cluster_name", data=jsonutils.dumps({"some": "data"}), verify=False, auth=client._auth, headers=self.headers) def test_add_host_to_cluster(self): client = ambari_client.AmbariClient(self.instance) resp = mock.Mock() resp.text = "" resp.status_code = 200 self.http_client.post.return_value = resp instance = mock.MagicMock() instance.fqdn.return_value = "i1" instance.cluster.name = "cl" client.add_host_to_cluster(instance) self.http_client.post.assert_called_with( "http://1.2.3.4:8080/api/v1/clusters/cl/hosts/i1", verify=False, auth=client._auth, headers=self.headers) def test_start_process_on_host(self): client = ambari_client.AmbariClient(self.instance) self.http_client.put.return_value = self.good_pending_resp client.wait_ambari_request = mock.MagicMock() instance = mock.MagicMock() instance.fqdn.return_value = "i1" instance.cluster.name = "cl" client.start_service_on_host(instance, "HDFS", 'STATE') self.http_client.put.assert_called_with( "http://1.2.3.4:8080/api/v1/clusters/" "cl/hosts/i1/host_components/HDFS", data=jsonutils.dumps( { "HostRoles": {"state": "STATE"}, "RequestInfo": { "context": "Starting service HDFS, " "moving to state STATE"} }), verify=False, auth=client._auth, headers=self.headers) def test_stop_process_on_host(self): client = ambari_client.AmbariClient(self.instance) check_mock = mock.MagicMock() check_mock.status_code = 200 check_mock.text = '{"HostRoles": {"state": "SOME_STATE"}}' self.http_client.get.return_value = check_mock self.http_client.put.return_value = self.good_pending_resp client.wait_ambari_request = mock.MagicMock() instance = mock.MagicMock() instance.fqdn.return_value = "i1" client.stop_process_on_host("cluster_name", instance, "p1") self.http_client.put.assert_called_with( "http://1.2.3.4:8080/api/v1/clusters/" "cluster_name/hosts/i1/host_components/p1", data=jsonutils.dumps( { "HostRoles": {"state": "INSTALLED"}, "RequestInfo": {"context": "Stopping p1"} }), verify=False, auth=client._auth, headers=self.headers) @mock.patch("sahara.plugins.ambari.client.context") def test_wait_ambari_request(self, mock_context): client = ambari_client.AmbariClient(self.instance) check_mock = mock.MagicMock() d1 = {"request_context": "r1", "request_status": "PENDING", "progress_percent": 42} d2 = {"request_context": "r1", "request_status": "COMPLETED", "progress_percent": 100} check_mock.side_effect = [d1, d2] client.check_request_status = check_mock client.wait_ambari_request("id1", "c1") check_mock.assert_has_calls([mock.call("c1", "id1"), mock.call("c1", "id1")]) @mock.patch("sahara.plugins.ambari.client.context") def test_wait_ambari_request_error(self, mock_context): client = ambari_client.AmbariClient(self.instance) check_mock = mock.MagicMock() d1 = {"request_context": "r1", "request_status": "ERROR", "progress_percent": 146} check_mock.return_value = d1 client.check_request_status = check_mock self.assertRaises(p_exc.HadoopProvisionError, client.wait_ambari_request, "id1", "c1") sahara-8.0.0/sahara/tests/unit/plugins/storm/0000775000175100017510000000000013245515027021213 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/storm/test_config_helper.py0000666000175100017510000000447613245514472025450 0ustar zuulzuul00000000000000# Copyright 2017 Massachusetts Open Cloud # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from testtools import testcase from sahara.plugins.storm import config_helper as s_config from sahara.plugins.storm import plugin as s_plugin class TestStormConfigHelper(testcase.TestCase): def test_generate_storm_config(self): STORM_092 = '0.9.2' STORM_101 = '1.0.1' STORM_110 = '1.1.0' tested_versions = [] master_hostname = "s-master" zk_hostnames = ["s-zoo"] configs_092 = s_config.generate_storm_config( master_hostname, zk_hostnames, STORM_092) self.assertIn('nimbus.host', configs_092.keys()) self.assertNotIn('nimbus.seeds', configs_092.keys()) tested_versions.append(STORM_092) configs_101 = s_config.generate_storm_config( master_hostname, zk_hostnames, STORM_101) self.assertNotIn('nimbus.host', configs_101.keys()) self.assertIn('nimbus.seeds', configs_101.keys()) self.assertIn('client.jartransformer.class', configs_101.keys()) self.assertEqual(configs_101['client.jartransformer.class'], 'org.apache.storm.hack.StormShadeTransformer') tested_versions.append(STORM_101) configs_110 = s_config.generate_storm_config( master_hostname, zk_hostnames, STORM_110) self.assertNotIn('nimbus.host', configs_110.keys()) self.assertIn('nimbus.seeds', configs_110.keys()) self.assertIn('client.jartransformer.class', configs_110.keys()) self.assertEqual(configs_110['client.jartransformer.class'], 'org.apache.storm.hack.StormShadeTransformer') tested_versions.append(STORM_110) storm = s_plugin.StormProvider() self.assertEqual(storm.get_versions(), tested_versions) sahara-8.0.0/sahara/tests/unit/plugins/storm/test_plugin.py0000666000175100017510000002561213245514472024135 0ustar zuulzuul00000000000000# Copyright (c) 2015 TellesNobrega # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara import conductor as cond from sahara import context from sahara.plugins import base as pb from sahara.plugins import exceptions as ex from sahara.plugins.storm import plugin as pl from sahara.service.edp.storm import engine from sahara.tests.unit import base from sahara.utils import edp conductor = cond.API class StormPluginTest(base.SaharaWithDbTestCase): def setUp(self): super(StormPluginTest, self).setUp() self.override_config("plugins", ["storm"]) self.master_host = "master" self.master_inst = "6789" self.storm_topology_name = 'topology1' pb.setup_plugins() def _make_master_instance(self, return_code=0): master = mock.Mock() master.execute_command.return_value = (return_code, self.storm_topology_name) master.hostname.return_value = self.master_host master.id = self.master_inst return master def _get_cluster(self, name, version): cluster_dict = { 'name': name, 'plugin_name': 'storm', 'hadoop_version': version, 'node_groups': []} return cluster_dict def test_validate_existing_ng_scaling(self): data = [ {'name': 'master', 'flavor_id': '42', 'count': 1, 'node_processes': ['nimbus']}, {'name': 'slave', 'flavor_id': '42', 'count': 1, 'node_processes': ['supervisor']}, {'name': 'zookeeper', 'flavor_id': '42', 'count': 1, 'node_processes': ['zookeeper']} ] cluster_data_092 = self._get_cluster('cluster_0.9.2', '0.9.2') cluster_data_101 = self._get_cluster('cluster_1.0.1', '1.0.1') cluster_data_110 = self._get_cluster('cluster_1.1.0', '1.1.0') cluster_data_092['node_groups'] = data cluster_data_101['node_groups'] = data cluster_data_110['node_groups'] = data clusters = [cluster_data_092, cluster_data_101, cluster_data_110] for cluster_data in clusters: cluster = conductor.cluster_create(context.ctx(), cluster_data) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) supervisor_id = [node.id for node in cluster.node_groups if node.name == 'supervisor'] self.assertIsNone( plugin._validate_existing_ng_scaling(cluster, supervisor_id)) @mock.patch("sahara.plugins.storm.plugin.utils") def test_validate(self, mock_utils): cluster_data = self._get_cluster('cluster', '0.9.2') cluster = conductor.cluster_create(context.ctx(), cluster_data) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) # number of nimbus nodes != 1 should raise an exception fake_ng = mock.Mock() fake_ng.count = 0 mock_ng = mock.Mock(return_value=[fake_ng]) mock_utils.get_node_groups = mock_ng self.assertRaises(ex.RequiredServiceMissingException, plugin.validate, cluster) mock_ng.assert_called_once_with(cluster, "nimbus") fake_ng.count = 2 self.assertRaises(ex.InvalidComponentCountException, plugin.validate, cluster) mock_ng.assert_called_with(cluster, "nimbus") self.assertEqual(2, mock_ng.call_count) # no supervisor should raise an exception fake_nimbus = mock.Mock() fake_nimbus.count = 1 fake_supervisor = mock.Mock() fake_supervisor.count = 0 mock_ng = mock.Mock(side_effect=[[fake_nimbus], [fake_supervisor]]) mock_utils.get_node_groups = mock_ng self.assertRaises(ex.InvalidComponentCountException, plugin.validate, cluster) mock_ng.assert_any_call(cluster, "nimbus") mock_ng.assert_any_call(cluster, "supervisor") self.assertEqual(2, mock_ng.call_count) # one nimbus and one or more supervisors should not raise an exception fake_nimbus.count = 1 fake_supervisor.count = 2 mock_ng = mock.Mock(side_effect=[[fake_nimbus], [fake_supervisor]]) mock_utils.get_node_groups = mock_ng plugin.validate(cluster) mock_ng.assert_any_call(cluster, "nimbus") mock_ng.assert_any_call(cluster, "supervisor") self.assertEqual(2, mock_ng.call_count) def test_validate_additional_ng_scaling(self): data = [ {'name': 'master', 'flavor_id': '42', 'count': 1, 'node_processes': ['nimbus']}, {'name': 'slave', 'flavor_id': '42', 'count': 1, 'node_processes': ['supervisor']}, {'name': 'zookeeper', 'flavor_id': '42', 'count': 1, 'node_processes': ['zookeeper']}, {'name': 'slave2', 'flavor_id': '42', 'count': 0, 'node_processes': ['supervisor']} ] cluster_data_092 = self._get_cluster('cluster_0.9.2', '0.9.2') cluster_data_101 = self._get_cluster('cluster_1.0.1', '1.0.1') cluster_data_110 = self._get_cluster('cluster_1.1.0', '1.1.0') cluster_data_092['node_groups'] = data cluster_data_101['node_groups'] = data cluster_data_110['node_groups'] = data clusters = [cluster_data_092, cluster_data_101, cluster_data_110] for cluster_data in clusters: cluster = conductor.cluster_create(context.ctx(), cluster_data) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) supervisor_id = [node.id for node in cluster.node_groups if node.name == 'supervisor'] self.assertIsNone( plugin._validate_additional_ng_scaling(cluster, supervisor_id)) def test_validate_existing_ng_scaling_raises(self): data = [ {'name': 'master', 'flavor_id': '42', 'count': 1, 'node_processes': ['nimbus']}, {'name': 'slave', 'flavor_id': '42', 'count': 1, 'node_processes': ['supervisor']}, {'name': 'zookeeper', 'flavor_id': '42', 'count': 1, 'node_processes': ['zookeeper']} ] cluster_data_092 = self._get_cluster('cluster_0.9.2', '0.9.2') cluster_data_101 = self._get_cluster('cluster_1.0.1', '1.0.1') cluster_data_110 = self._get_cluster('cluster_1.1.0', '1.1.0') cluster_data_092['node_groups'] = data cluster_data_101['node_groups'] = data cluster_data_110['node_groups'] = data clusters = [cluster_data_092, cluster_data_101, cluster_data_110] for cluster_data in clusters: cluster = conductor.cluster_create(context.ctx(), cluster_data) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) master_id = [node.id for node in cluster.node_groups if node.name == 'master'] self.assertRaises(ex.NodeGroupCannotBeScaled, plugin._validate_existing_ng_scaling, cluster, master_id) def test_validate_additional_ng_scaling_raises(self): data = [ {'name': 'master', 'flavor_id': '42', 'count': 1, 'node_processes': ['nimbus']}, {'name': 'slave', 'flavor_id': '42', 'count': 1, 'node_processes': ['supervisor']}, {'name': 'zookeeper', 'flavor_id': '42', 'count': 1, 'node_processes': ['zookeeper']}, {'name': 'master2', 'flavor_id': '42', 'count': 0, 'node_processes': ['nimbus']} ] cluster_data_092 = self._get_cluster('cluster_0.9.2', '0.9.2') cluster_data_101 = self._get_cluster('cluster_1.0.1', '1.0.1') cluster_data_110 = self._get_cluster('cluster_1.1.0', '1.1.0') cluster_data_092['node_groups'] = data cluster_data_101['node_groups'] = data cluster_data_110['node_groups'] = data clusters = [cluster_data_092, cluster_data_101, cluster_data_110] for cluster_data in clusters: cluster = conductor.cluster_create(context.ctx(), cluster_data) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) master_id = [node.id for node in cluster.node_groups if node.name == 'master2'] self.assertRaises(ex.NodeGroupCannotBeScaled, plugin._validate_existing_ng_scaling, cluster, master_id) def test_get_open_port(self): plugin_storm = pl.StormProvider() cluster = mock.Mock() ng = mock.Mock() ng.node_processes = ['nimbus'] cluster.node_groups = [ng] ng.cluster = cluster ports = plugin_storm.get_open_ports(ng) self.assertEqual([8080], ports) def _test_engine(self, version, job_type, eng): cluster_dict = self._get_cluster('demo', version) cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) self.assertIsInstance(plugin.get_edp_engine(cluster, job_type), eng) def test_plugin092_edp_storm_engine(self): self._test_engine('0.9.2', edp.JOB_TYPE_STORM, engine.StormJobEngine) def test_plugin092_edp_storm_pyleus_engine(self): self._test_engine('0.9.2', edp.JOB_TYPE_PYLEUS, engine.StormJobEngine) def test_plugin101_edp_storm_engine(self): self._test_engine('1.0.1', edp.JOB_TYPE_STORM, engine.StormJobEngine) def test_plugin101_edp_storm_pyleus_engine(self): self._test_engine('1.0.1', edp.JOB_TYPE_PYLEUS, engine.StormJobEngine) def test_plugin110_edp_storm_engine(self): self._test_engine('1.1.0', edp.JOB_TYPE_STORM, engine.StormJobEngine) def test_plugin110_edp_storm_pyleus_engine(self): self._test_engine('1.1.0', edp.JOB_TYPE_PYLEUS, engine.StormJobEngine) sahara-8.0.0/sahara/tests/unit/plugins/storm/__init__.py0000666000175100017510000000000013245514472023317 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/test_images.py0000666000175100017510000005006213245514472022735 0ustar zuulzuul00000000000000# Copyright (c) 2016 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from oslo_utils import uuidutils import yaml from sahara import exceptions as ex from sahara.plugins import exceptions as p_ex from sahara.plugins import images from sahara.tests.unit import base as b class TestImages(b.SaharaTestCase): def test_package_spec(self): cls = images.SaharaPackageValidator validator = cls.from_spec("java", {}, []) self.assertIsInstance(validator, cls) self.assertEqual(str(validator.packages[0]), "java") validator = cls.from_spec({"java": {"version": "8"}}, {}, []) self.assertIsInstance(validator, cls) self.assertEqual(str(validator.packages[0]), "java-8") validator = cls.from_spec( [{"java": {"version": "8"}}, "hadoop"], {}, []) self.assertIsInstance(validator, cls) self.assertEqual(str(validator.packages[0]), "java-8") self.assertEqual(str(validator.packages[1]), "hadoop") def test_script_spec(self): cls = images.SaharaScriptValidator resource_roots = ['tests/unit/plugins'] validator = cls.from_spec('test_images.py', {}, resource_roots) self.assertIsInstance(validator, cls) self.assertEqual(validator.env_vars, ['test_only', 'distro']) validator = cls.from_spec( {'test_images.py': {'env_vars': ['extra-file', 'user']}}, {}, resource_roots) self.assertIsInstance(validator, cls) self.assertEqual(validator.env_vars, ['test_only', 'distro', 'extra-file', 'user']) def test_all_spec(self): cls = images.SaharaAllValidator validator_map = images.SaharaImageValidatorBase.get_validator_map() validator = cls.from_spec( [{'package': {'java': {'version': '8'}}}, {'package': 'hadoop'}], validator_map, []) self.assertIsInstance(validator, cls) self.assertEqual(len(validator.validators), 2) self.assertEqual(validator.validators[0].packages[0].name, 'java') self.assertEqual(validator.validators[1].packages[0].name, 'hadoop') def test_any_spec(self): cls = images.SaharaAnyValidator validator_map = images.SaharaImageValidatorBase.get_validator_map() validator = cls.from_spec( [{'package': {'java': {'version': '8'}}}, {'package': 'hadoop'}], validator_map, []) self.assertIsInstance(validator, cls) self.assertEqual(len(validator.validators), 2) self.assertEqual(validator.validators[0].packages[0].name, 'java') self.assertEqual(validator.validators[1].packages[0].name, 'hadoop') def test_os_case_spec(self): cls = images.SaharaOSCaseValidator validator_map = images.SaharaImageValidatorBase.get_validator_map() spec = [ {'redhat': [{'package': 'nfs-utils'}]}, {'debian': [{'package': 'nfs-common'}]} ] validator = cls.from_spec(spec, validator_map, []) self.assertIsInstance(validator, cls) self.assertEqual(len(validator.distros), 2) self.assertEqual(validator.distros[0].distro, 'redhat') self.assertEqual(validator.distros[1].distro, 'debian') redhat, debian = ( validator.distros[os].validator.validators[0].packages[0].name for os in range(2)) self.assertEqual(redhat, 'nfs-utils') self.assertEqual(debian, 'nfs-common') def test_sahara_image_validator_spec(self): cls = images.SaharaImageValidator validator_map = images.SaharaImageValidatorBase.get_validator_map() resource_roots = ['tests/unit/plugins'] spec = """ arguments: java-version: description: The version of java. default: openjdk required: false choices: - openjdk - oracle-java validators: - os_case: - redhat: - package: nfs-utils - debian: - package: nfs-common - any: - all: - package: java-1.8.0-openjdk-devel - argument_set: argument_name: java-version value: 1.8.0 - all: - package: java-1.7.0-openjdk-devel - argument_set: argument_name: java-version value: 1.7.0 - script: test_images.py - package: - hadoop - hadoop-libhdfs - hadoop-native - hadoop-pipes - hadoop-sbin - hadoop-lzo - lzo - lzo-devel - hadoop-lzo-native - argument_case: argument_name: JAVA_VERSION cases: 1.7.0: - script: test_images.py 1.8.0: - script: test_images.py """ spec = yaml.safe_load(spec) validator = cls.from_spec(spec, validator_map, resource_roots) validators = validator.validators self.assertIsInstance(validator, cls) self.assertEqual(len(validators), 5) self.assertIsInstance(validators[0], images.SaharaOSCaseValidator) self.assertIsInstance(validators[1], images.SaharaAnyValidator) self.assertIsInstance(validators[2], images.SaharaScriptValidator) self.assertIsInstance(validators[3], images.SaharaPackageValidator) self.assertIsInstance( validators[4], images.SaharaArgumentCaseValidator) self.assertEqual(1, len(validator.arguments)) self.assertEqual(validator.arguments['java-version'].required, False) self.assertEqual(validator.arguments['java-version'].default, 'openjdk') self.assertEqual(validator.arguments['java-version'].description, 'The version of java.') self.assertEqual(validator.arguments['java-version'].choices, ['openjdk', 'oracle-java']) def test_package_validator_redhat(self): cls = images.SaharaPackageValidator image_arguments = {"distro": 'centos'} packages = [cls.Package("java", "8")] validator = images.SaharaPackageValidator(packages) remote = mock.Mock() validator.validate(remote, test_only=True, image_arguments=image_arguments) remote.execute_command.assert_called_with( "rpm -q java-8", run_as_root=True) image_arguments = {"distro": 'fedora'} packages = [cls.Package("java", "8"), cls.Package("hadoop")] validator = images.SaharaPackageValidator(packages) remote = mock.Mock() remote.execute_command.side_effect = ( ex.RemoteCommandException("So bad!")) try: validator.validate(remote, test_only=True, image_arguments=image_arguments) except p_ex.ImageValidationError as e: self.assertIn("So bad!", e.message) remote.execute_command.assert_called_with( "rpm -q java-8 hadoop", run_as_root=True) self.assertEqual(remote.execute_command.call_count, 1) image_arguments = {"distro": 'redhat'} packages = [cls.Package("java", "8"), cls.Package("hadoop")] validator = images.SaharaPackageValidator(packages) remote = mock.Mock() def side_effect(call, run_as_root=False): if "rpm" in call: raise ex.RemoteCommandException("So bad!") remote.execute_command.side_effect = side_effect try: validator.validate(remote, test_only=False, image_arguments=image_arguments) except p_ex.ImageValidationError as e: self.assertIn("So bad!", e.message) self.assertEqual(remote.execute_command.call_count, 3) calls = [mock.call("rpm -q java-8 hadoop", run_as_root=True), mock.call("yum install -y java-8 hadoop", run_as_root=True), mock.call("rpm -q java-8 hadoop", run_as_root=True)] remote.execute_command.assert_has_calls(calls) def test_package_validator_debian(self): cls = images.SaharaPackageValidator image_arguments = {"distro": 'ubuntu'} packages = [cls.Package("java", "8")] validator = images.SaharaPackageValidator(packages) remote = mock.Mock() validator.validate(remote, test_only=True, image_arguments=image_arguments) remote.execute_command.assert_called_with( "dpkg -s java-8", run_as_root=True) image_arguments = {"distro": 'ubuntu'} packages = [cls.Package("java", "8"), cls.Package("hadoop")] validator = images.SaharaPackageValidator(packages) remote = mock.Mock() remote.execute_command.side_effect = ( ex.RemoteCommandException("So bad!")) try: validator.validate(remote, test_only=True, image_arguments=image_arguments) except p_ex.ImageValidationError as e: self.assertIn("So bad!", e.message) remote.execute_command.assert_called_with( "dpkg -s java-8 hadoop", run_as_root=True) self.assertEqual(remote.execute_command.call_count, 1) image_arguments = {"distro": 'ubuntu'} packages = [cls.Package("java", "8"), cls.Package("hadoop")] validator = images.SaharaPackageValidator(packages) remote = mock.Mock() remote.execute_command.side_effect = ( ex.RemoteCommandException("So bad!")) try: validator.validate(remote, test_only=False, image_arguments=image_arguments) except p_ex.ImageValidationError as e: self.assertIn("So bad!", e.message) self.assertEqual(remote.execute_command.call_count, 2) calls = [mock.call("dpkg -s java-8 hadoop", run_as_root=True), mock.call("apt-get -y install java-8 hadoop", run_as_root=True)] remote.execute_command.assert_has_calls(calls) @mock.patch('oslo_utils.uuidutils.generate_uuid') def test_script_validator(self, uuid): hash_value = '00000000-0000-0000-0000-000000000000' uuidutils.generate_uuid.return_value = hash_value cls = images.SaharaScriptValidator image_arguments = {"distro": 'centos'} cmd = "It's dangerous to go alone. Run this." validator = cls(cmd, env_vars=image_arguments.keys(), output_var="distro") remote = mock.Mock( execute_command=mock.Mock( return_value=(0, 'fedora'))) validator.validate(remote, test_only=False, image_arguments=image_arguments) call = [mock.call('chmod +x /tmp/%(hash_value)s.sh' % {'hash_value': hash_value}, run_as_root=True), mock.call('/tmp/%(hash_value)s.sh' % {'hash_value': hash_value}, run_as_root=True)] remote.execute_command.assert_has_calls(call) self.assertEqual(image_arguments['distro'], 'fedora') def test_any_validator(self): cls = images.SaharaAnyValidator class FakeValidator(images.SaharaImageValidatorBase): def __init__(self, mock_validate): self.mock_validate = mock_validate def validate(self, remote, test_only=False, **kwargs): self.mock_validate(remote, test_only=test_only, **kwargs) # One success short circuits validation always_tells_the_truth = FakeValidator(mock.Mock()) validator = cls([always_tells_the_truth, always_tells_the_truth]) validator.validate(None, test_only=False) self.assertEqual(always_tells_the_truth.mock_validate.call_count, 1) # All failures fails, and calls with test_only=True on all first always_lies = FakeValidator( mock.Mock(side_effect=p_ex.ImageValidationError("Oh no!"))) validator = cls([always_lies, always_lies]) try: validator.validate(None, test_only=False) except p_ex.ImageValidationError: pass self.assertEqual(always_lies.mock_validate.call_count, 4) # But it fails after a first pass if test_only=True. always_lies = FakeValidator( mock.Mock(side_effect=p_ex.ImageValidationError("Oh no!"))) validator = cls([always_lies, always_lies]) try: validator.validate(None, test_only=True) except p_ex.ImageValidationError: pass self.assertEqual(always_lies.mock_validate.call_count, 2) # One failure doesn't end iteration. always_tells_the_truth = FakeValidator(mock.Mock()) always_lies = FakeValidator( mock.Mock(side_effect=p_ex.ImageValidationError("Oh no!"))) validator = cls([always_lies, always_tells_the_truth]) validator.validate(None, test_only=False) self.assertEqual(always_lies.mock_validate.call_count, 1) self.assertEqual(always_tells_the_truth.mock_validate.call_count, 1) def test_all_validator(self): cls = images.SaharaAllValidator # All pass always_tells_the_truth = mock.Mock() validator = cls([always_tells_the_truth, always_tells_the_truth]) validator.validate(None, test_only=False) self.assertEqual(always_tells_the_truth.validate.call_count, 2) always_tells_the_truth.validate.assert_called_with( None, test_only=False, image_arguments=None) # Second fails always_tells_the_truth = mock.Mock() always_lies = mock.Mock(validate=mock.Mock( side_effect=p_ex.ImageValidationError("Boom!"))) validator = cls([always_tells_the_truth, always_lies]) try: validator.validate(None, test_only=True) except p_ex.ImageValidationError: pass self.assertEqual(always_tells_the_truth.validate.call_count, 1) self.assertEqual(always_lies.validate.call_count, 1) always_tells_the_truth.validate.assert_called_with( None, test_only=True, image_arguments=None) always_lies.validate.assert_called_with( None, test_only=True, image_arguments=None) # First fails always_tells_the_truth = mock.Mock() always_lies = mock.Mock(validate=mock.Mock( side_effect=p_ex.ImageValidationError("Boom!"))) validator = cls([always_lies, always_tells_the_truth]) try: validator.validate(None, test_only=True, image_arguments={}) except p_ex.ImageValidationError: pass self.assertEqual(always_lies.validate.call_count, 1) always_lies.validate.assert_called_with( None, test_only=True, image_arguments={}) self.assertEqual(always_tells_the_truth.validate.call_count, 0) def test_os_case_validator(self): cls = images.SaharaOSCaseValidator Distro = images.SaharaOSCaseValidator._distro_tuple # First match wins and short circuits iteration centos = Distro("centos", mock.Mock()) redhat = Distro("redhat", mock.Mock()) distros = [centos, redhat] image_arguments = {images.SaharaImageValidator.DISTRO_KEY: "centos"} validator = cls(distros) validator.validate(None, test_only=False, image_arguments=image_arguments) self.assertEqual(centos.validator.validate.call_count, 1) self.assertEqual(redhat.validator.validate.call_count, 0) centos.validator.validate.assert_called_with( None, test_only=False, image_arguments=image_arguments) # Families match centos = Distro("centos", mock.Mock()) redhat = Distro("redhat", mock.Mock()) distros = [centos, redhat] image_arguments = {images.SaharaImageValidator.DISTRO_KEY: "fedora"} validator = cls(distros) validator.validate(None, test_only=False, image_arguments=image_arguments) self.assertEqual(centos.validator.validate.call_count, 0) self.assertEqual(redhat.validator.validate.call_count, 1) redhat.validator.validate.assert_called_with( None, test_only=False, image_arguments=image_arguments) # Non-matches do nothing centos = Distro("centos", mock.Mock()) redhat = Distro("redhat", mock.Mock()) distros = [centos, redhat] image_arguments = {images.SaharaImageValidator.DISTRO_KEY: "ubuntu"} validator = cls(distros) validator.validate(None, test_only=False, image_arguments=image_arguments) self.assertEqual(centos.validator.validate.call_count, 0) self.assertEqual(redhat.validator.validate.call_count, 0) def test_sahara_argument_case_validator(self): cls = images.SaharaArgumentCaseValidator # Match gets called image_arguments = {"argument": "value"} match = mock.Mock() nomatch = mock.Mock() cases = {"value": match, "another_value": nomatch} validator = cls("argument", cases) validator.validate(None, test_only=False, image_arguments=image_arguments) self.assertEqual(match.validate.call_count, 1) self.assertEqual(nomatch.validate.call_count, 0) match.validate.assert_called_with( None, test_only=False, image_arguments=image_arguments) # Non-matches do nothing image_arguments = {"argument": "value"} nomatch = mock.Mock() cases = {"some_value": nomatch, "another_value": nomatch} validator = cls("argument", cases) validator.validate(None, test_only=False, image_arguments=image_arguments) self.assertEqual(nomatch.validate.call_count, 0) def test_sahara_argument_set_validator(self): cls = images.SaharaArgumentSetterValidator # Old variable is overwritten image_arguments = {"argument": "value"} validator = cls("argument", "new_value") validator.validate(None, test_only=False, image_arguments=image_arguments) self.assertEqual(image_arguments["argument"], "new_value") # New variable is set image_arguments = {"argument": "value"} validator = cls("another_argument", "value") validator.validate(None, test_only=False, image_arguments=image_arguments) self.assertEqual(image_arguments, {"argument": "value", "another_argument": "value"}) def test_sahara_image_validator(self): cls = images.SaharaImageValidator sub_validator = mock.Mock(validate=mock.Mock()) remote = mock.Mock(get_os_distrib=mock.Mock( return_value="centos")) validator = cls(sub_validator, {}) validator.validate(remote, test_only=False, image_arguments={}) expected_map = {images.SaharaImageValidatorBase.DISTRO_KEY: "centos"} sub_validator.validate.assert_called_with( remote, test_only=False, image_arguments=expected_map) expected_map = {images.SaharaImageValidatorBase.DISTRO_KEY: "centos"} validator.validate(remote, test_only=True, image_arguments={}) sub_validator.validate.assert_called_with( remote, test_only=True, image_arguments=expected_map) sahara-8.0.0/sahara/tests/unit/plugins/mapr/0000775000175100017510000000000013245515027021006 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/mapr/test_distro.py0000666000175100017510000000261313245514472023732 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from sahara.plugins.mapr.domain import distro import sahara.tests.unit.base as b class TestDistro(b.SaharaTestCase): def __init__(self, *args, **kwds): super(TestDistro, self).__init__(*args, **kwds) self.install_cmd = 'foo_bar' self.separator = '-' self.distro = distro.Distro('foo', 'foo', self.install_cmd, self.separator) def test_create_install_cmd(self): pkgs = [('foo',), ('bar', 'version')] cmd = self.distro.create_install_cmd(pkgs) self.assertIsNotNone(cmd) parts = cmd.split(' ') self.assertEqual(self.install_cmd, parts[0]) self.assertEqual('foo', parts[1]) self.assertEqual('bar', parts[2].split(self.separator)[0]) self.assertEqual('version*', parts[2].split(self.separator)[1]) sahara-8.0.0/sahara/tests/unit/plugins/mapr/__init__.py0000666000175100017510000000000013245514472023112 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/mapr/test_config_files.py0000666000175100017510000001761713245514472025067 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import sahara.plugins.mapr.domain.configuration_file as conf_f import sahara.tests.unit.base as b class TestHadoopXML(b.SaharaTestCase): def __init__(self, *args, **kwds): super(TestHadoopXML, self).__init__(*args, **kwds) self.content = ''' key1 value1 key2 value2 ''' def test_remote_path(self): foo = conf_f.HadoopXML('foo') foo.remote_path = '/bar' self.assertEqual('/bar/foo', foo.remote_path) def test_parse(self): foo = conf_f.HadoopXML('foo') foo.parse(self.content) expected = {'key1': 'value1', 'key2': 'value2'} self.assertEqual(expected, foo._config_dict) def test_render(self): foo = conf_f.HadoopXML('foo') expected = {'ke1': 'value1', 'key2': 'value2'} foo._config_dict = expected actual = foo.render() bar = conf_f.HadoopXML('bar') bar.parse(actual) self.assertEqual(expected, bar._config_dict) def test_add_property(self): foo = conf_f.HadoopXML('foo') self.assertEqual(foo._config_dict, {}) foo.add_property('key1', 'value1') self.assertEqual(foo._config_dict, {'key1': 'value1'}) foo.add_property('key2', 'value2') expected = {'key1': 'value1', 'key2': 'value2'} self.assertEqual(expected, foo._config_dict) def test_get_config_value(self): foo = conf_f.HadoopXML('foo') foo._config_dict = {'foo': 'bar'} self.assertEqual('bar', foo._get_config_value('foo')) self.assertIsNone(foo._get_config_value('bar')) class TestRawFile(b.SaharaTestCase): def __init__(self, *args, **kwds): super(TestRawFile, self).__init__(*args, **kwds) self.content = 'some meaningful text' def test_remote_path(self): foo = conf_f.RawFile('foo') foo.remote_path = '/bar' self.assertEqual('/bar/foo', foo.remote_path) def test_parse(self): foo = conf_f.RawFile('foo') foo.parse(self.content) expected = {'content': self.content} self.assertEqual(expected, foo._config_dict) def test_render(self): foo = conf_f.RawFile('foo') expected = {'content': 'foo bar'} foo._config_dict = expected actual = foo.render() bar = conf_f.RawFile('bar') bar.parse(actual) self.assertEqual(expected, bar._config_dict) class TestPropertiesFile(b.SaharaTestCase): def __init__(self, *args, **kwds): super(TestPropertiesFile, self).__init__(*args, **kwds) self.content = ''' key1=value1 key2=value2 ''' def test_remote_path(self): foo = conf_f.PropertiesFile('foo') foo.remote_path = '/bar' self.assertEqual('/bar/foo', foo.remote_path) def test_parse(self): foo = conf_f.PropertiesFile('foo') foo.parse(self.content) expected = {'key1': 'value1', 'key2': 'value2'} self.assertEqual(expected, foo._config_dict) def test_render(self): foo = conf_f.PropertiesFile('foo') expected = {'ke1': 'value1', 'key2': 'value2'} foo._config_dict = expected actual = foo.render() bar = conf_f.PropertiesFile('bar') bar.parse(actual) self.assertEqual(expected, bar._config_dict) def test_add_property(self): foo = conf_f.PropertiesFile('foo') expected = {} self.assertEqual(expected, foo._config_dict) foo.add_property('key1', 'value1') expected = {'key1': 'value1'} self.assertEqual(expected, foo._config_dict) foo.add_property('key2', 'value2') expected = {'key1': 'value1', 'key2': 'value2'} self.assertEqual(expected, foo._config_dict) def test_get_config_value(self): foo = conf_f.PropertiesFile('foo') foo._config_dict = {'foo': 'bar'} self.assertEqual('bar', foo._get_config_value('foo')) self.assertIsNone(foo._get_config_value('bar')) class TestTemplateFile(b.SaharaTestCase): def __init__(self, *args, **kwds): super(TestTemplateFile, self).__init__(*args, **kwds) self.content = ''' key1={{ value1 }} key2={{ value2 }}''' self.rendered = ''' key1=value1 key2=value2''' def test_remote_path(self): foo = conf_f.TemplateFile('foo') foo.remote_path = '/bar' self.assertEqual('/bar/foo', foo.remote_path) def test_parse(self): foo = conf_f.TemplateFile('foo') foo.parse(self.content) self.assertIsNotNone(foo._template) def test_render(self): foo = conf_f.TemplateFile('foo') expected = {'value1': 'value1', 'value2': 'value2'} foo.parse(self.content) foo._config_dict = expected actual = foo.render() self.assertEqual(self.rendered, actual) def test_add_property(self): foo = conf_f.TemplateFile('foo') expected = {} self.assertEqual(expected, foo._config_dict) foo.add_property('key1', 'value1') expected = {'key1': 'value1'} self.assertEqual(expected, foo._config_dict) foo.add_property('key2', 'value2') expected = {'key1': 'value1', 'key2': 'value2'} self.assertEqual(expected, foo._config_dict) class TestEnvironmentConfig(b.SaharaTestCase): def __init__(self, *args, **kwds): super(TestEnvironmentConfig, self).__init__(*args, **kwds) self.content = ''' non export line export key1=value1 export key2=value2 export key ''' def test_remote_path(self): foo = conf_f.EnvironmentConfig('foo') foo.remote_path = '/bar' self.assertEqual('/bar/foo', foo.remote_path) def test_parse(self): foo = conf_f.EnvironmentConfig('foo') foo.parse(self.content) expected = {'key1': 'value1', 'key2': 'value2'} self.assertEqual(expected, foo._config_dict) def test_render(self): foo = conf_f.EnvironmentConfig('foo') expected = {'ke1': 'value1', 'key2': 'value2'} foo._config_dict = expected actual = foo.render() bar = conf_f.EnvironmentConfig('bar') bar.parse(actual) self.assertEqual(expected, bar._config_dict) def test_render_extra_properties(self): foo = conf_f.EnvironmentConfig('foo') foo.parse(self.content) foo.add_property('key3', 'value3') foo_content = foo.render() bar = conf_f.EnvironmentConfig('bar') bar.parse(foo_content) expected = {'key1': 'value1', 'key2': 'value2', 'key3': 'value3'} self.assertEqual(expected, bar._config_dict) def test_add_property(self): foo = conf_f.EnvironmentConfig('foo') self.assertEqual({}, foo._config_dict) foo.add_property('key1', 'value1') self.assertEqual({'key1': 'value1'}, foo._config_dict) foo.add_property('key2', 'value2') expected = {'key1': 'value1', 'key2': 'value2'} self.assertEqual(expected, foo._config_dict) def test_get_config_value(self): foo = conf_f.EnvironmentConfig('foo') foo._config_dict = {'foo': 'bar'} self.assertEqual('bar', foo._get_config_value('foo')) self.assertIsNone(foo._get_config_value('bar')) sahara-8.0.0/sahara/tests/unit/plugins/mapr/test_cluster_context.py0000666000175100017510000002671013245514472025657 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import testtools import sahara.exceptions as e from sahara.plugins.mapr.domain import node_process as np from sahara.plugins.mapr.services.management import management from sahara.plugins.mapr.services.maprfs import maprfs from sahara.plugins.mapr.services.oozie import oozie from sahara.plugins.mapr.services.swift import swift from sahara.plugins.mapr.services.yarn import yarn import sahara.plugins.mapr.versions.v5_2_0_mrv2.context as cc import sahara.plugins.mapr.versions.v5_2_0_mrv2.version_handler as handler from sahara.plugins import provisioning as p from sahara.tests.unit import base as b from sahara.tests.unit import testutils as tu MANAGEMENT_IP = '1.1.1.1' INTERNAL_IP = '1.1.1.2' class TestClusterContext(b.SaharaTestCase): def __init__(self, *args, **kwds): super(TestClusterContext, self).__init__(*args, **kwds) self.fake_np = np.NodeProcess('fake', 'foo', 'bar') def _get_context(self): i1 = tu.make_inst_dict('id_1', 'instance_1', MANAGEMENT_IP) i1['internal_ip'] = INTERNAL_IP master_proc = [ yarn.RESOURCE_MANAGER.ui_name, yarn.NODE_MANAGER.ui_name, yarn.HISTORY_SERVER.ui_name, maprfs.CLDB.ui_name, maprfs.FILE_SERVER.ui_name, oozie.OOZIE.ui_name, management.ZOOKEEPER.ui_name, ] master_ng = tu.make_ng_dict('master', 'large', master_proc, 1, [i1]) cluster_configs = { 'Service': { 'key': 'value', 'Service Version': '1.1', }, 'Oozie': { 'Oozie Version': '4.2.0', } } cluster = tu.create_cluster( name='test_cluster', tenant='large', plugin='mapr', version='5.2.0.mrv2', node_groups=[master_ng], cluster_configs=cluster_configs, ) self.ng = cluster.node_groups[0] self.instance = self.ng.instances[0] return cc.Context(cluster, handler.VersionHandler()) def test_get_oozie_server_uri(self): ctx = self._get_context() expected = 'http://%s:11000/oozie' % MANAGEMENT_IP self.assertEqual(expected, ctx.oozie_server_uri) def test_oozie_server(self): ctx = self._get_context() node_processes = ctx.oozie_server.node_group.node_processes self.assertIn(oozie.OOZIE.ui_name, node_processes) def test_oozie_http(self): ctx = self._get_context() expected = '%s:11000' % MANAGEMENT_IP self.assertEqual(expected, ctx.oozie_http) def test_configure_sh(self): ctx = self._get_context() conf_sh = ctx.configure_sh pattern = (r'^(\S+)\s+(-N (\S+))\s+(-C (\S+))\s+(-Z (\S+))\s+' r'(-no-autostart)\s+(-f)\s+\s(-HS (\S+))') self.assertRegex(conf_sh, pattern) self.assertIn('/opt/mapr/server/configure.sh', conf_sh) self.assertIn('-C %s' % INTERNAL_IP, conf_sh) self.assertIn('-Z %s' % INTERNAL_IP, conf_sh) self.assertIn('-HS %s' % INTERNAL_IP, conf_sh) self.assertIn('-no-autostart', conf_sh) self.assertIn('-N ' + ctx.cluster.name, conf_sh) def test_get_cluster_config_value(self): ctx = self._get_context() conf = p.Config('key', 'Service', 'cluster') self.assertEqual('value', ctx._get_cluster_config_value(conf)) not_set = p.Config('nonset', 'Service', 'cluster') self.assertIsNone(ctx._get_cluster_config_value(not_set)) def test_get_instances(self): ctx = self._get_context() instances = ctx.get_instances() self.assertEqual(1, len(instances)) rms1 = ctx.get_instances(yarn.RESOURCE_MANAGER) self.assertEqual(1, len(rms1)) rms2 = ctx.get_instances(yarn.RESOURCE_MANAGER.ui_name) self.assertEqual(1, len(rms2)) not_existing_1 = ctx.get_instances(self.fake_np) self.assertEqual(0, len(not_existing_1)) not_existing_2 = ctx.get_instances(self.fake_np.ui_name) self.assertEqual(0, len(not_existing_2)) def test_get_instance(self): ctx = self._get_context() instance_1 = ctx.get_instance(yarn.RESOURCE_MANAGER) self.assertIn(yarn.RESOURCE_MANAGER.ui_name, instance_1.node_group.node_processes) instance_2 = ctx.get_instance(yarn.RESOURCE_MANAGER) self.assertIn(yarn.RESOURCE_MANAGER.ui_name, instance_2.node_group.node_processes) self.assertIsNone(ctx.get_instance(self.fake_np)) def test_get_instances_ip(self): ctx = self._get_context() ip_list_1 = ctx.get_instances_ip(yarn.RESOURCE_MANAGER) self.assertEqual(1, len(ip_list_1)) self.assertIn(INTERNAL_IP, ip_list_1) ip_list_2 = ctx.get_instances_ip(yarn.RESOURCE_MANAGER.ui_name) self.assertEqual(1, len(ip_list_2)) self.assertIn(INTERNAL_IP, ip_list_2) empty_list = ctx.get_instances_ip(self.fake_np) self.assertEqual(0, len(empty_list)) def test_get_instance_ip(self): ctx = self._get_context() ip_1 = ctx.get_instance_ip(yarn.RESOURCE_MANAGER) self.assertEqual(INTERNAL_IP, ip_1) ip_2 = ctx.get_instance_ip(yarn.RESOURCE_MANAGER.ui_name) self.assertEqual(INTERNAL_IP, ip_2) none_ip = ctx.get_instance_ip(self.fake_np) self.assertIsNone(none_ip) def test_get_zookeeper_nodes_ip_with_port(self): ctx = self._get_context() expected = '%s:5181' % INTERNAL_IP actual = ctx.get_zookeeper_nodes_ip_with_port() self.assertEqual(expected, actual) management.ZK_CLIENT_PORT = '0000' expected = '%s:0000' % INTERNAL_IP actual = ctx.get_zookeeper_nodes_ip_with_port() self.assertEqual(expected, actual) def test_filter_instances(self): ctx = self._get_context() instances = ctx.get_instances() rsmngs = ctx.filter_instances(instances, yarn.RESOURCE_MANAGER) self.assertEqual(1, len(rsmngs)) not_existing_i = ctx.filter_instances(instances, self.fake_np) self.assertEqual(0, len(not_existing_i)) def test_check_for_process(self): ctx = self._get_context() instance = ctx.get_instance(yarn.RESOURCE_MANAGER) self.assertTrue(ctx.check_for_process(instance, yarn.RESOURCE_MANAGER)) self.assertTrue(ctx.check_for_process(instance, yarn.RESOURCE_MANAGER.ui_name)) self.assertFalse(ctx.check_for_process(instance, maprfs.NFS)) self.assertFalse(ctx.check_for_process(instance, maprfs.NFS.ui_name)) def test_get_chosen_service_version(self): ctx = self._get_context() version = ctx.get_chosen_service_version('Service') self.assertEqual('1.1', version) def test_get_cluster_services(self): ctx = self._get_context() actual_services = ctx.get_cluster_services() actual_services_names = map(lambda s: s.ui_name, actual_services) expected_services_names = [ yarn.YARN().ui_name, management.Management().ui_name, maprfs.MapRFS().ui_name, oozie.Oozie().ui_name, swift.Swift().ui_name, ] self.assertEqual(sorted(actual_services_names), sorted(expected_services_names)) def test_get_service(self): ctx = self._get_context() service = ctx.get_service(yarn.HISTORY_SERVER) self.assertEqual(yarn.YARN().ui_name, service.ui_name) with testtools.ExpectedException(e.InvalidDataException): ctx.get_service(self.fake_np) def test_get_service_name_by_node_process(self): ctx = self._get_context() s_name_1 = ctx.get_service_name_by_node_process(yarn.RESOURCE_MANAGER) self.assertEqual(yarn.YARN().ui_name, s_name_1) s_name_2 = ctx.get_service_name_by_node_process( yarn.RESOURCE_MANAGER.ui_name) self.assertEqual(yarn.YARN().ui_name, s_name_2) not_existing_np = np.NodeProcess('not_existing', 'NotExisting', 'foo') self.assertIsNone(ctx.get_service_name_by_node_process( not_existing_np)) self.assertIsNone(ctx.get_service_name_by_node_process( not_existing_np.ui_name)) def test_get_instances_count(self): ctx = self._get_context() self.assertEqual(1, ctx.get_instances_count()) self.assertEqual(1, ctx.get_instances_count(yarn.RESOURCE_MANAGER)) self.assertEqual(1, ctx.get_instances_count( yarn.RESOURCE_MANAGER.ui_name)) self.assertEqual(0, ctx.get_instances_count(self.fake_np)) self.assertEqual(0, ctx.get_instances_count( self.fake_np.ui_name)) def test_get_node_groups(self): ctx = self._get_context() all_ngs = ctx.get_node_groups() self.assertEqual(1, len(all_ngs)) self.assertEqual([self.ng], all_ngs) rm_ngs_1 = ctx.get_node_groups(yarn.RESOURCE_MANAGER) self.assertEqual(1, len(rm_ngs_1)) self.assertEqual([self.ng], rm_ngs_1) rm_ngs_2 = ctx.get_node_groups(yarn.RESOURCE_MANAGER.ui_name) self.assertEqual(1, len(rm_ngs_2)) self.assertEqual([self.ng], rm_ngs_2) empty_ngs = ctx.get_node_groups(self.fake_np) self.assertEqual(0, len(empty_ngs)) def test_get_cldb_nodes_ip(self): ctx = self._get_context() cldb_list_1 = ctx.get_cldb_nodes_ip() self.assertEqual(1, len(cldb_list_1.split(','))) self.assertIn(INTERNAL_IP, cldb_list_1) cldb_list_2 = ctx.get_cldb_nodes_ip() self.assertEqual(1, len(cldb_list_2.split(','))) self.assertIn(INTERNAL_IP, cldb_list_2) sep = ':' cldb_list_3 = ctx.get_cldb_nodes_ip(sep) self.assertEqual(1, len(cldb_list_3.split(sep))) self.assertIn(INTERNAL_IP, cldb_list_3) def test_get_zookeeper_nodes_ip(self): ctx = self._get_context() zk_list_1 = ctx.get_zookeeper_nodes_ip() self.assertEqual(1, len(zk_list_1.split(','))) self.assertIn(INTERNAL_IP, zk_list_1) zk_list_2 = ctx.get_zookeeper_nodes_ip() self.assertEqual(1, len(zk_list_2.split(','))) self.assertIn(INTERNAL_IP, zk_list_2) sep = ':' zk_list_3 = ctx.get_zookeeper_nodes_ip(sep) self.assertEqual(1, len(zk_list_3.split(sep))) self.assertIn(INTERNAL_IP, zk_list_3) def test_get_resourcemanager_ip(self): ctx = self._get_context() ip = ctx.get_resourcemanager_ip() self.assertEqual(INTERNAL_IP, ip) def test_get_historyserver_ip(self): ctx = self._get_context() self.assertTrue(ctx.has_control_nodes([self.instance])) def test_is_present(self): cluster_context = self._get_context() self.assertTrue(cluster_context.is_present(oozie.Oozie())) self.assertFalse(cluster_context.is_present(oozie.OozieV401())) self.assertTrue(cluster_context.is_present(oozie.OozieV420())) sahara-8.0.0/sahara/tests/unit/plugins/mapr/test_base_handler.py0000666000175100017510000001013013245514472025026 0ustar zuulzuul00000000000000# Copyright (c) 2015, MapR Technologies # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import six import sahara.plugins.mapr.abstract.cluster_context as abstract_context from sahara.plugins.mapr.domain import service as bs from sahara.plugins.mapr.services.management import management from sahara.plugins.mapr.services.maprfs import maprfs from sahara.plugins.mapr.services.oozie import oozie from sahara.plugins.mapr.services.yarn import yarn import sahara.plugins.mapr.versions.v5_2_0_mrv2.version_handler as handler from sahara.plugins import provisioning as p from sahara.tests.unit import base as b from sahara.tests.unit import testutils as tu class TestHandler(b.SaharaTestCase): def __init__(self, *args, **kwds): super(TestHandler, self).__init__(*args, **kwds) self.handler = handler.VersionHandler() self.cluster = self._get_cluster() def _get_cluster(self): i1 = tu.make_inst_dict('id_1', 'instance_1', '1.1.1.1') master_proc = [ yarn.RESOURCE_MANAGER.ui_name, yarn.NODE_MANAGER.ui_name, yarn.HISTORY_SERVER.ui_name, maprfs.CLDB.ui_name, maprfs.FILE_SERVER.ui_name, oozie.OOZIE.ui_name, management.ZOOKEEPER.ui_name, ] master_ng = tu.make_ng_dict('master', 'large', master_proc, 1, [i1]) cluster_configs = { 'Service': { 'key': 'value', 'Service Version': '1.1', }, } cluster = tu.create_cluster( name='test_cluster', tenant='large', plugin='mapr', version='5.2.0.mrv2', node_groups=[master_ng], cluster_configs=cluster_configs, ) self.ng = cluster.node_groups[0] self.instance = self.ng.instances[0] return cluster def test_get_context(self): self.assertIsInstance(self.handler.get_context(self._get_cluster()), abstract_context.AbstractClusterContext) def test_get_services(self): services = self.handler.get_services() for service in services: self.assertIsInstance(service, bs.Service) def test_get_np_dict(self): np_dict = self.handler.get_node_processes() self.assertIsNotNone(np_dict) self.assertIsInstance(np_dict, dict) for k, v in six.iteritems(np_dict): self.assertIsInstance(k, six.string_types) self.assertIsInstance(v, list) self.assertNotEqual(0, len(v)) def test_get_configs(self): configs = self.handler.get_configs() self.assertIsInstance(configs, list) for config in configs: self.assertIsInstance(config, p.Config) def test_get_version_configs(self): version_configs = self.handler._get_version_configs() self.assertIsInstance(version_configs, list) for conf in version_configs: self.assertIsInstance(conf, p.Config) self.assertNotEqual(0, len(conf.config_values)) self.assertEqual('dropdown', conf.config_type) def test_get_configs_dict(self): configs_dict = self.handler.get_configs_dict() self.assertIsInstance(configs_dict, dict) for k, v in six.iteritems(configs_dict): self.assertIsInstance(k, six.string_types) self.assertIsInstance(v, dict) def test_get_open_ports(self): ports_list = self.handler.get_open_ports(self.ng) self.assertIsNotNone(ports_list) self.assertIsInstance(ports_list, list) for port in ports_list: self.assertIsInstance(port, int) sahara-8.0.0/sahara/tests/unit/plugins/test_provisioning.py0000666000175100017510000001004413245514472024212 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from sahara import conductor as cond from sahara import context from sahara import exceptions as ex from sahara.plugins import provisioning as p from sahara.tests.unit import base conductor = cond.API class ProvisioningPluginBaseTest(testtools.TestCase): def test__map_to_user_inputs_success(self): c1, c2, c3, plugin = _build_configs_and_plugin() user_inputs = plugin._map_to_user_inputs(None, { 'at-1': { 'n-1': 'v-1', 'n-3': 'v-3', }, 'at-2': { 'n-2': 'v-2', }, }) self.assertEqual([ p.UserInput(c1, 'v-1'), p.UserInput(c2, 'v-2'), p.UserInput(c3, 'v-3'), ], user_inputs) def test__map_to_user_inputs_failure(self): c1, c2, c3, plugin = _build_configs_and_plugin() with testtools.ExpectedException(ex.ConfigurationError): plugin._map_to_user_inputs(None, { 'at-X': { 'n-1': 'v-1', }, }) with testtools.ExpectedException(ex.ConfigurationError): plugin._map_to_user_inputs(None, { 'at-1': { 'n-X': 'v-1', }, }) def _build_configs_and_plugin(): c1 = p.Config('n-1', 'at-1', 'cluster') c2 = p.Config('n-2', 'at-2', 'cluster') c3 = p.Config('n-3', 'at-1', 'node') class TestPlugin(TestEmptyPlugin): def get_configs(self, hadoop_version): return [c1, c2, c3] return c1, c2, c3, TestPlugin() class TestEmptyPlugin(p.ProvisioningPluginBase): def get_title(self): pass def get_versions(self): pass def get_configs(self, hadoop_version): pass def get_node_processes(self, hadoop_version): pass def configure_cluster(self, cluster): pass def start_cluster(self, cluster): pass class TestPluginDataCRUD(base.SaharaWithDbTestCase): def test_crud(self): ctx = context.ctx() data = conductor.plugin_create( ctx, {'name': 'fake', 'plugin_labels': {'enabled': True}}) self.assertIsNotNone(data) raised = None try: # duplicate entry, shouldn't work conductor.plugin_create(ctx, {'name': 'fake'}) except Exception as e: raised = e self.assertIsNotNone(raised) # not duplicated entry, other tenant ctx.tenant = "tenant_2" res = conductor.plugin_create(ctx, {'name': 'fake'}) conductor.plugin_create(ctx, {'name': 'guy'}) self.assertIsNotNone(res) self.assertEqual(2, len(conductor.plugin_get_all(ctx))) ctx.tenant = "tenant_1" data = conductor.plugin_get(ctx, 'fake') self.assertEqual('fake', data['name']) data = conductor.plugin_update( ctx, 'fake', {'version_labels': {'0.1': {'enabled': False}}}) data = conductor.plugin_get(ctx, 'fake') self.assertEqual( {'0.1': {'enabled': False}}, data.get('version_labels')) with testtools.ExpectedException(ex.NotFoundException): conductor.plugin_update(ctx, 'fake_not_found', {}) data = conductor.plugin_remove(ctx, 'fake') self.assertIsNone(data) data = conductor.plugin_get(ctx, 'fake') self.assertIsNone(data) with testtools.ExpectedException(ex.NotFoundException): conductor.plugin_remove(ctx, 'fake') sahara-8.0.0/sahara/tests/unit/plugins/spark/0000775000175100017510000000000013245515027021167 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/spark/test_config_helper.py0000666000175100017510000001053513245514472025415 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import copy import xml.dom.minidom as xml import mock from sahara.plugins.spark import config_helper as c_helper from sahara.swift import swift_helper as swift from sahara.tests.unit import base as test_base from sahara.utils import xmlutils class ConfigHelperUtilsTest(test_base.SaharaTestCase): def test_make_hadoop_path(self): storage_paths = ['/mnt/one', '/mnt/two'] paths = c_helper.make_hadoop_path(storage_paths, '/spam') expected = ['/mnt/one/spam', '/mnt/two/spam'] self.assertEqual(expected, paths) @mock.patch('sahara.plugins.utils.get_config_value_or_default') def test_cleanup_configs(self, get_config_value): getter = lambda plugin, key, cluster: plugin_configs[key] get_config_value.side_effect = getter plugin_configs = {"Minimum cleanup megabytes": 4096, "Minimum cleanup seconds": 86400, "Maximum cleanup seconds": 1209600} configs = c_helper.generate_job_cleanup_config(None) self.assertTrue(configs['valid']) expected = ["MINIMUM_CLEANUP_MEGABYTES=4096", "MINIMUM_CLEANUP_SECONDS=86400", "MAXIMUM_CLEANUP_SECONDS=1209600"] for config_value in expected: self.assertIn(config_value, configs['script']) self.assertIn("0 * * * * root /etc/hadoop/tmp-cleanup.sh", configs['cron'][0]) plugin_configs['Maximum cleanup seconds'] = 0 configs = c_helper.generate_job_cleanup_config(None) self.assertFalse(configs['valid']) self.assertNotIn(configs, 'script') self.assertNotIn(configs, 'cron') plugin_configs = {"Minimum cleanup megabytes": 0, "Minimum cleanup seconds": 0, "Maximum cleanup seconds": 1209600} configs = c_helper.generate_job_cleanup_config(None) self.assertFalse(configs['valid']) self.assertNotIn(configs, 'script') self.assertNotIn(configs, 'cron') @mock.patch("sahara.swift.utils.retrieve_auth_url") def test_generate_xml_configs(self, auth_url): auth_url.return_value = "http://localhost:5000/v2/" # Make a dict of swift configs to verify generated values swift_vals = c_helper.extract_name_values(swift.get_swift_configs()) # Make sure that all the swift configs are in core-site c = c_helper.generate_xml_configs({}, ['/mnt/one'], 'localhost', None) doc = xml.parseString(c['core-site']) configuration = doc.getElementsByTagName('configuration') properties = xmlutils.get_property_dict(configuration[0]) self.assertDictContainsSubset(swift_vals, properties) # Make sure that user values have precedence over defaults c = c_helper.generate_xml_configs( {'HDFS': {'fs.swift.service.sahara.tenant': 'fred'}}, ['/mnt/one'], 'localhost', None) doc = xml.parseString(c['core-site']) configuration = doc.getElementsByTagName('configuration') properties = xmlutils.get_property_dict(configuration[0]) mod_swift_vals = copy.copy(swift_vals) mod_swift_vals['fs.swift.service.sahara.tenant'] = 'fred' self.assertDictContainsSubset(mod_swift_vals, properties) # Make sure that swift configs are left out if not enabled c = c_helper.generate_xml_configs( {'HDFS': {'fs.swift.service.sahara.tenant': 'fred'}, 'general': {'Enable Swift': False}}, ['/mnt/one'], 'localhost', None) doc = xml.parseString(c['core-site']) configuration = doc.getElementsByTagName('configuration') properties = xmlutils.get_property_dict(configuration[0]) for key in mod_swift_vals.keys(): self.assertNotIn(key, properties) sahara-8.0.0/sahara/tests/unit/plugins/spark/test_plugin.py0000666000175100017510000002221513245514472024105 0ustar zuulzuul00000000000000# Copyright (c) 2013 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara import conductor as cond from sahara import context from sahara import exceptions as ex from sahara.plugins import base as pb from sahara.plugins import exceptions as pe from sahara.plugins.spark import plugin as pl from sahara.service.edp.spark import engine from sahara.tests.unit import base from sahara.tests.unit import testutils as tu from sahara.utils import edp conductor = cond.API class SparkPluginTest(base.SaharaWithDbTestCase): def setUp(self): super(SparkPluginTest, self).setUp() self.override_config("plugins", ["spark"]) pb.setup_plugins() def _init_cluster_dict(self, version): cluster_dict = { 'name': 'cluster', 'plugin_name': 'spark', 'hadoop_version': version, 'default_image_id': 'image'} return cluster_dict def test_plugin09_edp_engine_validation(self): cluster_dict = self._init_cluster_dict('0.9.1') job = mock.Mock() job.type = edp.JOB_TYPE_SPARK cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) edp_engine = plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK) with testtools.ExpectedException( ex.InvalidDataException, value_re="Spark 1.3.1 or higher required to run " "Spark jobs\nError ID: .*"): edp_engine.validate_job_execution(cluster, job, mock.Mock()) def test_plugin10_edp_engine(self): self._test_engine('1.3.1', edp.JOB_TYPE_SPARK, engine.SparkJobEngine) def test_plugin10_shell_engine(self): self._test_engine('1.3.1', edp.JOB_TYPE_SHELL, engine.SparkShellJobEngine) def test_plugin11_edp_engine(self): self._test_engine('1.6.0', edp.JOB_TYPE_SPARK, engine.SparkJobEngine) def test_plugin12_shell_engine(self): self._test_engine('1.6.0', edp.JOB_TYPE_SHELL, engine.SparkShellJobEngine) def test_plugin21_edp_engine(self): self._test_engine('2.1.0', edp.JOB_TYPE_SPARK, engine.SparkJobEngine) def test_plugin21_shell_engine(self): self._test_engine('2.1.0', edp.JOB_TYPE_SHELL, engine.SparkShellJobEngine) def test_plugin22_edp_engine(self): self._test_engine('2.2', edp.JOB_TYPE_SPARK, engine.SparkJobEngine) def test_plugin22_shell_engine(self): self._test_engine('2.2', edp.JOB_TYPE_SHELL, engine.SparkShellJobEngine) def _test_engine(self, version, job_type, eng): cluster_dict = self._init_cluster_dict(version) cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) self.assertIsInstance(plugin.get_edp_engine(cluster, job_type), eng) def test_plugin13_edp_engine(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'spark', 'hadoop_version': '1.3.1', 'default_image_id': 'image'} cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) self.assertIsInstance( plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK), engine.SparkJobEngine) def test_cleanup_configs(self): remote = mock.Mock() instance = mock.Mock() extra_conf = {'job_cleanup': { 'valid': True, 'script': 'script_text', 'cron': 'cron_text'}} instance.node_group.node_processes = ["master"] instance.node_group.id = id cluster_dict = self._init_cluster_dict('1.3.1') cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) plugin._push_cleanup_job(remote, cluster, extra_conf, instance) remote.write_file_to.assert_called_with( '/etc/hadoop/tmp-cleanup.sh', 'script_text') remote.execute_command.assert_called_with( 'sudo sh -c \'echo "cron_text" > /etc/cron.d/spark-cleanup\'') remote.reset_mock() instance.node_group.node_processes = ["worker"] plugin._push_cleanup_job(remote, cluster, extra_conf, instance) self.assertFalse(remote.called) remote.reset_mock() instance.node_group.node_processes = ["master"] extra_conf['job_cleanup']['valid'] = False plugin._push_cleanup_job(remote, cluster, extra_conf, instance) remote.execute_command.assert_called_with( 'sudo rm -f /etc/crond.d/spark-cleanup') class SparkValidationTest(base.SaharaTestCase): def setUp(self): super(SparkValidationTest, self).setUp() pb.setup_plugins() self.plugin = pl.SparkProvider() def test_validate(self): self.ng = [] self.ng.append(tu.make_ng_dict("nn", "f1", ["namenode"], 0)) self.ng.append(tu.make_ng_dict("ma", "f1", ["master"], 0)) self.ng.append(tu.make_ng_dict("sl", "f1", ["slave"], 0)) self.ng.append(tu.make_ng_dict("dn", "f1", ["datanode"], 0)) self._validate_case(1, 1, 3, 3) self._validate_case(1, 1, 3, 4) self._validate_case(1, 1, 4, 3) with testtools.ExpectedException(pe.InvalidComponentCountException): self._validate_case(2, 1, 3, 3) with testtools.ExpectedException(pe.InvalidComponentCountException): self._validate_case(1, 2, 3, 3) with testtools.ExpectedException(pe.InvalidComponentCountException): self._validate_case(0, 1, 3, 3) with testtools.ExpectedException(pe.RequiredServiceMissingException): self._validate_case(1, 0, 3, 3) cl = self._create_cluster( 1, 1, 3, 3, cluster_configs={'HDFS': {'dfs.replication': 4}}) with testtools.ExpectedException(pe.InvalidComponentCountException): self.plugin.validate(cl) def _create_cluster(self, *args, **kwargs): lst = [] for i in range(0, len(args)): self.ng[i]['count'] = args[i] lst.append(self.ng[i]) return tu.create_cluster("cluster1", "tenant1", "spark", "1.60", lst, **kwargs) def _validate_case(self, *args): cl = self._create_cluster(*args) self.plugin.validate(cl) class SparkProviderTest(base.SaharaTestCase): def setUp(self): super(SparkProviderTest, self).setUp() def test_supported_job_types(self): provider = pl.SparkProvider() res = provider.get_edp_job_types() self.assertEqual([edp.JOB_TYPE_SHELL, edp.JOB_TYPE_SPARK], res['1.3.1']) self.assertEqual([edp.JOB_TYPE_SHELL, edp.JOB_TYPE_SPARK], res['1.6.0']) self.assertEqual([edp.JOB_TYPE_SHELL, edp.JOB_TYPE_SPARK], res['2.1.0']) self.assertEqual([edp.JOB_TYPE_SHELL, edp.JOB_TYPE_SPARK], res['2.2']) def test_edp_config_hints(self): provider = pl.SparkProvider() res = provider.get_edp_config_hints(edp.JOB_TYPE_SHELL, "1.3.1") self.assertEqual({'configs': {}, 'args': [], 'params': {}}, res['job_config']) res = provider.get_edp_config_hints(edp.JOB_TYPE_SHELL, "1.6.0") self.assertEqual({'configs': {}, 'args': [], 'params': {}}, res['job_config']) res = provider.get_edp_config_hints(edp.JOB_TYPE_SPARK, "1.3.1") self.assertEqual({'args': [], 'configs': []}, res['job_config']) res = provider.get_edp_config_hints(edp.JOB_TYPE_SPARK, "1.6.0") self.assertEqual({'args': [], 'configs': []}, res['job_config']) res = provider.get_edp_config_hints(edp.JOB_TYPE_SPARK, "2.1.0") self.assertEqual({'args': [], 'configs': []}, res['job_config']) res = provider.get_edp_config_hints(edp.JOB_TYPE_SHELL, "2.1.0") self.assertEqual({'args': [], 'configs': {}, 'params': {}}, res['job_config']) res = provider.get_edp_config_hints(edp.JOB_TYPE_SPARK, "2.2") self.assertEqual({'args': [], 'configs': []}, res['job_config']) res = provider.get_edp_config_hints(edp.JOB_TYPE_SHELL, "2.2") self.assertEqual({'args': [], 'configs': {}, 'params': {}}, res['job_config']) sahara-8.0.0/sahara/tests/unit/plugins/spark/__init__.py0000666000175100017510000000000013245514472023273 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/0000775000175100017510000000000013245515027020605 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/0000775000175100017510000000000013245515027021657 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_plugin_utils_5110.py0000666000175100017510000000166213245514472026466 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_11_0 import plugin_utils as pu from sahara.tests.unit.plugins.cdh import base_plugin_utils_test class TestPluginUtilsV5110(base_plugin_utils_test.TestPluginUtilsHigherThanV5): def setUp(self): super(TestPluginUtilsV5110, self).setUp() self.plug_utils = pu.PluginUtilsV5110() self.version = "v5_11_0" sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_versionhandler.py0000666000175100017510000001334113245514472026322 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import six from sahara.plugins.cdh.v5_11_0 import edp_engine from sahara.plugins.cdh.v5_11_0 import versionhandler from sahara.tests.unit import base class VersionHandlerTestCase(base.SaharaTestCase): plugin_path = "sahara.plugins.cdh.v5_11_0." cloudera_utils_path = plugin_path + "cloudera_utils.ClouderaUtilsV5110." plugin_utils_path = plugin_path + "plugin_utils.PluginUtilsV5110." def setUp(self): super(VersionHandlerTestCase, self).setUp() self.vh = versionhandler.VersionHandler() def test_get_node_processes(self): processes = self.vh.get_node_processes() for k, v in six.iteritems(processes): for p in v: self.assertIsInstance(p, str) @mock.patch("sahara.conductor.API.cluster_update") @mock.patch("sahara.context.ctx") @mock.patch(plugin_path + "deploy.configure_cluster") @mock.patch(cloudera_utils_path + "get_cloudera_manager_info", return_value={"fake_cm_info": "fake"}) def test_config_cluster(self, get_cm_info, configure_cluster, ctx, cluster_update): cluster = mock.Mock() self.vh.configure_cluster(cluster) configure_cluster.assert_called_once_with(cluster) cluster_update.assert_called_once_with( ctx(), cluster, {'info': {"fake_cm_info": "fake"}}) @mock.patch(plugin_path + "deploy.start_cluster") def test_start_cluster(self, start_cluster): cluster = mock.Mock() self.vh._set_cluster_info = mock.Mock() self.vh.start_cluster(cluster) start_cluster.assert_called_once_with(cluster) self.vh._set_cluster_info.assert_called_once_with(cluster) @mock.patch(plugin_path + "deploy.decommission_cluster") def test_decommission_nodes(self, decommission_cluster): cluster = mock.Mock() instances = mock.Mock() self.vh.decommission_nodes(cluster, instances) decommission_cluster.assert_called_once_with(cluster, instances) @mock.patch(plugin_path + "deploy.scale_cluster") def test_scale_cluster(self, scale_cluster): cluster = mock.Mock() instances = mock.Mock() self.vh.scale_cluster(cluster, instances) scale_cluster.assert_called_once_with(cluster, instances) @mock.patch("sahara.conductor.API.cluster_update") @mock.patch("sahara.context.ctx") @mock.patch(cloudera_utils_path + "get_cloudera_manager_info", return_value={}) @mock.patch(plugin_utils_path + "get_hue") def test_set_cluster_info(self, get_hue, get_cloudera_manager_info, ctx, cluster_update): hue = mock.Mock() hue.get_ip_or_dns_name.return_value = "1.2.3.4" get_hue.return_value = hue cluster = mock.Mock() self.vh._set_cluster_info(cluster) info = {'info': {'Hue Dashboard': {'Web UI': 'http://1.2.3.4:8888'}}} cluster_update.assert_called_once_with(ctx(), cluster, info) @mock.patch("sahara.plugins.utils.get_instance") @mock.patch("sahara.plugins.utils.get_config_value_or_default") @mock.patch("sahara.service.edp.job_utils.get_plugin") def test_get_edp_engine(self, get_plugin, get_config_value_or_default, get_instance): cluster = mock.Mock() job_type = 'Java' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsInstance(ret, edp_engine.EdpOozieEngine) job_type = 'Spark' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsInstance(ret, edp_engine.EdpSparkEngine) job_type = 'unsupported' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsNone(ret) def test_get_edp_job_types(self): ret = self.vh.get_edp_job_types() expect = edp_engine.EdpOozieEngine.get_supported_job_types() + \ edp_engine.EdpSparkEngine.get_supported_job_types() self.assertEqual(expect, ret) @mock.patch(plugin_path + "edp_engine.EdpOozieEngine.get_possible_job_config", return_value={'job_config': {}}) def test_edp_config_hints(self, get_possible_job_config): job_type = mock.Mock() ret = self.vh.get_edp_config_hints(job_type) get_possible_job_config.assert_called_once_with(job_type) self.assertEqual(ret, {'job_config': {}}) @mock.patch(plugin_path + "deploy.get_open_ports", return_value=[1234]) def test_get_open_ports(self, get_open_ports): node_group = mock.Mock() ret = self.vh.get_open_ports(node_group) get_open_ports.assert_called_once_with(node_group) self.assertEqual(ret, [1234]) @mock.patch(plugin_utils_path + "recommend_configs") def test_recommend_configs(self, recommend_configs): cluster = mock.Mock() scaling = mock.Mock() self.vh.get_plugin_configs = mock.Mock() self.vh.recommend_configs(cluster, scaling) recommend_configs.assert_called_once_with(cluster, self.vh.get_plugin_configs(), scaling) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_config_helper.py0000666000175100017510000000167513245514472026112 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_11_0 import config_helper from sahara.tests.unit.plugins.cdh import base_config_helper_test as bcht class TestConfigHelperV5110(bcht.TestConfigHelper): def setUp(self): super(TestConfigHelperV5110, self).setUp() self.c_h = config_helper.ConfigHelperV5110() self.path_to_config = 'plugins/cdh/v5_11_0/resources/' sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_validation.py0000666000175100017510000000156113245514472025432 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corpration # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_11_0 import validation from sahara.tests.unit.plugins.cdh import base_validation_tests as bvt class ValidationTestCase(bvt.BaseValidationTestCase): def setUp(self): super(ValidationTestCase, self).setUp() self.module = validation.ValidatorV5110 sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/__init__.py0000666000175100017510000000000013245514472023763 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_spark_engine.py0000666000175100017510000000456713245514472025756 0ustar zuulzuul00000000000000# Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara import conductor as cond from sahara import context from sahara.plugins import base as pb from sahara.plugins import exceptions as ex from sahara.service.edp.spark import engine from sahara.tests.unit import base from sahara.utils import edp conductor = cond.API class SparkPluginTest(base.SaharaWithDbTestCase): def setUp(self): super(SparkPluginTest, self).setUp() self.override_config("plugins", ["cdh"]) pb.setup_plugins() def test_plugin_edp_engine_no_spark(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'cdh', 'hadoop_version': '5.11.0', 'default_image_id': 'image'} job = mock.Mock() job.type = edp.JOB_TYPE_SPARK cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) edp_engine = plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK) with testtools.ExpectedException( ex.InvalidComponentCountException, value_re="Hadoop cluster should contain 1 " "SPARK_YARN_HISTORY_SERVER component\(s\). Actual " "SPARK_YARN_HISTORY_SERVER count is 0\nError ID: .*"): edp_engine.validate_job_execution(cluster, job, mock.Mock()) def test_plugin_edp_engine(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'cdh', 'hadoop_version': '5.11.0', 'default_image_id': 'image'} cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) self.assertIsInstance( plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK), engine.SparkJobEngine) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_edp_engine.py0000666000175100017510000001576413245514472025407 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins import base as pb from sahara.plugins.cdh.v5_11_0 import edp_engine from sahara.plugins import exceptions as pl_ex from sahara.tests.unit import base as sahara_base from sahara.tests.unit.plugins.cdh import utils as c_u from sahara.utils import edp def get_cluster(version='5.11.0'): cluster = c_u.get_fake_cluster(plugin_name='CDH', hadoop_version=version) return cluster class EdpEngineTestV5110(sahara_base.SaharaTestCase): def setUp(self): super(EdpEngineTestV5110, self).setUp() pb.setup_plugins() def test_get_hdfs_user(self): eng = edp_engine.EdpOozieEngine(get_cluster()) self.assertEqual('hdfs', eng.get_hdfs_user()) @mock.patch('sahara.service.edp.hdfs_helper.create_dir_hadoop2') def test_create_hdfs_dir(self, create_dir_hadoop2): eng = edp_engine.EdpOozieEngine(get_cluster()) remote = mock.Mock() dir_name = mock.Mock() eng.create_hdfs_dir(remote, dir_name) create_dir_hadoop2.assert_called_once_with(remote, dir_name, eng.get_hdfs_user()) def test_get_oozie_server_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_oozie_server_uri(cluster) self.assertEqual("http://1.2.3.5:11000/oozie", uri) def test_get_name_node_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_name_node_uri(cluster) self.assertEqual("hdfs://master_inst.novalocal:8020", uri) # has HDFS_JOURNALNODE cluster = get_cluster() jns_node_group = mock.MagicMock() jns_node_group.node_processes = ['HDFS_JOURNALNODE'] jns_node_group.instances = [mock.Mock()] list.append(cluster.node_groups, jns_node_group) uri = eng.get_name_node_uri(cluster) self.assertEqual("hdfs://nameservice01", uri) def test_get_resource_manager_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_resource_manager_uri(cluster) self.assertEqual("master_inst.novalocal:8032", uri) def test_get_oozie_server(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) actual = eng.get_oozie_server(cluster) expect = cluster.node_groups[1].instances[0] self.assertEqual(expect, actual) @mock.patch('sahara.service.edp.oozie.engine.' 'OozieJobEngine.validate_job_execution') def test_validate_job_execution(self, c): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) eng.validate_job_execution(cluster, mock.Mock(), mock.Mock()) # more than one oozie server dict.__setitem__(cluster.node_groups[1], 'count', 2) self.assertRaises(pl_ex.InvalidComponentCountException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) @mock.patch( 'sahara.plugins.cdh.confighints_helper.get_possible_hive_config_from', return_value={}) def test_get_possible_job_config_hive(self, get_possible_hive_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_HIVE) get_possible_hive_config_from.assert_called_once_with( 'plugins/cdh/v5_11_0/resources/hive-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.cdh.v5_11_0.edp_engine.EdpOozieEngine') def test_get_possible_job_config_java(self, BaseCDHEdpOozieEngine): expected_config = {'job_config': {}} BaseCDHEdpOozieEngine.get_possible_job_config.return_value = ( expected_config) actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_JAVA) BaseCDHEdpOozieEngine.get_possible_job_config.assert_called_once_with( edp.JOB_TYPE_JAVA) self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.' 'get_possible_mapreduce_config_from', return_value={}) def test_get_possible_job_config_mapreduce( self, get_possible_mapreduce_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_MAPREDUCE) get_possible_mapreduce_config_from.assert_called_once_with( 'plugins/cdh/v5_11_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.' 'get_possible_mapreduce_config_from', return_value={}) def test_get_possible_job_config_mapreduce_streaming( self, get_possible_mapreduce_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_MAPREDUCE_STREAMING) get_possible_mapreduce_config_from.assert_called_once_with( 'plugins/cdh/v5_11_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.get_possible_pig_config_from', return_value={}) def test_get_possible_job_config_pig(self, get_possible_pig_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_PIG) get_possible_pig_config_from.assert_called_once_with( 'plugins/cdh/v5_11_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.cdh.v5_11_0.edp_engine.EdpOozieEngine') def test_get_possible_job_config_shell(self, BaseCDHEdpOozieEngine): expected_config = {'job_config': {}} BaseCDHEdpOozieEngine.get_possible_job_config.return_value = ( expected_config) actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_SHELL) BaseCDHEdpOozieEngine.get_possible_job_config.assert_called_once_with( edp.JOB_TYPE_SHELL) self.assertEqual(expected_config, actual_config) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_deploy.py0000666000175100017510000002364113245514472024577 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.cdh.v5_11_0 import deploy from sahara.tests.unit import base class DeployCDHV5110(base.SaharaTestCase): def setUp(self): super(DeployCDHV5110, self).setUp() self.master = mock.MagicMock() self.master.node_group.node_processes = [ "HDFS_NAMENODE", "YARN_RESOURCEMANAGER", "CLOUDERA_MANAGER", "SENTRY_SERVER", "YARN_NODEMANAGER", "ZOOKEEPER_SERVER", "OOZIE_SERVER", "YARN_JOBHISTORY", "HDFS_SECONDARYNAMENODE", "HIVE_METASTORE", "HIVE_SERVER2", "SPARK_YARN_HISTORY_SERVER", "HBASE_MASTER", "HBASE_REGIONSERVER", "HUE_SERVER", "KMS", "FLUME_AGENT", "SOLR_SERVER", "SQOOP_SERVER", "IMPALA_STATESTORE", "IMPALA_CATALOGSERVER", "IMPALAD", "KEY_VALUE_STORE_INDEXER", ] self.worker = mock.MagicMock() self.worker.node_group.node_processes = [ "HDFS_DATANODE", "HDFS_JOURNALNODE", "JOURNALNODE", "YARN_NODEMANAGER", "YARN_STANDBYRM", ] self.instances = [self.master, self.worker] self.cluster = mock.MagicMock() self.is_cdh_exists = mock.patch( "sahara.plugins.cdh.commands.is_pre_installed_cdh", return_value=False) self.is_cdh_exists.start() self._create_facade = mock.patch( "sahara.db.sqlalchemy.api._create_facade_lazily") self._create_facade.start() def tearDown(self): self.is_cdh_exists.stop() self._create_facade.stop() super(DeployCDHV5110, self).tearDown() @mock.patch("sahara.plugins.utils.get_instances") @mock.patch("sahara.plugins.cdh.v5_11_0.deploy.CU") def test_configure_cluster(self, mock_cu, mock_get_instances): mock_get_instances.return_value = self.instances deploy.configure_cluster(self.cluster) mock_cu.pu.configure_os.assert_called_once_with(self.instances) mock_cu.pu.install_packages.assert_called_once_with(self.instances, deploy.PACKAGES) mock_cu.pu.start_cloudera_agents.assert_called_once_with( self.instances) mock_cu.pu.start_cloudera_manager.assert_called_once_with(self.cluster) mock_cu.update_cloudera_password.assert_called_once_with(self.cluster) mock_cu.await_agents.assert_called_once_with(self.cluster, self.instances) mock_cu.create_mgmt_service.assert_called_once_with(self.cluster) mock_cu.create_services.assert_called_once_with(self.cluster) mock_cu.configure_services.assert_called_once_with(self.cluster) mock_cu.configure_instances.assert_called_once_with(self.instances, self.cluster) mock_cu.deploy_configs.assert_called_once_with(self.cluster) @mock.patch("sahara.plugins.cdh.v5_11_0.deploy.CU") def test__start_roles(self, mock_cu): deploy._start_roles(self.cluster, self.instances) mock_cu.get_service_by_role.assert_any_call('DATANODE', instance=self.worker) mock_cu.get_service_by_role.assert_any_call('NODEMANAGER', instance=self.master) mock_cu.get_service_by_role.assert_any_call('NODEMANAGER', instance=self.worker) self.assertEqual(mock_cu.start_roles.call_count, 3) @mock.patch("sahara.plugins.cdh.v5_11_0.deploy._start_roles") @mock.patch("sahara.plugins.cdh.v5_11_0.deploy.CU") def test_scale_cluster(self, mock_cu, mock__start_roles): deploy.scale_cluster(self.cluster, None) self.assertEqual(mock_cu.call_count, 0) self.assertEqual(mock__start_roles.call_count, 0) deploy.scale_cluster(self.cluster, self.instances) mock_cu.pu.configure_os.assert_called_once_with(self.instances) mock_cu.pu.install_packages.assert_called_once_with(self.instances, deploy.PACKAGES) mock_cu.pu.start_cloudera_agents.assert_called_once_with( self.instances) mock_cu.await_agents.assert_called_once_with(self.cluster, self.instances) mock_cu.configure_instances.assert_called_once_with(self.instances, self.cluster) mock_cu.update_configs.assert_called_once_with(self.instances) mock_cu.pu.configure_swift.assert_called_once_with(self.cluster, self.instances) mock_cu.refresh_datanodes.assert_called_once_with(self.cluster) mock__start_roles.assert_called_once_with(self.cluster, self.instances) @mock.patch("sahara.plugins.cdh.v5_11_0.deploy.CU") def test_decommission_cluster(self, mock_cu): deploy.decommission_cluster(self.cluster, self.instances) dns = [] dns_2 = [] nms = [] nms_2 = [] for i in self.instances: if 'HDFS_DATANODE' in i.node_group.node_processes: dns.append(mock_cu.pu.get_role_name(i, 'DATANODE')) dns_2.append(mock_cu.pu.get_role_name(i, 'HDFS_GATEWAY')) if 'YARN_NODEMANAGER' in i.node_group.node_processes: nms.append(mock_cu.pu.get_role_name(i, 'NODEMANAGER')) nms_2.append(mock_cu.pu.get_role_name(i, 'YARN_GATEWAY')) mock_cu.decommission_nodes.assert_any_call( self.cluster, 'DATANODE', dns, dns_2) mock_cu.decommission_nodes.assert_any_call( self.cluster, 'NODEMANAGER', nms, nms_2) mock_cu.delete_instances.assert_called_once_with(self.cluster, self.instances) mock_cu.refresh_datanodes.assert_called_once_with(self.cluster) mock_cu.refresh_yarn_nodes.assert_called_once_with(self.cluster) @mock.patch("sahara.plugins.cdh.v5_11_0.deploy.CU") def test__prepare_cluster(self, mock_cu): deploy._prepare_cluster(self.cluster) mock_cu.pu.install_extjs.assert_called_once_with(self.cluster) mock_cu.pu.configure_hive.assert_called_once_with(self.cluster) mock_cu.pu.configure_sentry.assert_called_once_with(self.cluster) @mock.patch("sahara.service.edp.hdfs_helper.create_hbase_common_lib") @mock.patch("sahara.plugins.cdh.v5_11_0.deploy.CU") def test__finish_cluster_starting(self, mock_cu, mock_create_hbase): deploy._finish_cluster_starting(self.cluster) mock_cu.pu.put_hive_hdfs_xml.assert_called_once_with(self.cluster) self.assertTrue(mock_create_hbase.called) mock_cu.start_service.assert_called_once_with( mock_cu.get_service_by_role('AGENT', self.cluster)) @mock.patch("sahara.plugins.cdh.v5_11_0.deploy._finish_cluster_starting") @mock.patch("sahara.plugins.cdh.v5_11_0.deploy._prepare_cluster") @mock.patch("sahara.plugins.cdh.v5_11_0.deploy.CU") def test_start_cluster(self, mock_cu, mock_prepare, mock_finish): jns_count = 0 for i in self.instances: if "HDFS_JOURNALNODE" in i.node_group.node_processes: jns_count += 1 mock_cu.pu.get_jns.return_value.__len__.return_value = jns_count deploy.start_cluster(self.cluster) mock_prepare.assert_called_once_with(self.cluster) mock_cu.first_run.assert_called_once_with(self.cluster) mock_cu.pu.configure_swift.assert_called_once_with(self.cluster) if jns_count > 0: mock_cu.enable_namenode_ha.assert_called_once_with(self.cluster) mock_cu.update_role_config.assert_any_call( mock_cu.pu.get_secondarynamenode(self.cluster), 'HDFS_NAMENODE' ) mock_cu.enable_resourcemanager_ha.assert_called_once_with(self.cluster) mock_cu.update_role_config.assert_any_call( mock_cu.pu.get_stdb_rm(self.cluster), 'YARN_STANDBYRM') mock_finish.assert_called_once_with(self.cluster) def test_get_open_ports(self): master_ports = [ 9000, 7180, 7182, 7183, 7432, 7184, 8084, 8086, 10101, 9997, 9996, 8087, 9998, 9999, 8085, 9995, 9994, 8020, 8022, 50070, 50470, 50090, 50495, 8030, 8031, 8032, 8033, 8088, 8040, 8041, 8042, 10020, 19888, 9083, 10000, 8888, 11000, 11001, 18088, 2181, 3181, 4181, 9010, 60000, 60020, 41414, 8038, 8983, 8984, 8005, 12000, 25020, 26000, 25010, 24000, 21050, 21000, 23000, 25000, 28000, 22000, 16000, 16001 ] deploy.get_open_ports(self.master.node_group) self.assertItemsEqual(master_ports, deploy.get_open_ports(self.master.node_group)) worker_ports = [ 9000, 50010, 1004, 50075, 1006, 50020, 8480, 8481, 8485, 8040, 8041, 8042, 8030, 8031, 8032, 8033, 8088 ] self.assertItemsEqual(worker_ports, deploy.get_open_ports(self.worker.node_group)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_plugin_utils.py0000666000175100017510000000166213245514472026020 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_11_0 import plugin_utils as pu from sahara.tests.unit.plugins.cdh import base_plugin_utils_test class TestPluginUtilsV5110(base_plugin_utils_test.TestPluginUtilsHigherThanV5): def setUp(self): super(TestPluginUtilsV5110, self).setUp() self.plug_utils = pu.PluginUtilsV5110() self.version = "v5_11_0" sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_11_0/test_config_helper_5110.py0000666000175100017510000001407513245514472026556 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json from sahara.plugins.cdh.v5_11_0 import config_helper from sahara.tests.unit import base from sahara.tests.unit.plugins.cdh import utils as ctu from sahara.utils import files as f c_h = config_helper.ConfigHelperV5110() path_to_config = 'plugins/cdh/v5_11_0/resources/' json_files = [ 'hdfs-service.json', 'hdfs-namenode.json', 'hdfs-datanode.json', 'hdfs-secondarynamenode.json', 'hdfs-gateway.json', 'hdfs-journalnode.json', 'yarn-service.json', 'yarn-resourcemanager.json', 'yarn-nodemanager.json', 'yarn-jobhistory.json', 'yarn-gateway.json', 'oozie-service.json', 'oozie-oozie_server.json', 'hive-service.json', 'hive-hivemetastore.json', 'hive-hiveserver2.json', 'hive-webhcat.json', 'hue-service.json', 'hue-hue_server.json', 'spark-service.json', 'spark-spark_yarn_history_server.json', 'zookeeper-service.json', 'zookeeper-server.json', 'hbase-service.json', 'hbase-master.json', 'hbase-regionserver.json', 'flume-service.json', 'flume-agent.json', 'sentry-service.json', 'sentry-sentry_server.json', 'solr-service.json', 'solr-solr_server.json', 'sqoop-service.json', 'sqoop-sqoop_server.json', 'ks_indexer-service.json', 'ks_indexer-hbase_indexer.json', 'impala-service.json', 'impala-catalogserver.json', 'impala-impalad.json', 'impala-statestore.json', 'kms-service.json', 'kms-kms.json', 'kafka-kafka_broker.json', 'kafka-kafka_mirror_maker.json', 'kafka-service.json' ] class ConfigHelperTestCase(base.SaharaTestCase): def test_get_ng_plugin_configs(self): actual_configs = c_h._get_ng_plugin_configs() expected_configs = [] for json_file in json_files: expected_configs += json.loads( f.get_file_text(path_to_config + json_file)) # compare names expected_names = set(i['name'] for i in expected_configs) actual_names = set(i.to_dict()['name'] for i in actual_configs) self.assertEqual(expected_names, actual_names) def test_get_cdh5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CDH5_REPO_URL.default_value, c_h.get_cdh5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CDH5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cdh5_repo_url(cluster)) def test_get_cdh5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CDH5_REPO_KEY_URL.default_value, c_h.get_cdh5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CDH5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cdh5_key_url(cluster)) def test_get_cm5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CM5_REPO_URL.default_value, c_h.get_cm5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CM5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cm5_repo_url(cluster)) def test_get_cm5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CM5_REPO_KEY_URL.default_value, c_h.get_cm5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CM5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cm5_key_url(cluster)) def test_is_swift_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(c_h.is_swift_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.ENABLE_SWIFT.name: False}}) self.assertFalse(c_h.is_swift_enabled(cluster)) def test_get_swift_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.DEFAULT_SWIFT_LIB_URL, c_h.get_swift_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.SWIFT_LIB_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_swift_lib_url(cluster)) def test_is_hbase_common_lib_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(c_h.is_hbase_common_lib_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.ENABLE_HBASE_COMMON_LIB.name: False}}) self.assertFalse(c_h.is_hbase_common_lib_enabled(cluster)) def test_get_extjs_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.DEFAULT_EXTJS_LIB_URL, c_h.get_extjs_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.EXTJS_LIB_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_extjs_lib_url(cluster)) def test_get_kms_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.KMS_REPO_KEY_URL.default_value, c_h.get_kms_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.KMS_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_kms_key_url(cluster)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/base_validation_tests.py0000666000175100017510000003370213245514472025537 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corpration # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import itertools import six from sahara.plugins import exceptions as ex from sahara.tests.unit import base from sahara.tests.unit import testutils as tu ivse = ex.InvalidVolumeSizeException icce = ex.InvalidComponentCountException rsme = ex.RequiredServiceMissingException icte = ex.InvalidClusterTopology nnce = ex.NameNodeHAConfigurationError rmce = ex.ResourceManagerHAConfigurationError def make_ng_dict_with_inst(counter, name, flavor, processes, count, instances=None, volumes_size=None, node_configs=None, **kwargs): if not instances: instances = [] for i in range(count): n = six.next(counter) instance = tu.make_inst_dict("id{0}".format(n), "fake_inst{0}".format(n), management_ip='1.2.3.{0}'.format(n)) instances.append(instance) return tu.make_ng_dict(name, flavor, processes, count, instances, volumes_size, node_configs, **kwargs) def get_fake_cluster_with_process(processes=None, provided_ng_list=None, **kwargs): processes = processes or {} provided_ng_list = provided_ng_list or [] inst_counter = itertools.count(start=0) ng_counter = itertools.count(start=0) ng_id_counter = itertools.count(start=0) # default mng_ng = ('manager_ng', 1, ['CLOUDERA_MANAGER'], 1) mst_ng = ('master_ng', 1, ['HDFS_NAMENODE', 'HDFS_SECONDARYNAMENODE', 'YARN_RESOURCEMANAGER', 'YARN_JOBHISTORY', ], 1) wkrs_ng = ('worker_ng', 1, ['HDFS_DATANODE', 'YARN_NODEMANAGER'], 3) basic_ng_list = [mng_ng, mst_ng, wkrs_ng] # if in default_ng_list, change it if 'CLOUDERA_MANAGER' in processes: if processes['CLOUDERA_MANAGER'] == 0: basic_ng_list.remove(mng_ng) else: processes['CLOUDERA_MANAGER'] -= 1 for process in mst_ng[2]: if process in processes: if processes[process] == 0: mst_ng[2].remove(process) else: processes[process] -= 1 # only consider processes set to 0 for process in wkrs_ng[2]: if process in processes: if processes[process] == 0: wkrs_ng[2].remove(process) other_ng_list = [] for process, count in six.iteritems(processes): if count: ng = ('service_ng{0}'.format(six.next(ng_counter)), 1, [process], count) other_ng_list.append(ng) ng_list = basic_ng_list + other_ng_list + provided_ng_list ng_dict_list = [make_ng_dict_with_inst( inst_counter, *args, id="ng_id{0}".format(six.next(ng_id_counter))) for args in ng_list] return tu.create_cluster('test_cluster', 1, 'cdh', '5', ng_dict_list, **kwargs) class BaseValidationTestCase(base.SaharaTestCase): def _test_cluster_validation(self, exception, processes, provided_ng_list=None, kwargs_dict=None): provided_ng_list = provided_ng_list or [] kwargs_dict = kwargs_dict or {} cluster = get_fake_cluster_with_process(processes, provided_ng_list, **kwargs_dict) if exception: self.assertRaises(exception, self.module.validate_cluster_creating, cluster) else: self.module.validate_cluster_creating(cluster) def setUp(self): super(BaseValidationTestCase, self).setUp() self.module = None def _get_test_cases(self): """return cases with [exception expected, progresses, configs].""" cases = [ [icce, {'CLOUDERA_MANAGER': 0}], [None, {'CLOUDERA_MANAGER': 1}], [icce, {'CLOUDERA_MANAGER': 2}], [icce, {'HDFS_NAMENODE': 0}], [None, {'HDFS_NAMENODE': 1}], [icce, {'HDFS_NAMENODE': 2}], [icce, {'HDFS_SECONDARYNAMENODE': 0}], [None, {'HDFS_SECONDARYNAMENODE': 1}], [icce, {'HDFS_SECONDARYNAMENODE': 2}], [None, {}, [], {'cluster_configs': {'HDFS': {'dfs_replication': 2}}}], [None, {}, [], {'cluster_configs': {'HDFS': {'dfs_replication': 3}}}], [icce, {}, [], {'cluster_configs': {'HDFS': {'dfs_replication': 4}}}], [ivse, {}, [( 'worker_ng_vol', 1, ['HDFS_DATANODE', 'YARN_NODEMANAGER'], 3, None, 20, {'DATANODE': {'dfs_datanode_du_reserved': 22548578304}})]], [None, {}, [( 'worker_ng_vol', 1, ['HDFS_DATANODE', 'YARN_NODEMANAGER'], 3, None, 22, {'DATANODE': {'dfs_datanode_du_reserved': 22548578304}})]], [None, {'YARN_RESOURCEMANAGER': 1}], [icce, {'YARN_RESOURCEMANAGER': 2}], [None, {'YARN_JOBHISTORY': 1}], [icce, {'YARN_JOBHISTORY': 2}], [rsme, {'YARN_JOBHISTORY': 0, 'YARN_RESOURCEMANAGER': 1}], [rsme, {'YARN_RESOURCEMANAGER': 0, 'YARN_NODEMANAGER': 3}], [None, {'YARN_RESOURCEMANAGER': 0, 'YARN_NODEMANAGER': 0}], [None, {'OOZIE_SERVER': 1}], [icce, {'OOZIE_SERVER': 2}], [rsme, {'YARN_NODEMANAGER': 0, 'OOZIE_SERVER': 1}], [rsme, {'YARN_JOBHISTORY': 0, 'OOZIE_SERVER': 1}], [rsme, {'HIVE_SERVER2': 0, 'HIVE_METASTORE': 1}], [rsme, {'HIVE_METASTORE': 0, 'HIVE_SERVER2': 1}], [rsme, {'HIVE_METASTORE': 0, 'HIVE_SERVER2': 0, 'HIVE_WEBHCAT': 1}], [None, {'HUE_SERVER': 1, 'OOZIE_SERVER': 1, 'HIVE_METASTORE': 1, 'HIVE_SERVER2': 1}], [icce, {'HUE_SERVER': 2, 'OOZIE_SERVER': 1, 'HIVE_METASTORE': 1, 'HIVE_SERVER2': 1}], [rsme, {'HUE_SERVER': 1, 'OOZIE_SERVER': 0}], [rsme, {'HUE_SERVER': 1, 'OOZIE_SERVER': 1, 'HIVE_METASTORE': 0}], [None, {'SPARK_YARN_HISTORY_SERVER': 0}], [None, {'SPARK_YARN_HISTORY_SERVER': 1}], [icce, {'SPARK_YARN_HISTORY_SERVER': 2}], [rsme, {'HBASE_MASTER': 1, 'ZOOKEEPER_SERVER': 0}], [icce, {'HBASE_MASTER': 1, 'ZOOKEEPER_SERVER': 1, 'HBASE_REGIONSERVER': 0}], [None, {'HBASE_MASTER': 1, 'ZOOKEEPER_SERVER': 1, 'HBASE_REGIONSERVER': 1}], [icce, {'HBASE_MASTER': 0, 'HBASE_REGIONSERVER': 1}], [None, {}] ] disable_anti_affinity = {'cluster_configs': {'general': { 'Require Anti Affinity': False}}} cases += [ [None, {'HDFS_JOURNALNODE': 3, 'ZOOKEEPER_SERVER': 1}, [], disable_anti_affinity], [rsme, {'HDFS_JOURNALNODE': 3, 'ZOOKEEPER_SERVER': 0}, [], disable_anti_affinity], [icce, {'HDFS_JOURNALNODE': 4, 'ZOOKEEPER_SERVER': 1}, [], disable_anti_affinity], [icce, {'HDFS_JOURNALNODE': 2, 'ZOOKEEPER_SERVER': 1}, [], disable_anti_affinity], [nnce, {'HDFS_JOURNALNODE': 3, 'ZOOKEEPER_SERVER': 1}, [], {'anti_affinity': ['HDFS_SECONDARYNAMENODE']}], [nnce, {'HDFS_JOURNALNODE': 3, 'ZOOKEEPER_SERVER': 1}, [], {'anti_affinity': ['HDFS_NAMENODE']}], [None, {'HDFS_JOURNALNODE': 3, 'ZOOKEEPER_SERVER': 1}, [], {'anti_affinity': ['HDFS_NAMENODE', 'HDFS_SECONDARYNAMENODE']}], [None, {'YARN_STANDBYRM': 1, 'ZOOKEEPER_SERVER': 1}, [], disable_anti_affinity], [icce, {'YARN_STANDBYRM': 2, 'ZOOKEEPER_SERVER': 1}, [], disable_anti_affinity], [rsme, {'YARN_STANDBYRM': 1, 'ZOOKEEPER_SERVER': 0}, [], disable_anti_affinity], [rmce, {'YARN_STANDBYRM': 1, 'ZOOKEEPER_SERVER': 1}, [], {'anti_affinity': ['YARN_RESOURCEMANAGER']}], [rmce, {'YARN_STANDBYRM': 1, 'ZOOKEEPER_SERVER': 1}, [], {'anti_affinity': ['YARN_STANDBYRM']}], [None, {'YARN_STANDBYRM': 1, 'ZOOKEEPER_SERVER': 1}, [], {'anti_affinity': ['YARN_STANDBYRM', 'YARN_RESOURCEMANAGER']}], ] cases += [ [None, {'FLUME_AGENT': 1}], [icce, {'ZOOKEEPER_SERVER': 1, 'SENTRY_SERVER': 2}], [None, {'ZOOKEEPER_SERVER': 1, 'SENTRY_SERVER': 1}], [rsme, {'ZOOKEEPER_SERVER': 0, 'SENTRY_SERVER': 1}], [None, {'ZOOKEEPER_SERVER': 1, 'SOLR_SERVER': 1}], [rsme, {'ZOOKEEPER_SERVER': 0, 'SOLR_SERVER': 1}], [None, {'YARN_NODEMANAGER': 1, 'YARN_JOBHISTORY': 1, 'SQOOP_SERVER': 1}], [rsme, {'YARN_NODEMANAGER': 0, 'YARN_JOBHISTORY': 1, 'SQOOP_SERVER': 1}], [rsme, {'YARN_NODEMANAGER': 1, 'YARN_JOBHISTORY': 0, 'SQOOP_SERVER': 1}], # HBASE_MASTER AND HBASE_REGIONSERVER depend circularly [None, {'ZOOKEEPER_SERVER': 1, 'SOLR_SERVER': 1, 'HBASE_MASTER': 1, 'HBASE_INDEXER': 1, 'HBASE_REGIONSERVER': 1}], [rsme, {'ZOOKEEPER_SERVER': 0, 'SOLR_SERVER': 1, 'HBASE_MASTER': 1, 'HBASE_INDEXER': 1, 'HBASE_REGIONSERVER': 1}], [rsme, {'ZOOKEEPER_SERVER': 1, 'SOLR_SERVER': 0, 'HBASE_MASTER': 1, 'HBASE_INDEXER': 1, 'HBASE_REGIONSERVER': 1}], [rsme, {'ZOOKEEPER_SERVER': 1, 'SOLR_SERVER': 1, 'HBASE_MASTER': 0, 'HBASE_INDEXER': 1}], ] worker_with_implama = ('worker_ng', 1, ['HDFS_DATANODE', 'YARN_NODEMANAGER', 'IMPALAD'], 3) cases += [ [None, {'IMPALA_CATALOGSERVER': 1, 'IMPALA_STATESTORE': 1, 'HIVE_METASTORE': 1, 'HIVE_SERVER2': 1, 'HDFS_DATANODE': 0, 'YARN_NODEMANAGER': 0}, [worker_with_implama]], [icte, {'IMPALA_CATALOGSERVER': 1, 'IMPALA_STATESTORE': 1, 'HIVE_METASTORE': 1, 'HIVE_SERVER2': 1}, []], [icte, {'IMPALA_CATALOGSERVER': 1, 'IMPALA_STATESTORE': 1, 'HIVE_METASTORE': 1, 'HIVE_SERVER2': 1}, [worker_with_implama]], [rsme, {'IMPALA_CATALOGSERVER': 1, 'IMPALA_STATESTORE': 0, 'HIVE_METASTORE': 1, 'HIVE_SERVER2': 1, 'HDFS_DATANODE': 0, 'YARN_NODEMANAGER': 0}, [worker_with_implama]], [rsme, {'IMPALA_CATALOGSERVER': 1, 'IMPALA_STATESTORE': 1, 'HIVE_METASTORE': 0, 'HIVE_SERVER2': 1, 'HDFS_DATANODE': 0, 'YARN_NODEMANAGER': 0}, [worker_with_implama]] ] cases += [ [None, {'KMS': 1}], [icce, {'KMS': 2}] ] return cases def test_validate_cluster_creating(self): cases = self._get_test_cases() for case in cases: self._test_cluster_validation(*case) def test_validate_additional_ng_scaling(self): # valid scaling cluster = get_fake_cluster_with_process({}) additional = {cluster.node_groups[2].id: 4} self.module.validate_additional_ng_scaling(cluster, additional) # node processes unscalable cluster = get_fake_cluster_with_process({}) additional = {cluster.node_groups[0].id: 2} self.assertRaises(ex.NodeGroupCannotBeScaled, self.module.validate_additional_ng_scaling, cluster, additional) # scaling without resourcemanager cluster = get_fake_cluster_with_process({"YARN_RESOURCEMANAGER": 0}) additional = {cluster.node_groups[2].id: 4} self.assertRaises(ex.NodeGroupCannotBeScaled, self.module.validate_additional_ng_scaling, cluster, additional) def test_validate_existing_ng_scaling(self): # valid scaling, add datanodes cluster = get_fake_cluster_with_process({}) existing = {cluster.node_groups[2].id: 4} self.module.validate_existing_ng_scaling(cluster, existing) # node processes unscalable cluster = get_fake_cluster_with_process({}) existing = {cluster.node_groups[0].id: 2} self.assertRaises(ex.NodeGroupCannotBeScaled, self.module.validate_existing_ng_scaling, cluster, existing) # datanode count less than replicas cfg = {'HDFS': {'dfs_replication': 3}} cluster = get_fake_cluster_with_process({}, cluster_configs=cfg) existing = {cluster.node_groups[2].id: 2} self.assertRaises(ex.ClusterCannotBeScaled, self.module.validate_existing_ng_scaling, cluster, existing) # datanode count more than replicas cfg = {'HDFS': {'dfs_replication': 2}} cluster = get_fake_cluster_with_process({}, cluster_configs=cfg) existing = {cluster.node_groups[2].id: 2} self.module.validate_existing_ng_scaling(cluster, existing) sahara-8.0.0/sahara/tests/unit/plugins/cdh/base_plugin_utils_test.py0000666000175100017510000003376413245514476025754 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.i18n import _ from sahara.tests.unit import base as b from sahara.tests.unit.plugins.cdh import utils as ctu from sahara.utils import files CONFIGURATION_SCHEMA = { 'node_configs': { 'yarn.scheduler.minimum-allocation-mb': ( 'RESOURCEMANAGER', 'yarn_scheduler_minimum_allocation_mb'), 'mapreduce.reduce.memory.mb': ( 'YARN_GATEWAY', 'mapreduce_reduce_memory_mb'), 'mapreduce.map.memory.mb': ( 'YARN_GATEWAY', 'mapreduce_map_memory_mb',), 'yarn.scheduler.maximum-allocation-mb': ( 'RESOURCEMANAGER', 'yarn_scheduler_maximum_allocation_mb'), 'yarn.app.mapreduce.am.command-opts': ( 'YARN_GATEWAY', 'yarn_app_mapreduce_am_command_opts'), 'yarn.nodemanager.resource.memory-mb': ( 'NODEMANAGER', 'yarn_nodemanager_resource_memory_mb'), 'mapreduce.task.io.sort.mb': ( 'YARN_GATEWAY', 'io_sort_mb'), 'mapreduce.map.java.opts': ( 'YARN_GATEWAY', 'mapreduce_map_java_opts'), 'mapreduce.reduce.java.opts': ( 'YARN_GATEWAY', 'mapreduce_reduce_java_opts'), 'yarn.app.mapreduce.am.resource.mb': ( 'YARN_GATEWAY', 'yarn_app_mapreduce_am_resource_mb') }, 'cluster_configs': { 'dfs.replication': ('HDFS', 'dfs_replication') } } def get_concrete_cluster(): cluster = ctu.get_fake_cluster() # add configs to cluster configs = {"SQOOP": {}, "HUE": {}, "general": {}, "KMS": {}, "HIVE": {}, "SOLR": {}, "FLUME": {}, "HDFS": {"dfs_replication": 1}, "KS_INDEXER": {}, "SPARK_ON_YARN": {}, "SENTRY": {}, "YARN": {}, "ZOOKEEPER": {}, "OOZIE": {}, "HBASE": {}, "IMPALA": {}} # cluster is immutable, a work around dict.__setitem__(cluster, "cluster_configs", configs) # add fake remotes to instances instances = [i for ng in cluster.node_groups for i in ng.instances] for i in instances: object.__setattr__(i, 'remote', mock.MagicMock()) # add cluster_id to each node group for ng in cluster.node_groups: dict.__setitem__(ng, "cluster_id", ng.cluster.id) # add extra config dict.__setitem__(cluster, "extra", {}) return cluster def get_fake_worker_instances(): ng = get_concrete_cluster().node_groups[2] return ng.instances class TestPluginUtils(b.SaharaTestCase): def setUp(self): super(TestPluginUtils, self).setUp() self.plug_utils = None @mock.patch('sahara.config.CONF.disable_event_log') @mock.patch('sahara.plugins.cdh.plugin_utils.' 'CDHPluginAutoConfigsProvider') def test_recommend_configs(self, provider, log_cfg): fake_plugin_utils = mock.Mock() fake_cluster = mock.Mock() self.plug_utils.recommend_configs( fake_cluster, fake_plugin_utils, False) self.assertEqual([mock.call(CONFIGURATION_SCHEMA, fake_plugin_utils, fake_cluster, False)], provider.call_args_list) @mock.patch('sahara.config.CONF.disable_event_log') @mock.patch('sahara.plugins.cdh.commands.install_packages') def test_install_packages(self, install_packages, log_cfg): packages = mock.Mock() instances = get_fake_worker_instances() self.plug_utils.install_packages(instances, packages) calls = [mock.call(i.remote().__enter__(), packages) for i in instances] install_packages.assert_has_calls(calls, any_order=False) @mock.patch('sahara.config.CONF.disable_event_log') @mock.patch('sahara.plugins.cdh.commands.start_agent') @mock.patch('sahara.plugins.cdh.commands.configure_agent') def test_start_cloudera_agents(self, configure_agent, start_agent, log_cfg): instances = get_fake_worker_instances() self.plug_utils.start_cloudera_agents(instances) cfg_calls = [mock.call(i.remote().__enter__(), 'manager_inst') for i in instances] start_calls = [mock.call(i.remote().__enter__()) for i in instances] configure_agent.assert_has_calls(cfg_calls, any_order=False) start_agent.assert_has_calls(start_calls, any_order=False) @mock.patch('sahara.config.CONF.disable_event_log') def test_put_hive_hdfs_xml(self, log_cfg): cluster = get_concrete_cluster() hive_server = cluster.node_groups[1].instances[0] self.plug_utils.put_hive_hdfs_xml(cluster) with hive_server.remote() as r: calls = [mock.call('sudo su - -c "hadoop fs -mkdir -p' ' /user/hdfs/conf" hdfs'), mock.call('sudo su - -c "hadoop fs -put' ' /etc/hive/conf/hive-site.xml' ' /user/hdfs/conf/hive-site.xml" hdfs')] r.execute_command.assert_has_calls(calls, any_order=False) @mock.patch('sahara.config.CONF.disable_event_log') def test_configure_swift(self, log_cfg): cluster = get_concrete_cluster() cluster.cluster_configs['general']['Enable Swift'] = True instances = [i for ng in cluster.node_groups for i in ng.instances] self.plug_utils.configure_swift(cluster) for i in instances: with i.remote() as r: cmd = r'ls /usr/lib/hadoop-mapreduce/hadoop-openstack.jar' # use any_call because the execute_command has a call: # call().__getitem__().__ne__(0) during the method r.execute_command.assert_any_call(cmd, raise_when_error=False) cmd = (r'sudo curl %s' r' -o /usr/lib/hadoop-mapreduce/hadoop-openstack.jar') cmd = cmd % self.plug_utils.c_helper.get_swift_lib_url(cluster) r.execute_command.call r.execute_command.assert_any_call(cmd) @mock.patch('sahara.config.CONF.disable_event_log') @mock.patch('uuid.uuid4') @mock.patch('sahara.conductor.API.cluster_update') @mock.patch('sahara.conductor.API.cluster_get') @mock.patch('castellan.key_manager.API') def test_configure_hive(self, keymanager, cluster_get, cluster_update, uuid4, log_cfg): cluster = get_concrete_cluster() manager = cluster.node_groups[0].instances[0] cluster_get.return_value = cluster db_password = 'a8f2939f-ff9f-4659-a333-abc012ee9b2d' uuid4.return_value = db_password create_db_script = files.get_file_text( 'plugins/cdh/db_resources/create_hive_db.sql' .format(version=self.version)) create_db_script = create_db_script % db_password self.plug_utils.configure_hive(cluster) with manager.remote() as r: cmd_exe_sql = ('PGPASSWORD=$(sudo head -1' ' /var/lib/cloudera-scm-server-db/data/' 'generated_password.txt) psql' ' -U cloudera-scm -h localhost -p 7432 -d scm -f' ' script_to_exec.sql') cmd_clean = 'rm script_to_exec.sql' self.assertEqual(create_db_script, r.write_file_to.call_args[0][1]) r.execute_command.assert_has_calls([mock.call(cmd_exe_sql), mock.call(cmd_clean)]) @mock.patch('sahara.config.CONF.disable_event_log') @mock.patch('sahara.plugins.cdh.commands.is_ubuntu_os') @mock.patch('sahara.plugins.cdh.commands.is_centos_os') @mock.patch('sahara.plugins.cdh.commands.update_repository') @mock.patch('sahara.plugins.cdh.commands.add_apt_key') @mock.patch('sahara.plugins.cdh.commands.write_ubuntu_repository') @mock.patch('sahara.plugins.cdh.commands.write_centos_repository') def test_configure_os(self, write_centos_repository, write_ubuntu_repository, add_apt_key, update_repository, is_centos_os, is_ubuntu_os, log_cfg): cluster = get_concrete_cluster() ubuntu_instance = cluster.node_groups[2].instances[0] centos_instance = cluster.node_groups[2].instances[1] instances = [ubuntu_instance, centos_instance] is_ubuntu_os.side_effect = \ lambda r: r is ubuntu_instance.remote().__enter__() is_centos_os.side_effect = \ lambda r: r is centos_instance.remote().__enter__() self.plug_utils.configure_os(instances) with ubuntu_instance.remote() as r: write_ubuntu_repository.assert_has_calls( [mock.call(r, self.plug_utils.c_helper.CDH5_UBUNTU_REPO, 'cdh'), mock.call(r, self.plug_utils.c_helper.CM5_UBUNTU_REPO, 'cm')], any_order=False) add_apt_key.assert_has_calls( [mock.call(r, self.plug_utils.c_helper. DEFAULT_CDH5_UBUNTU_REPO_KEY_URL), mock.call(r, self.plug_utils.c_helper. DEFAULT_CM5_UBUNTU_REPO_KEY_URL)], any_order=False) update_repository.assert_any_call(r) with centos_instance.remote() as r: write_centos_repository.assert_has_calls( [mock.call(r, self.plug_utils.c_helper.CDH5_CENTOS_REPO, 'cdh'), mock.call(r, self.plug_utils.c_helper.CM5_CENTOS_REPO, 'cm')], any_order=False) update_repository.assert_any_call(r) @mock.patch('sahara.config.CONF.disable_event_log') def test_install_extjs(self, log_cfg): cluster = get_concrete_cluster() oozie_server = cluster.node_groups[1].instances[0] self.plug_utils.install_extjs(cluster) with oozie_server.remote() as r: calls = [mock.call('ls /var/lib/oozie/ext-2.2', raise_when_error=False), mock.call("curl -L -o '/var/lib/oozie/extjs.zip'" " http://tarballs.openstack.org/sahara/dist/" "common-artifacts/ext-2.2.zip", run_as_root=True), mock.call('unzip /var/lib/oozie/extjs.zip' ' -d /var/lib/oozie', run_as_root=True)] r.execute_command.assert_has_calls(calls, any_order=True) @mock.patch('sahara.config.CONF.disable_event_log') @mock.patch('sahara.utils.poll_utils.plugin_option_poll') @mock.patch('sahara.plugins.cdh.commands.start_cloudera_db') @mock.patch('sahara.plugins.cdh.commands.start_manager') def test_start_cloudera_manager(self, start_manager, start_cloudera_db, plugin_option_poll, log_cfg): cluster = get_concrete_cluster() manager = cluster.node_groups[0].instances[0] self.plug_utils.start_cloudera_manager(cluster) with manager.remote() as r: start_manager.assert_called_once_with(r) start_cloudera_db.assert_called_once_with(r) call = [cluster, self.plug_utils._check_cloudera_manager_started, self.plug_utils.c_helper.AWAIT_MANAGER_STARTING_TIMEOUT, _("Await starting Cloudera Manager"), 2, {'manager': manager}] plugin_option_poll.assert_called_once_with(*call) def test_get_config_value(self): cluster = get_concrete_cluster() dfs_replication = self.plug_utils.get_config_value( 'HDFS', 'dfs_replication', cluster) self.assertEqual(1, dfs_replication) dfs_replication_default = self.plug_utils.get_config_value( 'HDFS', 'dfs_replication') self.assertEqual(3, dfs_replication_default) class TestPluginUtilsHigherThanV5(TestPluginUtils): @mock.patch('sahara.config.CONF.disable_event_log') @mock.patch('uuid.uuid4') @mock.patch('sahara.conductor.API.cluster_update') @mock.patch('sahara.conductor.API.cluster_get') @mock.patch('castellan.key_manager.API') def test_configure_sentry(self, keymanager, cluster_get, cluster_update, uuid4, cfg_log): cluster = get_concrete_cluster() manager = cluster.node_groups[0].instances[0] cluster_get.return_value = cluster db_password = 'a8f2939f-ff9f-4659-a333-abc012ee9b2d' uuid4.return_value = db_password create_db_script = files.get_file_text( 'plugins/cdh/db_resources/create_sentry_db.sql' .format(version=self.version)) create_db_script = create_db_script % db_password self.plug_utils.configure_sentry(cluster) with manager.remote() as r: cmd_exe_sql = ('PGPASSWORD=$(sudo head -1' ' /var/lib/cloudera-scm-server-db/data/' 'generated_password.txt) psql' ' -U cloudera-scm -h localhost -p 7432 -d scm -f' ' script_to_exec.sql') cmd_clean = 'rm script_to_exec.sql' self.assertEqual(create_db_script, r.write_file_to.call_args[0][1]) r.execute_command.assert_has_calls([mock.call(cmd_exe_sql), mock.call(cmd_clean)]) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/0000775000175100017510000000000013245515027021604 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_versionhandler.py0000666000175100017510000001333413245514472026251 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import six from sahara.plugins.cdh.v5_7_0 import edp_engine from sahara.plugins.cdh.v5_7_0 import versionhandler from sahara.tests.unit import base class VersionHandlerTestCase(base.SaharaTestCase): plugin_path = "sahara.plugins.cdh.v5_7_0." cloudera_utils_path = plugin_path + "cloudera_utils.ClouderaUtilsV570." plugin_utils_path = plugin_path + "plugin_utils.PluginUtilsV570." def setUp(self): super(VersionHandlerTestCase, self).setUp() self.vh = versionhandler.VersionHandler() def test_get_node_processes(self): processes = self.vh.get_node_processes() for k, v in six.iteritems(processes): for p in v: self.assertIsInstance(p, str) @mock.patch("sahara.conductor.API.cluster_update") @mock.patch("sahara.context.ctx") @mock.patch(plugin_path + "deploy.configure_cluster") @mock.patch(cloudera_utils_path + "get_cloudera_manager_info", return_value={"fake_cm_info": "fake"}) def test_config_cluster(self, get_cm_info, configure_cluster, ctx, cluster_update): cluster = mock.Mock() self.vh.configure_cluster(cluster) configure_cluster.assert_called_once_with(cluster) cluster_update.assert_called_once_with( ctx(), cluster, {'info': {"fake_cm_info": "fake"}}) @mock.patch(plugin_path + "deploy.start_cluster") def test_start_cluster(self, start_cluster): cluster = mock.Mock() self.vh._set_cluster_info = mock.Mock() self.vh.start_cluster(cluster) start_cluster.assert_called_once_with(cluster) self.vh._set_cluster_info.assert_called_once_with(cluster) @mock.patch(plugin_path + "deploy.decommission_cluster") def test_decommission_nodes(self, decommission_cluster): cluster = mock.Mock() instances = mock.Mock() self.vh.decommission_nodes(cluster, instances) decommission_cluster.assert_called_once_with(cluster, instances) @mock.patch(plugin_path + "deploy.scale_cluster") def test_scale_cluster(self, scale_cluster): cluster = mock.Mock() instances = mock.Mock() self.vh.scale_cluster(cluster, instances) scale_cluster.assert_called_once_with(cluster, instances) @mock.patch("sahara.conductor.API.cluster_update") @mock.patch("sahara.context.ctx") @mock.patch(cloudera_utils_path + "get_cloudera_manager_info", return_value={}) @mock.patch(plugin_utils_path + "get_hue") def test_set_cluster_info(self, get_hue, get_cloudera_manager_info, ctx, cluster_update): hue = mock.Mock() hue.get_ip_or_dns_name.return_value = "1.2.3.4" get_hue.return_value = hue cluster = mock.Mock() self.vh._set_cluster_info(cluster) info = {'info': {'Hue Dashboard': {'Web UI': 'http://1.2.3.4:8888'}}} cluster_update.assert_called_once_with(ctx(), cluster, info) @mock.patch("sahara.plugins.utils.get_instance") @mock.patch("sahara.plugins.utils.get_config_value_or_default") @mock.patch("sahara.service.edp.job_utils.get_plugin") def test_get_edp_engine(self, get_plugin, get_config_value_or_default, get_instance): cluster = mock.Mock() job_type = 'Java' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsInstance(ret, edp_engine.EdpOozieEngine) job_type = 'Spark' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsInstance(ret, edp_engine.EdpSparkEngine) job_type = 'unsupported' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsNone(ret) def test_get_edp_job_types(self): ret = self.vh.get_edp_job_types() expect = edp_engine.EdpOozieEngine.get_supported_job_types() + \ edp_engine.EdpSparkEngine.get_supported_job_types() self.assertEqual(expect, ret) @mock.patch(plugin_path + "edp_engine.EdpOozieEngine.get_possible_job_config", return_value={'job_config': {}}) def test_edp_config_hints(self, get_possible_job_config): job_type = mock.Mock() ret = self.vh.get_edp_config_hints(job_type) get_possible_job_config.assert_called_once_with(job_type) self.assertEqual(ret, {'job_config': {}}) @mock.patch(plugin_path + "deploy.get_open_ports", return_value=[1234]) def test_get_open_ports(self, get_open_ports): node_group = mock.Mock() ret = self.vh.get_open_ports(node_group) get_open_ports.assert_called_once_with(node_group) self.assertEqual(ret, [1234]) @mock.patch(plugin_utils_path + "recommend_configs") def test_recommend_configs(self, recommend_configs): cluster = mock.Mock() scaling = mock.Mock() self.vh.get_plugin_configs = mock.Mock() self.vh.recommend_configs(cluster, scaling) recommend_configs.assert_called_once_with(cluster, self.vh.get_plugin_configs(), scaling) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_config_helper.py0000666000175100017510000000167013245514472026032 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_7_0 import config_helper from sahara.tests.unit.plugins.cdh import base_config_helper_test as bcht class TestConfigHelperV570(bcht.TestConfigHelper): def setUp(self): super(TestConfigHelperV570, self).setUp() self.c_h = config_helper.ConfigHelperV570() self.path_to_config = 'plugins/cdh/v5_7_0/resources/' sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_validation.py0000666000175100017510000000155713245514472025364 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corpration # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_7_0 import validation from sahara.tests.unit.plugins.cdh import base_validation_tests as bvt class ValidationTestCase(bvt.BaseValidationTestCase): def setUp(self): super(ValidationTestCase, self).setUp() self.module = validation.ValidatorV570 sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/__init__.py0000666000175100017510000000000013245514472023710 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_config_helper_570.py0000666000175100017510000001407213245514472026425 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json from sahara.plugins.cdh.v5_7_0 import config_helper from sahara.tests.unit import base from sahara.tests.unit.plugins.cdh import utils as ctu from sahara.utils import files as f c_h = config_helper.ConfigHelperV570() path_to_config = 'plugins/cdh/v5_7_0/resources/' json_files = [ 'hdfs-service.json', 'hdfs-namenode.json', 'hdfs-datanode.json', 'hdfs-secondarynamenode.json', 'hdfs-gateway.json', 'hdfs-journalnode.json', 'yarn-service.json', 'yarn-resourcemanager.json', 'yarn-nodemanager.json', 'yarn-jobhistory.json', 'yarn-gateway.json', 'oozie-service.json', 'oozie-oozie_server.json', 'hive-service.json', 'hive-hivemetastore.json', 'hive-hiveserver2.json', 'hive-webhcat.json', 'hue-service.json', 'hue-hue_server.json', 'spark-service.json', 'spark-spark_yarn_history_server.json', 'zookeeper-service.json', 'zookeeper-server.json', 'hbase-service.json', 'hbase-master.json', 'hbase-regionserver.json', 'flume-service.json', 'flume-agent.json', 'sentry-service.json', 'sentry-sentry_server.json', 'solr-service.json', 'solr-solr_server.json', 'sqoop-service.json', 'sqoop-sqoop_server.json', 'ks_indexer-service.json', 'ks_indexer-hbase_indexer.json', 'impala-service.json', 'impala-catalogserver.json', 'impala-impalad.json', 'impala-statestore.json', 'kms-service.json', 'kms-kms.json', 'kafka-kafka_broker.json', 'kafka-kafka_mirror_maker.json', 'kafka-service.json' ] class ConfigHelperTestCase(base.SaharaTestCase): def test_get_ng_plugin_configs(self): actual_configs = c_h._get_ng_plugin_configs() expected_configs = [] for json_file in json_files: expected_configs += json.loads( f.get_file_text(path_to_config + json_file)) # compare names expected_names = set(i['name'] for i in expected_configs) actual_names = set(i.to_dict()['name'] for i in actual_configs) self.assertEqual(expected_names, actual_names) def test_get_cdh5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CDH5_REPO_URL.default_value, c_h.get_cdh5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CDH5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cdh5_repo_url(cluster)) def test_get_cdh5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CDH5_REPO_KEY_URL.default_value, c_h.get_cdh5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CDH5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cdh5_key_url(cluster)) def test_get_cm5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CM5_REPO_URL.default_value, c_h.get_cm5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CM5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cm5_repo_url(cluster)) def test_get_cm5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CM5_REPO_KEY_URL.default_value, c_h.get_cm5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CM5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cm5_key_url(cluster)) def test_is_swift_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(c_h.is_swift_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.ENABLE_SWIFT.name: False}}) self.assertFalse(c_h.is_swift_enabled(cluster)) def test_get_swift_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.DEFAULT_SWIFT_LIB_URL, c_h.get_swift_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.SWIFT_LIB_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_swift_lib_url(cluster)) def test_is_hbase_common_lib_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(c_h.is_hbase_common_lib_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.ENABLE_HBASE_COMMON_LIB.name: False}}) self.assertFalse(c_h.is_hbase_common_lib_enabled(cluster)) def test_get_extjs_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.DEFAULT_EXTJS_LIB_URL, c_h.get_extjs_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.EXTJS_LIB_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_extjs_lib_url(cluster)) def test_get_kms_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.KMS_REPO_KEY_URL.default_value, c_h.get_kms_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.KMS_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_kms_key_url(cluster)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_spark_engine.py0000666000175100017510000000456513245514472025701 0ustar zuulzuul00000000000000# Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara import conductor as cond from sahara import context from sahara.plugins import base as pb from sahara.plugins import exceptions as ex from sahara.service.edp.spark import engine from sahara.tests.unit import base from sahara.utils import edp conductor = cond.API class SparkPluginTest(base.SaharaWithDbTestCase): def setUp(self): super(SparkPluginTest, self).setUp() self.override_config("plugins", ["cdh"]) pb.setup_plugins() def test_plugin_edp_engine_no_spark(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'cdh', 'hadoop_version': '5.7.0', 'default_image_id': 'image'} job = mock.Mock() job.type = edp.JOB_TYPE_SPARK cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) edp_engine = plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK) with testtools.ExpectedException( ex.InvalidComponentCountException, value_re="Hadoop cluster should contain 1 " "SPARK_YARN_HISTORY_SERVER component\(s\). Actual " "SPARK_YARN_HISTORY_SERVER count is 0\nError ID: .*"): edp_engine.validate_job_execution(cluster, job, mock.Mock()) def test_plugin_edp_engine(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'cdh', 'hadoop_version': '5.7.0', 'default_image_id': 'image'} cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) self.assertIsInstance( plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK), engine.SparkJobEngine) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_edp_engine.py0000666000175100017510000002073713245514472025330 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara import exceptions as ex from sahara.plugins import base as pb from sahara.plugins.cdh.v5_7_0 import edp_engine from sahara.plugins import exceptions as pl_ex from sahara.tests.unit import base as sahara_base from sahara.tests.unit.plugins.cdh import utils as c_u from sahara.utils import edp def get_cluster(version='5.7.0'): cluster = c_u.get_fake_cluster(plugin_name='CDH', hadoop_version=version) return cluster class EdpEngineTestV570(sahara_base.SaharaTestCase): def setUp(self): super(EdpEngineTestV570, self).setUp() pb.setup_plugins() def test_get_hdfs_user(self): eng = edp_engine.EdpOozieEngine(get_cluster()) self.assertEqual('hdfs', eng.get_hdfs_user()) @mock.patch('sahara.service.edp.hdfs_helper.create_dir_hadoop2') def test_create_hdfs_dir(self, create_dir_hadoop2): eng = edp_engine.EdpOozieEngine(get_cluster()) remote = mock.Mock() dir_name = mock.Mock() eng.create_hdfs_dir(remote, dir_name) create_dir_hadoop2.assert_called_once_with(remote, dir_name, eng.get_hdfs_user()) def test_get_oozie_server_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_oozie_server_uri(cluster) self.assertEqual("http://1.2.3.5:11000/oozie", uri) def test_get_name_node_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_name_node_uri(cluster) self.assertEqual("hdfs://master_inst.novalocal:8020", uri) # has HDFS_JOURNALNODE cluster = get_cluster() jns_node_group = mock.MagicMock() jns_node_group.node_processes = ['HDFS_JOURNALNODE'] jns_node_group.instances = [mock.Mock()] list.append(cluster.node_groups, jns_node_group) uri = eng.get_name_node_uri(cluster) self.assertEqual("hdfs://nameservice01", uri) def test_get_resource_manager_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_resource_manager_uri(cluster) self.assertEqual("master_inst.novalocal:8032", uri) def test_get_oozie_server(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) actual = eng.get_oozie_server(cluster) expect = cluster.node_groups[1].instances[0] self.assertEqual(expect, actual) @mock.patch('sahara.service.edp.oozie.engine.' 'OozieJobEngine.validate_job_execution') def test_validate_job_execution(self, c): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) eng.validate_job_execution(cluster, mock.Mock(), mock.Mock()) # more than one oozie server dict.__setitem__(cluster.node_groups[1], 'count', 2) self.assertRaises(pl_ex.InvalidComponentCountException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) @mock.patch( 'sahara.plugins.cdh.confighints_helper.get_possible_hive_config_from', return_value={}) def test_get_possible_job_config_hive(self, get_possible_hive_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_HIVE) get_possible_hive_config_from.assert_called_once_with( 'plugins/cdh/v5_7_0/resources/hive-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.cdh.v5_7_0.edp_engine.EdpOozieEngine') def test_get_possible_job_config_java(self, BaseCDHEdpOozieEngine): expected_config = {'job_config': {}} BaseCDHEdpOozieEngine.get_possible_job_config.return_value = ( expected_config) actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_JAVA) BaseCDHEdpOozieEngine.get_possible_job_config.assert_called_once_with( edp.JOB_TYPE_JAVA) self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.' 'get_possible_mapreduce_config_from', return_value={}) def test_get_possible_job_config_mapreduce( self, get_possible_mapreduce_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_MAPREDUCE) get_possible_mapreduce_config_from.assert_called_once_with( 'plugins/cdh/v5_7_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.' 'get_possible_mapreduce_config_from', return_value={}) def test_get_possible_job_config_mapreduce_streaming( self, get_possible_mapreduce_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_MAPREDUCE_STREAMING) get_possible_mapreduce_config_from.assert_called_once_with( 'plugins/cdh/v5_7_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.get_possible_pig_config_from', return_value={}) def test_get_possible_job_config_pig(self, get_possible_pig_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_PIG) get_possible_pig_config_from.assert_called_once_with( 'plugins/cdh/v5_7_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.cdh.v5_7_0.edp_engine.EdpOozieEngine') def test_get_possible_job_config_shell(self, BaseCDHEdpOozieEngine): expected_config = {'job_config': {}} BaseCDHEdpOozieEngine.get_possible_job_config.return_value = ( expected_config) actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_SHELL) BaseCDHEdpOozieEngine.get_possible_job_config.assert_called_once_with( edp.JOB_TYPE_SHELL) self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.utils.get_config_value_or_default') @mock.patch('sahara.plugins.utils.get_instance') @mock.patch('sahara.service.edp.spark.engine.' 'SparkJobEngine.validate_job_execution') def test_spark_engine_validate_job_execution(self, validate_job_execution, get_instance, get_config_value_or_default): # version unsupported cluster = get_cluster(version='5.4.0') eng = edp_engine.EdpSparkEngine(cluster) self.assertRaises(ex.InvalidDataException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) # none yarn history server cluster = get_cluster() eng = edp_engine.EdpSparkEngine(cluster) self.assertRaises(pl_ex.InvalidComponentCountException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) # valid cluster = get_cluster() yarn_history_node_group = mock.Mock() yarn_history_node_group.node_processes = ['SPARK_YARN_HISTORY_SERVER'] yarn_history_node_group.count = 1 list.append(cluster.node_groups, yarn_history_node_group) eng = edp_engine.EdpSparkEngine(cluster) eng.validate_job_execution(cluster, mock.Mock(), mock.Mock()) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_deploy.py0000666000175100017510000002362313245514472024524 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.cdh.v5_7_0 import deploy from sahara.tests.unit import base class DeployCDHV570(base.SaharaTestCase): def setUp(self): super(DeployCDHV570, self).setUp() self.master = mock.MagicMock() self.master.node_group.node_processes = [ "HDFS_NAMENODE", "YARN_RESOURCEMANAGER", "CLOUDERA_MANAGER", "SENTRY_SERVER", "YARN_NODEMANAGER", "ZOOKEEPER_SERVER", "OOZIE_SERVER", "YARN_JOBHISTORY", "HDFS_SECONDARYNAMENODE", "HIVE_METASTORE", "HIVE_SERVER2", "SPARK_YARN_HISTORY_SERVER", "HBASE_MASTER", "HBASE_REGIONSERVER", "HUE_SERVER", "KMS", "FLUME_AGENT", "SOLR_SERVER", "SQOOP_SERVER", "IMPALA_STATESTORE", "IMPALA_CATALOGSERVER", "IMPALAD", "KEY_VALUE_STORE_INDEXER", ] self.worker = mock.MagicMock() self.worker.node_group.node_processes = [ "HDFS_DATANODE", "HDFS_JOURNALNODE", "JOURNALNODE", "YARN_NODEMANAGER", "YARN_STANDBYRM", ] self.instances = [self.master, self.worker] self.cluster = mock.MagicMock() self.is_cdh_exists = mock.patch( "sahara.plugins.cdh.commands.is_pre_installed_cdh", return_value=False) self.is_cdh_exists.start() self._create_facade = mock.patch( "sahara.db.sqlalchemy.api._create_facade_lazily") self._create_facade.start() def tearDown(self): self.is_cdh_exists.stop() self._create_facade.stop() super(DeployCDHV570, self).tearDown() @mock.patch("sahara.plugins.utils.get_instances") @mock.patch("sahara.plugins.cdh.v5_7_0.deploy.CU") def test_configure_cluster(self, mock_cu, mock_get_instances): mock_get_instances.return_value = self.instances deploy.configure_cluster(self.cluster) mock_cu.pu.configure_os.assert_called_once_with(self.instances) mock_cu.pu.install_packages.assert_called_once_with(self.instances, deploy.PACKAGES) mock_cu.pu.start_cloudera_agents.assert_called_once_with( self.instances) mock_cu.pu.start_cloudera_manager.assert_called_once_with(self.cluster) mock_cu.update_cloudera_password.assert_called_once_with(self.cluster) mock_cu.await_agents.assert_called_once_with(self.cluster, self.instances) mock_cu.create_mgmt_service.assert_called_once_with(self.cluster) mock_cu.create_services.assert_called_once_with(self.cluster) mock_cu.configure_services.assert_called_once_with(self.cluster) mock_cu.configure_instances.assert_called_once_with(self.instances, self.cluster) mock_cu.deploy_configs.assert_called_once_with(self.cluster) @mock.patch("sahara.plugins.cdh.v5_7_0.deploy.CU") def test__start_roles(self, mock_cu): deploy._start_roles(self.cluster, self.instances) mock_cu.get_service_by_role.assert_any_call('DATANODE', instance=self.worker) mock_cu.get_service_by_role.assert_any_call('NODEMANAGER', instance=self.master) mock_cu.get_service_by_role.assert_any_call('NODEMANAGER', instance=self.worker) self.assertEqual(mock_cu.start_roles.call_count, 3) @mock.patch("sahara.plugins.cdh.v5_7_0.deploy._start_roles") @mock.patch("sahara.plugins.cdh.v5_7_0.deploy.CU") def test_scale_cluster(self, mock_cu, mock__start_roles): deploy.scale_cluster(self.cluster, None) self.assertEqual(mock_cu.call_count, 0) self.assertEqual(mock__start_roles.call_count, 0) deploy.scale_cluster(self.cluster, self.instances) mock_cu.pu.configure_os.assert_called_once_with(self.instances) mock_cu.pu.install_packages.assert_called_once_with(self.instances, deploy.PACKAGES) mock_cu.pu.start_cloudera_agents.assert_called_once_with( self.instances) mock_cu.await_agents.assert_called_once_with(self.cluster, self.instances) mock_cu.configure_instances.assert_called_once_with(self.instances, self.cluster) mock_cu.update_configs.assert_called_once_with(self.instances) mock_cu.pu.configure_swift.assert_called_once_with(self.cluster, self.instances) mock_cu.refresh_datanodes.assert_called_once_with(self.cluster) mock__start_roles.assert_called_once_with(self.cluster, self.instances) @mock.patch("sahara.plugins.cdh.v5_7_0.deploy.CU") def test_decommission_cluster(self, mock_cu): deploy.decommission_cluster(self.cluster, self.instances) dns = [] dns_2 = [] nms = [] nms_2 = [] for i in self.instances: if 'HDFS_DATANODE' in i.node_group.node_processes: dns.append(mock_cu.pu.get_role_name(i, 'DATANODE')) dns_2.append(mock_cu.pu.get_role_name(i, 'HDFS_GATEWAY')) if 'YARN_NODEMANAGER' in i.node_group.node_processes: nms.append(mock_cu.pu.get_role_name(i, 'NODEMANAGER')) nms_2.append(mock_cu.pu.get_role_name(i, 'YARN_GATEWAY')) mock_cu.decommission_nodes.assert_any_call( self.cluster, 'DATANODE', dns, dns_2) mock_cu.decommission_nodes.assert_any_call( self.cluster, 'NODEMANAGER', nms, nms_2) mock_cu.delete_instances.assert_called_once_with(self.cluster, self.instances) mock_cu.refresh_datanodes.assert_called_once_with(self.cluster) mock_cu.refresh_yarn_nodes.assert_called_once_with(self.cluster) @mock.patch("sahara.plugins.cdh.v5_7_0.deploy.CU") def test__prepare_cluster(self, mock_cu): deploy._prepare_cluster(self.cluster) mock_cu.pu.install_extjs.assert_called_once_with(self.cluster) mock_cu.pu.configure_hive.assert_called_once_with(self.cluster) mock_cu.pu.configure_sentry.assert_called_once_with(self.cluster) @mock.patch("sahara.service.edp.hdfs_helper.create_hbase_common_lib") @mock.patch("sahara.plugins.cdh.v5_7_0.deploy.CU") def test__finish_cluster_starting(self, mock_cu, mock_create_hbase): deploy._finish_cluster_starting(self.cluster) mock_cu.pu.put_hive_hdfs_xml.assert_called_once_with(self.cluster) self.assertTrue(mock_create_hbase.called) mock_cu.start_service.assert_called_once_with( mock_cu.get_service_by_role('AGENT', self.cluster)) @mock.patch("sahara.plugins.cdh.v5_7_0.deploy._finish_cluster_starting") @mock.patch("sahara.plugins.cdh.v5_7_0.deploy._prepare_cluster") @mock.patch("sahara.plugins.cdh.v5_7_0.deploy.CU") def test_start_cluster(self, mock_cu, mock_prepare, mock_finish): jns_count = 0 for i in self.instances: if "HDFS_JOURNALNODE" in i.node_group.node_processes: jns_count += 1 mock_cu.pu.get_jns.return_value.__len__.return_value = jns_count deploy.start_cluster(self.cluster) mock_prepare.assert_called_once_with(self.cluster) mock_cu.first_run.assert_called_once_with(self.cluster) mock_cu.pu.configure_swift.assert_called_once_with(self.cluster) if jns_count > 0: mock_cu.enable_namenode_ha.assert_called_once_with(self.cluster) mock_cu.update_role_config.assert_any_call( mock_cu.pu.get_secondarynamenode(self.cluster), 'HDFS_NAMENODE' ) mock_cu.enable_resourcemanager_ha.assert_called_once_with(self.cluster) mock_cu.update_role_config.assert_any_call( mock_cu.pu.get_stdb_rm(self.cluster), 'YARN_STANDBYRM') mock_finish.assert_called_once_with(self.cluster) def test_get_open_ports(self): master_ports = [ 9000, 7180, 7182, 7183, 7432, 7184, 8084, 8086, 10101, 9997, 9996, 8087, 9998, 9999, 8085, 9995, 9994, 8020, 8022, 50070, 50470, 50090, 50495, 8030, 8031, 8032, 8033, 8088, 8040, 8041, 8042, 10020, 19888, 9083, 10000, 8888, 11000, 11001, 18088, 2181, 3181, 4181, 9010, 60000, 60020, 41414, 8038, 8983, 8984, 8005, 12000, 25020, 26000, 25010, 24000, 21050, 21000, 23000, 25000, 28000, 22000, 16000, 16001 ] deploy.get_open_ports(self.master.node_group) self.assertItemsEqual(master_ports, deploy.get_open_ports(self.master.node_group)) worker_ports = [ 9000, 50010, 1004, 50075, 1006, 50020, 8480, 8481, 8485, 8040, 8041, 8042, 8030, 8031, 8032, 8033, 8088 ] self.assertItemsEqual(worker_ports, deploy.get_open_ports(self.worker.node_group)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_plugin_utils.py0000666000175100017510000000165513245514472025747 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_7_0 import plugin_utils as pu from sahara.tests.unit.plugins.cdh import base_plugin_utils_test class TestPluginUtilsV570(base_plugin_utils_test.TestPluginUtilsHigherThanV5): def setUp(self): super(TestPluginUtilsV570, self).setUp() self.plug_utils = pu.PluginUtilsV570() self.version = "v5_7_0" sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_7_0/test_plugin_utils_570.py0000666000175100017510000000165513245514472026342 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_7_0 import plugin_utils as pu from sahara.tests.unit.plugins.cdh import base_plugin_utils_test class TestPluginUtilsV570(base_plugin_utils_test.TestPluginUtilsHigherThanV5): def setUp(self): super(TestPluginUtilsV570, self).setUp() self.plug_utils = pu.PluginUtilsV570() self.version = "v5_7_0" sahara-8.0.0/sahara/tests/unit/plugins/cdh/test_versionfactory.py0000666000175100017510000000320113245514472025274 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import abstractversionhandler as avh from sahara.plugins.cdh import versionfactory as vf from sahara.tests.unit import base class VersionFactoryTestCase(base.SaharaTestCase): def test_get_instance(self): self.assertFalse(vf.VersionFactory.initialized) factory = vf.VersionFactory.get_instance() self.assertIsInstance(factory, vf.VersionFactory) self.assertTrue(vf.VersionFactory.initialized) def test_get_versions(self): factory = vf.VersionFactory.get_instance() versions = factory.get_versions() expected_versions = self.get_support_versions() self.assertEqual(expected_versions, versions) def test_get_version_handler(self): factory = vf.VersionFactory.get_instance() versions = self.get_support_versions() for version in versions: hander = factory.get_version_handler(version) self.assertIsInstance(hander, avh.AbstractVersionHandler) def get_support_versions(self): return ['5.5.0', '5.7.0', '5.9.0', '5.11.0'] sahara-8.0.0/sahara/tests/unit/plugins/cdh/test_health.py0000666000175100017510000001054313245514472023473 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import six import testtools from sahara.plugins.cdh import health from sahara.service.health import health_check_base as base_health from sahara.tests.unit import base as unit_base class TestHealthCheck(unit_base.SaharaTestCase): def test_check_health_availability(self): cluster = mock.Mock(plugin_name='cdh', hadoop_version='5.5.0') self.assertTrue(health.ClouderaManagerHealthCheck( cluster, mock.Mock()).is_available()) cluster = mock.Mock(plugin_name='cdh', hadoop_version='5.5.0') self.assertTrue(health.ServiceHealthCheck( cluster, mock.Mock(), mock.Mock()).is_available()) def _base_negative_testcase(self, cdh_response_mock, return_value, msg, col, service, postfix=None): if not postfix: postfix = "" cdh_response_mock.return_value = return_value exc = (base_health.YellowHealthError if col == 'YELLOW' else base_health.RedHealthError) with testtools.ExpectedException(exc): try: pr = health.HealthStatusProvider(mock.Mock(), mock.Mock()) health.ServiceHealthCheck( mock.Mock(), provider=pr, service=service).check_health() except Exception as e: msg = "%s%s" % (msg, postfix) all_message = "Cluster health is %(col)s. Reason: %(msg)s" % { 'col': col, 'msg': msg} self.assertEqual(all_message, six.text_type(e)) raise @mock.patch('sahara.plugins.cdh.health.HealthStatusProvider.' 'get_cloudera_health') @mock.patch('sahara.plugins.cdh.health.HealthStatusProvider.' 'get_important_services') def test_health_calculation(self, important_stuff, cdh_response): important_stuff.return_value = ['yarn01', 'hdfs01', 'oozie01'] cdh_response.return_value = { 'yarn01': { 'summary': 'GOOD', }, 'hdfs01': { 'summary': 'GOOD', } } msg = ("Cloudera Manager has responded that service is in " "the %s state") pr = health.HealthStatusProvider(mock.Mock(), mock.Mock()) self.assertEqual( msg % 'GOOD', health.ServiceHealthCheck( mock.Mock(), provider=pr, service='hdfs01').check_health()) self._base_negative_testcase(cdh_response, { 'yarn01': {'summary': 'GOOD'}, 'hdfs01': {'summary': 'BAD'} }, msg % 'BAD', 'RED', service='hdfs01') self._base_negative_testcase(cdh_response, { 'yarn01': {'summary': 'CONCERNING'}, 'hdfs01': {'summary': 'BAD'} }, msg % 'CONCERNING', 'YELLOW', service='yarn01') # not important service, only yellow health self._base_negative_testcase(cdh_response, { 'yarn01': {'summary': 'CONCERNING'}, 'hdfs01': {'summary': 'BAD'}, 'some_service01': {'summary': 'BAD'} }, msg % 'BAD', 'YELLOW', service='some_service01') self._base_negative_testcase(cdh_response, { 'yarn01': {'summary': 'UNKNOWN_STATE'}, 'hdfs01': {'summary': 'BAD'}, 'some_service01': {'summary': 'BAD'} }, msg % 'UNKNOWN_STATE', 'RED', service='yarn01') # test additional info postfix = (". The following checks did not " "pass: SUPER_HEALTH_CHECK - BAD state") self._base_negative_testcase(cdh_response, { 'yarn01': {'summary': 'UNKNOWN_STATE'}, 'hdfs01': {'summary': 'BAD', 'checks': [ {'name': 'SUPER_HEALTH_CHECK', 'summary': 'BAD'}]}, 'some_service01': {'summary': 'BAD'} }, msg % 'BAD', 'RED', service='hdfs01', postfix=postfix) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/0000775000175100017510000000000013245515027021606 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_versionhandler.py0000666000175100017510000001333413245514472026253 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import six from sahara.plugins.cdh.v5_9_0 import edp_engine from sahara.plugins.cdh.v5_9_0 import versionhandler from sahara.tests.unit import base class VersionHandlerTestCase(base.SaharaTestCase): plugin_path = "sahara.plugins.cdh.v5_9_0." cloudera_utils_path = plugin_path + "cloudera_utils.ClouderaUtilsV590." plugin_utils_path = plugin_path + "plugin_utils.PluginUtilsV590." def setUp(self): super(VersionHandlerTestCase, self).setUp() self.vh = versionhandler.VersionHandler() def test_get_node_processes(self): processes = self.vh.get_node_processes() for k, v in six.iteritems(processes): for p in v: self.assertIsInstance(p, str) @mock.patch("sahara.conductor.API.cluster_update") @mock.patch("sahara.context.ctx") @mock.patch(plugin_path + "deploy.configure_cluster") @mock.patch(cloudera_utils_path + "get_cloudera_manager_info", return_value={"fake_cm_info": "fake"}) def test_config_cluster(self, get_cm_info, configure_cluster, ctx, cluster_update): cluster = mock.Mock() self.vh.configure_cluster(cluster) configure_cluster.assert_called_once_with(cluster) cluster_update.assert_called_once_with( ctx(), cluster, {'info': {"fake_cm_info": "fake"}}) @mock.patch(plugin_path + "deploy.start_cluster") def test_start_cluster(self, start_cluster): cluster = mock.Mock() self.vh._set_cluster_info = mock.Mock() self.vh.start_cluster(cluster) start_cluster.assert_called_once_with(cluster) self.vh._set_cluster_info.assert_called_once_with(cluster) @mock.patch(plugin_path + "deploy.decommission_cluster") def test_decommission_nodes(self, decommission_cluster): cluster = mock.Mock() instances = mock.Mock() self.vh.decommission_nodes(cluster, instances) decommission_cluster.assert_called_once_with(cluster, instances) @mock.patch(plugin_path + "deploy.scale_cluster") def test_scale_cluster(self, scale_cluster): cluster = mock.Mock() instances = mock.Mock() self.vh.scale_cluster(cluster, instances) scale_cluster.assert_called_once_with(cluster, instances) @mock.patch("sahara.conductor.API.cluster_update") @mock.patch("sahara.context.ctx") @mock.patch(cloudera_utils_path + "get_cloudera_manager_info", return_value={}) @mock.patch(plugin_utils_path + "get_hue") def test_set_cluster_info(self, get_hue, get_cloudera_manager_info, ctx, cluster_update): hue = mock.Mock() hue.get_ip_or_dns_name.return_value = "1.2.3.4" get_hue.return_value = hue cluster = mock.Mock() self.vh._set_cluster_info(cluster) info = {'info': {'Hue Dashboard': {'Web UI': 'http://1.2.3.4:8888'}}} cluster_update.assert_called_once_with(ctx(), cluster, info) @mock.patch("sahara.plugins.utils.get_instance") @mock.patch("sahara.plugins.utils.get_config_value_or_default") @mock.patch("sahara.service.edp.job_utils.get_plugin") def test_get_edp_engine(self, get_plugin, get_config_value_or_default, get_instance): cluster = mock.Mock() job_type = 'Java' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsInstance(ret, edp_engine.EdpOozieEngine) job_type = 'Spark' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsInstance(ret, edp_engine.EdpSparkEngine) job_type = 'unsupported' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsNone(ret) def test_get_edp_job_types(self): ret = self.vh.get_edp_job_types() expect = edp_engine.EdpOozieEngine.get_supported_job_types() + \ edp_engine.EdpSparkEngine.get_supported_job_types() self.assertEqual(expect, ret) @mock.patch(plugin_path + "edp_engine.EdpOozieEngine.get_possible_job_config", return_value={'job_config': {}}) def test_edp_config_hints(self, get_possible_job_config): job_type = mock.Mock() ret = self.vh.get_edp_config_hints(job_type) get_possible_job_config.assert_called_once_with(job_type) self.assertEqual(ret, {'job_config': {}}) @mock.patch(plugin_path + "deploy.get_open_ports", return_value=[1234]) def test_get_open_ports(self, get_open_ports): node_group = mock.Mock() ret = self.vh.get_open_ports(node_group) get_open_ports.assert_called_once_with(node_group) self.assertEqual(ret, [1234]) @mock.patch(plugin_utils_path + "recommend_configs") def test_recommend_configs(self, recommend_configs): cluster = mock.Mock() scaling = mock.Mock() self.vh.get_plugin_configs = mock.Mock() self.vh.recommend_configs(cluster, scaling) recommend_configs.assert_called_once_with(cluster, self.vh.get_plugin_configs(), scaling) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_config_helper.py0000666000175100017510000000167013245514472026034 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_9_0 import config_helper from sahara.tests.unit.plugins.cdh import base_config_helper_test as bcht class TestConfigHelperV590(bcht.TestConfigHelper): def setUp(self): super(TestConfigHelperV590, self).setUp() self.c_h = config_helper.ConfigHelperV590() self.path_to_config = 'plugins/cdh/v5_9_0/resources/' sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_validation.py0000666000175100017510000000155713245514472025366 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corpration # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_9_0 import validation from sahara.tests.unit.plugins.cdh import base_validation_tests as bvt class ValidationTestCase(bvt.BaseValidationTestCase): def setUp(self): super(ValidationTestCase, self).setUp() self.module = validation.ValidatorV590 sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/__init__.py0000666000175100017510000000000013245514472023712 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_config_helper_590.py0000666000175100017510000001407213245514472026431 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json from sahara.plugins.cdh.v5_9_0 import config_helper from sahara.tests.unit import base from sahara.tests.unit.plugins.cdh import utils as ctu from sahara.utils import files as f c_h = config_helper.ConfigHelperV590() path_to_config = 'plugins/cdh/v5_9_0/resources/' json_files = [ 'hdfs-service.json', 'hdfs-namenode.json', 'hdfs-datanode.json', 'hdfs-secondarynamenode.json', 'hdfs-gateway.json', 'hdfs-journalnode.json', 'yarn-service.json', 'yarn-resourcemanager.json', 'yarn-nodemanager.json', 'yarn-jobhistory.json', 'yarn-gateway.json', 'oozie-service.json', 'oozie-oozie_server.json', 'hive-service.json', 'hive-hivemetastore.json', 'hive-hiveserver2.json', 'hive-webhcat.json', 'hue-service.json', 'hue-hue_server.json', 'spark-service.json', 'spark-spark_yarn_history_server.json', 'zookeeper-service.json', 'zookeeper-server.json', 'hbase-service.json', 'hbase-master.json', 'hbase-regionserver.json', 'flume-service.json', 'flume-agent.json', 'sentry-service.json', 'sentry-sentry_server.json', 'solr-service.json', 'solr-solr_server.json', 'sqoop-service.json', 'sqoop-sqoop_server.json', 'ks_indexer-service.json', 'ks_indexer-hbase_indexer.json', 'impala-service.json', 'impala-catalogserver.json', 'impala-impalad.json', 'impala-statestore.json', 'kms-service.json', 'kms-kms.json', 'kafka-kafka_broker.json', 'kafka-kafka_mirror_maker.json', 'kafka-service.json' ] class ConfigHelperTestCase(base.SaharaTestCase): def test_get_ng_plugin_configs(self): actual_configs = c_h._get_ng_plugin_configs() expected_configs = [] for json_file in json_files: expected_configs += json.loads( f.get_file_text(path_to_config + json_file)) # compare names expected_names = set(i['name'] for i in expected_configs) actual_names = set(i.to_dict()['name'] for i in actual_configs) self.assertEqual(expected_names, actual_names) def test_get_cdh5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CDH5_REPO_URL.default_value, c_h.get_cdh5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CDH5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cdh5_repo_url(cluster)) def test_get_cdh5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CDH5_REPO_KEY_URL.default_value, c_h.get_cdh5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CDH5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cdh5_key_url(cluster)) def test_get_cm5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CM5_REPO_URL.default_value, c_h.get_cm5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CM5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cm5_repo_url(cluster)) def test_get_cm5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CM5_REPO_KEY_URL.default_value, c_h.get_cm5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CM5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cm5_key_url(cluster)) def test_is_swift_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(c_h.is_swift_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.ENABLE_SWIFT.name: False}}) self.assertFalse(c_h.is_swift_enabled(cluster)) def test_get_swift_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.DEFAULT_SWIFT_LIB_URL, c_h.get_swift_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.SWIFT_LIB_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_swift_lib_url(cluster)) def test_is_hbase_common_lib_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(c_h.is_hbase_common_lib_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.ENABLE_HBASE_COMMON_LIB.name: False}}) self.assertFalse(c_h.is_hbase_common_lib_enabled(cluster)) def test_get_extjs_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.DEFAULT_EXTJS_LIB_URL, c_h.get_extjs_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.EXTJS_LIB_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_extjs_lib_url(cluster)) def test_get_kms_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.KMS_REPO_KEY_URL.default_value, c_h.get_kms_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.KMS_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_kms_key_url(cluster)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_spark_engine.py0000666000175100017510000000456513245514472025703 0ustar zuulzuul00000000000000# Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara import conductor as cond from sahara import context from sahara.plugins import base as pb from sahara.plugins import exceptions as ex from sahara.service.edp.spark import engine from sahara.tests.unit import base from sahara.utils import edp conductor = cond.API class SparkPluginTest(base.SaharaWithDbTestCase): def setUp(self): super(SparkPluginTest, self).setUp() self.override_config("plugins", ["cdh"]) pb.setup_plugins() def test_plugin_edp_engine_no_spark(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'cdh', 'hadoop_version': '5.9.0', 'default_image_id': 'image'} job = mock.Mock() job.type = edp.JOB_TYPE_SPARK cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) edp_engine = plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK) with testtools.ExpectedException( ex.InvalidComponentCountException, value_re="Hadoop cluster should contain 1 " "SPARK_YARN_HISTORY_SERVER component\(s\). Actual " "SPARK_YARN_HISTORY_SERVER count is 0\nError ID: .*"): edp_engine.validate_job_execution(cluster, job, mock.Mock()) def test_plugin_edp_engine(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'cdh', 'hadoop_version': '5.9.0', 'default_image_id': 'image'} cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) self.assertIsInstance( plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK), engine.SparkJobEngine) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_edp_engine.py0000666000175100017510000002073713245514472025332 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara import exceptions as ex from sahara.plugins import base as pb from sahara.plugins.cdh.v5_9_0 import edp_engine from sahara.plugins import exceptions as pl_ex from sahara.tests.unit import base as sahara_base from sahara.tests.unit.plugins.cdh import utils as c_u from sahara.utils import edp def get_cluster(version='5.9.0'): cluster = c_u.get_fake_cluster(plugin_name='CDH', hadoop_version=version) return cluster class EdpEngineTestV590(sahara_base.SaharaTestCase): def setUp(self): super(EdpEngineTestV590, self).setUp() pb.setup_plugins() def test_get_hdfs_user(self): eng = edp_engine.EdpOozieEngine(get_cluster()) self.assertEqual('hdfs', eng.get_hdfs_user()) @mock.patch('sahara.service.edp.hdfs_helper.create_dir_hadoop2') def test_create_hdfs_dir(self, create_dir_hadoop2): eng = edp_engine.EdpOozieEngine(get_cluster()) remote = mock.Mock() dir_name = mock.Mock() eng.create_hdfs_dir(remote, dir_name) create_dir_hadoop2.assert_called_once_with(remote, dir_name, eng.get_hdfs_user()) def test_get_oozie_server_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_oozie_server_uri(cluster) self.assertEqual("http://1.2.3.5:11000/oozie", uri) def test_get_name_node_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_name_node_uri(cluster) self.assertEqual("hdfs://master_inst.novalocal:8020", uri) # has HDFS_JOURNALNODE cluster = get_cluster() jns_node_group = mock.MagicMock() jns_node_group.node_processes = ['HDFS_JOURNALNODE'] jns_node_group.instances = [mock.Mock()] list.append(cluster.node_groups, jns_node_group) uri = eng.get_name_node_uri(cluster) self.assertEqual("hdfs://nameservice01", uri) def test_get_resource_manager_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_resource_manager_uri(cluster) self.assertEqual("master_inst.novalocal:8032", uri) def test_get_oozie_server(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) actual = eng.get_oozie_server(cluster) expect = cluster.node_groups[1].instances[0] self.assertEqual(expect, actual) @mock.patch('sahara.service.edp.oozie.engine.' 'OozieJobEngine.validate_job_execution') def test_validate_job_execution(self, c): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) eng.validate_job_execution(cluster, mock.Mock(), mock.Mock()) # more than one oozie server dict.__setitem__(cluster.node_groups[1], 'count', 2) self.assertRaises(pl_ex.InvalidComponentCountException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) @mock.patch( 'sahara.plugins.cdh.confighints_helper.get_possible_hive_config_from', return_value={}) def test_get_possible_job_config_hive(self, get_possible_hive_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_HIVE) get_possible_hive_config_from.assert_called_once_with( 'plugins/cdh/v5_9_0/resources/hive-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.cdh.v5_9_0.edp_engine.EdpOozieEngine') def test_get_possible_job_config_java(self, BaseCDHEdpOozieEngine): expected_config = {'job_config': {}} BaseCDHEdpOozieEngine.get_possible_job_config.return_value = ( expected_config) actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_JAVA) BaseCDHEdpOozieEngine.get_possible_job_config.assert_called_once_with( edp.JOB_TYPE_JAVA) self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.' 'get_possible_mapreduce_config_from', return_value={}) def test_get_possible_job_config_mapreduce( self, get_possible_mapreduce_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_MAPREDUCE) get_possible_mapreduce_config_from.assert_called_once_with( 'plugins/cdh/v5_9_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.' 'get_possible_mapreduce_config_from', return_value={}) def test_get_possible_job_config_mapreduce_streaming( self, get_possible_mapreduce_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_MAPREDUCE_STREAMING) get_possible_mapreduce_config_from.assert_called_once_with( 'plugins/cdh/v5_9_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.get_possible_pig_config_from', return_value={}) def test_get_possible_job_config_pig(self, get_possible_pig_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_PIG) get_possible_pig_config_from.assert_called_once_with( 'plugins/cdh/v5_9_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.cdh.v5_9_0.edp_engine.EdpOozieEngine') def test_get_possible_job_config_shell(self, BaseCDHEdpOozieEngine): expected_config = {'job_config': {}} BaseCDHEdpOozieEngine.get_possible_job_config.return_value = ( expected_config) actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_SHELL) BaseCDHEdpOozieEngine.get_possible_job_config.assert_called_once_with( edp.JOB_TYPE_SHELL) self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.utils.get_config_value_or_default') @mock.patch('sahara.plugins.utils.get_instance') @mock.patch('sahara.service.edp.spark.engine.' 'SparkJobEngine.validate_job_execution') def test_spark_engine_validate_job_execution(self, validate_job_execution, get_instance, get_config_value_or_default): # version unsupported cluster = get_cluster(version='5.4.0') eng = edp_engine.EdpSparkEngine(cluster) self.assertRaises(ex.InvalidDataException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) # none yarn history server cluster = get_cluster() eng = edp_engine.EdpSparkEngine(cluster) self.assertRaises(pl_ex.InvalidComponentCountException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) # valid cluster = get_cluster() yarn_history_node_group = mock.Mock() yarn_history_node_group.node_processes = ['SPARK_YARN_HISTORY_SERVER'] yarn_history_node_group.count = 1 list.append(cluster.node_groups, yarn_history_node_group) eng = edp_engine.EdpSparkEngine(cluster) eng.validate_job_execution(cluster, mock.Mock(), mock.Mock()) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_deploy.py0000666000175100017510000002362313245514472024526 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.cdh.v5_9_0 import deploy from sahara.tests.unit import base class DeployCDHV590(base.SaharaTestCase): def setUp(self): super(DeployCDHV590, self).setUp() self.master = mock.MagicMock() self.master.node_group.node_processes = [ "HDFS_NAMENODE", "YARN_RESOURCEMANAGER", "CLOUDERA_MANAGER", "SENTRY_SERVER", "YARN_NODEMANAGER", "ZOOKEEPER_SERVER", "OOZIE_SERVER", "YARN_JOBHISTORY", "HDFS_SECONDARYNAMENODE", "HIVE_METASTORE", "HIVE_SERVER2", "SPARK_YARN_HISTORY_SERVER", "HBASE_MASTER", "HBASE_REGIONSERVER", "HUE_SERVER", "KMS", "FLUME_AGENT", "SOLR_SERVER", "SQOOP_SERVER", "IMPALA_STATESTORE", "IMPALA_CATALOGSERVER", "IMPALAD", "KEY_VALUE_STORE_INDEXER", ] self.worker = mock.MagicMock() self.worker.node_group.node_processes = [ "HDFS_DATANODE", "HDFS_JOURNALNODE", "JOURNALNODE", "YARN_NODEMANAGER", "YARN_STANDBYRM", ] self.instances = [self.master, self.worker] self.cluster = mock.MagicMock() self.is_cdh_exists = mock.patch( "sahara.plugins.cdh.commands.is_pre_installed_cdh", return_value=False) self.is_cdh_exists.start() self._create_facade = mock.patch( "sahara.db.sqlalchemy.api._create_facade_lazily") self._create_facade.start() def tearDown(self): self.is_cdh_exists.stop() self._create_facade.stop() super(DeployCDHV590, self).tearDown() @mock.patch("sahara.plugins.utils.get_instances") @mock.patch("sahara.plugins.cdh.v5_9_0.deploy.CU") def test_configure_cluster(self, mock_cu, mock_get_instances): mock_get_instances.return_value = self.instances deploy.configure_cluster(self.cluster) mock_cu.pu.configure_os.assert_called_once_with(self.instances) mock_cu.pu.install_packages.assert_called_once_with(self.instances, deploy.PACKAGES) mock_cu.pu.start_cloudera_agents.assert_called_once_with( self.instances) mock_cu.pu.start_cloudera_manager.assert_called_once_with(self.cluster) mock_cu.update_cloudera_password.assert_called_once_with(self.cluster) mock_cu.await_agents.assert_called_once_with(self.cluster, self.instances) mock_cu.create_mgmt_service.assert_called_once_with(self.cluster) mock_cu.create_services.assert_called_once_with(self.cluster) mock_cu.configure_services.assert_called_once_with(self.cluster) mock_cu.configure_instances.assert_called_once_with(self.instances, self.cluster) mock_cu.deploy_configs.assert_called_once_with(self.cluster) @mock.patch("sahara.plugins.cdh.v5_9_0.deploy.CU") def test__start_roles(self, mock_cu): deploy._start_roles(self.cluster, self.instances) mock_cu.get_service_by_role.assert_any_call('DATANODE', instance=self.worker) mock_cu.get_service_by_role.assert_any_call('NODEMANAGER', instance=self.master) mock_cu.get_service_by_role.assert_any_call('NODEMANAGER', instance=self.worker) self.assertEqual(mock_cu.start_roles.call_count, 3) @mock.patch("sahara.plugins.cdh.v5_9_0.deploy._start_roles") @mock.patch("sahara.plugins.cdh.v5_9_0.deploy.CU") def test_scale_cluster(self, mock_cu, mock__start_roles): deploy.scale_cluster(self.cluster, None) self.assertEqual(mock_cu.call_count, 0) self.assertEqual(mock__start_roles.call_count, 0) deploy.scale_cluster(self.cluster, self.instances) mock_cu.pu.configure_os.assert_called_once_with(self.instances) mock_cu.pu.install_packages.assert_called_once_with(self.instances, deploy.PACKAGES) mock_cu.pu.start_cloudera_agents.assert_called_once_with( self.instances) mock_cu.await_agents.assert_called_once_with(self.cluster, self.instances) mock_cu.configure_instances.assert_called_once_with(self.instances, self.cluster) mock_cu.update_configs.assert_called_once_with(self.instances) mock_cu.pu.configure_swift.assert_called_once_with(self.cluster, self.instances) mock_cu.refresh_datanodes.assert_called_once_with(self.cluster) mock__start_roles.assert_called_once_with(self.cluster, self.instances) @mock.patch("sahara.plugins.cdh.v5_9_0.deploy.CU") def test_decommission_cluster(self, mock_cu): deploy.decommission_cluster(self.cluster, self.instances) dns = [] dns_2 = [] nms = [] nms_2 = [] for i in self.instances: if 'HDFS_DATANODE' in i.node_group.node_processes: dns.append(mock_cu.pu.get_role_name(i, 'DATANODE')) dns_2.append(mock_cu.pu.get_role_name(i, 'HDFS_GATEWAY')) if 'YARN_NODEMANAGER' in i.node_group.node_processes: nms.append(mock_cu.pu.get_role_name(i, 'NODEMANAGER')) nms_2.append(mock_cu.pu.get_role_name(i, 'YARN_GATEWAY')) mock_cu.decommission_nodes.assert_any_call( self.cluster, 'DATANODE', dns, dns_2) mock_cu.decommission_nodes.assert_any_call( self.cluster, 'NODEMANAGER', nms, nms_2) mock_cu.delete_instances.assert_called_once_with(self.cluster, self.instances) mock_cu.refresh_datanodes.assert_called_once_with(self.cluster) mock_cu.refresh_yarn_nodes.assert_called_once_with(self.cluster) @mock.patch("sahara.plugins.cdh.v5_9_0.deploy.CU") def test__prepare_cluster(self, mock_cu): deploy._prepare_cluster(self.cluster) mock_cu.pu.install_extjs.assert_called_once_with(self.cluster) mock_cu.pu.configure_hive.assert_called_once_with(self.cluster) mock_cu.pu.configure_sentry.assert_called_once_with(self.cluster) @mock.patch("sahara.service.edp.hdfs_helper.create_hbase_common_lib") @mock.patch("sahara.plugins.cdh.v5_9_0.deploy.CU") def test__finish_cluster_starting(self, mock_cu, mock_create_hbase): deploy._finish_cluster_starting(self.cluster) mock_cu.pu.put_hive_hdfs_xml.assert_called_once_with(self.cluster) self.assertTrue(mock_create_hbase.called) mock_cu.start_service.assert_called_once_with( mock_cu.get_service_by_role('AGENT', self.cluster)) @mock.patch("sahara.plugins.cdh.v5_9_0.deploy._finish_cluster_starting") @mock.patch("sahara.plugins.cdh.v5_9_0.deploy._prepare_cluster") @mock.patch("sahara.plugins.cdh.v5_9_0.deploy.CU") def test_start_cluster(self, mock_cu, mock_prepare, mock_finish): jns_count = 0 for i in self.instances: if "HDFS_JOURNALNODE" in i.node_group.node_processes: jns_count += 1 mock_cu.pu.get_jns.return_value.__len__.return_value = jns_count deploy.start_cluster(self.cluster) mock_prepare.assert_called_once_with(self.cluster) mock_cu.first_run.assert_called_once_with(self.cluster) mock_cu.pu.configure_swift.assert_called_once_with(self.cluster) if jns_count > 0: mock_cu.enable_namenode_ha.assert_called_once_with(self.cluster) mock_cu.update_role_config.assert_any_call( mock_cu.pu.get_secondarynamenode(self.cluster), 'HDFS_NAMENODE' ) mock_cu.enable_resourcemanager_ha.assert_called_once_with(self.cluster) mock_cu.update_role_config.assert_any_call( mock_cu.pu.get_stdb_rm(self.cluster), 'YARN_STANDBYRM') mock_finish.assert_called_once_with(self.cluster) def test_get_open_ports(self): master_ports = [ 9000, 7180, 7182, 7183, 7432, 7184, 8084, 8086, 10101, 9997, 9996, 8087, 9998, 9999, 8085, 9995, 9994, 8020, 8022, 50070, 50470, 50090, 50495, 8030, 8031, 8032, 8033, 8088, 8040, 8041, 8042, 10020, 19888, 9083, 10000, 8888, 11000, 11001, 18088, 2181, 3181, 4181, 9010, 60000, 60020, 41414, 8038, 8983, 8984, 8005, 12000, 25020, 26000, 25010, 24000, 21050, 21000, 23000, 25000, 28000, 22000, 16000, 16001 ] deploy.get_open_ports(self.master.node_group) self.assertItemsEqual(master_ports, deploy.get_open_ports(self.master.node_group)) worker_ports = [ 9000, 50010, 1004, 50075, 1006, 50020, 8480, 8481, 8485, 8040, 8041, 8042, 8030, 8031, 8032, 8033, 8088 ] self.assertItemsEqual(worker_ports, deploy.get_open_ports(self.worker.node_group)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_plugin_utils_590.py0000666000175100017510000000165513245514472026346 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_9_0 import plugin_utils as pu from sahara.tests.unit.plugins.cdh import base_plugin_utils_test class TestPluginUtilsV590(base_plugin_utils_test.TestPluginUtilsHigherThanV5): def setUp(self): super(TestPluginUtilsV590, self).setUp() self.plug_utils = pu.PluginUtilsV590() self.version = "v5_9_0" sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_9_0/test_plugin_utils.py0000666000175100017510000000165513245514472025751 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_9_0 import plugin_utils as pu from sahara.tests.unit.plugins.cdh import base_plugin_utils_test class TestPluginUtilsV590(base_plugin_utils_test.TestPluginUtilsHigherThanV5): def setUp(self): super(TestPluginUtilsV590, self).setUp() self.plug_utils = pu.PluginUtilsV590() self.version = "v5_9_0" sahara-8.0.0/sahara/tests/unit/plugins/cdh/test_utils.py0000666000175100017510000000435513245514472023372 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh import plugin_utils as pu from sahara.tests.unit import base from sahara.tests.unit.plugins.cdh import utils as tu PU = pu.AbstractPluginUtils() class UtilsTestCase(base.SaharaTestCase): def test_get_manager(self): cluster = tu.get_fake_cluster() inst = PU.get_manager(cluster) self.assertEqual('id1', inst.instance_id) def test_get_namenode(self): cluster = tu.get_fake_cluster() inst = PU.get_namenode(cluster) self.assertEqual('id2', inst.instance_id) def test_get_secondarynamenode(self): cluster = tu.get_fake_cluster() inst = PU.get_secondarynamenode(cluster) self.assertEqual('id2', inst.instance_id) def test_get_resourcemanager(self): cluster = tu.get_fake_cluster() inst = PU.get_resourcemanager(cluster) self.assertEqual('id2', inst.instance_id) def test_get_datanodes(self): cluster = tu.get_fake_cluster() dns = PU.get_datanodes(cluster) ids = [dn.instance_id for dn in dns] self.assertEqual(sorted(['id00', 'id01', 'id02']), sorted(ids)) def test_get_nodemanagers(self): cluster = tu.get_fake_cluster() nms = PU.get_nodemanagers(cluster) ids = [nm.instance_id for nm in nms] self.assertEqual(sorted(['id00', 'id01', 'id02']), sorted(ids)) def test_get_historyserver(self): cluster = tu.get_fake_cluster() inst = PU.get_historyserver(cluster) self.assertEqual('id2', inst.instance_id) def test_get_oozie(self): cluster = tu.get_fake_cluster() inst = PU.get_oozie(cluster) self.assertEqual('id2', inst.instance_id) sahara-8.0.0/sahara/tests/unit/plugins/cdh/__init__.py0000666000175100017510000000000013245514472022711 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/0000775000175100017510000000000013245515027021602 5ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_config_helper_550.py0000666000175100017510000001407213245514472026421 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json from sahara.plugins.cdh.v5_5_0 import config_helper from sahara.tests.unit import base from sahara.tests.unit.plugins.cdh import utils as ctu from sahara.utils import files as f c_h = config_helper.ConfigHelperV550() path_to_config = 'plugins/cdh/v5_5_0/resources/' json_files = [ 'hdfs-service.json', 'hdfs-namenode.json', 'hdfs-datanode.json', 'hdfs-secondarynamenode.json', 'hdfs-gateway.json', 'hdfs-journalnode.json', 'yarn-service.json', 'yarn-resourcemanager.json', 'yarn-nodemanager.json', 'yarn-jobhistory.json', 'yarn-gateway.json', 'oozie-service.json', 'oozie-oozie_server.json', 'hive-service.json', 'hive-hivemetastore.json', 'hive-hiveserver2.json', 'hive-webhcat.json', 'hue-service.json', 'hue-hue_server.json', 'spark-service.json', 'spark-spark_yarn_history_server.json', 'zookeeper-service.json', 'zookeeper-server.json', 'hbase-service.json', 'hbase-master.json', 'hbase-regionserver.json', 'flume-service.json', 'flume-agent.json', 'sentry-service.json', 'sentry-sentry_server.json', 'solr-service.json', 'solr-solr_server.json', 'sqoop-service.json', 'sqoop-sqoop_server.json', 'ks_indexer-service.json', 'ks_indexer-hbase_indexer.json', 'impala-service.json', 'impala-catalogserver.json', 'impala-impalad.json', 'impala-statestore.json', 'kms-service.json', 'kms-kms.json', 'kafka-kafka_broker.json', 'kafka-kafka_mirror_maker.json', 'kafka-service.json' ] class ConfigHelperTestCase(base.SaharaTestCase): def test_get_ng_plugin_configs(self): actual_configs = c_h._get_ng_plugin_configs() expected_configs = [] for json_file in json_files: expected_configs += json.loads( f.get_file_text(path_to_config + json_file)) # compare names expected_names = set(i['name'] for i in expected_configs) actual_names = set(i.to_dict()['name'] for i in actual_configs) self.assertEqual(expected_names, actual_names) def test_get_cdh5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CDH5_REPO_URL.default_value, c_h.get_cdh5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CDH5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cdh5_repo_url(cluster)) def test_get_cdh5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CDH5_REPO_KEY_URL.default_value, c_h.get_cdh5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CDH5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cdh5_key_url(cluster)) def test_get_cm5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CM5_REPO_URL.default_value, c_h.get_cm5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CM5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cm5_repo_url(cluster)) def test_get_cm5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.CM5_REPO_KEY_URL.default_value, c_h.get_cm5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.CM5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_cm5_key_url(cluster)) def test_is_swift_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(c_h.is_swift_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.ENABLE_SWIFT.name: False}}) self.assertFalse(c_h.is_swift_enabled(cluster)) def test_get_swift_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.DEFAULT_SWIFT_LIB_URL, c_h.get_swift_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.SWIFT_LIB_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_swift_lib_url(cluster)) def test_is_hbase_common_lib_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(c_h.is_hbase_common_lib_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.ENABLE_HBASE_COMMON_LIB.name: False}}) self.assertFalse(c_h.is_hbase_common_lib_enabled(cluster)) def test_get_extjs_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.DEFAULT_EXTJS_LIB_URL, c_h.get_extjs_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.EXTJS_LIB_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_extjs_lib_url(cluster)) def test_get_kms_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(c_h.KMS_REPO_KEY_URL.default_value, c_h.get_kms_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={'general': {c_h.KMS_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', c_h.get_kms_key_url(cluster)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_versionhandler.py0000666000175100017510000001333413245514472026247 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import six from sahara.plugins.cdh.v5_5_0 import edp_engine from sahara.plugins.cdh.v5_5_0 import versionhandler from sahara.tests.unit import base class VersionHandlerTestCase(base.SaharaTestCase): plugin_path = "sahara.plugins.cdh.v5_5_0." cloudera_utils_path = plugin_path + "cloudera_utils.ClouderaUtilsV550." plugin_utils_path = plugin_path + "plugin_utils.PluginUtilsV550." def setUp(self): super(VersionHandlerTestCase, self).setUp() self.vh = versionhandler.VersionHandler() def test_get_node_processes(self): processes = self.vh.get_node_processes() for k, v in six.iteritems(processes): for p in v: self.assertIsInstance(p, str) @mock.patch("sahara.conductor.API.cluster_update") @mock.patch("sahara.context.ctx") @mock.patch(plugin_path + "deploy.configure_cluster") @mock.patch(cloudera_utils_path + "get_cloudera_manager_info", return_value={"fake_cm_info": "fake"}) def test_config_cluster(self, get_cm_info, configure_cluster, ctx, cluster_update): cluster = mock.Mock() self.vh.configure_cluster(cluster) configure_cluster.assert_called_once_with(cluster) cluster_update.assert_called_once_with( ctx(), cluster, {'info': {"fake_cm_info": "fake"}}) @mock.patch(plugin_path + "deploy.start_cluster") def test_start_cluster(self, start_cluster): cluster = mock.Mock() self.vh._set_cluster_info = mock.Mock() self.vh.start_cluster(cluster) start_cluster.assert_called_once_with(cluster) self.vh._set_cluster_info.assert_called_once_with(cluster) @mock.patch(plugin_path + "deploy.decommission_cluster") def test_decommission_nodes(self, decommission_cluster): cluster = mock.Mock() instances = mock.Mock() self.vh.decommission_nodes(cluster, instances) decommission_cluster.assert_called_once_with(cluster, instances) @mock.patch(plugin_path + "deploy.scale_cluster") def test_scale_cluster(self, scale_cluster): cluster = mock.Mock() instances = mock.Mock() self.vh.scale_cluster(cluster, instances) scale_cluster.assert_called_once_with(cluster, instances) @mock.patch("sahara.conductor.API.cluster_update") @mock.patch("sahara.context.ctx") @mock.patch(cloudera_utils_path + "get_cloudera_manager_info", return_value={}) @mock.patch(plugin_utils_path + "get_hue") def test_set_cluster_info(self, get_hue, get_cloudera_manager_info, ctx, cluster_update): hue = mock.Mock() hue.get_ip_or_dns_name.return_value = "1.2.3.4" get_hue.return_value = hue cluster = mock.Mock() self.vh._set_cluster_info(cluster) info = {'info': {'Hue Dashboard': {'Web UI': 'http://1.2.3.4:8888'}}} cluster_update.assert_called_once_with(ctx(), cluster, info) @mock.patch("sahara.plugins.utils.get_instance") @mock.patch("sahara.plugins.utils.get_config_value_or_default") @mock.patch("sahara.service.edp.job_utils.get_plugin") def test_get_edp_engine(self, get_plugin, get_config_value_or_default, get_instance): cluster = mock.Mock() job_type = 'Java' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsInstance(ret, edp_engine.EdpOozieEngine) job_type = 'Spark' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsInstance(ret, edp_engine.EdpSparkEngine) job_type = 'unsupported' ret = self.vh.get_edp_engine(cluster, job_type) self.assertIsNone(ret) def test_get_edp_job_types(self): ret = self.vh.get_edp_job_types() expect = edp_engine.EdpOozieEngine.get_supported_job_types() + \ edp_engine.EdpSparkEngine.get_supported_job_types() self.assertEqual(expect, ret) @mock.patch(plugin_path + "edp_engine.EdpOozieEngine.get_possible_job_config", return_value={'job_config': {}}) def test_edp_config_hints(self, get_possible_job_config): job_type = mock.Mock() ret = self.vh.get_edp_config_hints(job_type) get_possible_job_config.assert_called_once_with(job_type) self.assertEqual(ret, {'job_config': {}}) @mock.patch(plugin_path + "deploy.get_open_ports", return_value=[1234]) def test_get_open_ports(self, get_open_ports): node_group = mock.Mock() ret = self.vh.get_open_ports(node_group) get_open_ports.assert_called_once_with(node_group) self.assertEqual(ret, [1234]) @mock.patch(plugin_utils_path + "recommend_configs") def test_recommend_configs(self, recommend_configs): cluster = mock.Mock() scaling = mock.Mock() self.vh.get_plugin_configs = mock.Mock() self.vh.recommend_configs(cluster, scaling) recommend_configs.assert_called_once_with(cluster, self.vh.get_plugin_configs(), scaling) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_config_helper.py0000666000175100017510000000167013245514472026030 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_5_0 import config_helper from sahara.tests.unit.plugins.cdh import base_config_helper_test as bcht class TestConfigHelperV550(bcht.TestConfigHelper): def setUp(self): super(TestConfigHelperV550, self).setUp() self.c_h = config_helper.ConfigHelperV550() self.path_to_config = 'plugins/cdh/v5_5_0/resources/' sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_validation.py0000666000175100017510000000155713245514472025362 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corpration # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_5_0 import validation from sahara.tests.unit.plugins.cdh import base_validation_tests as bvt class ValidationTestCase(bvt.BaseValidationTestCase): def setUp(self): super(ValidationTestCase, self).setUp() self.module = validation.ValidatorV550 sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/__init__.py0000666000175100017510000000000013245514472023706 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_spark_engine.py0000666000175100017510000000456513245514472025677 0ustar zuulzuul00000000000000# Copyright (c) 2015 ISPRAS # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara import conductor as cond from sahara import context from sahara.plugins import base as pb from sahara.plugins import exceptions as ex from sahara.service.edp.spark import engine from sahara.tests.unit import base from sahara.utils import edp conductor = cond.API class SparkPluginTest(base.SaharaWithDbTestCase): def setUp(self): super(SparkPluginTest, self).setUp() self.override_config("plugins", ["cdh"]) pb.setup_plugins() def test_plugin_edp_engine_no_spark(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'cdh', 'hadoop_version': '5.5.0', 'default_image_id': 'image'} job = mock.Mock() job.type = edp.JOB_TYPE_SPARK cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) edp_engine = plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK) with testtools.ExpectedException( ex.InvalidComponentCountException, value_re="Hadoop cluster should contain 1 " "SPARK_YARN_HISTORY_SERVER component\(s\). Actual " "SPARK_YARN_HISTORY_SERVER count is 0\nError ID: .*"): edp_engine.validate_job_execution(cluster, job, mock.Mock()) def test_plugin_edp_engine(self): cluster_dict = { 'name': 'cluster', 'plugin_name': 'cdh', 'hadoop_version': '5.5.0', 'default_image_id': 'image'} cluster = conductor.cluster_create(context.ctx(), cluster_dict) plugin = pb.PLUGINS.get_plugin(cluster.plugin_name) self.assertIsInstance( plugin.get_edp_engine(cluster, edp.JOB_TYPE_SPARK), engine.SparkJobEngine) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_plugin_utils_550.py0000666000175100017510000000165513245514472026336 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_5_0 import plugin_utils as pu from sahara.tests.unit.plugins.cdh import base_plugin_utils_test class TestPluginUtilsV550(base_plugin_utils_test.TestPluginUtilsHigherThanV5): def setUp(self): super(TestPluginUtilsV550, self).setUp() self.plug_utils = pu.PluginUtilsV550() self.version = "v5_5_0" sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_edp_engine.py0000666000175100017510000002073713245514472025326 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara import exceptions as ex from sahara.plugins import base as pb from sahara.plugins.cdh.v5_5_0 import edp_engine from sahara.plugins import exceptions as pl_ex from sahara.tests.unit import base as sahara_base from sahara.tests.unit.plugins.cdh import utils as c_u from sahara.utils import edp def get_cluster(version='5.5.0'): cluster = c_u.get_fake_cluster(plugin_name='CDH', hadoop_version=version) return cluster class EdpEngineTestV550(sahara_base.SaharaTestCase): def setUp(self): super(EdpEngineTestV550, self).setUp() pb.setup_plugins() def test_get_hdfs_user(self): eng = edp_engine.EdpOozieEngine(get_cluster()) self.assertEqual('hdfs', eng.get_hdfs_user()) @mock.patch('sahara.service.edp.hdfs_helper.create_dir_hadoop2') def test_create_hdfs_dir(self, create_dir_hadoop2): eng = edp_engine.EdpOozieEngine(get_cluster()) remote = mock.Mock() dir_name = mock.Mock() eng.create_hdfs_dir(remote, dir_name) create_dir_hadoop2.assert_called_once_with(remote, dir_name, eng.get_hdfs_user()) def test_get_oozie_server_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_oozie_server_uri(cluster) self.assertEqual("http://1.2.3.5:11000/oozie", uri) def test_get_name_node_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_name_node_uri(cluster) self.assertEqual("hdfs://master_inst.novalocal:8020", uri) # has HDFS_JOURNALNODE cluster = get_cluster() jns_node_group = mock.MagicMock() jns_node_group.node_processes = ['HDFS_JOURNALNODE'] jns_node_group.instances = [mock.Mock()] list.append(cluster.node_groups, jns_node_group) uri = eng.get_name_node_uri(cluster) self.assertEqual("hdfs://nameservice01", uri) def test_get_resource_manager_uri(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) uri = eng.get_resource_manager_uri(cluster) self.assertEqual("master_inst.novalocal:8032", uri) def test_get_oozie_server(self): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) actual = eng.get_oozie_server(cluster) expect = cluster.node_groups[1].instances[0] self.assertEqual(expect, actual) @mock.patch('sahara.service.edp.oozie.engine.' 'OozieJobEngine.validate_job_execution') def test_validate_job_execution(self, c): cluster = get_cluster() eng = edp_engine.EdpOozieEngine(cluster) eng.validate_job_execution(cluster, mock.Mock(), mock.Mock()) # more than one oozie server dict.__setitem__(cluster.node_groups[1], 'count', 2) self.assertRaises(pl_ex.InvalidComponentCountException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) @mock.patch( 'sahara.plugins.cdh.confighints_helper.get_possible_hive_config_from', return_value={}) def test_get_possible_job_config_hive(self, get_possible_hive_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_HIVE) get_possible_hive_config_from.assert_called_once_with( 'plugins/cdh/v5_5_0/resources/hive-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.cdh.v5_5_0.edp_engine.EdpOozieEngine') def test_get_possible_job_config_java(self, BaseCDHEdpOozieEngine): expected_config = {'job_config': {}} BaseCDHEdpOozieEngine.get_possible_job_config.return_value = ( expected_config) actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_JAVA) BaseCDHEdpOozieEngine.get_possible_job_config.assert_called_once_with( edp.JOB_TYPE_JAVA) self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.' 'get_possible_mapreduce_config_from', return_value={}) def test_get_possible_job_config_mapreduce( self, get_possible_mapreduce_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_MAPREDUCE) get_possible_mapreduce_config_from.assert_called_once_with( 'plugins/cdh/v5_5_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.' 'get_possible_mapreduce_config_from', return_value={}) def test_get_possible_job_config_mapreduce_streaming( self, get_possible_mapreduce_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_MAPREDUCE_STREAMING) get_possible_mapreduce_config_from.assert_called_once_with( 'plugins/cdh/v5_5_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch( 'sahara.plugins.cdh.confighints_helper.get_possible_pig_config_from', return_value={}) def test_get_possible_job_config_pig(self, get_possible_pig_config_from): expected_config = {'job_config': {}} actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_PIG) get_possible_pig_config_from.assert_called_once_with( 'plugins/cdh/v5_5_0/resources/mapred-site.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.cdh.v5_5_0.edp_engine.EdpOozieEngine') def test_get_possible_job_config_shell(self, BaseCDHEdpOozieEngine): expected_config = {'job_config': {}} BaseCDHEdpOozieEngine.get_possible_job_config.return_value = ( expected_config) actual_config = edp_engine.EdpOozieEngine.get_possible_job_config( edp.JOB_TYPE_SHELL) BaseCDHEdpOozieEngine.get_possible_job_config.assert_called_once_with( edp.JOB_TYPE_SHELL) self.assertEqual(expected_config, actual_config) @mock.patch('sahara.plugins.utils.get_config_value_or_default') @mock.patch('sahara.plugins.utils.get_instance') @mock.patch('sahara.service.edp.spark.engine.' 'SparkJobEngine.validate_job_execution') def test_spark_engine_validate_job_execution(self, validate_job_execution, get_instance, get_config_value_or_default): # version unsupported cluster = get_cluster(version='5.4.0') eng = edp_engine.EdpSparkEngine(cluster) self.assertRaises(ex.InvalidDataException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) # none yarn history server cluster = get_cluster() eng = edp_engine.EdpSparkEngine(cluster) self.assertRaises(pl_ex.InvalidComponentCountException, eng.validate_job_execution, cluster, mock.Mock(), mock.Mock()) # valid cluster = get_cluster() yarn_history_node_group = mock.Mock() yarn_history_node_group.node_processes = ['SPARK_YARN_HISTORY_SERVER'] yarn_history_node_group.count = 1 list.append(cluster.node_groups, yarn_history_node_group) eng = edp_engine.EdpSparkEngine(cluster) eng.validate_job_execution(cluster, mock.Mock(), mock.Mock()) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_deploy.py0000666000175100017510000002362313245514472024522 0ustar zuulzuul00000000000000# Copyright (c) 2016 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.cdh.v5_5_0 import deploy from sahara.tests.unit import base class DeployCDHV550(base.SaharaTestCase): def setUp(self): super(DeployCDHV550, self).setUp() self.master = mock.MagicMock() self.master.node_group.node_processes = [ "HDFS_NAMENODE", "YARN_RESOURCEMANAGER", "CLOUDERA_MANAGER", "SENTRY_SERVER", "YARN_NODEMANAGER", "ZOOKEEPER_SERVER", "OOZIE_SERVER", "YARN_JOBHISTORY", "HDFS_SECONDARYNAMENODE", "HIVE_METASTORE", "HIVE_SERVER2", "SPARK_YARN_HISTORY_SERVER", "HBASE_MASTER", "HBASE_REGIONSERVER", "HUE_SERVER", "KMS", "FLUME_AGENT", "SOLR_SERVER", "SQOOP_SERVER", "IMPALA_STATESTORE", "IMPALA_CATALOGSERVER", "IMPALAD", "KEY_VALUE_STORE_INDEXER", ] self.worker = mock.MagicMock() self.worker.node_group.node_processes = [ "HDFS_DATANODE", "HDFS_JOURNALNODE", "JOURNALNODE", "YARN_NODEMANAGER", "YARN_STANDBYRM", ] self.instances = [self.master, self.worker] self.cluster = mock.MagicMock() self.is_cdh_exists = mock.patch( "sahara.plugins.cdh.commands.is_pre_installed_cdh", return_value=False) self.is_cdh_exists.start() self._create_facade = mock.patch( "sahara.db.sqlalchemy.api._create_facade_lazily") self._create_facade.start() def tearDown(self): self.is_cdh_exists.stop() self._create_facade.stop() super(DeployCDHV550, self).tearDown() @mock.patch("sahara.plugins.utils.get_instances") @mock.patch("sahara.plugins.cdh.v5_5_0.deploy.CU") def test_configure_cluster(self, mock_cu, mock_get_instances): mock_get_instances.return_value = self.instances deploy.configure_cluster(self.cluster) mock_cu.pu.configure_os.assert_called_once_with(self.instances) mock_cu.pu.install_packages.assert_called_once_with(self.instances, deploy.PACKAGES) mock_cu.pu.start_cloudera_agents.assert_called_once_with( self.instances) mock_cu.pu.start_cloudera_manager.assert_called_once_with(self.cluster) mock_cu.update_cloudera_password.assert_called_once_with(self.cluster) mock_cu.await_agents.assert_called_once_with(self.cluster, self.instances) mock_cu.create_mgmt_service.assert_called_once_with(self.cluster) mock_cu.create_services.assert_called_once_with(self.cluster) mock_cu.configure_services.assert_called_once_with(self.cluster) mock_cu.configure_instances.assert_called_once_with(self.instances, self.cluster) mock_cu.deploy_configs.assert_called_once_with(self.cluster) @mock.patch("sahara.plugins.cdh.v5_5_0.deploy.CU") def test__start_roles(self, mock_cu): deploy._start_roles(self.cluster, self.instances) mock_cu.get_service_by_role.assert_any_call('DATANODE', instance=self.worker) mock_cu.get_service_by_role.assert_any_call('NODEMANAGER', instance=self.master) mock_cu.get_service_by_role.assert_any_call('NODEMANAGER', instance=self.worker) self.assertEqual(mock_cu.start_roles.call_count, 3) @mock.patch("sahara.plugins.cdh.v5_5_0.deploy._start_roles") @mock.patch("sahara.plugins.cdh.v5_5_0.deploy.CU") def test_scale_cluster(self, mock_cu, mock__start_roles): deploy.scale_cluster(self.cluster, None) self.assertEqual(mock_cu.call_count, 0) self.assertEqual(mock__start_roles.call_count, 0) deploy.scale_cluster(self.cluster, self.instances) mock_cu.pu.configure_os.assert_called_once_with(self.instances) mock_cu.pu.install_packages.assert_called_once_with(self.instances, deploy.PACKAGES) mock_cu.pu.start_cloudera_agents.assert_called_once_with( self.instances) mock_cu.await_agents.assert_called_once_with(self.cluster, self.instances) mock_cu.configure_instances.assert_called_once_with(self.instances, self.cluster) mock_cu.update_configs.assert_called_once_with(self.instances) mock_cu.pu.configure_swift.assert_called_once_with(self.cluster, self.instances) mock_cu.refresh_datanodes.assert_called_once_with(self.cluster) mock__start_roles.assert_called_once_with(self.cluster, self.instances) @mock.patch("sahara.plugins.cdh.v5_5_0.deploy.CU") def test_decommission_cluster(self, mock_cu): deploy.decommission_cluster(self.cluster, self.instances) dns = [] dns_2 = [] nms = [] nms_2 = [] for i in self.instances: if 'HDFS_DATANODE' in i.node_group.node_processes: dns.append(mock_cu.pu.get_role_name(i, 'DATANODE')) dns_2.append(mock_cu.pu.get_role_name(i, 'HDFS_GATEWAY')) if 'YARN_NODEMANAGER' in i.node_group.node_processes: nms.append(mock_cu.pu.get_role_name(i, 'NODEMANAGER')) nms_2.append(mock_cu.pu.get_role_name(i, 'YARN_GATEWAY')) mock_cu.decommission_nodes.assert_any_call( self.cluster, 'DATANODE', dns, dns_2) mock_cu.decommission_nodes.assert_any_call( self.cluster, 'NODEMANAGER', nms, nms_2) mock_cu.delete_instances.assert_called_once_with(self.cluster, self.instances) mock_cu.refresh_datanodes.assert_called_once_with(self.cluster) mock_cu.refresh_yarn_nodes.assert_called_once_with(self.cluster) @mock.patch("sahara.plugins.cdh.v5_5_0.deploy.CU") def test__prepare_cluster(self, mock_cu): deploy._prepare_cluster(self.cluster) mock_cu.pu.install_extjs.assert_called_once_with(self.cluster) mock_cu.pu.configure_hive.assert_called_once_with(self.cluster) mock_cu.pu.configure_sentry.assert_called_once_with(self.cluster) @mock.patch("sahara.service.edp.hdfs_helper.create_hbase_common_lib") @mock.patch("sahara.plugins.cdh.v5_5_0.deploy.CU") def test__finish_cluster_starting(self, mock_cu, mock_create_hbase): deploy._finish_cluster_starting(self.cluster) mock_cu.pu.put_hive_hdfs_xml.assert_called_once_with(self.cluster) self.assertTrue(mock_create_hbase.called) mock_cu.start_service.assert_called_once_with( mock_cu.get_service_by_role('AGENT', self.cluster)) @mock.patch("sahara.plugins.cdh.v5_5_0.deploy._finish_cluster_starting") @mock.patch("sahara.plugins.cdh.v5_5_0.deploy._prepare_cluster") @mock.patch("sahara.plugins.cdh.v5_5_0.deploy.CU") def test_start_cluster(self, mock_cu, mock_prepare, mock_finish): jns_count = 0 for i in self.instances: if "HDFS_JOURNALNODE" in i.node_group.node_processes: jns_count += 1 mock_cu.pu.get_jns.return_value.__len__.return_value = jns_count deploy.start_cluster(self.cluster) mock_prepare.assert_called_once_with(self.cluster) mock_cu.first_run.assert_called_once_with(self.cluster) mock_cu.pu.configure_swift.assert_called_once_with(self.cluster) if jns_count > 0: mock_cu.enable_namenode_ha.assert_called_once_with(self.cluster) mock_cu.update_role_config.assert_any_call( mock_cu.pu.get_secondarynamenode(self.cluster), 'HDFS_NAMENODE' ) mock_cu.enable_resourcemanager_ha.assert_called_once_with(self.cluster) mock_cu.update_role_config.assert_any_call( mock_cu.pu.get_stdb_rm(self.cluster), 'YARN_STANDBYRM') mock_finish.assert_called_once_with(self.cluster) def test_get_open_ports(self): master_ports = [ 9000, 7180, 7182, 7183, 7432, 7184, 8084, 8086, 10101, 9997, 9996, 8087, 9998, 9999, 8085, 9995, 9994, 8020, 8022, 50070, 50470, 50090, 50495, 8030, 8031, 8032, 8033, 8088, 8040, 8041, 8042, 10020, 19888, 9083, 10000, 8888, 11000, 11001, 18088, 2181, 3181, 4181, 9010, 60000, 60020, 41414, 8038, 8983, 8984, 8005, 12000, 25020, 26000, 25010, 24000, 21050, 21000, 23000, 25000, 28000, 22000, 16000, 16001 ] deploy.get_open_ports(self.master.node_group) self.assertItemsEqual(master_ports, deploy.get_open_ports(self.master.node_group)) worker_ports = [ 9000, 50010, 1004, 50075, 1006, 50020, 8480, 8481, 8485, 8040, 8041, 8042, 8030, 8031, 8032, 8033, 8088 ] self.assertItemsEqual(worker_ports, deploy.get_open_ports(self.worker.node_group)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/v5_5_0/test_plugin_utils.py0000666000175100017510000000165513245514472025745 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Corporation. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.plugins.cdh.v5_5_0 import plugin_utils as pu from sahara.tests.unit.plugins.cdh import base_plugin_utils_test class TestPluginUtilsV550(base_plugin_utils_test.TestPluginUtilsHigherThanV5): def setUp(self): super(TestPluginUtilsV550, self).setUp() self.plug_utils = pu.PluginUtilsV550() self.version = "v5_5_0" sahara-8.0.0/sahara/tests/unit/plugins/cdh/test_confighints_helper.py0000666000175100017510000000435613245514472026105 0ustar zuulzuul00000000000000# Copyright (c) 2015 Intel Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins.cdh import confighints_helper as ch_helper from sahara.tests.unit import base as sahara_base class ConfigHintsHelperTest(sahara_base.SaharaTestCase): @mock.patch('sahara.utils.xmlutils.load_hadoop_xml_defaults', return_value=[]) def test_get_possible_hive_config_from(self, load_hadoop_xml_defaults): expected_config = { 'configs': [], 'params': {} } actual_config = ch_helper.get_possible_hive_config_from( 'sample-config.xml') load_hadoop_xml_defaults.assert_called_once_with('sample-config.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.utils.xmlutils.load_hadoop_xml_defaults', return_value=[]) def test_get_possible_mapreduce_config_from( self, load_hadoop_xml_defaults): expected_config = { 'configs': [], } actual_config = ch_helper.get_possible_mapreduce_config_from( 'sample-config.xml') load_hadoop_xml_defaults.assert_any_call('sample-config.xml') self.assertEqual(expected_config, actual_config) @mock.patch('sahara.utils.xmlutils.load_hadoop_xml_defaults', return_value=[]) def test_get_possible_pig_config_from( self, load_hadoop_xml_defaults): expected_config = { 'configs': [], 'args': [], 'params': {} } actual_config = ch_helper.get_possible_pig_config_from( 'sample-config.xml') load_hadoop_xml_defaults.assert_called_once_with('sample-config.xml') self.assertEqual(expected_config, actual_config) sahara-8.0.0/sahara/tests/unit/plugins/cdh/utils.py0000666000175100017510000000364113245514472022330 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from sahara.tests.unit import testutils as tu def get_fake_cluster(**kwargs): mng = tu.make_inst_dict('id1', 'manager_inst', management_ip='1.2.3.4') mng_ng = tu.make_ng_dict('manager_ng', 1, ['CLOUDERA_MANAGER'], 1, [mng]) mst = tu.make_inst_dict('id2', 'master_inst', management_ip='1.2.3.5') mst_ng = tu.make_ng_dict('master_ng', 1, ['HDFS_NAMENODE', 'HDFS_SECONDARYNAMENODE', 'YARN_RESOURCEMANAGER', 'YARN_JOBHISTORY', 'HIVE_SERVER2', 'HIVE_METASTORE', 'OOZIE_SERVER'], 1, [mst]) wkrs = _get_workers() wkrs_ng = tu.make_ng_dict('worker_ng', 1, ['HDFS_DATANODE', 'YARN_NODEMANAGER'], len(wkrs), wkrs) return tu.create_cluster('test_cluster', 1, 'cdh', '5', [mng_ng, mst_ng, wkrs_ng], **kwargs) def _get_workers(): workers = [] for i in range(3): w = tu.make_inst_dict('id0%d' % i, 'worker-0%d' % i, management_ip='1.2.3.1%d' % i) workers.append(w) return workers sahara-8.0.0/sahara/tests/unit/plugins/cdh/base_config_helper_test.py0000666000175100017510000001534213245514472026026 0ustar zuulzuul00000000000000# Copyright (c) 2016 Intel Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_serialization import jsonutils as json from sahara.tests.unit import base from sahara.tests.unit.plugins.cdh import utils as ctu from sahara.utils import files as f json_files = [ 'hdfs-service.json', 'hdfs-namenode.json', 'hdfs-datanode.json', 'hdfs-secondarynamenode.json', 'hdfs-gateway.json', 'hdfs-journalnode.json', 'yarn-service.json', 'yarn-resourcemanager.json', 'yarn-nodemanager.json', 'yarn-jobhistory.json', 'yarn-gateway.json', 'oozie-service.json', 'oozie-oozie_server.json', 'hive-service.json', 'hive-hivemetastore.json', 'hive-hiveserver2.json', 'hive-webhcat.json', 'hue-service.json', 'hue-hue_server.json', 'spark-service.json', 'spark-spark_yarn_history_server.json', 'zookeeper-service.json', 'zookeeper-server.json', 'hbase-service.json', 'hbase-master.json', 'hbase-regionserver.json', 'flume-service.json', 'flume-agent.json', 'sentry-service.json', 'sentry-sentry_server.json', 'solr-service.json', 'solr-solr_server.json', 'sqoop-service.json', 'sqoop-sqoop_server.json', 'ks_indexer-service.json', 'ks_indexer-hbase_indexer.json', 'impala-service.json', 'impala-catalogserver.json', 'impala-impalad.json', 'impala-statestore.json', 'kms-service.json', 'kms-kms.json', 'kafka-kafka_broker.json', 'kafka-kafka_mirror_maker.json', 'kafka-service.json' ] class TestConfigHelper(base.SaharaTestCase): def setUp(self): super(TestConfigHelper, self).setUp() self.path_to_config = None self.c_h = None def test_get_ng_plugin_configs(self): actual_configs = self.c_h._get_ng_plugin_configs() expected_configs = [] for json_file in json_files: expected_configs += json.loads( f.get_file_text(self.path_to_config + json_file)) # compare names expected_names = set(i['name'] for i in expected_configs) actual_names = set(i.to_dict()['name'] for i in actual_configs) self.assertEqual(expected_names, actual_names) def test_get_cdh5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(self.c_h.CDH5_REPO_URL.default_value, self.c_h.get_cdh5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.CDH5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', self.c_h.get_cdh5_repo_url(cluster)) def test_get_cdh5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(self.c_h.CDH5_REPO_KEY_URL.default_value, self.c_h.get_cdh5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.CDH5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', self.c_h.get_cdh5_key_url(cluster)) def test_get_cm5_repo_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(self.c_h.CM5_REPO_URL.default_value, self.c_h.get_cm5_repo_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.CM5_REPO_URL.name: 'spam'}}) self.assertEqual('spam', self.c_h.get_cm5_repo_url(cluster)) def test_get_cm5_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(self.c_h.CM5_REPO_KEY_URL.default_value, self.c_h.get_cm5_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.CM5_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', self.c_h.get_cm5_key_url(cluster)) def test_is_swift_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(self.c_h.is_swift_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.ENABLE_SWIFT.name: False}}) self.assertFalse(self.c_h.is_swift_enabled(cluster)) def test_get_swift_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(self.c_h.DEFAULT_SWIFT_LIB_URL, self.c_h.get_swift_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.SWIFT_LIB_URL.name: 'spam'}}) self.assertEqual('spam', self.c_h.get_swift_lib_url(cluster)) def test_is_hbase_common_lib_enabled(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(self.c_h.is_hbase_common_lib_enabled(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.ENABLE_HBASE_COMMON_LIB.name: False}}) self.assertFalse(self.c_h.is_hbase_common_lib_enabled(cluster)) def test_get_extjs_lib_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(self.c_h.DEFAULT_EXTJS_LIB_URL, self.c_h.get_extjs_lib_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.EXTJS_LIB_URL.name: 'spam'}}) self.assertEqual('spam', self.c_h.get_extjs_lib_url(cluster)) def test_get_kms_key_url(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertEqual(self.c_h.KMS_REPO_KEY_URL.default_value, self.c_h.get_kms_key_url(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.KMS_REPO_KEY_URL.name: 'spam'}}) self.assertEqual('spam', self.c_h.get_kms_key_url(cluster)) def test_get_required_anti_affinity(self): cluster = ctu.get_fake_cluster(cluster_configs={}) self.assertTrue(self.c_h.get_required_anti_affinity(cluster)) cluster = ctu.get_fake_cluster( cluster_configs={ 'general': {self.c_h.REQUIRE_ANTI_AFFINITY.name: False}}) self.assertFalse(self.c_h.get_required_anti_affinity(cluster)) sahara-8.0.0/sahara/tests/unit/plugins/cdh/test_cloudera_utils.py0000666000175100017510000000721013245514472025241 0ustar zuulzuul00000000000000# Copyright (c) 2014 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock import testtools from sahara.plugins.cdh import cloudera_utils as cu from sahara.tests.unit import base from sahara.tests.unit.plugins.cdh import utils as ctu CU = cu.ClouderaUtils() class ClouderaUtilsTestCase(base.SaharaTestCase): @mock.patch('sahara.plugins.cdh.cloudera_utils.ClouderaUtils.' 'get_cloudera_cluster') def test_get_service(self, mock_get_cl_cluster): self.assertRaises(ValueError, CU.get_service_by_role, 'NAMENODE') cluster = ctu.get_fake_cluster() inst = cluster.node_groups[0].instances[0] mock_get_cl_cluster.return_value = None self.assertRaises(ValueError, CU.get_service_by_role, 'spam', cluster) self.assertRaises(ValueError, CU.get_service_by_role, 'spam', instance=inst) mock_get_cl_cluster.reset_mock() mock_get_service = mock.MagicMock() mock_get_service.get_service.return_value = mock.Mock() mock_get_cl_cluster.return_value = mock_get_service CU.get_service_by_role('NAMENODE', cluster) args = ((CU.HDFS_SERVICE_NAME,),) self.assertEqual(args, mock_get_service.get_service.call_args) mock_get_service.reset_mock() CU.get_service_by_role('JOBHISTORY', instance=inst) args = ((CU.YARN_SERVICE_NAME,),) self.assertEqual(args, mock_get_service.get_service.call_args) mock_get_service.reset_mock() CU.get_service_by_role('OOZIE_SERVER', cluster) args = ((CU.OOZIE_SERVICE_NAME,),) self.assertEqual(args, mock_get_service.get_service.call_args) def test_get_role_name(self): inst_mock = mock.Mock() inst_mock.hostname.return_value = 'spam-host' self.assertEqual('eggs_spam_host', CU.pu.get_role_name(inst_mock, 'eggs')) @mock.patch('sahara.plugins.cdh.cloudera_utils.ClouderaUtils.' 'get_cloudera_cluster') def test_get_service_by_role(self, get_cloudera_cluster): class Ob(object): def get_service(self, x): return x get_cloudera_cluster.return_value = Ob() roles = ['NAMENODE', 'DATANODE', 'SECONDARYNAMENODE', 'HDFS_GATEWAY', 'RESOURCEMANAGER', 'NODEMANAGER', 'JOBHISTORY', 'YARN_GATEWAY', 'OOZIE_SERVER', 'HIVESERVER2', 'HIVEMETASTORE', 'WEBHCAT', 'HUE_SERVER', 'SPARK_YARN_HISTORY_SERVER', 'SERVER', 'MASTER', 'REGIONSERVER'] resps = ['hdfs01', 'hdfs01', 'hdfs01', 'hdfs01', 'yarn01', 'yarn01', 'yarn01', 'yarn01', 'oozie01', 'hive01', 'hive01', 'hive01', 'hue01', 'spark_on_yarn01', 'zookeeper01', 'hbase01', 'hbase01'] provider = cu.ClouderaUtils() cluster = mock.Mock() for (role, resp) in zip(roles, resps): self.assertEqual( resp, provider.get_service_by_role(role, cluster=cluster)) with testtools.ExpectedException(ValueError): provider.get_service_by_role('cat', cluster=cluster) sahara-8.0.0/sahara/tests/unit/plugins/test_utils.py0000666000175100017510000001647613245514472022643 0ustar zuulzuul00000000000000# Copyright (c) 2015 Mirantis Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara.plugins import exceptions as ex from sahara.plugins import utils as pu from sahara.tests.unit import base as b class FakeInstance(object): def __init__(self, _id, node_processes=None): self.id = _id self.node_processes = node_processes or [] @property def node_group(self): return self def __eq__(self, other): return self.id == other.id class FakeNodeGroup(object): def __init__(self, node_processes, instances=None): self.node_processes = node_processes self.instances = instances or [] self.count = len(self.instances) def __eq__(self, other): return self.node_processes == other.node_processes class TestPluginUtils(b.SaharaTestCase): def setUp(self): super(TestPluginUtils, self).setUp() self.cluster = mock.Mock() self.cluster.node_groups = [ FakeNodeGroup(["node_process1"], [FakeInstance("1")]), FakeNodeGroup(["node_process2"], [FakeInstance("2")]), FakeNodeGroup(["node_process3"], [FakeInstance("3")]), ] def test_get_node_groups(self): res = pu.get_node_groups(self.cluster) self.assertEqual([ FakeNodeGroup(["node_process1"]), FakeNodeGroup(["node_process2"]), FakeNodeGroup(["node_process3"]), ], res) res = pu.get_node_groups(self.cluster, "node_process1") self.assertEqual([ FakeNodeGroup(["node_process1"]) ], res) res = pu.get_node_groups(self.cluster, "node_process") self.assertEqual([], res) def test_get_instances_count(self): res = pu.get_instances_count(self.cluster) self.assertEqual(3, res) res = pu.get_instances_count(self.cluster, "node_process1") self.assertEqual(1, res) def test_get_instances(self): res = pu.get_instances(self.cluster) self.assertEqual([ FakeInstance("1"), FakeInstance("2"), FakeInstance("3")], res) res = pu.get_instances(self.cluster, "node_process1") self.assertEqual([FakeInstance("1")], res) def test_get_instance(self): self.assertRaises(ex.InvalidComponentCountException, pu.get_instance, self.cluster, None) res = pu.get_instance(self.cluster, "node_process") self.assertIsNone(res) res = pu.get_instance(self.cluster, "node_process1") self.assertEqual(FakeInstance("1"), res) def test_generate_host_names(self): node = mock.Mock() node.hostname = mock.Mock(return_value="host_name") res = pu.generate_host_names([node, node]) self.assertEqual("host_name\nhost_name", res) def test_generate_fqdn_host_names(self): node = mock.Mock() node.fqdn = mock.Mock(return_value="fqdn") res = pu.generate_fqdn_host_names([node, node]) self.assertEqual("fqdn\nfqdn", res) def test_get_port_from_address(self): res = pu.get_port_from_address("0.0.0.0:8000") self.assertEqual(8000, res) res = pu.get_port_from_address("http://localhost:8000/resource") self.assertEqual(8000, res) res = pu.get_port_from_address("http://192.168.1.101:10000") self.assertEqual(10000, res) res = pu.get_port_from_address("mydomain") self.assertIsNone(res) def test_instances_with_services(self): inst = [FakeInstance("1", ["nodeprocess1"]), FakeInstance("2", ["nodeprocess2"])] node_processes = ["nodeprocess"] res = pu.instances_with_services(inst, node_processes) self.assertEqual([], res) node_processes = ["nodeprocess1"] res = pu.instances_with_services(inst, node_processes) self.assertEqual([FakeInstance("1", ["nodeprocess1"])], res) @mock.patch("sahara.plugins.utils.plugins_base") def test_get_config_value_or_default(self, mock_plugins_base): # no config self.assertRaises(RuntimeError, pu.get_config_value_or_default) config = mock.Mock() config.applicable_target = "service" config.name = "name" config.default_value = "default_value" # cluster has the config cluster = mock.Mock() cluster.cluster_configs = {"service": {"name": "name"}} cluster.plugin_name = "plugin_name" cluster.hadoop_version = "hadoop_version" res = pu.get_config_value_or_default(cluster=cluster, config=config) self.assertEqual("name", res) # node group has the config cluster.cluster_configs = {} node_group1 = mock.Mock() node_group2 = mock.Mock() node_group1.configuration = mock.Mock(return_value={"service": {}}) node_group2.configuration = mock.Mock( return_value={"service": {"name": "name"}}) cluster.node_groups = [node_group1, node_group2] res = pu.get_config_value_or_default(cluster=cluster, config=config) self.assertEqual("name", res) # cluster doesn't have the config, neither the node groups # so it returns the default value cluster.node_groups = [] res = pu.get_config_value_or_default(cluster=cluster, config=config) self.assertEqual("default_value", res) # no config specified, but there's a config for the plugin # with this service and name mock_get_all_configs = mock.Mock(return_value=[config]) mock_plugin = mock.Mock() mock_plugin.get_all_configs = mock_get_all_configs mock_get_plugin = mock.Mock(return_value=mock_plugin) mock_PLUGINS = mock.Mock() mock_PLUGINS.get_plugin = mock_get_plugin mock_plugins_base.PLUGINS = mock_PLUGINS res = pu.get_config_value_or_default(cluster=cluster, service="service", name="name") self.assertEqual("default_value", res) mock_get_plugin.assert_called_once_with("plugin_name") mock_get_all_configs.assert_called_once_with("hadoop_version") # no config especified and no existing config for this plugin # with this service or name cluster.plugin_name = "plugin_name2" cluster.hadoop_version = "hadoop_version2" self.assertRaises(RuntimeError, pu.get_config_value_or_default, cluster=cluster, service="newService", name="name") mock_get_plugin.assert_called_with("plugin_name2") self.assertEqual(2, mock_get_plugin.call_count) mock_get_all_configs.assert_called_with("hadoop_version2") self.assertEqual(2, mock_get_all_configs.call_count) sahara-8.0.0/sahara/tests/unit/plugins/__init__.py0000666000175100017510000000000013245514472022153 0ustar zuulzuul00000000000000sahara-8.0.0/sahara/tests/unit/plugins/test_kerberos.py0000666000175100017510000001150113245514472023277 0ustar zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import mock from sahara import context from sahara.plugins import kerberos as krb from sahara.tests.unit import base ADD_PRINCIPAL_SCRIPT = """#!/bin/bash mkdir -p /tmp/sahara-kerberos/ kadmin -p sahara/admin <